Introduction

Torc is a distributed workflow orchestration system for managing computational pipelines ranging from simple workflows needing to parallelize independent jobs to complex workflows with job dependencies, mixed resource requirements, and multiple stages.

Key Features

• Declarative Workflow Definitions — Define workflows in YAML, JSON, JSON5, or KDL
• Automatic Dependency Resolution — Dependencies inferred from file and data relationships
• Distributed Execution — Run jobs across local machines, HPC clusters, and networked compute nodes
• Resource Management — Track CPU and memory usage across all jobs
• Automatic Failure Recovery — Detect OOM/timeout failures and retry with adjusted resources
• Fault Tolerance — Resume workflows after failures without losing progress
• AI-assisted configuration and management — Use AI tools like Claude Code and Copilot to configure workflows and diagnose problems.

Job Parameterization

Create parameter sweeps with simple syntax:

jobs:
  - name: job_{index}
    command: bash work.sh {index}
    parameters:
      index: "1:100"

This expands to 100 jobs.

Implicit Dependencies

Dependencies between jobs are automatically inferred from file relationships. Consider this diamond workflow where one job fans out to parallel jobs, which then converge:

name: diamond_workflow
jobs:
  - name: preprocess
    command: "preprocess.sh -i ${files.input.f1} -o ${files.output.f2} -o ${files.output.f3}"

  - name: work1
    command: "work.sh -i ${files.input.f2} -o ${files.output.f4}"

  - name: work2
    command: "work.sh -i ${files.input.f3} -o ${files.output.f5}"

  - name: postprocess
    command: "postprocess.sh -i ${files.input.f4} -i ${files.input.f5} -o ${files.output.f6}"

files:
  - name: f1
    path: input.json
  - name: f2
    path: intermediate_a.json
  - name: f3
    path: intermediate_b.json
  - name: f4
    path: result_a.json
  - name: f5
    path: result_b.json
  - name: f6
    path: final_output.json

Torc analyzes which jobs produce and consume each file, automatically building the dependency graph:

flowchart TD
    f1([input.json])
    preprocess[preprocess]
    f2([intermediate_a.json])
    f3([intermediate_b.json])
    work1[work1]
    work2[work2]
    f4([result_a.json])
    f5([result_b.json])
    postprocess[postprocess]
    f6([final_output.json])

    f1 --> preprocess
    preprocess --> f2 & f3
    f2 --> work1
    f3 --> work2
    work1 --> f4
    work2 --> f5
    f4 & f5 --> postprocess
    postprocess --> f6

    style f1 fill:#d4edda,stroke:#28a745,color:#155724
    style f2 fill:#d4edda,stroke:#28a745,color:#155724
    style f3 fill:#d4edda,stroke:#28a745,color:#155724
    style f4 fill:#d4edda,stroke:#28a745,color:#155724
    style f5 fill:#d4edda,stroke:#28a745,color:#155724
    style f6 fill:#d4edda,stroke:#28a745,color:#155724
    style preprocess fill:#4a9eff,color:#fff
    style work1 fill:#4a9eff,color:#fff
    style work2 fill:#4a9eff,color:#fff
    style postprocess fill:#4a9eff,color:#fff

No explicit depends_on declarations needed — Torc infers that work1 and work2 depend on preprocess, and postprocess waits for both to complete.

Who Should Use Torc?

Torc is designed for:

• HPC Users who need to parallelize jobs across cluster resources
• Computational Scientists running parameter sweeps and simulations
• Data Engineers building complex data processing pipelines
• ML/AI Researchers managing training workflows and hyperparameter searches
• Anyone who needs reliable, resumable workflow orchestration

Documentation Structure

This documentation is organized to help you find what you need quickly:

Core Documentation (for all users):

• Getting Started — Installation and local quick start
• Core Concepts — Architecture, job states, and dependencies
• Working with Workflows — Creating and managing workflows
• Tutorials — Step-by-step workflow patterns
• Monitoring & Debugging — TUI, reports, and troubleshooting
• Reference — CLI commands, formats, and configuration

Specialized Topics (below the separator in the sidebar):

• HPC & Slurm — Running on compute clusters
• Fault Tolerance & Recovery — Failure handlers, checkpointing, automatic recovery
• Administration & Security — Server deployment, authentication, access control
• Design & Architecture — Internal design for contributors

Next Steps

• New to Torc? Start with Quick Start (Local)
• Want to understand how it works? Read the Architecture Overview
• Ready to create workflows? Jump to Creating Workflows
• On an HPC cluster? See Quick Start (HPC)

Getting Started

Torc is a distributed workflow orchestration system for managing complex computational pipelines with job dependencies, resource requirements, and distributed execution.

Torc uses a client-server architecture where a central server manages workflow state and coordination, while clients create workflows and job runners execute tasks on compute resources.

How You Interact with Torc

Choose the interface that fits your workflow:

• CLI — Primary interface which provides access to all torc functionality
• Dashboard — Web UI (torc-dash) for visual configuration and monitoring
• TUI — Terminal User Interface (torc tui) for visual monitoring in a terminal
• AI Assistants — Use Claude Code or GitHub Copilot to manage workflows through natural language. "Create a workflow with 10 parallel jobs" or "Why did job 5 fail?"
• Spec Files — YAML, KDL, or JSON5 files for version-controlled workflow definitions
• Python/Julia APIs — Libraries for programmatic workflow generation

All interfaces work with the same server — mix and match as needed.

Architecture

flowchart LR
    subgraph you["You"]
        CLI["torc CLI"]
    end

    subgraph server["Server"]
        API["torc-server"]
    end

    subgraph workers["Workers"]
        W1["Job Runner"]
        W2["Job Runner"]
    end

    CLI -->|"create workflow"| API
    CLI -->|"start workers"| W1
    CLI -->|"start workers"| W2
    W1 -->|"claim & report"| API
    W2 -->|"claim & report"| API

    style CLI fill:#4a9eff,color:#fff
    style API fill:#28a745,color:#fff
    style W1 fill:#ffc107,color:#000
    style W2 fill:#ffc107,color:#000

1. You create workflows and start job runners via the CLI
2. Server tracks workflow state and job dependencies
3. Workers poll the server, claim ready jobs, execute them, and report results

Example Files

The repository includes ready-to-run workflow specifications in YAML, JSON5, and KDL formats:

Browse all examples:

• YAML examples
• JSON5 examples
• KDL examples
• Python examples

See the examples README for the complete list.

Choose Your Execution Mode

Torc supports three fundamentally different execution environments. Choose the one that matches your use case:

Local Execution

Best for: Development, testing, small-scale workflows on your workstation or a single server

• Jobs run directly on the machine where you start the job runner
• No scheduler needed — simple setup with torc run
• Resource management via local CPU/memory/GPU tracking
• → Quick Start (Local)

HPC/Slurm

Best for: Large-scale computations on institutional HPC clusters

• Jobs submitted to Slurm scheduler for compute node allocation
• Automatic resource matching to partitions/QOS
• Built-in profiles for common HPC systems
• → Quick Start (HPC/Slurm)

Remote Workers

Best for: Distributed execution across multiple machines you control via SSH

• Jobs distributed to remote workers over SSH
• No HPC scheduler required — you manage the machines
• Flexible heterogeneous resources (mix of CPU/GPU machines)
• → Quick Start (Remote Workers)

All three modes:

• Share the same workflow specification format
• Use the same server API for coordination
• Support the same monitoring tools (CLI, TUI, Dashboard)
• Can be used together (e.g., develop locally, deploy to HPC)

Continue to the Quick Start guide to run your first workflow.

Installation

Precompiled Binaries (Recommended)

1. Download the appropriate archive for your platform from the releases page:

   • Linux: torc-<version>-x86_64-unknown-linux-gnu.tar.gz
   • macOS (Intel): torc-<version>-x86_64-apple-darwin.tar.gz
   • macOS (Apple Silicon): torc-<version>-aarch64-apple-darwin.tar.gz

2. Extract the archive:

   # For .tar.gz files
   tar -xzf torc-<version>-<platform>.tar.gz
   
   # For .zip files
   unzip torc-<version>-<platform>.zip
   

3. Add the binaries to a directory in your system PATH:

   # Option 1: Copy to an existing PATH directory
   cp torc* ~/.local/bin/
   
   # Option 2: Add the extracted directory to your PATH
   export PATH="/path/to/extracted/torc:$PATH"
   

   To make the PATH change permanent, add the export line to your shell configuration file (~/.bashrc, ~/.zshrc, etc.).

macOS users: The precompiled binaries are not signed with an Apple Developer certificate. macOS Gatekeeper will block them by default. To allow the binaries to run, remove the quarantine attribute after downloading:

xattr -cr /path/to/torc*

Alternatively, you can right-click each binary and select "Open" to add a security exception.

Site-Specific Installations

Some HPC facilities maintain pre-installed Torc binaries and shared servers. Check if your site is listed below.

NREL Kestrel

Pre-installed binaries are available at:

/scratch/dthom/torc/
├── 0.8.0/
├── ...
└── latest -> 0.8.0  (symlink to current version)

Recommended: Use the latest directory. Torc maintains backwards compatibility, so you'll automatically receive updates and bug fixes without changing your configuration.

Add to your PATH:

export PATH="/scratch/dthom/torc/latest:$PATH"

Or add to your ~/.bashrc for persistence:

echo 'export PATH="/scratch/dthom/torc/latest:$PATH"' >> ~/.bashrc

Shared server: A torc-server instance runs on a dedicated VM within the Kestrel environment. Contact Daniel Thom for access credentials and the server URL. Once you have access:

export TORC_API_URL="http://<server-address>/torc-service/v1"
export TORC_PASSWORD="<your-password>"

Building from Source

Prerequisites

• Rust 1.70 or later
• SQLite 3.35 or later (usually included with Rust via sqlx)

Clone the Repository

git clone https://github.com/NREL/torc.git
cd torc

Building All Components

Note that the file .env designates the database URL as ./db/sqlite/dev.db Change as desired or set the environment variable DATABASE_URL.

Initialize the database

# Install sqlx-cli if needed
cargo install sqlx-cli --no-default-features --features sqlite
sqlx database setup

Build everything (server, client, dashboard, job runners):

# Development build
cargo build --workspace

# Release build (optimized, recommended)
cargo build --workspace --release

Build individual components:

# Server
cargo build --release -p torc-server

# Client CLI
cargo build --release -p torc

# Web Dashboard
cargo build --release -p torc-dash

# Slurm job runner
cargo build --release -p torc-slurm-job-runner

Binaries will be in target/release/.

Required: Add this directory to your system path or copy the binaries to a directory already in your path (e.g., ~/.local/bin/).

Python Client

The Python client provides programmatic workflow management for Python users.

Prerequisites

• Python 3.11 or later

Installation

pip install torc-client

The pytorc command will be available after installation.

Julia Client

The Julia client provides programmatic workflow management for Julia users.

Prerequisites

• Julia 1.10 or later

Installation

Since the package is not yet registered in the Julia General registry, install it directly from GitHub:

using Pkg
Pkg.add(url="https://github.com/NREL/torc.git", subdir="julia_client/Torc")

Then use it in your code:

using Torc

For Developers

Running Tests

Run all tests

cargo test -- --test-threads=1

# Run specific test
cargo test --test test_workflow_manager test_initialize_files_with_updated_files

# Run with debug logging
RUST_LOG=debug cargo test -- --nocapture

Setting Up the Server

Start the server:

# Development mode
cargo run -p torc-server -- run

# Production mode (release build)
./target/release/torc-server run

# Custom port
./target/release/torc-server run --port 8080

Server will start on http://localhost:8080.

When running small workflows for testing and demonstration purposes, we recommend setting this option so that the server detects job completions faster than the default value of 30 seconds.

./target/release/torc-server run --completion-check-interval-secs 5

Quick Start (Local)

This guide walks you through creating and running your first Torc workflow with local execution. Jobs run directly on the current machine, making this ideal for testing, development, or non-HPC environments.

For running workflows on HPC clusters with Slurm, see Quick Start (HPC).

Start the Server

Start a Torc server with a local database. Setting --completion-check-interval-secs ensures job completions are processed quickly (use this for personal servers, not shared deployments).

torc-server run --database torc.db --completion-check-interval-secs 5

Test the Connection

In a new terminal, verify the client can connect:

torc workflows list

Create a Workflow

Save this as workflow.yaml:

name: hello_world
description: Simple hello world workflow

jobs:
  - name: job 1
    command: echo "Hello from torc!"
  - name: job 2
    command: echo "Hello again from torc!"

Note: Torc also accepts .json, .json5 and .kdl workflow specifications. See Workflow Specification Formats for details.

Run the Workflow

Run jobs locally with a short poll interval for demo purposes:

torc run workflow.yaml --poll-interval 1

This creates the workflow, initializes it, and runs all jobs on the current machine.

View Results

torc results list

Or use the TUI for an interactive view:

torc tui

Example: Diamond Workflow

A workflow with fan-out and fan-in dependencies:

name: diamond_workflow
description: Example workflow with implicit dependencies

jobs:
  - name: preprocess
    command: "bash tests/scripts/preprocess.sh -i ${files.input.f1} -o ${files.output.f2} -o ${files.output.f3}"

  - name: work1
    command: "bash tests/scripts/work.sh -i ${files.input.f2} -o ${files.output.f4}"

  - name: work2
    command: "bash tests/scripts/work.sh -i ${files.input.f3} -o ${files.output.f5}"

  - name: postprocess
    command: "bash tests/scripts/postprocess.sh -i ${files.input.f4} -i ${files.input.f5} -o ${files.output.f6}"

files:
  - name: f1
    path: f1.json
  - name: f2
    path: f2.json
  - name: f3
    path: f3.json
  - name: f4
    path: f4.json
  - name: f5
    path: f5.json
  - name: f6
    path: f6.json

Dependencies are automatically inferred from file inputs/outputs:

• work1 and work2 wait for preprocess (depend on its output files)
• postprocess waits for both work1 and work2 to complete

More Examples

The examples directory contains many more workflow examples in YAML, JSON5, and KDL formats.

Next Steps

• Quick Start (HPC) - Run workflows on Slurm clusters
• Creating Workflows - Detailed workflow creation guide
• Terminal UI - Interactive workflow monitoring

Quick Start (HPC)

This guide walks you through running your first Torc workflow on an HPC cluster with Slurm. Jobs are submitted to Slurm and run on compute nodes.

For local execution (testing, development, or non-HPC environments), see Quick Start (Local).

Prerequisites

• Access to an HPC cluster with Slurm
• A Slurm account/allocation for submitting jobs
• Torc installed (see Installation)

Start the Server

On the login node, start a Torc server with a local database:

torc-server run --database torc.db --completion-check-interval-secs 5

Note: For larger deployments, your team may provide a shared Torc server. In that case, skip this step and set TORC_API_URL to the shared server address.

Check Your HPC Profile

Torc includes built-in profiles for common HPC systems. Check if your system is detected:

torc hpc detect

If detected, you'll see your HPC system name. To see available partitions:

torc hpc partitions <profile-name>

Note: If your HPC system isn't detected, see Custom HPC Profile or request built-in support.

Create a Workflow with Resource Requirements

Save this as workflow.yaml:

name: hpc_hello_world
description: Simple HPC workflow

resource_requirements:
  - name: small
    num_cpus: 4
    memory: 8g
    runtime: PT30M

jobs:
  - name: job1
    command: echo "Hello from compute node!" && hostname
    resource_requirements: small

  - name: job2
    command: echo "Hello again!" && hostname
    resource_requirements: small
    depends_on: [job1]

Key differences from local workflows:

• resource_requirements: Define CPU, memory, and runtime needs
• Jobs reference these requirements by name
• Torc matches requirements to appropriate Slurm partitions

Submit the Workflow

Submit with your Slurm account:

torc submit-slurm --account <your-account> workflow.yaml

Torc will:

1. Detect your HPC system
2. Match job requirements to appropriate partitions
3. Generate Slurm scheduler configurations
4. Create and submit the workflow

Monitor Progress

Check workflow status:

torc workflows list
torc jobs list <workflow-id>

Or use the interactive TUI:

torc tui

Check Slurm queue:

squeue --me

View Results

Once jobs complete:

torc results list <workflow-id>

Job output is stored in the output/ directory by default.

Example: Multi-Stage Pipeline

A more realistic workflow with different resource requirements per stage:

name: analysis_pipeline
description: Data processing pipeline

resource_requirements:
  - name: light
    num_cpus: 4
    memory: 8g
    runtime: PT30M

  - name: compute
    num_cpus: 32
    memory: 64g
    runtime: PT2H

  - name: gpu
    num_cpus: 8
    num_gpus: 1
    memory: 32g
    runtime: PT1H

jobs:
  - name: preprocess
    command: python preprocess.py
    resource_requirements: light

  - name: train
    command: python train.py
    resource_requirements: gpu
    depends_on: [preprocess]

  - name: evaluate
    command: python evaluate.py
    resource_requirements: compute
    depends_on: [train]

Torc stages resource allocation based on dependencies:

• preprocess resources are allocated at workflow start
• train resources are allocated when preprocess completes
• evaluate resources are allocated when train completes

This prevents wasting allocation time on resources that aren't needed yet.

Preview Before Submitting

For production workflows, preview the generated Slurm configuration first:

torc slurm generate --account <your-account> workflow.yaml

This shows what schedulers and actions Torc will create without submitting anything.

Next Steps

• Slurm Workflows — How Torc manages Slurm
• Resource Requirements — All resource options
• HPC Profiles — Managing HPC configurations
• Working with Slurm — Advanced Slurm configuration
• Debugging Slurm Workflows — Troubleshooting

Quick Start (Remote Workers)

This guide walks you through running a Torc workflow on multiple remote machines via SSH. Jobs are distributed across workers without requiring an HPC scheduler like Slurm.

For local execution, see Quick Start (Local). For HPC/Slurm execution, see Quick Start (HPC).

Prerequisites

• SSH key-based authentication to all remote machines (no password prompts)
• Torc installed on all machines with matching versions
• Torc server accessible from all machines

Start the Server

Start a Torc server that's accessible from the remote machines. This typically means binding to a network interface (not just localhost):

torc-server run --database torc.db --host 0.0.0.0 --port 8080

Create a Worker File

Create a file listing the remote machines. Each line contains one machine in the format [user@]hostname[:port]:

# workers.txt
worker1.example.com
alice@worker2.example.com
admin@192.168.1.10:2222

Lines starting with # are comments. Empty lines are ignored.

Create a Workflow

Save this as workflow.yaml:

name: distributed_hello
description: Distributed hello world workflow

jobs:
  - name: job 1
    command: echo "Hello from $(hostname)!"
  - name: job 2
    command: echo "Hello again from $(hostname)!"
  - name: job 3
    command: echo "And once more from $(hostname)!"

Create the Workflow on the Server

torc workflows create workflow.yaml

Note the workflow ID in the output.

Run Workers on Remote Machines

Start workers on all remote machines. Each worker will poll for available jobs and execute them:

torc remote run --workers workers.txt <workflow-id> --poll-interval 5

This will:

1. Check SSH connectivity to all machines
2. Verify all machines have the same torc version
3. Start a worker process on each machine (detached via nohup)
4. Report which workers started successfully

Check Worker Status

Monitor which workers are still running:

torc remote status <workflow-id>

View Workflow Progress

Check job status from any machine:

torc jobs list <workflow-id>

Or use the interactive TUI:

torc tui

Collect Logs

After the workflow completes, collect logs from all workers:

torc remote collect-logs <workflow-id> --local-output-dir ./logs

This creates a tarball for each worker containing:

• Worker logs: torc_worker_<workflow_id>.log
• Job stdout/stderr: job_stdio/job_*.o and job_stdio/job_*.e
• Resource utilization data (if enabled): resource_utilization/resource_metrics_*.db

Stop Workers

If you need to stop workers before the workflow completes:

torc remote stop <workflow-id>

Add --force to send SIGKILL instead of SIGTERM.

Next Steps

• Remote Workers Guide - Detailed configuration and troubleshooting
• Creating Workflows - Workflow specification format
• Resource Monitoring - Track CPU/memory usage per job

Core Concepts

This section covers the fundamental concepts you need to understand to work effectively with Torc.

• Architecture Overview - How Torc's components work together
• Workflow Definition - Structure of workflows and jobs
• Job State Transitions - How jobs move through their lifecycle
• Dependency Resolution - How job dependencies are managed
• Job Runners - Executing jobs locally and on clusters
• Parallelization Strategies - Running jobs in parallel
• Workflow Reinitialization - Restarting workflows

Architecture

Overview

Torc uses a client-server architecture where a central server manages workflow state and coordination, while clients create workflows and job runners execute tasks on compute resources.

flowchart TB
    subgraph ui["User Interfaces"]
        DASH["torc-dash<br/>(web)"]
        TUI["torc tui<br/>(terminal)"]
        CLI["torc CLI"]
    end

    subgraph server["Server (torc-server)"]
        API["HTTP API"]
        DB[(SQLite)]
        API <--> DB
    end

    subgraph workers["Job Runners"]
        W1["Runner 1"]
        W2["Runner 2"]
        WN["Runner N"]
    end

    DASH -->|"HTTP"| API
    DASH -->|"executes"| CLI
    TUI -->|"HTTP"| API
    TUI -->|"executes"| CLI
    CLI -->|"HTTP"| API
    W1 -->|"claim jobs"| API
    W2 -->|"claim jobs"| API
    WN -->|"claim jobs"| API

    style DASH fill:#17a2b8,color:#fff
    style TUI fill:#17a2b8,color:#fff
    style CLI fill:#4a9eff,color:#fff
    style API fill:#28a745,color:#fff
    style DB fill:#28a745,color:#fff
    style W1 fill:#ffc107,color:#000
    style W2 fill:#ffc107,color:#000
    style WN fill:#ffc107,color:#000

Key Components:

Workflow Definition

A workflow is a collection of jobs with dependencies. You define workflows in YAML, JSON5, or JSON files.

Minimal Example

name: hello_world
jobs:
  - name: greet
    command: echo "Hello, World!"

That's it. One job, no dependencies.

Jobs with Dependencies

name: two_stage
jobs:
  - name: prepare
    command: ./prepare.sh

  - name: process
    command: ./process.sh
    depends_on: [prepare]

The process job waits for prepare to complete.

Job Parameterization

Create multiple jobs from a single definition using parameters:

name: parameter_sweep
jobs:
  - name: task_{i}
    command: ./run.sh --index {i}
    parameters:
      i: "1:10"

This expands to 10 jobs: task_1, task_2, ..., task_10.

Parameter Formats

Format Specifiers

Control how values appear in names:

- name: job_{i:03d}      # job_001, job_002, ...
  parameters:
    i: "1:100"

- name: lr_{lr:.4f}      # lr_0.0010, lr_0.0100, ...
  parameters:
    lr: "[0.001,0.01,0.1]"

Resource Requirements

Specify what resources each job needs:

name: gpu_workflow

resource_requirements:
  - name: gpu_job
    num_cpus: 8
    num_gpus: 1
    memory: 16g
    runtime: PT2H

jobs:
  - name: train
    command: python train.py
    resource_requirements: gpu_job

Resource requirements are used for:

• Local execution: ensuring jobs don't exceed available resources
• HPC/Slurm: requesting appropriate allocations

Complete Example

name: data_pipeline
description: Process data in parallel, then aggregate

resource_requirements:
  - name: worker
    num_cpus: 4
    memory: 8g
    runtime: PT1H

jobs:
  - name: process_{i}
    command: python process.py --chunk {i} --output results/chunk_{i}.json
    resource_requirements: worker
    parameters:
      i: "1:10"

  - name: aggregate
    command: python aggregate.py --input results/ --output final.json
    resource_requirements: worker
    depends_on:
      - process_{i}
    parameters:
      i: "1:10"

This creates:

• 10 parallel process_* jobs
• 1 aggregate job that waits for all 10 to complete

Failure Recovery Options

Control how Torc handles job failures:

Default Behavior

By default, jobs that fail without a matching failure handler use Failed status:

name: my_workflow
jobs:
  - name: task
    command: ./run.sh  # If this fails, status = Failed

AI-Assisted Recovery (Opt-in)

Enable intelligent classification of ambiguous failures:

name: ml_training
use_pending_failed: true  # Enable AI-assisted recovery

jobs:
  - name: train_model
    command: python train.py

With use_pending_failed: true:

• Jobs without matching failure handlers get PendingFailed status
• AI agent can analyze stderr and decide whether to retry or fail
• See AI-Assisted Recovery for details

See Also

• Workflow Specification Formats — Complete syntax reference
• Job Parameterization — Advanced parameter options
• Dependency Resolution — How dependencies work

Job State Transitions

Jobs progress through a defined lifecycle:

stateDiagram-v2
    [*] --> uninitialized
    uninitialized --> ready: initialize_jobs
    uninitialized --> blocked: has dependencies

    blocked --> ready: dependencies met
    ready --> pending: runner claims
    pending --> running: execution starts

    running --> completed: exit 0
    running --> failed: exit != 0 (handler match + max retries)
    running --> pending_failed: exit != 0 (no handler match)
    running --> ready: exit != 0 (failure handler retry)
    running --> canceled: user cancels
    running --> terminated: system terminates

    pending_failed --> failed: AI classifies as permanent
    pending_failed --> ready: AI classifies as transient
    pending_failed --> uninitialized: reset-status

    completed --> [*]
    failed --> [*]
    canceled --> [*]
    terminated --> [*]

    classDef waiting fill:#6c757d,color:#fff
    classDef ready fill:#17a2b8,color:#fff
    classDef active fill:#ffc107,color:#000
    classDef success fill:#28a745,color:#fff
    classDef error fill:#dc3545,color:#fff
    classDef stopped fill:#6f42c1,color:#fff
    classDef classification fill:#fd7e14,color:#fff

    class uninitialized,blocked waiting
    class ready ready
    class pending,running active
    class completed success
    class failed error
    class canceled,terminated stopped
    class pending_failed classification

State Descriptions

• uninitialized (0) - Job created but dependencies not evaluated
• blocked (1) - Waiting for dependencies to complete
• ready (2) - All dependencies satisfied, ready for execution
• pending (3) - Job claimed by runner
• running (4) - Currently executing
• completed (5) - Finished successfully (exit code 0)
• failed (6) - Finished with error (exit code != 0)
• canceled (7) - Explicitly canceled by user or torc. Never executed.
• terminated (8) - Explicitly terminated by system, such as at wall-time timeout
• pending_failed (10) - Job failed without a matching failure handler. Awaiting AI-assisted classification to determine if the error is transient (retry) or permanent (fail). See AI-Assisted Recovery.

Dependency Resolution

Torc supports two types of dependencies: explicit (declared via depends_on) and implicit (inferred from file/data relationships). For a hands-on tutorial, see Diamond Workflow with File Dependencies.

Explicit Dependencies

Declared via depends_on:

jobs:
  - name: preprocess
    command: preprocess.sh
  - name: analyze
    command: analyze.sh
    depends_on:
      - preprocess

Implicit Dependencies via Files

Torc automatically infers dependencies from file relationships. When one job outputs a file and another job inputs it, the dependency is created automatically:

jobs:
  - name: preprocess
    command: "preprocess.sh -o ${files.output.intermediate}"
  - name: analyze
    command: "analyze.sh -i ${files.input.intermediate}"

files:
  - name: intermediate
    path: data/intermediate.json

This creates a diamond workflow pattern:

flowchart TD
    input([input.json])
    preprocess[preprocess]
    intermediate([intermediate.json])
    analyze[analyze]
    output([output.json])

    input --> preprocess
    preprocess --> intermediate
    intermediate --> analyze
    analyze --> output

    style input fill:#d4edda,stroke:#28a745,color:#155724
    style intermediate fill:#d4edda,stroke:#28a745,color:#155724
    style output fill:#d4edda,stroke:#28a745,color:#155724
    style preprocess fill:#4a9eff,color:#fff
    style analyze fill:#4a9eff,color:#fff

No explicit depends_on needed — Torc infers that analyze depends on preprocess because they share the intermediate file.

Implicit Dependencies via User Data

User data works like files but stores JSON in the database instead of the filesystem:

jobs:
  - name: generate_config
    command: |
      torc user-data update ${user_data.output.config} --data '{"lr": 0.001}'
  - name: run_simulation
    command: |
      CONFIG=$(torc user-data get ${user_data.input.config} | jq '.data')
      python simulate.py --config "$CONFIG"

user_data:
  - name: config

flowchart LR
    gen[generate_config]
    config[(config)]
    sim[run_simulation]

    gen -->|writes| config
    config -->|reads| sim

    style config fill:#fff3cd,stroke:#ffc107,color:#856404
    style gen fill:#4a9eff,color:#fff
    style sim fill:#4a9eff,color:#fff

Use user data for small configuration objects; use files for large datasets.

Resolution Process

During workflow creation, the server:

1. Resolves all names to IDs
2. Stores explicit dependencies in job_depends_on
3. Stores file/user_data relationships in junction tables
4. During initialize_jobs, queries junction tables to add implicit dependencies

Dependency Graph Evaluation

When initialize is called:

1. All jobs start in uninitialized state
2. Server builds complete dependency graph from explicit and implicit dependencies
3. Jobs with no unsatisfied dependencies are marked ready
4. Jobs waiting on dependencies are marked blocked
5. As jobs complete, blocked jobs are re-evaluated and may become ready

Variable Substitution Syntax

In workflow specification files (YAML, JSON5, KDL), use these patterns to reference files and user data in job commands:

Example:

jobs:
  - name: process
    command: "python process.py -i ${files.input.raw} -o ${files.output.result}"

See Workflow Specification Formats for complete syntax details.

Job Runners

Job runners are worker processes that execute jobs on compute resources.

Job Runner Modes

Torc supports three execution modes:

1. Local Runner (torc run) - Runs jobs on the local machine with resource tracking
2. HPC/Slurm Runner (torc submit-slurm) - Runs jobs on Slurm-allocated compute nodes
3. Remote Workers (torc remote run) - Distributes jobs across SSH-accessible machines

Local Runner

The local runner executes jobs directly on the current machine. Start it with:

torc run <workflow-id>

HPC/Slurm Runner

For HPC clusters, jobs run on Slurm-allocated compute nodes. The torc-slurm-job-runner binary is launched by Slurm on each allocated node and polls the server for work.

Remote Workers

Remote workers enable distributed execution without a scheduler. The torc remote run command SSH-es into multiple machines and starts a torc run process on each:

torc remote run workers.txt <workflow-id>

Each remote worker runs as a detached process and polls the server for jobs, just like the local runner. The server coordinates job distribution to prevent double-allocation.

Job Allocation Strategies

The job runner supports two different strategies for retrieving and executing jobs:

Resource-Based Allocation (Default)

Used when: --max-parallel-jobs is NOT specified

Behavior:

• Retrieves jobs from the server via the command claim_jobs_based_on_resources
• Server filters jobs based on available compute node resources (CPU, memory, GPU)
• Only returns jobs that fit within the current resource capacity
• Prevents resource over-subscription and ensures jobs have required resources
• Defaults to requiring one CPU and 1 MB of memory for each job.

Use cases:

• When you want parallelization based on one CPU per job.
• When you have heterogeneous jobs with different resource requirements and want intelligent resource management.

Example 1: Run jobs at queue depth of num_cpus:

parameters:
  i: "1:100"
jobs:
  - name: "work_{i}"
    command: bash my_script.sh {i}
    use_parameters:
    - i

Example 2: Resource-based parallelization:

resource_requirements:
  - name: "work_resources"
    num_cpus: 32
    memory: "200g"
    runtime: "PT4H"
    num_nodes: 1
    
parameters:
  i: "1:100"
jobs:
  - name: "work_{i}"
    command: bash my_script.sh {i}
    resource_requirements: work_resources  
    use_parameters:
    - i

Simple Queue-Based Allocation

Used when: --max-parallel-jobs is specified

Behavior:

• Retrieves jobs from the server via the command claim_next_jobs
• Server returns the next N ready jobs from the queue (up to the specified limit)
• Ignores job resource requirements completely
• Simply limits the number of concurrent jobs

Use cases: When all jobs have similar resource needs or when the resource bottleneck is not tracked by Torc, such as network or storage I/O. This is the only way to run jobs at a queue depth higher than the number of CPUs in the worker.

Example:

torc run $WORKFLOW_ID \
  --max-parallel-jobs 10 \
  --output-dir ./results

Job Runner Workflow

The job runner executes a continuous loop with these steps:

flowchart TD
    Start([Start]) --> CheckStatus[Check workflow status]
    CheckStatus --> IsComplete{Workflow complete<br/>or canceled?}
    IsComplete -->|Yes| End([Exit])
    IsComplete -->|No| MonitorJobs[Monitor running jobs]
    MonitorJobs --> CompleteFinished[Complete finished jobs<br/>Update server status]
    CompleteFinished --> ExecuteActions[Execute workflow actions<br/>e.g., schedule Slurm allocations]
    ExecuteActions --> ClaimJobs[Claim new jobs from server]
    ClaimJobs --> ResourceCheck{Allocation<br/>strategy?}
    ResourceCheck -->|Resource-based| ClaimResources[claim_jobs_based_on_resources<br/>Filter by CPU/memory/GPU]
    ResourceCheck -->|Queue-based| ClaimQueue[claim_next_jobs<br/>Up to max-parallel-jobs]
    ClaimResources --> StartJobs
    ClaimQueue --> StartJobs
    StartJobs[Start claimed jobs] --> ForEachJob[For each job:<br/>1. Call start_job<br/>2. Execute command<br/>3. Record stdout/stderr]
    ForEachJob --> Sleep[Sleep for poll interval]
    Sleep --> CheckStatus

    style Start fill:#10b981,stroke:#059669,color:#fff
    style End fill:#ef4444,stroke:#dc2626,color:#fff
    style IsComplete fill:#f59e0b,stroke:#d97706,color:#fff
    style ResourceCheck fill:#f59e0b,stroke:#d97706,color:#fff
    style CheckStatus fill:#3b82f6,stroke:#2563eb,color:#fff
    style MonitorJobs fill:#3b82f6,stroke:#2563eb,color:#fff
    style CompleteFinished fill:#3b82f6,stroke:#2563eb,color:#fff
    style ExecuteActions fill:#3b82f6,stroke:#2563eb,color:#fff
    style ClaimJobs fill:#3b82f6,stroke:#2563eb,color:#fff
    style StartJobs fill:#3b82f6,stroke:#2563eb,color:#fff
    style ForEachJob fill:#3b82f6,stroke:#2563eb,color:#fff
    style Sleep fill:#6b7280,stroke:#4b5563,color:#fff
    style ClaimResources fill:#8b5cf6,stroke:#7c3aed,color:#fff
    style ClaimQueue fill:#ec4899,stroke:#db2777,color:#fff

1. Check workflow status - Poll server to check if workflow is complete or canceled
2. Monitor running jobs - Check status of currently executing jobs
3. Execute workflow actions - Check for and execute any pending workflow actions, such as scheduling new Slurm allocations.
4. Claim new jobs - Request ready jobs from server based on allocation strategy:
   • Resource-based: claim_jobs_based_on_resources
   • Queue-based: claim_next_jobs
5. Start jobs - For each claimed job:
   • Call start_job to mark job as started in database
   • Execute job command in a non-blocking subprocess
   • Record stdout/stderr output to files
6. Complete jobs - When running jobs finish:
   • Call complete_job with exit code and result
   • Server updates job status and automatically marks dependent jobs as ready
7. Sleep and repeat - Wait for job completion poll interval, then repeat loop

The runner continues until the workflow is complete or canceled.

Resource Management (Resource-Based Allocation Only)

When using resource-based allocation (default), the local job runner tracks:

• Number of CPUs in use
• Memory allocated to running jobs
• GPUs in use
• Job runtime limits

When a ready job is retrieved, the runner checks if sufficient resources are available before executing it.

Parallelization Strategies

Torc provides flexible parallelization strategies to accommodate different workflow patterns and resource allocation scenarios. Understanding these strategies helps you optimize job execution for your specific use case.

Overview

Torc supports two primary approaches to parallel job execution:

1. Resource-aware allocation - Define per-job resource requirements and let runners intelligently select jobs that fit available resources
2. Queue-depth parallelism - Control the number of concurrent jobs without resource tracking

The choice between these approaches depends on your workflow characteristics and execution environment.

Use Case 1: Resource-Aware Job Allocation

This strategy is ideal for heterogeneous workflows where jobs have varying resource requirements (CPU, memory, GPU, runtime). The server intelligently allocates jobs based on available compute node resources.

How It Works

When you define resource requirements for each job:

resource_requirements:
  - name: small
    num_cpus: 2
    num_gpus: 0
    memory: 4g
    runtime: PT30M

  - name: large
    num_cpus: 16
    num_gpus: 2
    memory: 128g
    runtime: PT8H

jobs:
  - name: preprocessing
    command: ./preprocess.sh
    resource_requirements: small

  - name: model_training
    command: python train.py
    resource_requirements: large

The job runner pulls jobs from the server by detecting its available resources automatically.

torc run $WORKFLOW_ID

The server's GET /workflows/{id}/claim_jobs_based_on_resources endpoint:

1. Receives the runner's resource capacity
2. Queries the ready queue for jobs that fit within those resources
3. Returns a set of jobs that can run concurrently without over-subscription
4. Updates job status from ready to pending atomically

Job Allocation Ambiguity: Two Approaches

When you have multiple compute nodes or schedulers with different capabilities, there are two ways to handle job allocation:

Approach 1: Sort Method (Flexible but Potentially Ambiguous)

How it works:

• Jobs do NOT specify a particular scheduler/compute node
• The server uses a job_sort_method parameter to prioritize jobs when allocating
• Any runner with sufficient resources can claim any ready job

Available sort methods: Define the field job_sort_method in the workflow specification file (YAML/JSON/KDL)

• gpus_runtime_memory - Prioritize jobs by GPU count (desc), then runtime (desc), then memory (desc)
• gpus_memory_runtime - Prioritize jobs by GPU count (desc), then memory (desc), then runtime (desc)
• none - No sorting, jobs selected in queue order

Tradeoffs:

✅ Advantages:

• Maximum flexibility - any runner can execute any compatible job
• Better resource utilization - if GPU runner is idle, it can pick up CPU-only jobs
• Simpler workflow specifications - no need to explicitly map jobs to schedulers
• Fault tolerance - if one runner fails, others can pick up its jobs

❌ Disadvantages:

• Ambiguity - no guarantee GPU jobs go to GPU runners
• Potential inefficiency - high-memory jobs might land on low-memory nodes if timing is unlucky
• Requires careful sort method selection
• Less predictable job placement

When to use:

• Homogeneous or mostly-homogeneous compute resources
• Workflows where job placement flexibility is valuable
• When you want runners to opportunistically pick up work
• Development and testing environments

Approach 2: Scheduler ID (Deterministic but Less Flexible)

How it works:

• Define scheduler configurations in your workflow spec
• Assign each job a specific scheduler_id
• Runners provide their scheduler_config_id when requesting jobs
• Server only returns jobs matching that scheduler ID

Example workflow specification:

slurm_schedulers:
  - name: gpu_cluster
    partition: gpu
    account: myproject

  - name: highmem_cluster
    partition: highmem
    account: myproject

jobs:
  - name: model_training
    command: python train.py
    resource_requirements: large
    slurm_scheduler: gpu_cluster     # Binds to specific scheduler

  - name: large_analysis
    command: ./analyze.sh
    resource_requirements: highmem
    slurm_scheduler: highmem_cluster

Example runner invocation:

# GPU runner - only pulls jobs assigned to gpu_cluster
torc-slurm-job-runner $WORKFLOW_ID \
  --scheduler-config-id 1 \
  --num-cpus 32 \
  --num-gpus 8

# High-memory runner - only pulls jobs assigned to highmem_cluster
torc-slurm-job-runner $WORKFLOW_ID \
  --scheduler-config-id 2 \
  --num-cpus 64 \
  --memory-gb 512

Tradeoffs:

✅ Advantages:

• Zero ambiguity - jobs always run on intended schedulers
• Predictable job placement
• Prevents GPU jobs from landing on CPU-only nodes
• Clear workflow specification - explicit job→scheduler mapping
• Better for heterogeneous clusters (GPU vs CPU vs high-memory)

❌ Disadvantages:

• Less flexibility - idle runners can't help other queues
• Potential resource underutilization - GPU runner sits idle while CPU queue is full
• More complex workflow specifications
• If a scheduler fails, its jobs remain stuck until that scheduler returns

When to use:

• Highly heterogeneous compute resources (GPU clusters, high-memory nodes, specialized hardware)
• Production workflows requiring predictable job placement
• Multi-cluster environments
• When job-resource matching is critical (e.g., GPU-only codes, specific hardware requirements)
• Slurm or HPC scheduler integrations

Choosing Between Sort Method and Scheduler ID

You can also mix approaches: Use scheduler_id for jobs with strict requirements, leave it NULL for flexible jobs.

Use Case 2: Queue-Depth Parallelism

This strategy is ideal for workflows with homogeneous resource requirements where you simply want to control the level of parallelism.

How It Works

Instead of tracking resources, you specify a maximum number of concurrent jobs:

torc run $WORKFLOW_ID \
  --max-parallel-jobs 10 \
  --output-dir ./results

or with Slurm:

torc slurm schedule-nodes $WORKFLOW_ID \
  --scheduler-config-id 1 \
  --num-hpc-jobs 4 \
  --max-parallel-jobs 8

Server behavior:

The GET /workflows/{id}/claim_next_jobs endpoint:

1. Accepts limit parameter specifying maximum jobs to return
2. Ignores all resource requirements
3. Returns the next N ready jobs from the queue
4. Updates their status from ready to pending

Runner behavior:

• Maintains a count of running jobs
• When count falls below max_parallel_jobs, requests more work
• Does NOT track CPU, memory, GPU, or other resources
• Simply enforces the concurrency limit

Ignoring Resource Consumption

This is a critical distinction: when using --max-parallel-jobs, the runner completely ignores current resource consumption.

Normal resource-aware mode:

Runner has: 32 CPUs, 128 GB memory
Job A needs: 16 CPUs, 64 GB
Job B needs: 16 CPUs, 64 GB
Job C needs: 16 CPUs, 64 GB

Runner starts Job A and Job B (resources fully allocated)
Job C waits until resources free up

Queue-depth mode with --max-parallel-jobs 3:

Runner has: 32 CPUs, 128 GB memory (IGNORED)
Job A needs: 16 CPUs, 64 GB (IGNORED)
Job B needs: 16 CPUs, 64 GB (IGNORED)
Job C needs: 16 CPUs, 64 GB (IGNORED)

Runner starts Job A, Job B, and Job C simultaneously
Total requested: 48 CPUs, 192 GB (exceeds node capacity!)
System may: swap, OOM, or throttle performance

When to Use Queue-Depth Parallelism

✅ Use queue-depth parallelism when:

1. All jobs have similar resource requirements

   # All jobs use ~4 CPUs, ~8GB memory
   jobs:
     - name: process_file_1
       command: ./process.sh file1.txt
     - name: process_file_2
       command: ./process.sh file2.txt
     # ... 100 similar jobs
   

2. Resource requirements are negligible compared to node capacity

   • Running 100 lightweight Python scripts on a 64-core machine
   • I/O-bound jobs that don't consume much CPU/memory

3. Jobs are I/O-bound or sleep frequently

   • Data download jobs
   • Jobs waiting on external services
   • Polling or monitoring tasks

4. You want simplicity over precision

   • Quick prototypes
   • Testing workflows
   • Simple task queues

5. Jobs self-limit their resource usage

   • Application has built-in thread pools
   • Container resource limits
   • OS-level cgroups or resource controls

❌ Avoid queue-depth parallelism when:

1. Jobs have heterogeneous resource requirements

   • Mix of 2-CPU and 32-CPU jobs
   • Some jobs need 4GB, others need 128GB

2. Resource contention causes failures

   • Out-of-memory errors
   • CPU thrashing
   • GPU memory exhaustion

3. You need efficient bin-packing

   • Maximizing node utilization
   • Complex resource constraints

4. Jobs are compute-intensive

   • CPU-bound numerical simulations
   • Large matrix operations
   • Video encoding

Queue-Depth Parallelism in Practice

Example 1: Slurm with Queue Depth

# Schedule 4 Slurm nodes, each running up to 8 concurrent jobs
torc slurm schedule-nodes $WORKFLOW_ID \
  --scheduler-config-id 1 \
  --num-hpc-jobs 4 \
  --max-parallel-jobs 8

This creates 4 Slurm job allocations. Each allocation runs a worker that:

• Pulls up to 8 jobs at a time
• Runs them concurrently
• Requests more when any job completes

Total concurrency: up to 32 jobs (4 nodes × 8 jobs/node)

Example 2: Local Runner with Queue Depth

# Run up to 20 jobs concurrently on local machine
torc-job-runner $WORKFLOW_ID \
  --max-parallel-jobs 20 \
  --output-dir ./output

Example 3: Mixed Approach

You can even run multiple runners with different strategies:

# Terminal 1: Resource-aware runner for large jobs
torc run $WORKFLOW_ID \
  --num-cpus 32 \
  --memory-gb 256

# Terminal 2: Queue-depth runner for small jobs
torc run $WORKFLOW_ID \
  --max-parallel-jobs 50

The ready queue serves both runners. The resource-aware runner gets large jobs that fit its capacity, while the queue-depth runner gets small jobs for fast parallel execution.

Performance Characteristics

Resource-aware allocation:

• Query complexity: O(jobs in ready queue)
• Requires computing resource sums
• Slightly slower due to filtering and sorting
• Better resource utilization

Queue-depth allocation:

• Query complexity: O(1) with limit
• Simple LIMIT clause, no resource computation
• Faster queries
• Simpler logic

For workflows with thousands of ready jobs, queue-depth allocation has lower overhead.

Best Practices

1. Start with resource-aware allocation for new workflows

   • Better default behavior
   • Prevents resource over-subscription
   • Easier to debug resource issues

2. Use scheduler_id for production multi-cluster workflows

   • Explicit job placement
   • Predictable resource usage
   • Better for heterogeneous resources

3. Use sort_method for flexible single-cluster workflows

   • Simpler specifications
   • Better resource utilization
   • Good for homogeneous resources

4. Use queue-depth parallelism for homogeneous task queues

   • Many similar jobs
   • I/O-bound workloads
   • When simplicity matters more than precision

5. Monitor resource usage when switching strategies

   • Check for over-subscription
   • Verify expected parallelism
   • Look for resource contention

6. Test with small workflows first

   • Validate job allocation behavior
   • Check resource accounting
   • Ensure jobs run on intended schedulers

Summary

Choose the strategy that best matches your workflow characteristics and execution environment. You can even mix strategies across different runners for maximum flexibility.

Workflow Reinitialization

When you modify input files or configuration after a workflow has run, you need a way to re-execute only the affected jobs. Reinitialization handles this by detecting what changed and marking the appropriate jobs for re-execution.

When to Use Reinitialization

Use torc workflows reinitialize when:

• Input files changed — You modified an input file and want dependent jobs to rerun
• Configuration updated — You changed user_data parameters
• Output files missing — Output files were deleted and need regeneration
• Job definition changed — You modified a job's command or other attributes
• Iterative development — You're refining a workflow and need quick iteration

Basic Usage

# Preview what would change (recommended first step)
torc workflows reinitialize <workflow_id> --dry-run

# Reinitialize the workflow
torc workflows reinitialize <workflow_id>

# Force reinitialization even with warnings
torc workflows reinitialize <workflow_id> --force

How Change Detection Works

Reinitialization detects changes through three mechanisms:

1. File Modification Times

For files tracked in the workflow, Torc compares the current st_mtime (modification time) against the stored value. If a file was modified since the last run, jobs that use it as input are marked for re-execution.

# Modify an input file
echo "new data" > input.json

# Reinitialize detects the change
torc workflows reinitialize <workflow_id>
# Output: Reset 3 jobs due to changed inputs

2. Job Attribute and User Data Hashing

Torc computes SHA256 hashes of critical job attributes (such as the command) and user_data input values. If any hash differs from the stored value, the job is marked for re-execution. This detects changes like modified commands, updated scripts, or changed configuration parameters.

3. Missing Output Files

If a job's output file no longer exists on disk, the job is marked for re-execution regardless of whether inputs changed.

The Reinitialization Process

When you run reinitialize, Torc performs these steps:

1. Bump run_id — Increments the workflow's run counter for tracking
2. Reset workflow status — Clears the previous run's completion state
3. Check file modifications — Compares current st_mtime values to stored values
4. Check missing outputs — Identifies jobs whose output files no longer exist
5. Check user_data changes — Computes and compares input hashes
6. Mark affected jobs — Sets jobs needing re-execution to uninitialized
7. Re-evaluate dependencies — Runs initialize_jobs to set jobs to ready or blocked

Dependency Propagation

When a job is marked for re-execution, all downstream jobs that depend on its outputs are also marked. This ensures the entire dependency chain is re-executed:

preprocess (input changed) → marked for rerun
    ↓
process (depends on preprocess output) → also marked
    ↓
postprocess (depends on process output) → also marked

Dry Run Mode

Always use --dry-run first to preview changes without modifying anything:

torc workflows reinitialize <workflow_id> --dry-run

Example output:

Dry run: 5 jobs would be reset due to changed inputs
  - preprocess
  - analyze_batch_1
  - analyze_batch_2
  - merge_results
  - generate_report

Retrying Failed Jobs

Important: Reinitialization does not automatically retry failed jobs. To retry failed jobs, use reset-status:

# Reset failed jobs to ready status, then reinitialize to check for other changes
torc workflows reset-status <workflow_id> --failed-only --reinitialize

# Or just reset failed jobs without reinitialization
torc workflows reset-status <workflow_id> --failed-only

Comparison with Full Reset

Working with Workflows

This section covers how to create, configure, and manage workflows.

• Creating Workflows - Getting started with workflow creation
• Workflow Specification Formats - JSON, YAML, and other formats
• Visualizing Workflow Structure - Viewing workflow graphs
• Exporting and Importing Workflows - Moving workflows between systems
• Archiving Workflows - Long-term workflow storage

How to Create Workflows

This guide shows different methods for creating Torc workflows, from the most common (specification files) to more advanced approaches (CLI, API).

Using Workflow Specification Files (Recommended)

The easiest way to create workflows is with specification files. Torc supports YAML, JSON5, and KDL formats.

Create from a YAML File

torc workflows create workflow.yaml

Create from JSON5 or KDL

torc workflows create workflow.json5
torc workflows create workflow.kdl

Torc detects the format from the file extension.

Create and Run in One Step

For quick iteration, combine creation and execution:

# Create and run locally
torc run workflow.yaml

# Create and submit to Slurm
torc submit workflow.yaml

For format syntax and examples, see the Workflow Specification Formats guide. For a complete reference of all fields, see the Workflow Specification Reference.

Using the CLI (Step by Step)

For programmatic workflow construction or when you need fine-grained control, create workflows piece by piece using the CLI.

Step 1: Create an Empty Workflow

torc workflows new \
  --name "my_workflow" \
  --description "My test workflow"

Output:

Successfully created workflow:
  ID: 1
  Name: my_workflow
  User: dthom
  Description: My test workflow

Note the workflow ID (1) for subsequent commands.

Step 2: Add Resource Requirements

torc resource-requirements create \
  --name "small" \
  --num-cpus 1 \
  --memory "1g" \
  --runtime "PT10M" \
  1  # workflow ID

Output:

Successfully created resource requirements:
  ID: 2
  Workflow ID: 1
  Name: small

Step 3: Add Files (Optional)

torc files create \
  --name "input_file" \
  --path "/data/input.txt" \
  1  # workflow ID

Step 4: Add Jobs

torc jobs create \
  --name "process_data" \
  --command "python process.py" \
  --resource-requirements-id 2 \
  --input-file-ids 1 \
  1  # workflow ID

Step 5: Initialize and Run

# Initialize the workflow (resolves dependencies)
torc workflows initialize-jobs 1

# Run the workflow
torc run 1

Using the Python API

For complex programmatic workflow construction, use the Python client:

from torc import make_api
from torc.openapi_client import (
    WorkflowModel,
    JobModel,
    ResourceRequirementsModel,
)

# Connect to the server
api = make_api("http://localhost:8080/torc-service/v1")

# Create workflow
workflow = api.create_workflow(WorkflowModel(
    name="my_workflow",
    user="myuser",
    description="Programmatically created workflow",
))

# Add resource requirements
rr = api.create_resource_requirements(ResourceRequirementsModel(
    workflow_id=workflow.id,
    name="small",
    num_cpus=1,
    memory="1g",
    runtime="PT10M",
))

# Add jobs
api.create_job(JobModel(
    workflow_id=workflow.id,
    name="job1",
    command="echo 'Hello World'",
    resource_requirements_id=rr.id,
))

print(f"Created workflow {workflow.id}")

For more details, see the Map Python Functions tutorial.

Using the Julia API

The Julia client provides similar functionality for programmatic workflow construction:

using Torc
import Torc: APIClient

# Connect to the server
api = make_api("http://localhost:8080/torc-service/v1")

# Create workflow
workflow = send_api_command(
    api,
    APIClient.create_workflow,
    APIClient.WorkflowModel(;
        name = "my_workflow",
        user = get_user(),
        description = "Programmatically created workflow",
    ),
)

# Add resource requirements
rr = send_api_command(
    api,
    APIClient.create_resource_requirements,
    APIClient.ResourceRequirementsModel(;
        workflow_id = workflow.id,
        name = "small",
        num_cpus = 1,
        memory = "1g",
        runtime = "PT10M",
    ),
)

# Add jobs
send_api_command(
    api,
    APIClient.create_job,
    APIClient.JobModel(;
        workflow_id = workflow.id,
        name = "job1",
        command = "echo 'Hello World'",
        resource_requirements_id = rr.id,
    ),
)

println("Created workflow $(workflow.id)")

The Julia client also supports map_function_to_jobs for mapping a function across parameters, similar to the Python client.

Choosing a Method

Common Tasks

Validate a Workflow File Without Creating

Use --dry-run to validate a workflow specification without creating it on the server:

torc workflows create --dry-run workflow.yaml

Example output:

Workflow Validation Results
===========================

Workflow: my_workflow
Description: A sample workflow

Components to be created:
  Jobs: 100 (expanded from 1 parameterized job specs)
  Files: 5
  User data records: 2
  Resource requirements: 2
  Slurm schedulers: 2
  Workflow actions: 3

Submission: Ready for scheduler submission (has on_workflow_start schedule_nodes action)

Validation: PASSED

For programmatic use (e.g., in scripts or the dashboard), get JSON output:

torc -f json workflows create --dry-run workflow.yaml

What Validation Checks

The dry-run performs comprehensive validation:

Structural Checks:

• Valid file format (YAML, JSON5, KDL, or JSON)
• Required fields present
• Parameter expansion (shows expanded job count vs. original spec count)

Reference Validation:

• depends_on references existing jobs
• depends_on_regexes patterns are valid and match at least one job
• resource_requirements references exist
• scheduler references exist
• input_files and output_files reference defined files
• input_user_data and output_user_data reference defined user data
• All regex patterns (*_regexes fields) are valid

Duplicate Detection:

• Duplicate job names
• Duplicate file names
• Duplicate user data names
• Duplicate resource requirement names
• Duplicate scheduler names

Dependency Analysis:

• Circular dependency detection (reports all jobs in the cycle)

Action Validation:

• Actions reference existing jobs and schedulers
• schedule_nodes actions have required scheduler and scheduler_type

Scheduler Configuration:

• Slurm scheduler node requirements are valid
• Warns about heterogeneous schedulers without jobs_sort_method (see below)

Heterogeneous Scheduler Warning

When you have multiple Slurm schedulers with different resource profiles (memory, GPUs, walltime, partition) and jobs without explicit scheduler assignments, the validation warns about potential suboptimal job-to-node matching:

Warnings (1):
  - Workflow has 3 schedulers with different memory (mem), walltime but 10 job(s)
    have no explicit scheduler assignment and jobs_sort_method is not set. The
    default sort method 'gpus_runtime_memory' will be used (jobs sorted by GPUs,
    then runtime, then memory). If this doesn't match your workload, consider
    setting jobs_sort_method explicitly to 'gpus_memory_runtime' (prioritize
    memory over runtime) or 'none' (no sorting).

This warning helps you avoid situations where:

• Long-walltime nodes pull short-runtime jobs
• High-memory nodes pull low-memory jobs
• GPU nodes pull non-GPU jobs

Solutions:

1. Set jobs_sort_method explicitly in your workflow spec
2. Assign jobs to specific schedulers using the scheduler field on each job
3. Accept the default gpus_runtime_memory sorting if it matches your workload

Bypassing Validation

To create a workflow despite validation warnings:

torc workflows create --skip-checks workflow.yaml

Note: This bypasses scheduler node validation checks (which are treated as errors), but does not bypass all errors. Errors such as missing references or circular dependencies will always prevent creation.

List Available Workflows

torc workflows list

Delete a Workflow

torc workflows delete <workflow_id>

View Workflow Details

torc workflows get <workflow_id>

Defining File Dependencies

Jobs often need to read input files and produce output files. Torc can automatically infer job dependencies from these file relationships using variable substitution:

files:
  - name: raw_data
    path: /data/raw.csv
  - name: processed_data
    path: /data/processed.csv

jobs:
  - name: preprocess
    command: "python preprocess.py -o ${files.output.raw_data}"

  - name: analyze
    command: "python analyze.py -i ${files.input.raw_data} -o ${files.output.processed_data}"

Key concepts:

• ${files.input.NAME} - References a file this job reads (creates a dependency on the job that outputs it)
• ${files.output.NAME} - References a file this job writes (satisfies dependencies for downstream jobs)

In the example above, analyze automatically depends on preprocess because it needs raw_data as input, which preprocess produces as output.

For a complete walkthrough, see Tutorial: Diamond Workflow.

Next Steps

• Tutorial: Diamond Workflow - Learn file-based dependencies with the fan-out/fan-in pattern
• Workflow Specification Formats - Detailed format reference
• Workflow Specification Reference - Complete field reference for all data models
• Job Parameterization - Generate multiple jobs from templates
• Tutorial: Many Independent Jobs - Your first workflow

Workflow Specification Formats

Torc supports three workflow specification formats: YAML, JSON5, and KDL. All formats provide the same functionality with different syntaxes to suit different preferences and use cases.

Format Overview

YAML Format

Best for: Most workflows, especially those using multi-line commands.

File Extension: .yaml or .yml

Example:

name: data_processing_workflow
user: datauser
description: Multi-stage data processing pipeline

# File definitions
files:
  - name: raw_data
    path: /data/input/raw_data.csv
  - name: processed_data
    path: /data/output/processed_data.csv

# Resource requirements
resource_requirements:
  - name: small_job
    num_cpus: 2
    num_gpus: 0
    num_nodes: 1
    memory: 4g
    runtime: PT30M

# Jobs
jobs:
  - name: download_data
    command: wget https://example.com/data.csv -O ${files.output.raw_data}
    resource_requirements: small_job

  - name: process_data
    command: python process.py ${files.input.raw_data} -o ${files.output.processed_data}
    resource_requirements: small_job
    depends_on:
      - download_data

# Workflow actions
actions:
  - trigger_type: on_workflow_start
    action_type: run_commands
    commands:
      - mkdir -p /data/input /data/output
      - echo "Workflow started"

Advantages:

• Most widely used configuration format
• Excellent for complex workflows with many jobs
• Clean, readable syntax without brackets

Disadvantages:

• Indentation-sensitive
• Can be verbose for deeply nested structures

JSON5 Format

Best for: Programmatic workflow generation and JSON compatibility.

File Extension: .json5

Example:

{
  name: "data_processing_workflow",
  user: "datauser",
  description: "Multi-stage data processing pipeline",

  // File definitions
  files: [
    {name: "raw_data", path: "/data/input/raw_data.csv"},
    {name: "processed_data", path: "/data/output/processed_data.csv"},
  ],

  // Resource requirements
  resource_requirements: [
    {
      name: "small_job",
      num_cpus: 2,
      num_gpus: 0,
      num_nodes: 1,
      memory: "4g",
      runtime: "PT30M",
    },
  ],

  // Jobs
  jobs: [
    {
      name: "download_data",
      command: "wget https://example.com/data.csv -O ${files.output.raw_data}",
      resource_requirements: "small_job",
    },
    {
      name: "process_data",
      command: "python process.py ${files.input.raw_data} -o ${files.output.processed_data}",
      resource_requirements: "small_job",
      depends_on: ["download_data"],
    },
  ],

  // Workflow actions
  actions: [
    {
      trigger_type: "on_workflow_start",
      action_type: "run_commands",
      commands: [
        "mkdir -p /data/input /data/output",
        "echo 'Workflow started'",
      ],
    },
  ],
}

Advantages:

• JSON-compatible (easy programmatic manipulation)
• Supports comments and trailing commas
• Familiar to JavaScript/JSON users

Disadvantages:

• More verbose than YAML
• More brackets and commas than YAML

KDL Format

Best for: Simple to moderate workflows with clean syntax.

File Extension: .kdl

Example:

name "data_processing_workflow"
user "datauser"
description "Multi-stage data processing pipeline"

// File definitions
file "raw_data" path="/data/input/raw_data.csv"
file "processed_data" path="/data/output/processed_data.csv"

// Resource requirements
resource_requirements "small_job" {
    num_cpus 2
    num_gpus 0
    num_nodes 1
    memory "4g"
    runtime "PT30M"
}

// Jobs
job "download_data" {
    command "wget https://example.com/data.csv -O ${files.output.raw_data}"
    resource_requirements "small_job"
}

job "process_data" {
    command "python process.py ${files.input.raw_data} -o ${files.output.processed_data}"
    resource_requirements "small_job"
    depends_on_job "download_data"
}

// Workflow actions
action {
    trigger_type "on_workflow_start"
    action_type "run_commands"
    command "mkdir -p /data/input /data/output"
    command "echo 'Workflow started'"
}

Advantages:

• Clean, minimal syntax
• No indentation requirements
• Supports all core Torc features

Disadvantages:

• Less familiar to most users
• Boolean values use special syntax (#true, #false)

KDL-Specific Syntax Notes

1. Boolean values: Use #true and #false (not true or false)

   resource_monitor {
       enabled #true
       generate_plots #false
   }
   

2. Repeated child nodes: Use multiple statements

   action {
       command "echo 'First command'"
       command "echo 'Second command'"
   }
   

3. User data: Requires child nodes for properties

   user_data "metadata" {
       is_ephemeral #true
       data "{\"key\": \"value\"}"
   }
   

Common Features Across All Formats

Variable Substitution

All formats support the same variable substitution syntax:

• ${files.input.NAME} - Input file path
• ${files.output.NAME} - Output file path
• ${user_data.input.NAME} - Input user data
• ${user_data.output.NAME} - Output user data

Supported Fields

All formats support:

• Workflow metadata: name, user, description
• Jobs: name, command, dependencies, resource requirements
• Files: name, path, modification time
• User data: name, data (JSON), ephemeral flag
• Resource requirements: CPUs, GPUs, memory, runtime
• Slurm schedulers: account, partition, walltime, etc.
• Workflow actions: triggers, action types, commands
• Resource monitoring: enabled, granularity, sampling interval

Examples Directory

The Torc repository includes comprehensive examples in all three formats:

examples/
├── yaml/     # All workflows (15 examples)
├── json/     # All workflows (15 examples)
└── kdl/      # Non-parameterized workflows (9 examples)

Compare the same workflow in different formats to choose your preference:

• sample_workflow.yaml
• sample_workflow.json5
• sample_workflow.kdl

See the examples directory for the complete collection.

Creating Workflows

All formats use the same command:

torc workflows create examples/yaml/sample_workflow.yaml
torc workflows create examples/json/sample_workflow.json5
torc workflows create examples/kdl/sample_workflow.kdl

Or use the quick execution commands:

# Create and run locally
torc run examples/yaml/sample_workflow.yaml

# Create and submit to scheduler
torc submit examples/yaml/workflow_actions_data_pipeline.yaml

Recommendations

Start with YAML if you're unsure.

Switch to JSON5 if you need to programmatically generate workflows or prefer JSON syntax.

Try KDL if you prefer minimal syntax.

All three formats are fully supported and maintained. Choose based on your workflow complexity and personal preference.

Visualizing Workflow Structure

Understanding how your workflow will execute—which jobs run in parallel, how dependencies create stages, and when Slurm allocations are requested—is essential for debugging and optimization. Torc provides several tools for visualizing workflow structure.

Execution Plan Command

The torc workflows execution-plan command analyzes a workflow and displays its execution stages, showing how jobs are grouped and when schedulers allocate resources.

Basic Usage

# From a specification file
torc workflows execution-plan workflow.yaml

# From an existing workflow
torc workflows execution-plan <workflow_id>

Example Output

For a workflow with two independent processing pipelines that merge at the end:

Workflow: two_subgraph_pipeline
Total Jobs: 15

▶ Stage 1: Workflow Start
  Scheduler Allocations:
    • prep_sched (slurm) - 1 allocation(s)
  Jobs Becoming Ready:
    • prep_a
    • prep_b

→ Stage 2: When jobs 'prep_a', 'prep_b' complete
  Scheduler Allocations:
    • work_a_sched (slurm) - 1 allocation(s)
    • work_b_sched (slurm) - 1 allocation(s)
  Jobs Becoming Ready:
    • work_a_{1..5}
    • work_b_{1..5}

→ Stage 3: When 10 jobs complete
  Scheduler Allocations:
    • post_a_sched (slurm) - 1 allocation(s)
    • post_b_sched (slurm) - 1 allocation(s)
  Jobs Becoming Ready:
    • post_a
    • post_b

→ Stage 4: When jobs 'post_a', 'post_b' complete
  Scheduler Allocations:
    • final_sched (slurm) - 1 allocation(s)
  Jobs Becoming Ready:
    • final

Total Stages: 4

What the Execution Plan Shows

1. Stages: Groups of jobs that become ready at the same time based on dependency resolution
2. Scheduler Allocations: Which Slurm schedulers request resources at each stage (for workflows with Slurm configuration)
3. Jobs Becoming Ready: Which jobs transition to "ready" status at each stage
4. Subgraphs: Independent branches of the workflow that can execute in parallel

Workflows Without Slurm Schedulers

For workflows without pre-defined Slurm schedulers, the execution plan shows the job stages without scheduler information:

torc workflows execution-plan workflow_no_slurm.yaml

Workflow: my_pipeline
Total Jobs: 10

▶ Stage 1: Workflow Start
  Jobs Becoming Ready:
    • preprocess

→ Stage 2: When job 'preprocess' completes
  Jobs Becoming Ready:
    • work_{1..5}

→ Stage 3: When 5 jobs complete
  Jobs Becoming Ready:
    • postprocess

Total Stages: 3

This helps you understand the workflow topology before adding Slurm configuration with torc slurm generate.

Use Cases

• Validate workflow structure: Ensure dependencies create the expected execution order
• Identify parallelism: See which jobs can run concurrently
• Debug slow workflows: Find stages that serialize unnecessarily
• Plan Slurm allocations: Understand when resources will be requested
• Verify auto-generated schedulers: Check that torc slurm generate created appropriate staging

DAG Visualization in the Dashboard

The web dashboard provides interactive DAG (Directed Acyclic Graph) visualization.

Viewing the DAG

1. Navigate to the Details tab
2. Select a workflow
3. Click View DAG in the Visualization section

DAG Types

The dashboard supports three DAG visualization types:

DAG Features

• Color-coded nodes: Jobs are colored by status (ready, running, completed, failed, etc.)
• Interactive: Zoom, pan, and click nodes for details
• Layout: Automatic hierarchical layout using Dagre algorithm
• Legend: Status color reference

TUI DAG View

The terminal UI (torc tui) also includes DAG visualization:

1. Select a workflow
2. Press d to toggle the DAG view
3. Use arrow keys to navigate

Comparing Visualization Tools

Example: Analyzing a Complex Workflow

Consider a workflow with preprocessing, parallel work, and aggregation:

# First, view the execution plan
torc workflows execution-plan examples/subgraphs/subgraphs_workflow.yaml

# If no schedulers, generate them
torc slurm generate --account myproject examples/subgraphs/subgraphs_workflow_no_slurm.yaml

# View the plan again to see scheduler allocations
torc workflows execution-plan examples/subgraphs/subgraphs_workflow.yaml

The execution plan helps you verify that:

• Independent subgraphs are correctly identified
• Stages align with your expected execution order
• Slurm allocations are timed appropriately

See Also

• Web Dashboard — Full dashboard documentation
• Slurm Workflows — Understanding Slurm integration
• Workflow Actions — How actions trigger scheduler allocations
• Subgraphs Example — Complete example with multiple subgraphs

How to Export and Import Workflows

This guide shows how to export workflows to portable JSON files and import them into the same or different Torc servers. This is useful for:

• Backup and restore: Save workflow definitions for disaster recovery
• Migration: Move workflows between development, staging, and production environments
• Sharing: Share workflow templates with teammates or the community
• Duplication: Create copies of workflows for testing or experimentation

Exporting Workflows

Basic Export

Export a workflow to a JSON file:

torc workflows export 123 --output my_workflow.json

This creates a self-contained JSON document containing:

• Workflow metadata
• All jobs with their dependencies
• Files and user data
• Resource requirements
• Slurm and local schedulers
• Workflow actions

Export with Results

Include job results (stdout, stderr, return codes) in the export:

torc workflows export 123 --output my_workflow.json --include-results

Export with Events

Include workflow events (job status changes, scheduler events):

torc workflows export 123 --output my_workflow.json --include-events

Export with Everything

Include both results and events:

torc workflows export 123 --output my_workflow.json --include-results --include-events

Export to Stdout

Omit --output to write to stdout (useful for piping):

torc workflows export 123 > my_workflow.json

JSON Output Format

Use --format json for machine-readable output with export statistics:

torc workflows export 123 --output my_workflow.json --format json

Output:

{
  "success": true,
  "workflow_id": 123,
  "workflow_name": "my_workflow",
  "output_file": "my_workflow.json",
  "jobs": 5,
  "files": 3,
  "user_data": 2,
  "results": 0,
  "events": 0
}

Importing Workflows

Basic Import

Import a workflow from a JSON file:

torc workflows import my_workflow.json

Output:

Successfully imported workflow:
  Workflow ID: 456
  Name: my_workflow
  Jobs: 5
  Files: 3
  User data: 2

Import with Custom Name

Override the workflow name during import:

torc workflows import my_workflow.json --name "new_workflow_name"

Skip Results During Import

If the export includes results but you don't want to import them:

torc workflows import my_workflow.json --skip-results

Skip Events During Import

If the export includes events but you don't want to import them:

torc workflows import my_workflow.json --skip-events

JSON Output Format

Use --format json for machine-readable output:

torc workflows import my_workflow.json --format json

Output:

{
  "success": true,
  "workflow_id": 456,
  "workflow_name": "my_workflow",
  "jobs": 5,
  "files": 3,
  "user_data": 2
}

How Import Works

ID Remapping

When importing, all entity IDs are remapped to new IDs assigned by the target server. This ensures no conflicts with existing workflows. Cross-references between entities (e.g., job dependencies on files) are automatically updated to use the new IDs.

Job Status Reset

Imported jobs always start in the uninitialized status, regardless of their status in the exported file. After import, you need to initialize and run the workflow:

# Initialize the imported workflow
torc workflows initialize 456

# Run locally
torc workflows run 456

# Or submit to scheduler
torc workflows submit 456

Default Resource Requirements

Each workflow automatically gets a "default" resource requirements entry. During import, the exported "default" resource requirements are mapped to the new workflow's default entry.

Export Format

The export format is a versioned JSON document. Here's the structure:

{
  "export_version": "1.0",
  "exported_at": "2024-01-15T10:30:00Z",
  "workflow": { ... },
  "files": [ ... ],
  "user_data": [ ... ],
  "resource_requirements": [ ... ],
  "slurm_schedulers": [ ... ],
  "local_schedulers": [ ... ],
  "jobs": [ ... ],
  "workflow_actions": [ ... ],
  "results": [ ... ],
  "events": [ ... ]
}

The results and events fields are only present when --include-results or --include-events are specified.

Common Workflows

Backup All Active Workflows

for id in $(torc workflows list --format json | jq -r '.items[].id'); do
  torc workflows export $id --output "backup_workflow_${id}.json"
done

Migrate to Another Server

# On source server
torc workflows export 123 --output workflow.json

# On target server (different TORC_API_URL)
export TORC_API_URL="http://new-server:8080/torc-service/v1"
torc workflows import workflow.json

Clone a Workflow for Testing

# Export existing workflow
torc workflows export 123 --output original.json

# Import as a new workflow with different name
torc workflows import original.json --name "test_copy"

Troubleshooting

Import Fails with "File not found"

Ensure the export file exists and the path is correct:

ls -la my_workflow.json
torc workflows import ./my_workflow.json

Import Fails with API Error

Check that:

1. The Torc server is running and accessible
2. You have permission to create workflows
3. The export file is valid JSON (not corrupted)

Validate the export file:

python -m json.tool my_workflow.json > /dev/null && echo "Valid JSON"

Jobs Not Running After Import

Imported jobs start in uninitialized status. You must initialize the workflow:

torc workflows initialize 456

Then check job status:

torc jobs list 456

Jobs should now show ready or blocked status depending on their dependencies.

Archiving Workflows

Workflow archiving provides a way to hide completed or inactive workflows from default list views while preserving all workflow data and execution history. Archived workflows remain fully accessible but don't clutter everyday workflow management operations.

Purpose and Motivation

As projects mature and accumulate workflows over time, the list of active workflows can become difficult to navigate. Archiving addresses this by:

• Reducing visual clutter - Completed workflows no longer appear in default list views
• Preserving historical data - All workflow data, jobs, results, and logs remain accessible
• Improving usability - Users can focus on active workflows without losing access to past work
• Maintaining audit trails - Archived workflows can be retrieved for analysis, debugging, or compliance

Archiving is particularly useful for:

• Completed experiments that may need future reference
• Successful production runs that serve as historical records
• Development workflows that are no longer active but contain valuable examples
• Workflows from completed projects that need to be retained for documentation

How It Works

When you archive a workflow, it's marked with an "archived" flag. This flag controls whether the workflow appears in default list views:

• Active workflows (not archived): Appear in standard workflows list commands
• Archived workflows: Hidden from default lists but accessible with the --archived-only flag

The archive status is just metadata - it doesn't affect the workflow's data, results, or any other functionality.

Archiving Workflows

Use the workflows archive command to archive or unarchive workflows:

# Archive a specific workflow
torc workflows archive true <workflow_id>

# Archive multiple workflows
torc workflows archive true 123 456 789

# Interactive selection (prompts user to choose)
torc workflows archive true

# With JSON output
torc --format json workflows archive true <workflow_id>

The command will output confirmation messages:

Successfully archived workflow 123
Successfully archived workflow 456
Successfully archived workflow 789

Unarchiving Workflows

To restore an archived workflow to active status, use the same command with false:

# Unarchive a specific workflow
torc workflows archive false <workflow_id>

# Unarchive multiple workflows
torc workflows archive false 123 456 789

# Interactive selection
torc workflows archive false

Output:

Successfully unarchived workflow 123

Viewing Workflows

Default Behavior

By default, the workflows list command shows only non-archived workflows:

# Shows active (non-archived) workflows only
torc workflows list

# Shows active workflows for a specific user
torc workflows list --user alice

Viewing Archived Workflows

Use the --archived-only flag to see archived workflows:

# List only archived workflows for current user
torc workflows list --archived-only

Viewing All Workflows

Use the --include-archived flag to see all workflows:

torc workflows list --include-archived

Accessing Specific Workflows

You can always access a workflow directly by its ID, regardless of archive status:

# Get details of any workflow (archived or not)
torc workflows get <workflow_id>

# Check workflow status
torc workflows status <workflow_id>

Impact on Workflow Operations

Operations Restricted on Archived Workflows

Certain workflow operations are not allowed on archived workflows to prevent accidental modifications:

• ❌ Status reset: Cannot use workflows reset-status on archived workflows
  • Error message: "Cannot reset archived workflow status. Unarchive the workflow first."
  • To reset status, unarchive the workflow first, then reset

Interactive Selection Behavior

When commands prompt for interactive workflow selection (when workflow ID is not specified), archived workflows are excluded by default:

# These will NOT show archived workflows in the interactive menu
torc-client workflows delete
torc-client workflows status
torc-client workflows initialize

This prevents accidentally operating on archived workflows while still allowing explicit access by ID.

Archive vs. Delete

Understanding when to archive versus delete workflows:

Archive when:

• Workflow completed successfully and may need future reference
• Results should be preserved for reproducibility or compliance
• Workflow represents a milestone or important historical run
• You want to declutter lists but maintain data integrity

Delete when:

• Workflow failed and results are not useful
• Workflow was created for testing purposes only
• Data is no longer needed and storage space is a concern
• Workflow contains errors that would confuse future users

Common Use Cases

Completed Experiments

After completing an experiment and validating results:

# Archive the completed experiment
torc-client workflows archive true 123

# Later, if you need to reference it
torc-client workflows get 123
torc-client results list 123

Development Cleanup

Clean up development workflows while preserving examples:

# Delete test workflows
torc-client workflows delete 301 302 303

# Archive useful development examples
torc-client workflows archive true 304 305

Periodic Maintenance

Regularly archive old workflows to keep lists manageable:

# List workflows, identify completed ones
torc-client workflows list

# Archive workflows from completed projects
torc workflows archive true 401 402 403 404 405

Best Practices

When to Archive

1. After successful completion - Archive workflows once they've completed successfully and been validated
2. Project milestones - Archive workflows representing project phases or releases
3. Regular cleanup - Establish periodic archiving of workflows older than a certain timeframe
4. Before major changes - Archive working versions before making significant modifications

Summary

Workflow archiving provides a simple, reversible way to hide completed or inactive workflows from default views while preserving all data and functionality. It's designed for long-term workflow management in active projects where historical data is valuable but visual clutter is undesirable.

Key points:

• Archive workflows with: torc workflows archive true <id>
• Unarchive workflows with: torc workflows archive false <id>
• Archived workflows are hidden from default lists but remain fully functional
• View archived workflows with: torc workflows list --archived-only
• Archiving is reversible and does not affect data storage
• Use archiving for completed workflows; use deletion for unwanted data

How-Tos

Step-by-step guides for common tasks.

• Track Workflow Status - Monitoring workflow progress
• Cancel a Workflow - Stopping running workflows
• View Job Logs - Accessing job output
• Debug a Failed Job - Troubleshooting failures
• Check Resource Utilization - Monitoring resource usage
• View Resource Plots - Visualizing resource metrics
• Parameterize Jobs with Files - Using file-based parameters

How to Track Workflow Status

Monitor a running workflow's progress using the CLI, TUI, or dashboard.

Quick Status Check

torc reports summary <workflow_id>

Example output:

Workflow 42: data_pipeline

Jobs by Status:
  Completed:  45
  Running:     5
  Ready:      10
  Blocked:    40

Continuous Monitoring

Watch status update every 10 seconds:

watch -n 10 torc reports summary <workflow_id>

Interactive TUI

Launch the terminal UI for a visual dashboard:

torc tui

The TUI shows:

• Job status breakdown with progress bars
• Running job details
• Failed job information
• Real-time updates

List Individual Jobs

View job-level status:

# All jobs
torc jobs list <workflow_id>

# Filter by status
torc jobs list <workflow_id> --status running
torc jobs list <workflow_id> --status failed

Check Completion

Verify if a workflow has finished:

torc workflows is-complete <workflow_id>

For scripting:

if torc -f json workflows is-complete "$WORKFLOW_ID" | jq -e '.is_complete' > /dev/null; then
    echo "Workflow complete"
fi

See Also

• Terminal UI (TUI) — Interactive monitoring
• Web Dashboard — Visual workflow management
• Workflow Reports — Generate summary reports

How to Cancel a Workflow

Stop a running workflow and terminate its jobs.

Cancel a Workflow

torc workflows cancel <workflow_id>

This:

• Marks the workflow as canceled
• Stops claiming new jobs
• Sends SIGKILL to all running processes
• Sends scancel to all active or pending Slurm allocations

Check Cancellation Status

Verify the workflow was canceled:

torc workflows status <workflow_id>

Or check completion status:

torc workflows is-complete <workflow_id>

Output:

Workflow 42 completion status:
  Is Complete: true
  Is Canceled: true

Restart After Cancellation

To resume a canceled workflow:

# Reinitialize to reset canceled jobs
torc workflows reinitialize <workflow_id>

# Run again
torc workflows run <workflow_id>
torc workflows submit <workflow_id>

Jobs that completed before cancellation remain completed.

See Also

• Track Workflow Status — Monitor workflow progress
• Workflow Reinitialization — Resume after issues

How to View Job Logs

Find and read the stdout/stderr output from job execution.

Find Log File Paths

torc reports results <workflow_id>
torc reports results <workflow_id> --job-id 15

Output includes:

{
  "job_id": 15,
  "job_name": "work_2",
  "status": "Failed",
  "run_id": 1,
  "return_code": 137,
  "completion_time": "2026-01-06T20:30:00.200Z",
  "exec_time_minutes": 0.09313333333333332,
  "compute_node_id": 47,
  "job_stdout": "output/job_stdio/job_wf43_j15_r1_a1.o",
  "job_stderr": "output/job_stdio/job_wf43_j15_r1_a1.e",
  "compute_node_type": "slurm"
},

Read Logs Directly

Once you have the path, view the logs:

# View stdout
cat output/job_stdio/job_wf43_j15_r1_a1.o

# View stderr
cat output/job_stdio/job_wf43_j15_r1_a1.e

# Follow logs in real-time (for running jobs)
tail -f output/job_stdio/job_wf43_j15_r1_a1.*

Default Log Location

By default, logs are stored in the output directory:

output/
└── job_stdio/
    ├── job_wf<workflow_id>_j<job_id>_r<run_id>_a<attempt_id>.o
    ├── job_wf<workflow_id>_j<job_id>_r<run_id>_a<attempt_id>.3

The output directory can be configured via the run/submit CLI options.

View Logs for Failed Jobs

Quickly find logs for failed jobs:

# Get failed job IDs
torc jobs list <workflow_id> --status failed

# Then view each job's logs
torc reports results <workflow_id> --job-id <failed_job_id>

View Logs in TUI or Dashboard

You can also view job logs interactively:

• TUI — Run torc tui and select a job to view its stdout/stderr in the interface. See Terminal UI.
• Dashboard — The web dashboard displays job logs when you click on a job. See Web Dashboard.

See Also

• Working with Logs — Log configuration and management
• Debug a Failed Job — Full debugging workflow

How to Debug a Failed Job

Systematically diagnose why a job failed.

Step 1: Identify the Failed Job

torc jobs list <workflow_id> --status failed

Note the job ID and name.

Step 2: Check the Exit Code

torc results get <workflow_id> --job-id <job_id>

Common exit codes:

Step 3: Read the Logs

# Get log paths
torc reports results <workflow_id> --job-id <job_id>

# View stderr (usually contains error messages)
cat output/job_stdio/job_wf43_j15_r1_a1.e

# View stdout
cat output/job_stdio/job_wf43_j15_r1_a1.o

Step 4: Check Resource Usage

Did the job exceed its resource limits?

torc reports check-resource-utilization <workflow_id>

Look for:

• Memory exceeded — Job was likely OOM-killed (exit code 137)
• Runtime exceeded — Job was terminated for running too long

Step 5: Reproduce Locally

Get the exact command that was run:

torc jobs get <job_id>

Try running it manually to see the error:

# Copy the command from the output and run it
python process.py --input data.csv

Common Fixes

Step 6: Fix and Retry

After fixing the issue:

# Reinitialize to reset failed jobs
torc workflows reset-status --failed --reinitialize <workflow_id>

# Run again
torc workflows run <workflow_id>
torc submit-slurm <workflow_id>

See Also

• View Job Logs — Finding log files
• Check Resource Utilization — Resource analysis
• Debugging Workflows — Comprehensive debugging guide

How to Check Resource Utilization

Compare actual resource usage against specified requirements to identify jobs that exceeded their limits.

Quick Start

torc reports check-resource-utilization <workflow_id>

Example output:

⚠ Found 2 resource over-utilization violations:

Job ID | Job Name    | Resource | Specified | Peak Used | Over-Utilization
-------|-------------|----------|-----------|-----------|------------------
15     | train_model | Memory   | 8.00 GB   | 10.50 GB  | +31.3%
15     | train_model | Runtime  | 2h 0m 0s  | 2h 45m 0s | +37.5%

Show All Jobs

Include jobs that stayed within limits:

torc reports check-resource-utilization <workflow_id> --all

Check a Specific Run

For workflows that have been reinitialized multiple times:

torc reports check-resource-utilization <workflow_id> --run-id 2

Adjusting Requirements

When jobs exceed their limits, update your workflow specification with a buffer:

resource_requirements:
  - name: training
    memory: 12g       # 10.5 GB peak + 15% buffer
    runtime: PT3H     # 2h 45m actual + buffer

Guidelines:

• Memory: Add 10-20% above peak usage
• Runtime: Add 15-30% above actual duration
• CPU: Round up to next core count

See Also

• Resource Monitoring — Enable and configure monitoring
• Resource Requirements Reference — Specification format

How to View Resource Utilization Plots

Generate interactive visualizations of CPU and memory usage over time.

Prerequisites

Enable time series monitoring in your workflow specification:

resource_monitor:
  granularity: "time_series"
  sample_interval_seconds: 2

This creates a SQLite database with resource samples at regular intervals.

Generate Plots

After your workflow completes, generate plots from the collected data:

torc plot-resources output/resource_utilization/resource_metrics_*.db -o plots/

This creates:

• Individual job plots — CPU, memory, and process count over time for each job
• Overview plots — Comparison across all jobs
• Summary dashboard — Bar charts of peak and average usage

Plot Specific Jobs

Generate plots for only certain jobs:

torc plot-resources output/resource_utilization/resource_metrics_*.db \
  -o plots/ \
  --job-ids 15,16

View the Plots

Open the generated HTML files in your browser:

open plots/job_15_resources.html

Query Data Directly

For custom analysis, query the SQLite database:

sqlite3 -table output/resource_utilization/resource_metrics_1_1.db

-- View samples for a specific job
SELECT timestamp, cpu_percent, memory_bytes
FROM job_resource_samples
WHERE job_id = 1
ORDER BY timestamp;

See Also

• Resource Monitoring — Configuration options
• Resource Monitoring Database — Database schema reference

How to Parameterize Jobs with Files

Process multiple input files by combining parameterization with file path templating.

Basic Pattern

Use a parameter to generate jobs for each file:

name: process_files

jobs:
  - name: process_{dataset}
    command: python process.py --input data/{dataset}.csv --output results/{dataset}.json
    parameters:
      dataset: "[train, test, validation]"

This creates 3 jobs:

• process_train → processes data/train.csv
• process_test → processes data/test.csv
• process_validation → processes data/validation.csv

With File Dependencies

Combine parameterization with explicit file definitions for dependency tracking:

name: file_pipeline

files:
  - name: raw_{dataset}
    path: data/{dataset}.csv
  - name: processed_{dataset}
    path: results/{dataset}.json

jobs:
  - name: process_{dataset}
    command: python process.py -i ${files.input.raw_{dataset}} -o ${files.output.processed_{dataset}}
    parameters:
      dataset: "[train, test, validation]"

  - name: aggregate
    command: python aggregate.py --input results/ --output summary.json
    depends_on:
      - process_{dataset}
    parameters:
      dataset: "[train, test, validation]"

The aggregate job automatically waits for all process_* jobs to complete.

Processing Numbered Files

Use range syntax for numbered file sequences:

jobs:
  - name: convert_{i:03d}
    command: ffmpeg -i video_{i:03d}.mp4 -o audio_{i:03d}.mp3
    parameters:
      i: "1:100"

Creates jobs for video_001.mp4 through video_100.mp4.

Multi-Dimensional Sweeps

Combine multiple parameters for Cartesian product expansion:

jobs:
  - name: analyze_{region}_{year}
    command: python analyze.py --region {region} --year {year} --output results/{region}_{year}.json
    parameters:
      region: "[north, south, east, west]"
      year: "2020:2024"

Creates 20 jobs (4 regions × 5 years).

See Also

• Simple Parameterization — Basic parameter tutorial
• Advanced Parameterization — Multi-dimensional sweeps
• Job Parameterization Reference — Complete syntax

Tutorials

Hands-on tutorials to learn Torc by example.

• Many Independent Jobs - Running large batches of independent jobs
• Diamond Workflow - Building workflows with dependencies
• Simple Parameterization - Creating parameterized job sweeps
• Advanced Parameterization - Complex parameter combinations
• Multi-Stage Workflows with Barriers - Coordinating workflow stages
• User Data Dependencies - Passing data between jobs

Tutorial 1: Many Independent Jobs

This tutorial teaches you how to create and run a workflow with many independent parallel jobs using Torc's parameterization feature.

Learning Objectives

By the end of this tutorial, you will:

• Understand how to define parameterized jobs that expand into multiple instances
• Learn how Torc executes independent jobs in parallel
• Know how to monitor job execution and view results

Prerequisites

• Torc server running (see Installation)
• Basic familiarity with YAML syntax

Use Cases

This pattern is ideal for:

• Parameter sweeps: Testing different configurations
• Monte Carlo simulations: Running many independent trials
• Batch processing: Processing many files with the same logic
• Embarrassingly parallel workloads: Any task that can be split into independent units

Step 1: Start the Torc Server

First, ensure the Torc server is running:

torc-server run

By default, the server listens on port 8080, making the API URL http://localhost:8080/torc-service/v1.

If you use a custom port, set the environment variable:

export TORC_API_URL="http://localhost:8100/torc-service/v1"

Step 2: Create the Workflow Specification

Save the following as hundred_jobs.yaml:

name: hundred_jobs_parallel
description: 100 independent jobs that can run in parallel

jobs:
  - name: job_{i:03d}
    command: |
      echo "Running job {i}"
      sleep $((RANDOM % 10 + 1))
      echo "Job {i} completed"
    resource_requirements: minimal
    parameters:
      i: "1:100"

resource_requirements:
  - name: minimal
    num_cpus: 1
    num_gpus: 0
    num_nodes: 1
    memory: 1g
    runtime: PT5M

Understanding the Specification

Let's break down the key elements:

• name: job_{i:03d}: The {i:03d} is a parameter placeholder. The :03d format specifier means "3-digit zero-padded integer", so jobs will be named job_001, job_002, ..., job_100.

• parameters: i: "1:100": This defines a parameter i that ranges from 1 to 100 (inclusive). Torc will create one job for each value.

• resource_requirements: minimal: Each job uses the "minimal" resource profile defined below.

When Torc processes this specification, it expands the single job definition into 100 separate jobs, each with its own parameter value substituted.

Step 3: Run the Workflow

Create and run the workflow in one command:

torc run hundred_jobs.yaml

This command:

1. Creates the workflow on the server
2. Expands the parameterized job into 100 individual jobs
3. Initializes the dependency graph (in this case, no dependencies)
4. Starts executing jobs in parallel

You'll see output showing the workflow ID and progress.

Step 4: Monitor Execution

While the workflow runs, you can monitor progress:

# Check workflow status
torc workflows status <workflow_id>

# List jobs and their states
torc jobs list <workflow_id>

# Or use the interactive TUI
torc tui

Since all 100 jobs are independent (no dependencies between them), Torc will run as many in parallel as your system resources allow.

Step 5: View Results

After completion, check the results:

torc results list <workflow_id>

This shows return codes, execution times, and resource usage for each job.

How It Works

When you run this workflow, Torc:

1. Expands parameters: The single job definition becomes 100 jobs (job_001 through job_100)
2. Marks all as ready: Since there are no dependencies, all jobs start in the "ready" state
3. Executes in parallel: The job runner claims and executes jobs based on available resources
4. Tracks completion: Each job's return code and metrics are recorded

The job runner respects the resource requirements you specified. With num_cpus: 1 per job, if your machine has 8 CPUs, approximately 8 jobs will run simultaneously.

What You Learned

In this tutorial, you learned how to:

• ✅ Use parameter expansion (parameters: i: "1:100") to generate multiple jobs from one definition
• ✅ Use format specifiers ({i:03d}) for consistent naming
• ✅ Run independent parallel jobs with torc run
• ✅ Monitor workflow progress and view results

Example Files

See hundred_jobs_parameterized.yaml for a ready-to-run version of this workflow.

Next Steps

• Tutorial 2: Diamond Workflow - Learn how to create job dependencies using files
• Tutorial 4: Simple Parameterization - Explore more parameter expansion options
• Multi-Stage Workflows with Barriers - Scale to thousands of jobs efficiently

Tutorial 2: Diamond Workflow with File Dependencies

This tutorial teaches you how to create workflows where job dependencies are automatically inferred from file inputs and outputs—a core concept in Torc called implicit dependencies.

Learning Objectives

By the end of this tutorial, you will:

• Understand how Torc infers job dependencies from file relationships
• Learn the "diamond" workflow pattern (fan-out and fan-in)
• Know how to use file variable substitution (${files.input.*} and ${files.output.*})
• See how jobs automatically unblock when their input files become available

Prerequisites

• Completed Tutorial 1: Many Independent Jobs
• Torc server running

The Diamond Pattern

The "diamond" pattern is a common workflow structure where:

1. One job produces multiple outputs (fan-out)
2. Multiple jobs process those outputs in parallel
3. A final job combines all results (fan-in)

graph TD
    Input(["input.txt"]) --> Preprocess["preprocess"]
    Preprocess --> Int1(["intermediate1.txt"])
    Preprocess --> Int2(["intermediate2.txt"])

    Int1 --> Work1["work1"]
    Int2 --> Work2["work2"]

    Work1 --> Result1(["result1.txt"])
    Work2 --> Result2(["result2.txt"])

    Result1 --> Postprocess["postprocess"]
    Result2 --> Postprocess

    Postprocess --> Output(["output.txt"])

    style Input fill:#d4edda,stroke:#28a745,color:#155724
    style Int1 fill:#d4edda,stroke:#28a745,color:#155724
    style Int2 fill:#d4edda,stroke:#28a745,color:#155724
    style Result1 fill:#d4edda,stroke:#28a745,color:#155724
    style Result2 fill:#d4edda,stroke:#28a745,color:#155724
    style Output fill:#d4edda,stroke:#28a745,color:#155724
    style Preprocess fill:#4a9eff,color:#fff
    style Work1 fill:#4a9eff,color:#fff
    style Work2 fill:#4a9eff,color:#fff
    style Postprocess fill:#4a9eff,color:#fff

Notice that we never explicitly say "work1 depends on preprocess"—Torc figures this out automatically because work1 needs intermediate1.txt as input, and preprocess produces it as output.

Step 1: Create the Workflow Specification

Save as diamond.yaml:

name: diamond_workflow
description: Diamond workflow demonstrating fan-out and fan-in

jobs:
  - name: preprocess
    command: |
      cat ${files.input.input_file} |
      awk '{print $1}' > ${files.output.intermediate1}
      cat ${files.input.input_file} |
      awk '{print $2}' > ${files.output.intermediate2}
    resource_requirements: small

  - name: work1
    command: |
      cat ${files.input.intermediate1} |
      sort | uniq > ${files.output.result1}
    resource_requirements: medium

  - name: work2
    command: |
      cat ${files.input.intermediate2} |
      sort | uniq > ${files.output.result2}
    resource_requirements: medium

  - name: postprocess
    command: |
      paste ${files.input.result1} ${files.input.result2} > ${files.output.final_output}
    resource_requirements: small

files:
  - name: input_file
    path: /tmp/input.txt

  - name: intermediate1
    path: /tmp/intermediate1.txt

  - name: intermediate2
    path: /tmp/intermediate2.txt

  - name: result1
    path: /tmp/result1.txt

  - name: result2
    path: /tmp/result2.txt

  - name: final_output
    path: /tmp/output.txt

resource_requirements:
  - name: small
    num_cpus: 1
    num_gpus: 0
    num_nodes: 1
    memory: 1g
    runtime: PT10M

  - name: medium
    num_cpus: 4
    num_gpus: 0
    num_nodes: 1
    memory: 4g
    runtime: PT30M

Understanding File Variable Substitution

The key concept here is file variable substitution:

• ${files.input.filename} - References a file this job reads (creates a dependency)
• ${files.output.filename} - References a file this job writes (satisfies dependencies)

When Torc processes the workflow:

1. It sees preprocess outputs intermediate1 and intermediate2
2. It sees work1 inputs intermediate1 → dependency created
3. It sees work2 inputs intermediate2 → dependency created
4. It sees postprocess inputs result1 and result2 → dependencies created

This is more maintainable than explicit depends_on declarations because:

• Dependencies are derived from actual data flow
• Adding a new intermediate step automatically updates dependencies
• The workflow specification documents the data flow

Step 2: Create Input Data

# Create test input file
echo -e "apple red\nbanana yellow\ncherry red\ndate brown" > /tmp/input.txt

Step 3: Create and Initialize the Workflow

# Create the workflow and capture the ID
WORKFLOW_ID=$(torc workflows create diamond.yaml -f json | jq -r '.id')
echo "Created workflow: $WORKFLOW_ID"

# Ensure the input file timestamp is current
touch /tmp/input.txt

# Initialize the workflow (builds dependency graph)
torc workflows initialize-jobs $WORKFLOW_ID

The initialize-jobs command is where Torc:

1. Analyzes file input/output relationships
2. Builds the dependency graph
3. Marks jobs with satisfied dependencies as "ready"

Step 4: Observe Dependency Resolution

# Check job statuses
torc jobs list $WORKFLOW_ID

Expected output:

╭────┬──────────────┬─────────┬────────╮
│ ID │ Name         │ Status  │ ...    │
├────┼──────────────┼─────────┼────────┤
│ 1  │ preprocess   │ ready   │ ...    │
│ 2  │ work1        │ blocked │ ...    │
│ 3  │ work2        │ blocked │ ...    │
│ 4  │ postprocess  │ blocked │ ...    │
╰────┴──────────────┴─────────┴────────╯

Only preprocess is ready because:

• Its only input (input_file) already exists
• The others are blocked waiting for files that don't exist yet

Step 5: Run the Workflow

torc run $WORKFLOW_ID

Watch the execution unfold:

1. preprocess runs first - Creates intermediate1.txt and intermediate2.txt
2. work1 and work2 unblock - Their input files now exist
3. work1 and work2 run in parallel - They have no dependency on each other
4. postprocess unblocks - Both result1.txt and result2.txt exist
5. postprocess runs - Creates the final output

Step 6: Verify Results

cat /tmp/output.txt

You should see the combined, sorted, unique values from both columns of the input.

How Implicit Dependencies Work

Torc determines job order through file relationships:

The dependency graph is built automatically from these relationships. If you later add a validation step between preprocess and work1, you only need to update the file references—the dependencies adjust automatically.

What You Learned

In this tutorial, you learned:

• ✅ How to define files with files: section and reference them in jobs
• ✅ How ${files.input.*} creates implicit dependencies
• ✅ How ${files.output.*} satisfies dependencies for downstream jobs
• ✅ The diamond pattern: fan-out → parallel processing → fan-in
• ✅ How Torc automatically determines execution order from data flow

When to Use File Dependencies vs Explicit Dependencies

Use file dependencies when:

• Jobs actually read/write files
• Data flow defines the natural ordering
• You want self-documenting workflows

Use explicit depends_on when:

• Dependencies are logical, not data-based
• Jobs communicate through side effects
• You need precise control over ordering

Example Files

See the diamond workflow examples in all three formats:

• diamond_workflow.yaml
• diamond_workflow.json5
• diamond_workflow.kdl

A Python version is also available: diamond_workflow.py

Next Steps

• Tutorial 3: User Data Dependencies - Pass JSON data between jobs without files
• Tutorial 4: Simple Parameterization - Combine file dependencies with parameter expansion

Tutorial: Simple Job Parameterization

This tutorial teaches you how to create parameter sweeps—generating multiple related jobs from a single job definition using Torc's parameterization feature.

Learning Objectives

By the end of this tutorial, you will:

• Understand how parameterization expands one job definition into many jobs
• Learn the different parameter formats (lists, ranges)
• Know how to use format specifiers for consistent naming

Prerequisites

• Completed Tutorial 1: Many Independent Jobs
• Torc server running

Why Parameterization?

Without parameterization, running 10 similar jobs requires 10 separate definitions. With parameterization, you write one definition and Torc expands it:

# Without parameterization: 10 separate definitions
jobs:
  - name: task_1
    command: ./process.sh --index 1
  - name: task_2
    command: ./process.sh --index 2
  # ... 8 more ...

# With parameterization: 1 definition
jobs:
  - name: task_{i}
    command: ./process.sh --index {i}
    parameters:
      i: "1:10"

Step 1: Create a Simple Parameterized Workflow

Save as parallel_tasks.yaml:

name: parallel_tasks
description: Run 10 parallel tasks

jobs:
  - name: task_{i}
    command: |
      echo "Processing task {i}"
      sleep 2
      echo "Task {i} complete"
    parameters:
      i: "1:10"

This expands to 10 jobs: task_1, task_2, ..., task_10.

Step 2: Create and Run the Workflow

WORKFLOW_ID=$(torc workflows create parallel_tasks.yaml -f json | jq -r '.id')
echo "Created workflow: $WORKFLOW_ID"

torc run $WORKFLOW_ID

All 10 jobs run in parallel since they have no dependencies.

Step 3: Verify the Expansion

torc jobs list $WORKFLOW_ID -f json | jq -r '.jobs[].name' | sort

Output:

task_1
task_10
task_2
task_3
task_4
task_5
task_6
task_7
task_8
task_9

Adding Dependencies Between Parameterized Jobs

Now let's create a workflow where parameterized jobs have dependencies:

name: process_and_aggregate
description: Process chunks then aggregate results

jobs:
  - name: process_{i}
    command: |
      echo "Processing chunk {i}" > results/chunk_{i}.txt
    parameters:
      i: "1:5"

  - name: aggregate
    command: |
      cat results/chunk_*.txt > results/final.txt
      echo "Aggregated all chunks"
    depends_on:
      - process_{i}
    parameters:
      i: "1:5"

This creates:

• 5 parallel process_* jobs
• 1 aggregate job that waits for all 5 to complete

The aggregate job lists all process_{i} jobs as dependencies, so Torc expands this to depend on all 5 process jobs.

Understanding the Specification

Parameter Syntax:

• {lr} - Simple substitution with the parameter value
• {lr:.4f} - Format specifier: 4 decimal places (e.g., 0.0010 not 0.001)

Parameter Values:

• "[0.0001,0.0005,0.001,0.005,0.01]" - A list of 5 specific values

File Parameterization: Notice that both jobs AND files have parameters:. When Torc expands:

• Each train_lr{lr:.4f} job gets a corresponding model_lr{lr:.4f} file
• The file dependencies are matched by parameter value

Dependency Flow:

1. train_lr0.0001 → outputs model_lr0.0001 → unblocks evaluate_lr0.0001
2. train_lr0.0005 → outputs model_lr0.0005 → unblocks evaluate_lr0.0005
3. (and so on for each learning rate)
4. All evaluate_* jobs → unblock compare_results

Step 2: Create and Initialize the Workflow

WORKFLOW_ID=$(torc workflows create learning_rate_sweep.yaml -f json | jq -r '.id')
echo "Created workflow: $WORKFLOW_ID"

torc workflows initialize-jobs $WORKFLOW_ID

Step 3: Verify the Expansion

# Count jobs (should be 11: 5 train + 5 evaluate + 1 compare)
torc jobs list $WORKFLOW_ID -f json | jq '.jobs | length'

List the job names:

torc jobs list $WORKFLOW_ID -f json | jq -r '.jobs[].name' | sort

Output:

compare_results
evaluate_lr0.0001
evaluate_lr0.0005
evaluate_lr0.0010
evaluate_lr0.0050
evaluate_lr0.0100
train_lr0.0001
train_lr0.0005
train_lr0.0010
train_lr0.0050
train_lr0.0100

Notice:

• One job per parameter value for train_* and evaluate_*
• Only one compare_results job (it has the parameter for dependencies, but doesn't expand because its name has no {lr})

Step 4: Check Dependencies

torc jobs list $WORKFLOW_ID

Expected statuses:

• All train_* jobs: ready (no input dependencies)
• All evaluate_* jobs: blocked (waiting for corresponding model file)
• compare_results: blocked (waiting for all metrics files)

Step 5: Run the Workflow

torc run $WORKFLOW_ID

Execution flow:

1. All 5 training jobs run in parallel - They have no dependencies on each other
2. Each evaluation unblocks independently - When train_lr0.0001 finishes, evaluate_lr0.0001 can start (doesn't wait for other training jobs)
3. Compare runs last - Only after all 5 evaluations complete

This is more efficient than a simple two-stage workflow because evaluations can start as soon as their specific training job completes.

Parameter Format Reference

List Format

Explicit list of values:

parameters:
  lr: "[0.0001,0.0005,0.001,0.005,0.01]"  # Numbers
  opt: "['adam','sgd','rmsprop']"          # Strings (note the quotes)

Range Format

For integer or float sequences:

parameters:
  i: "1:100"        # Integers 1 to 100 (inclusive)
  i: "0:100:10"     # Integers 0, 10, 20, ..., 100 (with step)
  lr: "0.0:1.0:0.1" # Floats 0.0, 0.1, 0.2, ..., 1.0

Format Specifiers

Control how values appear in names:

How Parameterization and File Dependencies Interact

When both jobs and files are parameterized with the same parameter:

jobs:
  - name: train_{i}
    output_files: [model_{i}]
    parameters:
      i: "1:3"

  - name: eval_{i}
    input_files: [model_{i}]
    parameters:
      i: "1:3"

files:
  - name: model_{i}
    path: /models/model_{i}.pt
    parameters:
      i: "1:3"

Torc creates these relationships:

• train_1 → model_1 → eval_1
• train_2 → model_2 → eval_2
• train_3 → model_3 → eval_3

Each chain is independent—eval_2 doesn't wait for train_1.

Parameter Modes: Product vs Zip

By default, multiple parameters create a Cartesian product (all combinations). For paired parameters, use parameter_mode: zip:

jobs:
  # Default (product): 3 × 3 = 9 jobs
  - name: train_{dataset}_{model}
    command: python train.py --dataset={dataset} --model={model}
    parameters:
      dataset: "['cifar10', 'mnist', 'imagenet']"
      model: "['resnet', 'vgg', 'transformer']"

  # Zip mode: 3 paired jobs (cifar10+resnet, mnist+vgg, imagenet+transformer)
  - name: paired_{dataset}_{model}
    command: python train.py --dataset={dataset} --model={model}
    parameters:
      dataset: "['cifar10', 'mnist', 'imagenet']"
      model: "['resnet', 'vgg', 'transformer']"
    parameter_mode: zip

Use zip mode when parameters have a 1:1 correspondence (e.g., input/output file pairs, pre-determined configurations).

See Parameterization Reference for details.

What You Learned

In this tutorial, you learned:

• ✅ How to use parameters: to expand one job definition into many
• ✅ List format ("[a,b,c]") and range format ("1:100")
• ✅ Format specifiers ({i:03d}, {lr:.4f}) for consistent naming
• ✅ How parameterized files create one-to-one dependencies
• ✅ The efficiency of parameter-matched dependencies (each chain runs independently)
• ✅ The difference between product (default) and zip parameter modes

Next Steps

• Tutorial 5: Advanced Parameterization - Multi-dimensional grid searches
• Multi-Stage Workflows with Barriers - Scale to thousands of parameterized jobs

Tutorial 5: Advanced Multi-Dimensional Parameterization

This tutorial teaches you how to create multi-dimensional parameter sweeps—grid searches over multiple hyperparameters that generate all combinations automatically.

Learning Objectives

By the end of this tutorial, you will:

• Understand how multiple parameters create a Cartesian product (all combinations)
• Learn to structure complex workflows with data preparation, training, and aggregation stages
• Know how to combine parameterization with explicit dependencies
• See patterns for running large grid searches on HPC systems

Prerequisites

• Completed Tutorial 4: Simple Parameterization
• Torc server running
• Understanding of file dependencies

Multi-Dimensional Parameters: Cartesian Product

When a job has multiple parameters, Torc creates the Cartesian product—every combination of values:

parameters:
  lr: "[0.001,0.01]"   # 2 values
  bs: "[16,32]"        # 2 values

This generates 2 × 2 = 4 jobs:

• lr=0.001, bs=16
• lr=0.001, bs=32
• lr=0.01, bs=16
• lr=0.01, bs=32

With three parameters:

parameters:
  lr: "[0.0001,0.001,0.01]"  # 3 values
  bs: "[16,32,64]"            # 3 values
  opt: "['adam','sgd']"       # 2 values

This generates 3 × 3 × 2 = 18 jobs.

Step 1: Create the Workflow Specification

Save as grid_search.yaml:

name: hyperparameter_grid_search
description: 3D grid search over learning rate, batch size, and optimizer

jobs:
  # Data preparation (runs once, no parameters)
  - name: prepare_data
    command: python prepare_data.py --output=/data/processed.pkl
    resource_requirements: data_prep
    output_files:
      - training_data

  # Training jobs (one per parameter combination)
  - name: train_lr{lr:.4f}_bs{bs}_opt{opt}
    command: |
      python train.py \
        --data=/data/processed.pkl \
        --learning-rate={lr} \
        --batch-size={bs} \
        --optimizer={opt} \
        --output=/models/model_lr{lr:.4f}_bs{bs}_opt{opt}.pt \
        --metrics=/results/metrics_lr{lr:.4f}_bs{bs}_opt{opt}.json
    resource_requirements: gpu_training
    input_files:
      - training_data
    output_files:
      - model_lr{lr:.4f}_bs{bs}_opt{opt}
      - metrics_lr{lr:.4f}_bs{bs}_opt{opt}
    parameters:
      lr: "[0.0001,0.001,0.01]"
      bs: "[16,32,64]"
      opt: "['adam','sgd']"

  # Aggregate results (depends on ALL training jobs via file dependencies)
  - name: aggregate_results
    command: |
      python aggregate.py \
        --input-dir=/results \
        --output=/results/summary.csv
    resource_requirements: minimal
    input_files:
      - metrics_lr{lr:.4f}_bs{bs}_opt{opt}
    parameters:
      lr: "[0.0001,0.001,0.01]"
      bs: "[16,32,64]"
      opt: "['adam','sgd']"

  # Find best model (explicit dependency, no parameters)
  - name: select_best_model
    command: |
      python select_best.py \
        --summary=/results/summary.csv \
        --output=/results/best_config.json
    resource_requirements: minimal
    depends_on:
      - aggregate_results

files:
  - name: training_data
    path: /data/processed.pkl

  - name: model_lr{lr:.4f}_bs{bs}_opt{opt}
    path: /models/model_lr{lr:.4f}_bs{bs}_opt{opt}.pt
    parameters:
      lr: "[0.0001,0.001,0.01]"
      bs: "[16,32,64]"
      opt: "['adam','sgd']"

  - name: metrics_lr{lr:.4f}_bs{bs}_opt{opt}
    path: /results/metrics_lr{lr:.4f}_bs{bs}_opt{opt}.json
    parameters:
      lr: "[0.0001,0.001,0.01]"
      bs: "[16,32,64]"
      opt: "['adam','sgd']"

resource_requirements:
  - name: data_prep
    num_cpus: 8
    memory: 32g
    runtime: PT1H

  - name: gpu_training
    num_cpus: 8
    num_gpus: 1
    memory: 16g
    runtime: PT4H

  - name: minimal
    num_cpus: 1
    memory: 2g
    runtime: PT10M

Understanding the Structure

Four-stage workflow:

1. prepare_data (1 job) - No parameters, runs once
2. train_* (18 jobs) - Parameterized, all depend on prepare_data
3. aggregate_results (1 job) - Has parameters only for file dependency matching
4. select_best_model (1 job) - Explicit dependency on aggregate_results

Key insight: Why aggregate_results has parameters

The aggregate_results job won't expand into multiple jobs (its name has no {}). However, it needs parameters: to match the parameterized input_files. This tells Torc: "this job depends on ALL 18 metrics files."

Step 2: Create and Initialize the Workflow

WORKFLOW_ID=$(torc workflows create grid_search.yaml -f json | jq -r '.id')
echo "Created workflow: $WORKFLOW_ID"

torc workflows initialize-jobs $WORKFLOW_ID

Step 3: Verify the Expansion

Count the jobs:

torc jobs list $WORKFLOW_ID -f json | jq '.jobs | length'

Expected: 21 jobs (1 prepare + 18 training + 1 aggregate + 1 select)

List the training jobs:

torc jobs list $WORKFLOW_ID -f json | jq -r '.jobs[] | select(.name | startswith("train_")) | .name' | sort

Output (18 training jobs):

train_lr0.0001_bs16_optadam
train_lr0.0001_bs16_optsgd
train_lr0.0001_bs32_optadam
train_lr0.0001_bs32_optsgd
train_lr0.0001_bs64_optadam
train_lr0.0001_bs64_optsgd
train_lr0.0010_bs16_optadam
train_lr0.0010_bs16_optsgd
train_lr0.0010_bs32_optadam
train_lr0.0010_bs32_optsgd
train_lr0.0010_bs64_optadam
train_lr0.0010_bs64_optsgd
train_lr0.0100_bs16_optadam
train_lr0.0100_bs16_optsgd
train_lr0.0100_bs32_optadam
train_lr0.0100_bs32_optsgd
train_lr0.0100_bs64_optadam
train_lr0.0100_bs64_optsgd

Step 4: Examine the Dependency Graph

torc jobs list $WORKFLOW_ID

Initial states:

• prepare_data: ready (no dependencies)
• All train_*: blocked (waiting for training_data file)
• aggregate_results: blocked (waiting for all 18 metrics files)
• select_best_model: blocked (waiting for aggregate_results)

Step 5: Run the Workflow

For local execution:

torc run $WORKFLOW_ID

Execution flow:

1. prepare_data runs and produces training_data
2. All 18 train_* jobs unblock and run in parallel (resource-limited)
3. aggregate_results waits for all training jobs, then runs
4. select_best_model runs last

Step 6: Monitor Progress

# Check status summary
torc workflows status $WORKFLOW_ID

# Watch job completion in real-time
watch -n 10 'torc jobs list-by-status $WORKFLOW_ID'

# Or use the TUI
torc tui

Step 7: Retrieve Results

After completion:

# View best configuration
cat /results/best_config.json

# View summary of all runs
cat /results/summary.csv

Scaling Considerations

Job Count Growth

Multi-dimensional parameters grow exponentially:

Dependency Count

Without barriers, dependencies also grow quickly. In this tutorial:

• 18 training jobs each depend on 1 file = 18 dependencies
• 1 aggregate job depends on 18 files = 18 dependencies
• Total: ~36 dependencies

For larger sweeps (1000+ jobs), consider the barrier pattern to reduce dependencies from O(n²) to O(n).

Common Patterns

Mixing Fixed and Parameterized Jobs

jobs:
  # Fixed job (no parameters)
  - name: setup
    command: ./setup.sh

  # Parameterized jobs depend on fixed job
  - name: experiment_{i}
    command: ./run.sh {i}
    depends_on: [setup]
    parameters:
      i: "1:100"

Aggregating Parameterized Results

Use the file dependency pattern shown in this tutorial:

- name: aggregate
  input_files:
    - result_{i}    # Matches all parameterized result files
  parameters:
    i: "1:100"      # Same parameters as producer jobs

Nested Parameter Sweeps

For workflows with multiple independent sweeps:

jobs:
  # Sweep 1
  - name: sweep1_job_{a}
    parameters:
      a: "1:10"

  # Sweep 2 (independent of sweep 1)
  - name: sweep2_job_{b}
    parameters:
      b: "1:10"

What You Learned

In this tutorial, you learned:

• ✅ How multiple parameters create a Cartesian product of jobs
• ✅ How to structure multi-stage workflows (prep → train → aggregate → select)
• ✅ How to use parameters in file dependencies to collect all outputs
• ✅ How to mix parameterized and non-parameterized jobs
• ✅ Scaling considerations for large grid searches

Example Files

See these example files for hyperparameter sweep patterns:

• hyperparameter_sweep.yaml - Basic 3×3×2 grid search
• hyperparameter_sweep_shared_params.yaml - Grid search with shared parameter definitions

Next Steps

• Multi-Stage Workflows with Barriers - Essential for scaling to thousands of jobs
• Working with Slurm - Deploy grid searches on HPC clusters
• Resource Monitoring - Track resource usage across your sweep

Multi-Stage Workflows with Barriers

This tutorial teaches you how to efficiently structure workflows with multiple stages using the barrier pattern. This is essential for scaling workflows to thousands of jobs.

Learning Objectives

By the end of this tutorial, you will:

• Understand the quadratic dependency problem in multi-stage workflows
• Use barrier jobs to efficiently synchronize between stages
• Scale workflows to thousands of jobs with minimal overhead
• Know when to use barriers vs. direct dependencies

Prerequisites

• Basic understanding of Torc workflows
• Completed the Many Independent Jobs tutorial
• Completed the Simple Parameterization tutorial

The Problem: Quadratic Dependencies

Let's start with a common but inefficient pattern. Suppose you want to:

1. Stage 1: Run 1000 preprocessing jobs in parallel
2. Stage 2: Run 1000 analysis jobs, but only after ALL stage 1 jobs complete
3. Stage 3: Run a final aggregation job

Naive Approach (DON'T DO THIS!)

name: "Inefficient Multi-Stage Workflow"
description: "This creates 1,000,000 dependencies!"

jobs:
  # Stage 1: 1000 preprocessing jobs
  - name: "preprocess_{i:03d}"
    command: "python preprocess.py --id {i}"
    parameters:
      i: "0:999"

  # Stage 2: Each analysis job waits for ALL preprocessing jobs
  - name: "analyze_{i:03d}"
    command: "python analyze.py --id {i}"
    depends_on_regexes: ["^preprocess_.*"]  # ⚠️ Creates 1,000,000 dependencies!
    parameters:
      i: "0:999"

  # Stage 3: Final aggregation
  - name: "final_report"
    command: "python generate_report.py"
    depends_on_regexes: ["^analyze_.*"]  # ⚠️ Creates 1,000 more dependencies

Why This is Bad

When Torc expands this workflow:

• Each of the 1000 analyze_* jobs gets a dependency on each of the 1000 preprocess_* jobs
• Total dependencies: 1000 × 1000 = 1,000,000 relationships
• Workflow creation takes minutes instead of seconds
• Database becomes bloated with dependency records
• Job initialization is slow

The Solution: Barrier Jobs

A barrier job is a lightweight synchronization point that:

• Depends on all jobs from the previous stage (using a regex)
• Is depended upon by all jobs in the next stage
• Reduces dependencies from O(n²) to O(n)

flowchart LR
    subgraph stage1["Stage 1"]
        P1["preprocess_1"]
        P2["preprocess_2"]
        PN["preprocess_N"]
    end

    B1(["barrier"])

    subgraph stage2["Stage 2"]
        A1["analyze_1"]
        A2["analyze_2"]
        AN["analyze_N"]
    end

    P1 --> B1
    P2 --> B1
    PN --> B1
    B1 --> A1
    B1 --> A2
    B1 --> AN

    style P1 fill:#4a9eff,color:#fff
    style P2 fill:#4a9eff,color:#fff
    style PN fill:#4a9eff,color:#fff
    style B1 fill:#28a745,color:#fff
    style A1 fill:#ffc107,color:#000
    style A2 fill:#ffc107,color:#000
    style AN fill:#ffc107,color:#000

Instead of N×N dependencies (every stage 2 job depending on every stage 1 job), you get 2N dependencies (N into the barrier, N out of the barrier).

Efficient Approach (DO THIS!)

name: "Efficient Multi-Stage Workflow"
description: "Uses barrier pattern with only ~3000 dependencies"

jobs:
  # ═══════════════════════════════════════════════════════════
  # STAGE 1: Preprocessing (1000 parallel jobs)
  # ═══════════════════════════════════════════════════════════
  - name: "preprocess_{i:03d}"
    command: "python preprocess.py --id {i} --output data/stage1_{i:03d}.json"
    resource_requirements: "medium"
    parameters:
      i: "0:999"

  # ═══════════════════════════════════════════════════════════
  # BARRIER: Wait for ALL stage 1 jobs
  # ═══════════════════════════════════════════════════════════
  - name: "barrier_stage1_complete"
    command: "echo 'Stage 1 complete: 1000 files preprocessed' && date"
    resource_requirements: "tiny"
    depends_on_regexes: ["^preprocess_.*"]  # ✓ 1000 dependencies

  # ═══════════════════════════════════════════════════════════
  # STAGE 2: Analysis (1000 parallel jobs)
  # ═══════════════════════════════════════════════════════════
  - name: "analyze_{i:03d}"
    command: "python analyze.py --input data/stage1_{i:03d}.json --output data/stage2_{i:03d}.csv"
    resource_requirements: "large"
    depends_on: ["barrier_stage1_complete"]  # ✓ 1000 dependencies (one per job)
    parameters:
      i: "0:999"

  # ═══════════════════════════════════════════════════════════
  # BARRIER: Wait for ALL stage 2 jobs
  # ═══════════════════════════════════════════════════════════
  - name: "barrier_stage2_complete"
    command: "echo 'Stage 2 complete: 1000 analyses finished' && date"
    resource_requirements: "tiny"
    depends_on_regexes: ["^analyze_.*"]  # ✓ 1000 dependencies

  # ═══════════════════════════════════════════════════════════
  # STAGE 3: Final report (single job)
  # ═══════════════════════════════════════════════════════════
  - name: "final_report"
    command: "python generate_report.py --output final_report.html"
    resource_requirements: "medium"
    depends_on: ["barrier_stage2_complete"]  # ✓ 1 dependency

resource_requirements:
  - name: "tiny"
    num_cpus: 1
    num_gpus: 0
    num_nodes: 1
    memory: "100m"
    runtime: "PT1M"

  - name: "medium"
    num_cpus: 4
    num_gpus: 0
    num_nodes: 1
    memory: "4g"
    runtime: "PT30M"

  - name: "large"
    num_cpus: 16
    num_gpus: 1
    num_nodes: 1
    memory: "32g"
    runtime: "PT2H"

Dependency Breakdown

Without barriers:

• Stage 1 → Stage 2: 1000 × 1000 = 1,000,000 dependencies
• Stage 2 → Stage 3: 1000 = 1,000 dependencies
• Total: 1,001,000 dependencies

With barriers:

• Stage 1 → Barrier 1: 1,000 dependencies
• Barrier 1 → Stage 2: 1,000 dependencies
• Stage 2 → Barrier 2: 1,000 dependencies
• Barrier 2 → Stage 3: 1 dependency
• Total: 3,001 dependencies ← 333× improvement!

Step-by-Step: Creating Your First Barrier Workflow

Let's create a simple 2-stage workflow.

Step 1: Create the Workflow Spec

Create barrier_demo.yaml:

name: "Barrier Pattern Demo"
description: "Simple demonstration of the barrier pattern"

jobs:
  # Stage 1: Generate 100 data files
  - name: "generate_data_{i:02d}"
    command: "echo 'Data file {i}' > output/data_{i:02d}.txt"
    parameters:
      i: "0:99"

  # Barrier: Wait for all data generation
  - name: "data_generation_complete"
    command: "echo 'All 100 data files generated' && ls -l output/ | wc -l"
    depends_on_regexes: ["^generate_data_.*"]

  # Stage 2: Process each data file
  - name: "process_data_{i:02d}"
    command: "cat output/data_{i:02d}.txt | wc -w > output/processed_{i:02d}.txt"
    depends_on: ["data_generation_complete"]
    parameters:
      i: "0:99"

  # Final barrier and report
  - name: "processing_complete"
    command: "echo 'All 100 files processed' && cat output/processed_*.txt | awk '{sum+=$1} END {print sum}'"
    depends_on_regexes: ["^process_data_.*"]

Step 2: Create the Output Directory

mkdir -p output

Step 3: Create the Workflow

torc workflows create barrier_demo.yaml

You should see output like:

Created workflow with ID: 1
- Created 100 stage 1 jobs
- Created 1 barrier job
- Created 100 stage 2 jobs
- Created 1 final barrier
Total: 202 jobs, 201 dependencies

Compare this to 10,000 dependencies without barriers!

Step 4: Run the Workflow

torc workflows run 1

Step 5: Monitor Progress

torc tui

You'll see:

1. All 100 generate_data_* jobs run in parallel
2. Once they finish, data_generation_complete executes
3. Then all 100 process_data_* jobs run in parallel
4. Finally, processing_complete executes

Making Effective Barrier Jobs

1. Keep Barriers Lightweight

Barriers should be quick and cheap:

✓ GOOD - Lightweight logging
- name: "stage1_complete"
  command: "echo 'Stage 1 done' && date"
  resource_requirements: "tiny"

✗ BAD - Heavy computation
- name: "stage1_complete"
  command: "python expensive_validation.py"  # Don't do this!
  resource_requirements: "large"

If you need validation, create a separate job:

# Barrier - lightweight
- name: "stage1_complete"
  command: "echo 'Stage 1 done'"
  resource_requirements: "tiny"
  depends_on_regexes: ["^stage1_.*"]

# Validation - heavier
- name: "validate_stage1"
  command: "python validate_all_outputs.py"
  resource_requirements: "medium"
  depends_on: ["stage1_complete"]

# Stage 2 depends on validation passing
- name: "stage2_job_{i}"
  command: "python stage2.py {i}"
  depends_on: ["validate_stage1"]
  parameters:
    i: "0:999"

2. Use Descriptive Names

Names should clearly indicate what stage completed:

✓ GOOD
- name: "barrier_preprocessing_complete"
- name: "barrier_training_complete"
- name: "all_simulations_finished"

✗ BAD
- name: "barrier1"
- name: "sync"
- name: "wait"

3. Add Useful Information

Make barriers informative:

- name: "preprocessing_complete"
  command: |
    echo "════════════════════════════════════════"
    echo "Preprocessing Complete: $(date)"
    echo "Files generated: $(ls output/stage1_*.json | wc -l)"
    echo "Total size: $(du -sh output/)"
    echo "Proceeding to analysis stage..."
    echo "════════════════════════════════════════"
  depends_on_regexes: ["^preprocess_.*"]

4. Be Careful with Regex Patterns

Ensure your regex matches exactly what you intend:

✓ GOOD - Anchored patterns
depends_on_regexes: ["^stage1_job_.*"]      # Matches "stage1_job_001", "stage1_job_042"
depends_on_regexes: ["^preprocess_\\d+$"]   # Matches "preprocess_0", "preprocess_999"

✗ BAD - Too broad
depends_on_regexes: ["stage1"]              # Matches "my_stage1_test" (unintended!)
depends_on_regexes: [".*"]                  # Matches EVERYTHING (disaster!)

Test your regex before deploying:

# Python regex tester
python3 -c "import re; print(re.match(r'^stage1_job_.*', 'stage1_job_001'))"

When NOT to Use Barriers

Barriers are not always the right solution:

1. One-to-One Dependencies

When each job in stage 2 only needs its corresponding stage 1 job:

# DON'T use a barrier here
jobs:
  - name: "preprocess_{i}"
    command: "preprocess.py {i}"
    output_files: ["data_{i}.json"]
    parameters:
      i: "0:99"

  # Each analysis only needs its own preprocessed file
  - name: "analyze_{i}"
    command: "analyze.py {i}"
    input_files: ["data_{i}.json"]  # ✓ Automatic dependency via files
    parameters:
      i: "0:99"

The file dependency system already handles this efficiently!

2. Specific Dependencies in DAGs

When you have a directed acyclic graph (DAG) with specific paths:

# Diamond pattern - specific dependencies
jobs:
  - name: "fetch_data"
    command: "fetch.py"

  - name: "process_weather"
    command: "process_weather.py"
    depends_on: ["fetch_data"]

  - name: "process_traffic"
    command: "process_traffic.py"
    depends_on: ["fetch_data"]

  - name: "generate_report"
    command: "report.py"
    depends_on: ["process_weather", "process_traffic"]  # ✓ Specific dependencies

Don't force this into stages - the specific dependencies are clearer!

3. Small Workflows

For small workflows (< 100 jobs), the overhead of barriers isn't worth it:

# Only 10 jobs - barriers not needed
jobs:
  - name: "job_{i}"
    command: "process.py {i}"
    depends_on_regexes: ["^prepare_.*"]  # This is fine for 10 jobs
    parameters:
      i: "0:9"

Scaling to Thousands of Jobs

The barrier pattern scales beautifully. Let's compare performance:

As you can see, barriers become essential for large-scale workflows.

Complete Example

See multi_stage_barrier_pattern.yaml for a comprehensive example with:

• 3 distinct stages (1000 → 1000 → 100 jobs)
• Informative barrier jobs with progress logging
• Different resource requirements per stage
• Comments explaining the pattern

Summary

✓ Use barrier jobs when all jobs in one stage must complete before any job in the next stage starts

✓ Use file/data dependencies for one-to-one job relationships

✓ Use specific dependencies for DAG patterns with clear paths

✓ Keep barriers lightweight - just logging and simple checks

✓ Use descriptive names to track workflow progress

The barrier pattern is your key to scaling Torc workflows from hundreds to thousands of jobs efficiently!

Next Steps

• Try modifying the demo workflow to have 3 or more stages
• Experiment with adding validation logic to barrier jobs
• Check out Advanced Parameterization for creating complex multi-stage pipelines
• Learn about Workflow Actions for conditional execution between stages

Tutorial 3: User Data Dependencies

This tutorial teaches you how to pass structured data (JSON) between jobs using Torc's user_data feature—an alternative to file-based dependencies that stores data directly in the database.

Learning Objectives

By the end of this tutorial, you will:

• Understand what user_data is and when to use it instead of files
• Learn how to define user_data entries and reference them in jobs
• Know how to update user_data from within a job
• See how user_data creates implicit dependencies (like files)

Prerequisites

• Completed Tutorial 2: Diamond Workflow
• Torc server running
• jq command-line tool installed (for JSON parsing)

What is User Data?

User data is Torc's mechanism for passing small, structured data between jobs without creating actual files. The data is stored in the Torc database and can be:

• JSON objects (configurations, parameters)
• Arrays
• Simple values (strings, numbers)

Like files, user_data creates implicit dependencies: a job that reads user_data will be blocked until the job that writes it completes.

User Data vs Files

Step 1: Create the Workflow Specification

Save as user_data_workflow.yaml:

name: config_pipeline
description: Jobs that pass configuration via user_data

jobs:
  - name: generate_config
    command: |
      echo '{"learning_rate": 0.001, "batch_size": 32, "epochs": 10}' > /tmp/config.json
      torc user-data update ${user_data.output.ml_config} \
        --data "$(cat /tmp/config.json)"
    resource_requirements: minimal

  - name: train_model
    command: |
      echo "Training with config:"
      torc user-data get ${user_data.input.ml_config} | jq '.data'
      # In a real workflow: python train.py --config="${user_data.input.ml_config}"
    resource_requirements: gpu_large

  - name: evaluate_model
    command: |
      echo "Evaluating with config:"
      torc user-data get ${user_data.input.ml_config} | jq '.data'
      # In a real workflow: python evaluate.py --config="${user_data.input.ml_config}"
    resource_requirements: gpu_small

user_data:
  - name: ml_config
    data: null  # Will be populated by generate_config job

resource_requirements:
  - name: minimal
    num_cpus: 1
    memory: 1g
    runtime: PT5M

  - name: gpu_small
    num_cpus: 4
    num_gpus: 1
    memory: 16g
    runtime: PT1H

  - name: gpu_large
    num_cpus: 8
    num_gpus: 2
    memory: 32g
    runtime: PT4H

Understanding the Specification

Key elements:

• user_data: section - Defines data entries, similar to files:
• data: null - Initial value; will be populated by a job
• ${user_data.output.ml_config} - Job will write to this user_data (creates it)
• ${user_data.input.ml_config} - Job reads from this user_data (creates dependency)

The dependency flow:

1. generate_config outputs ml_config → runs first
2. train_model and evaluate_model input ml_config → blocked until step 1 completes
3. After generate_config finishes, both become ready and can run in parallel

Step 2: Create and Initialize the Workflow

# Create the workflow
WORKFLOW_ID=$(torc workflows create user_data_workflow.yaml -f json | jq -r '.id')
echo "Created workflow: $WORKFLOW_ID"

# Initialize jobs
torc workflows initialize-jobs $WORKFLOW_ID

Step 3: Check Initial State

Before running, examine the user_data:

# Check user_data - should be null
torc user-data list $WORKFLOW_ID

Output:

╭────┬───────────┬──────┬─────────────╮
│ ID │ Name      │ Data │ Workflow ID │
├────┼───────────┼──────┼─────────────┤
│ 1  │ ml_config │ null │ 1           │
╰────┴───────────┴──────┴─────────────╯

Check job statuses:

torc jobs list $WORKFLOW_ID

You should see:

• generate_config: ready (no input dependencies)
• train_model: blocked (waiting for ml_config)
• evaluate_model: blocked (waiting for ml_config)

Step 4: Run the Workflow

torc run $WORKFLOW_ID

Step 5: Observe the Data Flow

After generate_config completes, check the updated user_data:

torc user-data list $WORKFLOW_ID -f json | jq '.[] | {name, data}'

Output:

{
  "name": "ml_config",
  "data": {
    "learning_rate": 0.001,
    "batch_size": 32,
    "epochs": 10
  }
}

The data is now stored in the database. At this point:

• train_model and evaluate_model unblock
• Both can read the configuration and run in parallel

Step 6: Verify Completion

After the workflow completes:

torc results list $WORKFLOW_ID

All three jobs should show return code 0.

How User Data Dependencies Work

The mechanism is identical to file dependencies:

Torc substitutes these variables with the actual user_data ID at runtime, and the torc user-data CLI commands use that ID to read/write the data.

Accessing User Data in Your Code

From within a job, you can:

Read user_data:

# Get the full record
torc user-data get $USER_DATA_ID

# Get just the data field
torc user-data get $USER_DATA_ID | jq '.data'

# Save to a file for your application
torc user-data get $USER_DATA_ID | jq '.data' > config.json

Write user_data:

# Update with JSON data
torc user-data update $USER_DATA_ID --data '{"key": "value"}'

# Update from a file
torc user-data update $USER_DATA_ID --data "$(cat results.json)"

What You Learned

In this tutorial, you learned:

• ✅ What user_data is: structured data stored in the Torc database
• ✅ When to use it: configurations, parameters, metadata (not large files)
• ✅ How to define user_data entries with the user_data: section
• ✅ How ${user_data.input.*} and ${user_data.output.*} create dependencies
• ✅ How to read and write user_data from within jobs

Common Patterns

Dynamic Configuration Generation

jobs:
  - name: analyze_data
    command: |
      # Analyze data and determine optimal parameters
      OPTIMAL_LR=$(python analyze.py --find-optimal-lr)
      torc user-data update ${user_data.output.optimal_params} \
        --data "{\"learning_rate\": $OPTIMAL_LR}"

Collecting Results from Multiple Jobs

jobs:
  - name: worker_{i}
    command: |
      RESULT=$(python process.py --id {i})
      torc user-data update ${user_data.output.result_{i}} --data "$RESULT"
    parameters:
      i: "1:10"

  - name: aggregate
    command: |
      # Collect all results
      for i in $(seq 1 10); do
        torc user-data get ${user_data.input.result_$i} >> all_results.json
      done
      python aggregate.py all_results.json

Next Steps

• Tutorial 4: Simple Parameterization - Create parameter sweeps
• Tutorial 5: Advanced Parameterization - Multi-dimensional grid searches

Monitoring & Debugging

Tools and techniques for monitoring workflows and debugging issues.

• Web Dashboard - Browser-based workflow monitoring
• Terminal UI (TUI) - Interactive terminal interface
• Workflow Reports - Generating workflow summaries
• Resource Monitoring - Tracking resource usage
• Debugging Workflows - Troubleshooting workflow issues
• Working with Logs - Accessing and analyzing logs

Web Dashboard (torc-dash)

The Torc Dashboard (torc-dash) provides a modern web-based interface for monitoring and managing workflows, offering an intuitive alternative to the command-line interface.

Overview

torc-dash is a Rust-based web application that allows you to:

• Monitor workflows and jobs with real-time status updates
• Create and run workflows by uploading specification files (YAML, JSON, JSON5, KDL)
• Visualize workflow DAGs with interactive dependency graphs
• Debug failed jobs with integrated log file viewer
• Generate resource plots from time series monitoring data
• Manage torc-server start/stop in standalone mode
• Live event streaming via Server-Sent Events (SSE) for real-time job and compute node events

Installation

Building from Source

torc-dash is built as part of the Torc workspace:

# Build torc-dash
cargo build --release -p torc-dash

# Binary location
./target/release/torc-dash

Prerequisites

• A running torc-server (or use --standalone mode to auto-start one)
• The torc CLI binary in your PATH (for workflow execution features)

Running the Dashboard

Quick Start (Standalone Mode)

The easiest way to get started is standalone mode, which automatically starts torc-server:

torc-dash --standalone

This will:

1. Start torc-server on an automatically-detected free port
2. Start the dashboard on http://127.0.0.1:8090
3. Configure the dashboard to connect to the managed server

Connecting to an Existing Server

If you already have torc-server running:

# Use default API URL (http://localhost:8080/torc-service/v1)
torc-dash

# Specify custom API URL
torc-dash --api-url http://myserver:9000/torc-service/v1

# Or use environment variable
export TORC_API_URL="http://myserver:9000/torc-service/v1"
torc-dash

Command-Line Options

Options:
  -p, --port <PORT>           Dashboard port [default: 8090]
      --host <HOST>           Dashboard host [default: 127.0.0.1]
  -a, --api-url <API_URL>     Torc server API URL [default: http://localhost:8080/torc-service/v1]
      --torc-bin <PATH>       Path to torc CLI binary [default: torc]
      --torc-server-bin       Path to torc-server binary [default: torc-server]
      --standalone            Auto-start torc-server alongside dashboard
      --server-port <PORT>    Server port in standalone mode (0 = auto-detect) [default: 0]
      --database <PATH>       Database path for standalone server
      --completion-check-interval-secs <SECS>  Server polling interval [default: 5]

Features

Workflows Tab

The main workflows view provides:

• Workflow list with ID, name, timestamp, user, and description
• Create Workflow button to upload new workflow specifications
• Quick actions for each workflow:
  • View details and DAG visualization
  • Initialize/reinitialize workflow
  • Run locally or submit to scheduler
  • Delete workflow

Creating Workflows

Click "Create Workflow" to open the creation dialog:

1. Upload a file: Drag and drop or click to select a workflow specification file
   • Supports YAML, JSON, JSON5, and KDL formats
2. Or enter a file path: Specify a path on the server filesystem
3. Click "Create" to register the workflow

Details Tab

Explore workflow components with interactive tables:

• Jobs: View all jobs with status, name, command, and dependencies
• Files: Input/output files with paths and timestamps
• User Data: Key-value data passed between jobs
• Results: Execution results with return codes and resource metrics
• Compute Nodes: Available compute resources
• Resource Requirements: CPU, memory, GPU specifications
• Schedulers: Slurm scheduler configurations

Features:

• Workflow selector: Filter by workflow
• Column sorting: Click headers to sort
• Row filtering: Type in filter boxes (supports column:value syntax)
• Auto-refresh: Toggle automatic updates

DAG Visualization

Click "View" on any workflow to see an interactive dependency graph:

• Nodes represent jobs, colored by status
• Edges show dependencies (file-based and explicit)
• Zoom, pan, and click nodes for details
• Legend shows status colors

Debugging Tab

Investigate failed jobs with the integrated debugger:

1. Select a workflow
2. Configure output directory (where logs are stored)
3. Toggle "Show only failed jobs" to focus on problems
4. Click "Generate Report" to fetch results
5. Click any job row to view its log files:
   • stdout: Standard output from the job
   • stderr: Error output and stack traces
   • Copy file paths with one click

Events Tab (SSE Live Streaming)

Monitor workflow activity in real-time using Server-Sent Events (SSE):

• Live event streaming - events appear instantly without polling
• Connection status indicator - shows Live/Reconnecting/Disconnected status
• Event types displayed:
  • job_started / job_completed / job_failed - Job lifecycle events
  • compute_node_started / compute_node_stopped - Worker node lifecycle
  • workflow_started / workflow_reinitialized - Workflow initialization events
  • scheduler_node_created - Slurm scheduler events
• Clear button to reset the event list
• Auto-reconnect on connection loss

Resource Plots Tab

Visualize CPU and memory usage over time:

1. Enter a base directory containing resource database files
2. Click "Scan for Databases" to find .db files
3. Select databases to plot
4. Click "Generate Plots" for interactive Plotly charts

Requires workflows run with granularity: "time_series" in resource_monitor config.

Configuration Tab

Server Management

Start and stop torc-server directly from the dashboard:

• Server Port: Port to listen on (0 = auto-detect free port)
• Database Path: SQLite database file location
• Completion Check Interval: How often to check for job completions
• Log Level: Server logging verbosity

Click "Start Server" to launch, "Stop Server" to terminate.

API Configuration

• API URL: Torc server endpoint
• Test Connection: Verify connectivity

Settings are saved to browser local storage.

Common Usage Patterns

Running a Workflow

1. Navigate to Workflows tab
2. Click Create Workflow
3. Upload your specification file
4. Click Create
5. Click Initialize on the new workflow
6. Click Run Locally (or Submit for Slurm)
7. Monitor progress in the Details tab or Events tab

Debugging a Failed Workflow

1. Go to the Debugging tab
2. Select the workflow
3. Check "Show only failed jobs"
4. Click Generate Report
5. Click on a failed job row
6. Review the stderr tab for error messages
7. Check stdout for context

Monitoring Active Jobs

1. Open Details tab
2. Select "Jobs" and your workflow
3. Enable Auto-refresh
4. Watch job statuses update in real-time

Security Considerations

1. Network Access: By default, binds to 127.0.0.1 (localhost only)
2. Remote Access: Use --host 0.0.0.0 with caution; consider a reverse proxy with HTTPS
3. Authentication: Torc server supports htpasswd-based authentication (see Authentication)

Troubleshooting

Cannot Connect to Server

• Verify torc-server is running: curl http://localhost:8080/torc-service/v1/workflows
• Check the API URL in Configuration tab
• In standalone mode, check server output for startup errors

Workflow Creation Fails

• Ensure workflow specification is valid YAML/JSON/KDL
• Check file paths are accessible from the server
• Review browser console for error details

Resource Plots Not Showing

• Verify workflow used granularity: "time_series" mode
• Confirm .db files exist in the specified directory
• Check that database files contain data

Standalone Mode Server Won't Start

• Verify torc-server binary is in PATH or specify --torc-server-bin
• Check if the port is already in use
• Review console output for error messages

Architecture

torc-dash is a self-contained Rust binary with:

• Axum web framework for HTTP server
• Embedded static assets (HTML, CSS, JavaScript)
• API proxy to forward requests to torc-server
• CLI integration for workflow operations

The frontend uses vanilla JavaScript with:

• Cytoscape.js for DAG visualization
• Plotly.js for resource charts
• Custom components for tables and forms

Next Steps

• Dashboard Deployment Tutorial - Detailed deployment scenarios
• Authentication - Secure your deployment
• Server Deployment - Production server configuration

Terminal User Interface (TUI)

The Torc TUI provides a full-featured terminal interface for managing workflows, designed for HPC users working in terminal-over-SSH environments.

Quick Start

# Option 1: Connect to an existing server
torc-server run &   # Start server in background
torc tui            # Launch the TUI

# Option 2: Standalone mode (auto-starts server)
torc tui --standalone

# Option 3: Start TUI without server (manual connection)
torc tui            # Shows warning, use 'S' to start server

Standalone Mode

Use --standalone for single-machine development or testing:

# Basic standalone mode
torc tui --standalone

# Custom port
torc tui --standalone --port 8090

# Custom database location
torc tui --standalone --database /path/to/workflows.db

In standalone mode, the TUI automatically starts a torc-server process with the specified configuration.

Features

• Workflow Management: Create, initialize, run, submit, cancel, reset, and delete workflows
• Job Management: View details, logs, cancel, terminate, or retry jobs
• Real-time Monitoring: Auto-refresh, manual refresh, color-coded status
• Live Event Streaming: Server-Sent Events (SSE) for real-time job and compute node events
• Server Management: Start/stop torc-server directly from the TUI
• File Viewing: Preview workflow files with search and navigation
• DAG Visualization: Text-based dependency graph

Interface Overview

When the TUI starts, you'll see:

┌─ Torc Management Console ────────────────────────────────────────┐
│ ?: help | n: new | i: init | I: reinit | R: reset | x: run ...  │
└──────────────────────────────────────────────────────────────────┘
┌─ Server ─────────────────────────────────────────────────────────┐
│ http://localhost:8080/torc-service/v1  S: start | K: stop | O: output │
└──────────────────────────────────────────────────────────────────┘
┌─ User Filter ────────────────────────────────────────────────────┐
│ Current: yourname  (press 'w' to change, 'a' for all users)     │
└──────────────────────────────────────────────────────────────────┘
┌─ Workflows [FOCUSED] ────────────────────────────────────────────┐
│ >> 1  | my-workflow    | yourname | Example workflow            │
│    2  | data-pipeline  | yourname | Data processing pipeline    │
└──────────────────────────────────────────────────────────────────┘

Basic Navigation

Workflow Actions

Select a workflow and use these keys:

All destructive actions show a confirmation dialog.

Handling Existing Output Files

When initializing or re-initializing a workflow, if existing output files are detected, the TUI will show a confirmation dialog listing the files that will be deleted. Press y to proceed with --force or n to cancel.

Job Management

Navigate to the Jobs tab (→ then Tab if needed) to manage individual jobs:

Job Status Colors

• Green: Completed
• Yellow: Running
• Red: Failed
• Magenta: Canceled/Terminated
• Blue: Pending/Scheduled
• Cyan: Ready
• Gray: Blocked

Log Viewer

Press l on a job to view its logs:

File Viewer

Navigate to the Files tab and press Enter on a file to view its contents. The file viewer supports:

• Files up to 1MB
• Binary files show a hex dump preview
• Same navigation keys as the log viewer

Events Tab (SSE Live Streaming)

The Events tab shows real-time events from the server using Server-Sent Events (SSE). When you select a workflow and navigate to the Events tab, the TUI establishes a live connection to receive events as they occur.

Event types displayed:

• job_started / job_completed / job_failed - Job lifecycle events
• compute_node_started / compute_node_stopped - Worker node lifecycle
• workflow_started / workflow_reinitialized - Workflow initialization events
• scheduler_node_created - Slurm scheduler events

Features:

• Events appear instantly without polling
• Newest events shown at the top
• Filter by Event Type or Data content using f
• Title shows "[SSE Live]" indicator when connected

Server Management

The TUI can start and manage a torc-server instance:

The server status indicator in the connection bar shows:

• ● (green): Server is running (managed by TUI)
• ○ (yellow): Server was started but has stopped
• No indicator: External server (not managed by TUI)

Connection Settings

Auto-Refresh

Press A to toggle auto-refresh (30-second interval). When enabled, the workflow list and details refresh automatically.

Configuration

The TUI respects Torc's layered configuration system:

1. Interactive changes in TUI (press u to change server URL)
2. Environment variables (TORC_CLIENT__API_URL)
3. Local config file (./torc.toml)
4. User config file (~/.config/torc/config.toml)
5. System config file (/etc/torc/config.toml)
6. Default values

Troubleshooting

"Could not connect to server"

1. Ensure the Torc server is running: torc-server run
2. Check the server URL: press u to update if needed
3. Verify network connectivity

"No log content available"

Logs may not be available if:

• The job hasn't run yet
• You're on a different machine than where jobs ran
• The output directory is in a different location

Screen rendering issues

• Ensure your terminal supports UTF-8 and 256 colors
• Try resizing your terminal window
• Press r to force a refresh

TUI vs Web Dashboard

Choose the TUI for: SSH sessions, HPC environments, quick operations, low-bandwidth connections.

Choose torc-dash for: Rich visualizations, resource plots, team dashboards.

How to Generate Workflow Reports

This guide shows how to check workflow completion status and generate summary reports.

Check if a Workflow is Complete

Before generating reports, verify that your workflow has finished:

torc workflows is-complete <workflow_id>

If you omit the workflow ID, you'll be prompted to select from your workflows:

torc workflows is-complete

Example output:

Workflow 42 completion status:
  Is Complete: true
  Is Canceled: false
  Needs Completion Script: false

For JSON output:

torc -f json workflows is-complete <workflow_id>

Generate a Workflow Summary

Once a workflow is complete, generate a summary report:

torc reports summary <workflow_id>

If you omit the workflow ID, you'll be prompted to select from your workflows:

torc reports summary

Example output:

Workflow Summary
================

Workflow ID: 42
Name: data_processing_pipeline
User: jsmith

Job Status (total: 100):
  Completed:     95 ✓
  Failed:        5 ✗

Total Execution Time: 2h 30m 15s
Walltime:             3h 15m 42s

If all jobs succeeded:

Workflow Summary
================

Workflow ID: 42
Name: simulation_run
User: jsmith

Job Status (total: 50):
  Completed:     50 ✓

Total Execution Time: 45m 30s

✓ All jobs completed successfully!

Only non-zero status counts are displayed.

Continuous Monitoring

This command can be very convenient, but be mindful of your workflow size (number of jobs) and network load if you are using a shared server.

watch -n 10 torc reports summary <workflow_id>

JSON Output

This is useful for scripts:

torc -f json reports summary <workflow_id>

{
  "workflow_id": 42,
  "workflow_name": "data_processing_pipeline",
  "workflow_user": "jsmith",
  "total_jobs": 100,
  "jobs_by_status": {
    "uninitialized": 0,
    "blocked": 0,
    "ready": 0,
    "pending": 0,
    "running": 0,
    "completed": 95,
    "failed": 5,
    "canceled": 0,
    "terminated": 0,
    "disabled": 0
  },
  "total_exec_time_minutes": 150.25,
  "total_exec_time_formatted": "2h 30m 15s",
  "walltime_seconds": 11742.0,
  "walltime_formatted": "3h 15m 42s"
}

Use in Scripts

Combine these commands in automation scripts:

#!/bin/bash
WORKFLOW_ID=$1

# Check completion status
if torc -f json workflows is-complete "$WORKFLOW_ID" | jq -e '.is_complete' > /dev/null; then
    echo "Workflow complete, generating summary..."
    torc -f json reports summary "$WORKFLOW_ID" > "summary_${WORKFLOW_ID}.json"
else
    echo "Workflow not yet complete"
    exit 1
fi

Check Resource Utilization

After a workflow completes, check if any jobs exceeded their resource limits:

torc reports check-resource-utilization <workflow_id>

Example output when jobs stayed within limits:

Resource Utilization Report for Workflow 42
===========================================

All 50 jobs completed within resource limits.

Example output when jobs exceeded limits:

Resource Utilization Report for Workflow 42
===========================================

Jobs exceeding resource limits:

Job ID  Name           Memory Limit  Peak Memory  Status
------  -------------  ------------  -----------  ------
123     train_model_1  16g           18.2g        EXCEEDED
124     train_model_2  16g           17.8g        EXCEEDED

Recommendation: Increase memory allocation for affected jobs.

This helps identify jobs that may have been killed due to out-of-memory conditions or that are at risk of failure in future runs.

Related Commands

• torc workflows status <id> - View current job status counts
• torc results list <id> - List individual job results
• torc reports check-resource-utilization <id> - Check for resource violations
• torc reports results <id> - Generate detailed results with log file paths

Next Steps

• Resource Monitoring - Track CPU and memory usage
• Debugging Workflows - Troubleshoot failed jobs

How to Monitor Resource Usage

This guide shows how to track CPU and memory usage of your workflow jobs and identify resource requirement mismatches.

Enable Resource Monitoring

Resource monitoring is enabled by default for all workflows. To explicitly configure it, add a resource_monitor section to your workflow specification:

name: "My Workflow"

resource_monitor:
  enabled: true
  granularity: "summary"       # or "time_series"
  sample_interval_seconds: 5

jobs:
  # ... your jobs

To disable monitoring when creating a workflow:

torc workflows create my_workflow.yaml --no-resource-monitoring

View Summary Metrics

For workflows using summary mode (default), view resource metrics with:

torc results list <workflow_id>

The output includes columns for peak and average CPU and memory usage.

Check for Resource Violations

Use check-resource-utilization to identify jobs that exceeded their specified requirements:

# Check latest run
torc reports check-resource-utilization <workflow_id>

# Check a specific run
torc reports check-resource-utilization <workflow_id> --run-id <run_id>

# Show all jobs, not just violations
torc reports check-resource-utilization <workflow_id> --all

Example output:

⚠ Found 3 resource over-utilization violations:

Job ID | Job Name         | Resource | Specified | Peak Used | Over-Utilization
-------|------------------|----------|-----------|-----------|------------------
15     | train_model      | Memory   | 8.00 GB   | 10.50 GB  | +31.3%
15     | train_model      | Runtime  | 2h 0m 0s  | 2h 45m 0s | +37.5%
16     | large_preprocess | CPU      | 800%      | 950.5%    | +18.8%

Adjust Resource Requirements

After identifying violations, update your workflow specification:

# Before: job used 10.5 GB but was allocated 8 GB
resource_requirements:
  - name: training
    memory: 8g
    runtime: PT2H

# After: increased with buffer
resource_requirements:
  - name: training
    memory: 12g       # 10.5 GB peak + 15% buffer
    runtime: PT3H     # 2h 45m actual + buffer

Guidelines for buffers:

• Memory: Add 10-20% above peak usage
• Runtime: Add 15-30% above actual duration
• CPU: Round up to next core count

Enable Time Series Monitoring

For detailed resource analysis over time, switch to time series mode:

resource_monitor:
  granularity: "time_series"
  sample_interval_seconds: 2

This creates a SQLite database with samples at regular intervals.

Generate Resource Plots

Create interactive visualizations from time series data:

# Generate all plots
torc plot-resources output/resource_utilization/resource_metrics_*.db \
  -o plots/

# Generate plots for specific jobs
torc plot-resources output/resource_utilization/resource_metrics_*.db \
  -o plots/ \
  --job-ids 15,16

The tool generates:

• Individual job plots showing CPU, memory, and process count over time
• Overview plots comparing all jobs
• Summary dashboard with bar charts

Query Time Series Data Directly

Access the SQLite database for custom analysis:

sqlite3 -table output/resource_utilization/resource_metrics_1_1.db

-- View samples for a specific job
SELECT job_id, timestamp, cpu_percent, memory_bytes, num_processes
FROM job_resource_samples
WHERE job_id = 1
ORDER BY timestamp;

-- View job metadata
SELECT * FROM job_metadata;

Troubleshooting

No metrics recorded

• Check that monitoring wasn't disabled with --no-resource-monitoring
• Ensure jobs run long enough for at least one sample (default: 5 seconds)

Time series database not created

• Verify the output directory is writable
• Confirm granularity: "time_series" is set in the workflow spec

Missing child process metrics

• Decrease sample_interval_seconds to catch short-lived processes

Next Steps

• Resource Monitoring Reference - Configuration options and database schema
• Managing Resources - Define job resource requirements

Debugging Workflows

When workflows fail or produce unexpected results, Torc provides comprehensive debugging tools to help you identify and resolve issues. The primary debugging tools are:

• torc results list: Prints a table of return codes for each job execution (non-zero means failure)
• torc reports results: Generates a detailed JSON report containing job results and all associated log file paths
• torc logs analyze <output-dir>: Analyzes log files for known error patterns (see Working with Logs)
• torc-dash Debug tab: Interactive web interface for visual debugging with log file viewer

Overview

Torc automatically captures return codes and multiple log files for each job execution:

• Job stdout/stderr: Output from your job commands
• Job runner logs: Internal logs from the Torc job runner
• Slurm logs: Additional logs when using Slurm scheduler (see Debugging Slurm Workflows)

The reports results command consolidates all this information into a single JSON report, making it easy to locate and examine relevant log files for debugging.

Quick Start

View the job return codes in a table:

torc results list <workflow_id>

Results for workflow ID 2:
╭────┬────────┬───────┬────────┬─────────────┬───────────┬──────────┬────────────┬──────────────────────────┬────────╮
│ ID │ Job ID │ WF ID │ Run ID │ Return Code │ Exec Time │ Peak Mem │ Peak CPU % │ Completion Time          │ Status │
├────┼────────┼───────┼────────┼─────────────┼───────────┼──────────┼────────────┼──────────────────────────┼────────┤
│ 4  │ 6      │ 2     │ 1      │ 1           │ 1.01      │ 73.8MB   │ 21.9%      │ 2025-11-13T13:35:43.289Z │ Done   │
│ 5  │ 4      │ 2     │ 1      │ 0           │ 1.01      │ 118.1MB  │ 301.3%     │ 2025-11-13T13:35:43.393Z │ Done   │
│ 6  │ 5      │ 2     │ 1      │ 0           │ 1.01      │ 413.6MB  │ 19.9%      │ 2025-11-13T13:35:43.499Z │ Done   │
╰────┴────────┴───────┴────────┴─────────────┴───────────┴──────────┴────────────┴──────────────────────────┴────────╯

Total: 3 results

View only failed jobs:

torc results list <workflow_id> --failed

Results for workflow ID 2:
╭────┬────────┬───────┬────────┬─────────────┬───────────┬──────────┬────────────┬──────────────────────────┬────────╮
│ ID │ Job ID │ WF ID │ Run ID │ Return Code │ Exec Time │ Peak Mem │ Peak CPU % │ Completion Time          │ Status │
├────┼────────┼───────┼────────┼─────────────┼───────────┼──────────┼────────────┼──────────────────────────┼────────┤
│ 4  │ 6      │ 2     │ 1      │ 1           │ 1.01      │ 73.8MB   │ 21.9%      │ 2025-11-13T13:35:43.289Z │ Done   │
╰────┴────────┴───────┴────────┴─────────────┴───────────┴──────────┴────────────┴──────────────────────────┴────────╯

Generate a debugging report for a workflow:

# Generate report for a specific workflow
torc reports results <workflow_id>

# Specify custom output directory (default: "output")
torc reports results <workflow_id> --output-dir /path/to/output

# Include all workflow runs (default: only latest run)
torc reports results <workflow_id> --all-runs

# Interactive workflow selection (if workflow_id omitted)
torc reports results

The command outputs a comprehensive JSON report to stdout. Redirect it to a file for easier analysis:

torc reports results <workflow_id> > debug_report.json

Report Structure

Top-Level Fields

The JSON report includes workflow-level information:

{
  "workflow_id": 123,
  "workflow_name": "my_pipeline",
  "workflow_user": "researcher",
  "all_runs": false,
  "total_results": 5,
  "results": [...]
}

Fields:

• workflow_id: Unique identifier for the workflow
• workflow_name: Human-readable workflow name
• workflow_user: Owner of the workflow
• all_runs: Whether report includes all historical runs or just the latest
• total_results: Number of job results in the report
• results: Array of individual job result records

Job Result Records

Each entry in the results array contains detailed information about a single job execution:

{
  "job_id": 456,
  "job_name": "preprocess_data",
  "status": "Done",
  "run_id": 1,
  "return_code": 0,
  "completion_time": "2024-01-15T14:30:00.000Z",
  "exec_time_minutes": 5.2,
  "compute_node_id": 789,
  "compute_node_type": "local",
  "job_stdout": "output/job_stdio/job_456.o",
  "job_stderr": "output/job_stdio/job_456.e",
  "job_runner_log": "output/job_runner_hostname_123_1.log"
}

Core Fields:

• job_id: Unique identifier for the job
• job_name: Human-readable job name from workflow spec
• status: Job status (Done, Terminated, Failed, etc.)
• run_id: Workflow run number (increments on reinitialization)
• return_code: Exit code from job command (0 = success)
• completion_time: ISO 8601 timestamp when job completed
• exec_time_minutes: Duration of job execution in minutes

Compute Node Fields:

• compute_node_id: ID of the compute node that executed the job
• compute_node_type: Type of compute node ("local" or "slurm")

Log File Paths

The report includes paths to all log files associated with each job. The specific files depend on the compute node type.

Local Runner Log Files

For jobs executed by the local job runner (compute_node_type: "local"):

{
  "job_stdout": "output/job_stdio/job_456.o",
  "job_stderr": "output/job_stdio/job_456.e",
  "job_runner_log": "output/job_runner_hostname_123_1.log"
}

Log File Descriptions:

1. job_stdout (output/job_stdio/job_<workflow_id>_<job_id>_<run_id>.o):

   • Standard output from your job command
   • Contains print statements, normal program output
   • Use for: Checking expected output, debugging logic errors

2. job_stderr (output/job_stdio/job_<workflow_id>_<job_id>_<run_id>.e):

   • Standard error from your job command
   • Contains error messages, warnings, stack traces
   • Use for: Investigating crashes, exceptions, error messages

3. job_runner_log (output/job_runner_<hostname>_<workflow_id>_<run_id>.log):

   • Internal Torc job runner logging
   • Shows job lifecycle events, resource monitoring, process management
   • Use for: Understanding Torc's job execution behavior, timing issues

Log path format conventions:

• Job stdio logs use job ID in filename
• Runner logs use hostname, workflow ID, and run ID
• All paths are relative to the specified --output-dir

Slurm Runner Log Files

For jobs executed via Slurm scheduler (compute_node_type: "slurm"), additional log files are available including Slurm stdout/stderr, environment logs, and dmesg logs.

See Debugging Slurm Workflows for detailed information about Slurm-specific log files and debugging tools.

Using the torc-dash Debugging Tab

The torc-dash web interface provides an interactive Debugging tab for visual debugging of workflow jobs. This is often the quickest way to investigate failed jobs without using command-line tools.

Accessing the Debugging Tab

1. Start torc-dash (standalone mode recommended for quick setup):

   torc-dash --standalone
   

2. Open your browser to http://localhost:8090

3. Select a workflow from the dropdown in the sidebar

4. Click the Debugging tab in the navigation

Features

Job Results Report

The Debug tab provides a report generator with the following options:

• Output Directory: Specify where job logs are stored (default: output). This must match the directory used during workflow execution.

• Include all runs: Check this to see results from all workflow runs, not just the latest. Useful for comparing job behavior across reinitializations.

• Show only failed jobs: Filter to display only jobs with non-zero return codes. This is checked by default to help you focus on problematic jobs.

Click Generate Report to fetch job results from the server.

Job Results Table

After generating a report, the Debug tab displays an interactive table showing:

• Job ID: Unique identifier for the job
• Job Name: Human-readable name from the workflow spec
• Status: Job completion status (Done, Terminated, etc.)
• Return Code: Exit code (0 = success, non-zero = failure)
• Execution Time: Duration in minutes
• Run ID: Which workflow run the result is from

Click any row to select a job and view its log files.

Log File Viewer

When you select a job from the table, the Log File Viewer displays:

• stdout tab: Standard output from the job command

  • Shows print statements and normal program output
  • Useful for checking expected behavior and debugging logic

• stderr tab: Standard error from the job command

  • Shows error messages, warnings, and stack traces
  • Primary location for investigating crashes and exceptions

Each tab includes:

• Copy Path button: Copy the full file path to clipboard
• File path display: Shows where the log file is located
• Scrollable content viewer: Dark-themed viewer for easy reading

Quick Debugging Workflow with torc-dash

1. Open torc-dash and select your workflow from the sidebar
2. Go to the Debugging tab
3. Ensure "Show only failed jobs" is checked
4. Click Generate Report
5. Click on a failed job in the results table
6. Review the stderr tab for error messages
7. Check the stdout tab for context about what the job was doing

When to Use torc-dash vs CLI

Use torc-dash Debugging tab when:

• You want a visual, interactive debugging experience
• You need to quickly scan multiple failed jobs
• You're investigating jobs and want to easily switch between stdout/stderr
• You prefer not to construct jq queries manually

Use CLI tools (torc reports results) when:

• You need to automate failure detection in CI/CD
• You want to save reports for archival or version control
• You're working on a remote server without browser access
• You need to process results programmatically

Common Debugging Workflows

Investigating Failed Jobs

When a job fails, follow these steps:

1. Generate the debug report:

   torc reports results <workflow_id> > debug_report.json
   

2. Find the failed job using jq or similar tool:

   # Find jobs with non-zero return codes
   jq '.results[] | select(.return_code != 0)' debug_report.json
   
   # Find jobs with specific status
   jq '.results[] | select(.status == "Done")' debug_report.json
   

3. Check the job's stderr for error messages:

   # Extract stderr path for a specific job
   STDERR_PATH=$(jq -r '.results[] | select(.job_name == "my_failing_job") | .job_stderr' debug_report.json)
   
   # View the error output
   cat "$STDERR_PATH"
   

4. Review job stdout for context:

   STDOUT_PATH=$(jq -r '.results[] | select(.job_name == "my_failing_job") | .job_stdout' debug_report.json)
   cat "$STDOUT_PATH"
   

5. Check runner logs for execution issues:

   LOG_PATH=$(jq -r '.results[] | select(.job_name == "my_failing_job") | .job_runner_log' debug_report.json)
   cat "$LOG_PATH"
   

Searching Log Files with Grep

Torc's log messages use a structured key=value format that makes them easy to search with grep. This is especially useful for tracing specific jobs or workflows across multiple log files.

Search for all log entries related to a specific workflow:

# Find all log lines for workflow 123
grep -r "workflow_id=123" output/

# Find all log lines for workflow 123 in job runner logs only
grep -r "workflow_id=123" output/job_runner_*.log

Search for a specific job:

# Find all log lines for job 456
grep -r "job_id=456" output/

# Find log lines for job 456 with more context (2 lines before/after)
grep -r -C 2 "job_id=456" output/

Combine workflow and job searches:

# Find log lines for job 456 in workflow 123
grep -r "workflow_id=123" output/ | grep "job_id=456"

# Alternative using extended regex
grep -rE "workflow_id=123.*job_id=456" output/

Search for specific runs or attempts:

# Find all log lines for run 2 of workflow 123
grep -r "workflow_id=123" output/ | grep "run_id=2"

# Find retry attempts for a specific job
grep -r "job_id=456" output/ | grep "attempt_id="

# Find entries for a specific compute node
grep -r "compute_node_id=789" output/

Common log message patterns to search for:

# Find job start events
grep -r "Job started workflow_id=" output/

# Find job completion events
grep -r "Job completed workflow_id=" output/

# Find failed jobs
grep -r "status=failed" output/

# Find all job process completions with return codes
grep -r "Job process completed" output/ | grep "return_code="

Tip: Redirect grep output to a file for easier analysis of large result sets:

grep -r "workflow_id=123" output/ > workflow_123_logs.txt

Example: Complete Debugging Session

# 1. Generate report
torc reports results 123 > report.json

# 2. Check overall success/failure counts
echo "Total jobs: $(jq '.total_results' report.json)"
echo "Failed jobs: $(jq '[.results[] | select(.return_code != 0)] | length' report.json)"

# 3. List all failed jobs with their names
jq -r '.results[] | select(.return_code != 0) | "\(.job_id): \(.job_name) (exit code: \(.return_code))"' report.json

# Output:
# 456: process_batch_2 (exit code: 1)
# 789: validate_results (exit code: 2)

# 4. Examine stderr for first failure
jq -r '.results[] | select(.job_id == 456) | .job_stderr' report.json | xargs cat

# Output might show:
# FileNotFoundError: [Errno 2] No such file or directory: 'input/batch_2.csv'

# 5. Check if job dependencies completed successfully
# (The missing file might be an output from a previous job)
jq -r '.results[] | select(.job_name == "generate_batch_2") | "\(.status) (exit code: \(.return_code))"' report.json

Debugging Across Multiple Runs

When a workflow has been reinitialized multiple times, compare runs to identify regressions:

# Generate report with all historical runs
torc reports results <workflow_id> --all-runs > full_history.json

# Compare return codes across runs for a specific job
jq -r '.results[] | select(.job_name == "flaky_job") | "Run \(.run_id): exit code \(.return_code)"' full_history.json

# Output:
# Run 1: exit code 0
# Run 2: exit code 1
# Run 3: exit code 0
# Run 4: exit code 1

# Extract stderr paths for failed runs
jq -r '.results[] | select(.job_name == "flaky_job" and .return_code != 0) | "Run \(.run_id): \(.job_stderr)"' full_history.json

Log File Missing Warnings

The reports results command automatically checks for log file existence and prints warnings to stderr if files are missing:

Warning: job stdout log file does not exist for job 456: output/job_stdio/job_456.o
Warning: job runner log file does not exist for job 456: output/job_runner_host1_123_1.log

Common causes of missing log files:

1. Wrong output directory: Ensure --output-dir matches the directory used during workflow execution
2. Logs not yet written: Job may still be running or failed to start
3. Logs cleaned up: Files may have been manually deleted
4. Path mismatch: Output directory moved or renamed after execution

Solution: Verify the output directory and ensure it matches what was passed to torc run or torc slurm schedule-nodes.

Output Directory Management

The --output-dir parameter must match the directory used during workflow execution:

Local Runner

# Execute workflow with specific output directory
torc run <workflow_id> /path/to/my_output

# Generate report using the same directory
torc reports results <workflow_id> --output-dir /path/to/my_output

Slurm Scheduler

# Submit jobs to Slurm with output directory
torc slurm schedule-nodes <workflow_id> --output-dir /path/to/my_output

# Generate report using the same directory
torc reports results <workflow_id> --output-dir /path/to/my_output

Default behavior: If --output-dir is not specified, both the runner and reports command default to ./output.

Best Practices

1. Generate reports after each run: Create a debug report immediately after workflow execution for easier troubleshooting

2. Archive reports with logs: Store the JSON report alongside log files for future reference

   torc reports results "$WF_ID" > "output/report_${WF_ID}_$(date +%Y%m%d_%H%M%S).json"
   

3. Use version control: Commit debug reports for important workflow runs to track changes over time

4. Automate failure detection: Use the report in CI/CD pipelines to automatically detect and report failures

5. Check warnings: Pay attention to warnings about missing log files - they often indicate configuration issues

6. Combine with resource monitoring: Use reports results for log files and reports check-resource-utilization for performance issues

   # Check if job failed due to resource constraints
   torc reports check-resource-utilization "$WF_ID"
   torc reports results "$WF_ID" > report.json
   

7. Filter large reports: For workflows with many jobs, filter the report to focus on relevant jobs

   # Only include failed jobs in filtered report
   jq '{workflow_id, workflow_name, results: [.results[] | select(.return_code != 0)]}' report.json
   

Troubleshooting Common Issues

"Output directory does not exist" Error

Cause: The specified --output-dir path doesn't exist.

Solution: Verify the directory exists and the path is correct:

ls -ld output/  # Check if directory exists
torc reports results <workflow_id> --output-dir "$(pwd)/output"

Empty Results Array

Cause: No job results exist for the workflow (jobs not yet executed or initialized).

Solution: Check workflow status and ensure jobs have been completed:

torc workflows status <workflow_id>
torc results list <workflow_id>  # Verify results exist

All Log Paths Show Warnings

Cause: Output directory mismatch between execution and report generation.

Solution: Verify the output directory used during execution:

# Check where logs actually are
find . -name "job_*.o" -o -name "job_runner_*.log"

# Use correct output directory in report
torc reports results <workflow_id> --output-dir <correct_path>

Related Commands

• torc results list: View summary of job results in table format
• torc workflows status: Check overall workflow status
• torc reports results: Generate debug report with all log file paths
• torc reports check-resource-utilization: Analyze resource usage and find over-utilized jobs
• torc jobs list: View all jobs and their current status
• torc-dash: Launch web interface with interactive Debugging tab
• torc tui: Launch terminal UI for workflow monitoring

See Also

• Working with Logs — Bundling and analyzing logs
• Debugging Slurm Workflows — Slurm-specific debugging tools

Working with Logs

Torc provides tools for bundling and analyzing workflow logs. These are useful for:

• Sharing logs with colleagues for help debugging
• Archiving completed workflow logs for later reference
• Scanning for errors across all log files at once

Log File Overview

Torc generates several types of log files during workflow execution:

For detailed information about log file contents, see Debugging Workflows and Debugging Slurm Workflows.

Bundling Logs

The torc logs bundle command packages all logs for a workflow into a compressed tarball:

# Bundle all logs for a workflow
torc logs bundle <workflow_id>

# Specify custom output directory (where logs are located)
torc logs bundle <workflow_id> --output-dir /path/to/output

# Save bundle to a specific directory
torc logs bundle <workflow_id> --bundle-dir /path/to/bundles

This creates a wf<id>.tar.gz file containing:

• All job stdout/stderr files (job_wf*_j*_r*.o/e)
• Job runner logs (job_runner_*.log)
• Slurm output files (slurm_output_wf*_sl*.o/e)
• Slurm environment logs (slurm_env_wf*_sl*.log)
• dmesg logs (dmesg_slurm_wf*_sl*.log)
• Bundle metadata (workflow info, collection timestamp)

Example: Sharing Logs

# Bundle workflow logs
torc logs bundle 123 --bundle-dir ./bundles

# Share the bundle
ls ./bundles/
# wf123.tar.gz

# Recipient can extract and analyze
tar -xzf wf123.tar.gz
torc logs analyze wf123/

Analyzing Logs

The torc logs analyze command scans log files for known error patterns:

# Analyze a log bundle tarball
torc logs analyze wf123.tar.gz

# Analyze a log directory directly (auto-detects workflow if only one present)
torc logs analyze output/

# Analyze a directory with multiple workflows (specify which one)
torc logs analyze output/ --workflow-id 123

Detected Error Patterns

The analyzer scans for common failure patterns including:

Memory Errors:

• Out of memory, OOM kills
• std::bad_alloc (C++)
• MemoryError (Python)

Slurm Errors:

• Time limit exceeded
• Node failures
• Preemption

GPU/CUDA Errors:

• CUDA out of memory
• GPU memory exceeded

Crashes:

• Segmentation faults
• Bus errors
• Signal kills

Python Errors:

• Tracebacks
• Import errors

File System Errors:

• No space left on device
• Permission denied

Network Errors:

• Connection refused/timed out

Example Output

Log Analysis Results
====================

Analyzing: output/

Files with detected errors:

  output/job_stdio/job_wf123_j456_r1.e
    Line 42: MemoryError: Unable to allocate 8.00 GiB
    Severity: critical
    Type: Python Memory Error

  output/slurm_output_wf123_sl789.e
    Line 15: slurmstepd: error: Detected 1 oom-kill event(s)
    Severity: critical
    Type: Out of Memory (OOM) Kill

Summary:
  Total files scanned: 24
  Files with errors: 2
  Error types found: MemoryError, OOM Kill

Excluding Files

Environment variable files (slurm_env_*.log) are automatically excluded from error analysis since they contain configuration data, not error logs.

Workflow: Bundle and Share

A common pattern when asking for help:

# 1. Bundle the workflow logs
torc logs bundle <workflow_id>

# 2. Analyze locally first to understand the issue
torc logs analyze wf<id>.tar.gz

# 3. Share the bundle with your colleague/support
#    They can extract and analyze:
tar -xzf wf<id>.tar.gz
torc logs analyze wf<id>/

Related Commands

• torc reports results: Generate JSON report with all log file paths
• torc results list: View summary table of job return codes
• torc slurm parse-logs: Parse Slurm logs for error patterns (Slurm-specific)
• torc slurm sacct: Collect Slurm accounting data

See Also

• Debugging Workflows — General debugging workflow and log file details
• Debugging Slurm Workflows — Slurm-specific debugging tools

Reference

Comprehensive reference documentation.

• CLI Reference - Command-line interface documentation
• Job Parameterization - Parameter syntax and options
• Resource Requirements - Specifying job resources
• Environment Variables - Configuration via environment
• Resource Monitoring Database - Metrics database schema
• Configuration - Configuration file reference
• OpenAPI Specification - REST API documentation

CLI Reference

This documentation is automatically generated from the CLI help text.

To regenerate, run:

cargo run --bin generate-cli-docs --features "client,tui,plot_resources"

Command-Line Help for torc

This document contains the help content for the torc command-line program.

Command Overview:

• CLI Reference
• Command-Line Help for torc
  • torc - Subcommands: - Options:
  • torc run - Arguments: - Options:
  • torc submit - Arguments: - Options:
  • torc submit-slurm - Arguments: - Options:
  • torc watch - Arguments: - Options:
  • torc workflows - Subcommands:
  • torc workflows create - Arguments: - Options:
  • torc workflows create-slurm - Arguments: - Options:
  • torc workflows new - Options:
  • torc workflows list - Options:
  • torc workflows get - Arguments: - Options:
  • torc workflows update - Arguments: - Options:
  • torc workflows cancel - Arguments:
  • torc workflows delete - Arguments: - Options:
  • torc workflows archive - Arguments:
  • torc workflows submit - Arguments: - Options:
  • torc workflows run - Arguments: - Options:
  • torc workflows initialize - Arguments: - Options:
  • torc workflows reinitialize - Arguments:
    • Options:
  • torc workflows status - Arguments: - Options:
  • torc workflows reset-status - Arguments:
    • Options:
  • torc workflows execution-plan - Arguments:
  • torc workflows list-actions - Arguments:
    • Options:
  • torc workflows is-complete - Arguments:
  • torc workflows sync-status - Arguments: - Options:
  • torc compute-nodes - Subcommands:
  • torc compute-nodes get - Arguments:
  • torc compute-nodes list - Arguments: - Options:
  • torc files - Subcommands:
  • torc files create - Arguments: - Options:
  • torc files list - Arguments: - Options:
  • torc files get - Arguments:
  • torc files update - Arguments: - Options:
  • torc files delete - Arguments:
  • torc files list-required-existing - Arguments:
  • torc jobs - Subcommands:
  • torc jobs create - Arguments: - Options:
  • torc jobs create-from-file - Arguments: - Options:
  • torc jobs list - Arguments: - Options:
  • torc jobs get - Arguments:
  • torc jobs update - Arguments: - Options:
  • torc jobs delete - Arguments:
  • torc jobs delete-all - Arguments:
  • torc jobs list-resource-requirements - Arguments: - Options:
  • torc job-dependencies - Subcommands:
  • torc job-dependencies job-job - Arguments: - Options:
  • torc job-dependencies job-file - Arguments: - Options:
  • torc job-dependencies job-user-data - Arguments: - Options:
  • torc resource-requirements - Subcommands:
  • torc resource-requirements create - Arguments: - Options:
  • torc resource-requirements list - Arguments: - Options:
  • torc resource-requirements get - Arguments:
  • torc resource-requirements update - Arguments: - Options:
  • torc resource-requirements delete - Arguments:
  • torc events - Subcommands:
  • torc events create - Arguments: - Options:
  • torc events list - Arguments: - Options:
  • torc events monitor - Arguments: - Options:
  • torc events get-latest-event - Arguments:
  • torc events delete - Arguments:
  • torc results - Subcommands:
  • torc results list - Arguments: - Options:
  • torc results get - Arguments:
  • torc results delete - Arguments:
  • torc user-data - Subcommands:
  • torc user-data create - Arguments: - Options:
  • torc user-data list - Arguments: - Options:
  • torc user-data get - Arguments:
  • torc user-data update - Arguments: - Options:
  • torc user-data delete - Arguments:
  • torc user-data delete-all - Arguments:
  • torc user-data list-missing - Arguments:
  • torc slurm - Subcommands:
  • torc slurm create - Arguments: - Options:
  • torc slurm update - Arguments: - Options:
  • torc slurm list - Arguments: - Options:
  • torc slurm get - Arguments:
  • torc slurm delete - Arguments:
  • torc slurm schedule-nodes - Arguments: - Options:
  • torc slurm parse-logs - Arguments: - Options:
  • torc slurm sacct - Arguments: - Options:
  • torc slurm generate - Arguments: - Options:
  • torc slurm regenerate - Arguments: - Options:
  • torc remote - Subcommands:
  • torc remote add-workers - Arguments:
  • torc remote add-workers-from-file - Arguments:
  • torc remote remove-worker - Arguments:
  • torc remote list-workers - Arguments:
  • torc remote run - Arguments: - Options:
  • torc remote status - Arguments: - Options:
  • torc remote stop - Arguments: - Options:
  • torc remote collect-logs - Arguments: - Options:
  • torc remote delete-logs - Arguments: - Options:
  • torc scheduled-compute-nodes - Subcommands:
  • torc scheduled-compute-nodes get - Arguments:
  • torc scheduled-compute-nodes list - Arguments: - Options:
  • torc scheduled-compute-nodes list-jobs - Arguments:
  • torc hpc - Subcommands:
  • torc hpc list
  • torc hpc detect
  • torc hpc show - Arguments:
  • torc hpc partitions - Arguments: - Options:
  • torc hpc match - Options:
  • torc reports - Subcommands:
  • torc reports check-resource-utilization - Arguments: - Options:
  • torc reports results - Arguments: - Options:
  • torc reports summary - Arguments:
  • torc config - Subcommands:
  • torc config show - Options:
  • torc config paths
  • torc config init - Options:
  • torc config validate
  • torc tui - Options:
  • torc plot-resources - Arguments: - Options:
  • torc completions - Arguments:

torc

Torc workflow orchestration system

Usage: torc [OPTIONS] <COMMAND>

Subcommands:

• run — Run a workflow locally (create from spec file or run existing workflow by ID)
• submit — Submit a workflow to scheduler (create from spec file or submit existing workflow by ID)
• submit-slurm — Submit a workflow to Slurm with auto-generated schedulers
• watch — Watch a workflow and automatically recover from failures
• workflows — Workflow management commands
• compute-nodes — Compute node management commands
• files — File management commands
• jobs — Job management commands
• job-dependencies — Job dependency and relationship queries
• resource-requirements — Resource requirements management commands
• events — Event management commands
• results — Result management commands
• user-data — User data management commands
• slurm — Slurm scheduler commands
• remote — Remote worker execution commands (SSH-based distributed execution)
• scheduled-compute-nodes — Scheduled compute node management commands
• hpc — HPC system profiles and partition information
• reports — Generate reports and analytics
• config — Manage configuration files and settings
• tui — Interactive terminal UI for managing workflows
• plot-resources — Generate interactive HTML plots from resource monitoring data
• completions — Generate shell completions

Options:

• --log-level <LOG_LEVEL> — Log level (error, warn, info, debug, trace)

• -f, --format <FORMAT> — Output format (table or json)

• --log-level <LOG_LEVEL> — Log level (error, warn, info, debug, trace)

• -f, --format <FORMAT> — Output format (table or json)

  Default value: table

• --url <URL> — URL of torc server

• --username <USERNAME> — Username for basic authentication

• --password <PASSWORD> — Password for basic authentication (will prompt if username provided but password not)

• --url <URL> — URL of torc server

• --username <USERNAME> — Username for basic authentication

• --password <PASSWORD> — Password for basic authentication (will prompt if username provided but password not)

torc run

Run a workflow locally (create from spec file or run existing workflow by ID)

Usage: torc run [OPTIONS] <WORKFLOW_SPEC_OR_ID>

Arguments:

• <WORKFLOW_SPEC_OR_ID> — Path to workflow spec file (JSON/JSON5/YAML) or workflow ID
• <WORKFLOW_SPEC_OR_ID> — Path to workflow spec file (JSON/JSON5/YAML) or workflow ID

Options:

• --max-parallel-jobs <MAX_PARALLEL_JOBS> — Maximum number of parallel jobs to run concurrently

• --num-cpus <NUM_CPUS> — Number of CPUs available

• --memory-gb <MEMORY_GB> — Memory in GB

• --num-gpus <NUM_GPUS> — Number of GPUs available

• -p, --poll-interval <POLL_INTERVAL> — Job completion poll interval in seconds

• -o, --output-dir <OUTPUT_DIR> — Output directory for jobs

• --skip-checks — Skip validation checks (e.g., scheduler node requirements). Use with caution

• --max-parallel-jobs <MAX_PARALLEL_JOBS> — Maximum number of parallel jobs to run concurrently

• --num-cpus <NUM_CPUS> — Number of CPUs available

• --memory-gb <MEMORY_GB> — Memory in GB

• --num-gpus <NUM_GPUS> — Number of GPUs available

• -p, --poll-interval <POLL_INTERVAL> — Job completion poll interval in seconds

• -o, --output-dir <OUTPUT_DIR> — Output directory for jobs

• --skip-checks — Skip validation checks (e.g., scheduler node requirements). Use with caution

  Default value: false

torc submit

Submit a workflow to scheduler (create from spec file or submit existing workflow by ID)

Requires workflow to have an on_workflow_start action with schedule_nodes. For Slurm workflows without pre-configured schedulers, use submit-slurm instead. Requires workflow to have an on_workflow_start action with schedule_nodes. For Slurm workflows without pre-configured schedulers, use submit-slurm instead.

Usage: torc submit [OPTIONS] <WORKFLOW_SPEC_OR_ID>

Arguments:

• <WORKFLOW_SPEC_OR_ID> — Path to workflow spec file (JSON/JSON5/YAML) or workflow ID
• <WORKFLOW_SPEC_OR_ID> — Path to workflow spec file (JSON/JSON5/YAML) or workflow ID

Options:

• -i, --ignore-missing-data — Ignore missing data (defaults to false)

• -i, --ignore-missing-data — Ignore missing data (defaults to false)

  Default value: false

• --skip-checks — Skip validation checks (e.g., scheduler node requirements). Use with caution

• --skip-checks — Skip validation checks (e.g., scheduler node requirements). Use with caution

  Default value: false

torc submit-slurm

Submit a workflow to Slurm with auto-generated schedulers

Automatically generates Slurm schedulers based on job resource requirements and HPC profile.

WARNING: This command uses heuristics to generate schedulers and workflow actions. For complex workflows with unusual dependency patterns, the generated configuration may not be optimal and could waste allocation time. WARNING: This command uses heuristics to generate schedulers and workflow actions. For complex workflows with unusual dependency patterns, the generated configuration may not be optimal and could waste allocation time.

RECOMMENDED: Preview the generated configuration first with:

torc slurm generate --account workflow.yaml

Review the schedulers and actions to ensure they are appropriate for your workflow before submitting. You can save the output and submit manually: Review the schedulers and actions to ensure they are appropriate for your workflow before submitting. You can save the output and submit manually:

torc slurm generate --account -o workflow_with_schedulers.yaml workflow.yaml torc submit workflow_with_schedulers.yaml torc slurm generate --account -o workflow_with_schedulers.yaml workflow.yaml torc submit workflow_with_schedulers.yaml

Usage: torc submit-slurm [OPTIONS] --account <ACCOUNT> <WORKFLOW_SPEC>

Arguments:

• <WORKFLOW_SPEC> — Path to workflow spec file (JSON/JSON5/YAML/KDL)
• <WORKFLOW_SPEC> — Path to workflow spec file (JSON/JSON5/YAML/KDL)

Options:

• --account <ACCOUNT> — Slurm account to use for allocations

• --hpc-profile <HPC_PROFILE> — HPC profile to use (auto-detected if not specified)

• --single-allocation — Bundle all nodes into a single Slurm allocation per scheduler

• --account <ACCOUNT> — Slurm account to use for allocations

• --hpc-profile <HPC_PROFILE> — HPC profile to use (auto-detected if not specified)

• --single-allocation — Bundle all nodes into a single Slurm allocation per scheduler

  By default, creates one Slurm allocation per node (N×1 mode), which allows jobs to start as nodes become available and provides better fault tolerance. By default, creates one Slurm allocation per node (N×1 mode), which allows jobs to start as nodes become available and provides better fault tolerance.

  With this flag, creates one large allocation with all nodes (1×N mode), which requires all nodes to be available simultaneously but uses a single sbatch.

• -i, --ignore-missing-data — Ignore missing data (defaults to false) With this flag, creates one large allocation with all nodes (1×N mode), which requires all nodes to be available simultaneously but uses a single sbatch.

• -i, --ignore-missing-data — Ignore missing data (defaults to false)

  Default value: false

• --skip-checks — Skip validation checks (e.g., scheduler node requirements). Use with caution

• --skip-checks — Skip validation checks (e.g., scheduler node requirements). Use with caution

  Default value: false

torc watch

Watch a workflow and automatically recover from failures.

Monitors a workflow until completion. With --recover, automatically diagnoses failures, adjusts resource requirements, and resubmits jobs.

Usage: torc watch [OPTIONS] <WORKFLOW_ID>

Usage Modes

1. Basic monitoring (no recovery):

   torc watch 123
   

   Reports failures and exits. Use for manual intervention or AI-assisted recovery.

2. With automatic recovery (--recover):

   torc watch 123 --recover
   

   Automatically diagnoses OOM/timeout failures, adjusts resources, and retries. Runs until all jobs complete or max retries exceeded.

3. With auto-scheduling (--auto-schedule):

   torc watch 123 --auto-schedule
   

   Automatically submits new Slurm allocations when retry jobs are waiting. Essential for workflows using failure handlers that create retry jobs.

Arguments

• <WORKFLOW_ID> — Workflow ID to watch

Options

Polling:

• -p, --poll-interval <POLL_INTERVAL> — Poll interval in seconds (default: 60)
• -o, --output-dir <OUTPUT_DIR> — Output directory for job files (default: output)
• -s, --show-job-counts — Show job counts by status during polling. WARNING: Can cause high server load for large workflows.

Recovery:

• -r, --recover — Enable automatic failure recovery
• -m, --max-retries <MAX_RETRIES> — Maximum number of recovery attempts (default: 3)
• --memory-multiplier <MEMORY_MULTIPLIER> — Memory multiplier for OOM failures (default: 1.5)
• --runtime-multiplier <RUNTIME_MULTIPLIER> — Runtime multiplier for timeout failures (default: 1.5)
• --retry-unknown — Also retry jobs with unknown failure causes (not just OOM or timeout)
• --recovery-hook <RECOVERY_HOOK> — Custom recovery script for unknown failures. The workflow ID is passed as an argument and via TORC_WORKFLOW_ID environment variable.

Auto-scheduling:

• --auto-schedule — Automatically schedule new compute nodes when needed
• --auto-schedule-threshold <N> — Minimum retry jobs before auto-scheduling when schedulers exist (default: 5)
• --auto-schedule-cooldown <SECONDS> — Cooldown between auto-schedule attempts (default: 1800 / 30 min)
• --auto-schedule-stranded-timeout <SECONDS> — Schedule stranded jobs after this timeout even if below threshold (default: 7200 / 2 hrs). Set to 0 to disable.

Auto-Scheduling Behavior

When --auto-schedule is enabled:

1. No schedulers available: Immediately submits new allocations if ready jobs exist.
2. Threshold exceeded: If retry jobs (attempt_id > 1) exceed --auto-schedule-threshold while schedulers are running, submits additional allocations after cooldown.
3. Stranded jobs: If retry jobs are below threshold but waiting longer than --auto-schedule-stranded-timeout, schedules anyway to prevent indefinite waiting.

Examples

# Basic: watch until completion, report failures
torc watch 123

# Recovery: automatically fix OOM/timeout failures
torc watch 123 --recover

# Recovery with aggressive resource increases
torc watch 123 --recover --memory-multiplier 2.0 --runtime-multiplier 2.0

# Recovery including unknown failures (transient errors)
torc watch 123 --recover --retry-unknown

# Auto-schedule: ensure retry jobs get scheduled
torc watch 123 --auto-schedule

# Full production setup: recovery + auto-scheduling
torc watch 123 --recover --auto-schedule

# Custom auto-schedule settings
torc watch 123 --auto-schedule \
    --auto-schedule-threshold 10 \
    --auto-schedule-cooldown 3600 \
    --auto-schedule-stranded-timeout 14400

See Also

• torc recover — One-shot recovery (no continuous monitoring)
• Automatic Failure Recovery — Detailed guide

torc recover

Recover a Slurm workflow from failures (one-shot).

Diagnoses job failures (OOM, timeout), adjusts resource requirements, and resubmits jobs. Use after a workflow has completed with failures. For continuous monitoring, use torc watch --recover instead.

Usage: torc recover [OPTIONS] <WORKFLOW_ID>

Arguments

• <WORKFLOW_ID> — Workflow ID to recover

Options

• -o, --output-dir <OUTPUT_DIR> — Output directory for job files (default: output)
• --memory-multiplier <MEMORY_MULTIPLIER> — Memory multiplier for OOM failures (default: 1.5)
• --runtime-multiplier <RUNTIME_MULTIPLIER> — Runtime multiplier for timeout failures (default: 1.4)
• --retry-unknown — Also retry jobs with unknown failure causes
• --recovery-hook <RECOVERY_HOOK> — Custom recovery script for unknown failures
• --dry-run — Show what would be done without making any changes

When to Use

Use torc recover for:

• One-shot recovery after a workflow has completed with failures
• Manual investigation before retrying (use --dry-run first)
• Workflows where you want to inspect failures before retrying

Use torc watch --recover instead for:

• Continuous monitoring of long-running workflows
• Fully automated recovery without manual intervention
• Production workflows that should self-heal

Examples

# Basic recovery
torc recover 123

# Dry run to preview changes without modifying anything
torc recover 123 --dry-run

# Custom resource multipliers
torc recover 123 --memory-multiplier 2.0 --runtime-multiplier 1.5

# Also retry unknown failures (not just OOM/timeout)
torc recover 123 --retry-unknown

# With custom recovery hook for domain-specific fixes
torc recover 123 --recovery-hook 'bash fix-cluster.sh'

See Also

• torc watch --recover — Continuous monitoring with automatic recovery
• Automatic Failure Recovery — Detailed guide

torc workflows

Workflow management commands

Usage: torc workflows <COMMAND>

Subcommands:

• create — Create a workflow from a specification file (supports JSON, JSON5, YAML, and KDL formats)
• create-slurm — Create a workflow with auto-generated Slurm schedulers
• new — Create a new empty workflow
• list — List workflows
• get — Get a specific workflow by ID
• update — Update an existing workflow
• cancel — Cancel a workflow and all associated Slurm jobs
• delete — Delete one or more workflows
• archive — Archive or unarchive one or more workflows
• submit — Submit a workflow: initialize if needed and schedule nodes for on_workflow_start actions This command requires the workflow to have an on_workflow_start action with schedule_nodes
• run — Run a workflow locally on the current node
• initialize — Initialize a workflow, including all job statuses
• reinitialize — Reinitialize a workflow. This will reinitialize all jobs with a status of canceled, submitting, pending, or terminated. Jobs with a status of done will also be reinitialized if an input_file or user_data record has changed
• status — Get workflow status
• reset-status — Reset workflow and job status
• execution-plan — Show the execution plan for a workflow specification or existing workflow
• list-actions — List workflow actions and their statuses (useful for debugging action triggers)
• is-complete — Check if a workflow is complete
• create — Create a workflow from a specification file (supports JSON, JSON5, YAML, and KDL formats)
• create-slurm — Create a workflow with auto-generated Slurm schedulers
• new — Create a new empty workflow
• list — List workflows
• get — Get a specific workflow by ID
• update — Update an existing workflow
• cancel — Cancel a workflow and all associated Slurm jobs
• delete — Delete one or more workflows
• archive — Archive or unarchive one or more workflows
• submit — Submit a workflow: initialize if needed and schedule nodes for on_workflow_start actions This command requires the workflow to have an on_workflow_start action with schedule_nodes
• run — Run a workflow locally on the current node
• initialize — Initialize a workflow, including all job statuses
• reinitialize — Reinitialize a workflow. This will reinitialize all jobs with a status of canceled, submitting, pending, or terminated. Jobs with a status of done will also be reinitialized if an input_file or user_data record has changed
• status — Get workflow status
• reset-status — Reset workflow and job status
• execution-plan — Show the execution plan for a workflow specification or existing workflow
• list-actions — List workflow actions and their statuses (useful for debugging action triggers)
• is-complete — Check if a workflow is complete
• export — Export a workflow to a portable JSON file
• import — Import a workflow from an exported JSON file
• sync-status — Synchronize job statuses with Slurm (detect and fail orphaned jobs)

torc workflows create

Create a workflow from a specification file (supports JSON, JSON5, YAML, and KDL formats)

Usage: torc workflows create [OPTIONS] --user <USER> <FILE>

Arguments:

• <FILE> — Path to specification file containing WorkflowSpec

• <FILE> — Path to specification file containing WorkflowSpec

  Supported formats: - JSON (.json): Standard JSON format - JSON5 (.json5): JSON with comments and trailing commas - YAML (.yaml, .yml): Human-readable YAML format - KDL (.kdl): KDL document format Supported formats: - JSON (.json): Standard JSON format - JSON5 (.json5): JSON with comments and trailing commas - YAML (.yaml, .yml): Human-readable YAML format - KDL (.kdl): KDL document format

  Format is auto-detected from file extension, with fallback parsing attempted Format is auto-detected from file extension, with fallback parsing attempted

Options:

• -u, --user <USER> — User that owns the workflow (defaults to USER environment variable)

• --no-resource-monitoring — Disable resource monitoring (default: enabled with summary granularity and 5s sample rate)

• -u, --user <USER> — User that owns the workflow (defaults to USER environment variable)

• --no-resource-monitoring — Disable resource monitoring (default: enabled with summary granularity and 5s sample rate)

  Default value: false

• --skip-checks — Skip validation checks (e.g., scheduler node requirements). Use with caution

• --skip-checks — Skip validation checks (e.g., scheduler node requirements). Use with caution

  Default value: false

• --dry-run — Validate the workflow specification without creating it (dry-run mode) Returns a summary of what would be created including job count after parameter expansion

• --dry-run — Validate the workflow specification without creating it (dry-run mode) Returns a summary of what would be created including job count after parameter expansion

torc workflows create-slurm

Create a workflow with auto-generated Slurm schedulers

Automatically generates Slurm schedulers based on job resource requirements and HPC profile. For Slurm workflows without pre-configured schedulers. Automatically generates Slurm schedulers based on job resource requirements and HPC profile. For Slurm workflows without pre-configured schedulers.

Usage: torc workflows create-slurm [OPTIONS] --account <ACCOUNT> --user <USER> <FILE>

Arguments:

• <FILE> — Path to specification file containing WorkflowSpec
• <FILE> — Path to specification file containing WorkflowSpec

Options:

• --account <ACCOUNT> — Slurm account to use for allocations

• --hpc-profile <HPC_PROFILE> — HPC profile to use (auto-detected if not specified)

• --single-allocation — Bundle all nodes into a single Slurm allocation per scheduler

• --account <ACCOUNT> — Slurm account to use for allocations

• --hpc-profile <HPC_PROFILE> — HPC profile to use (auto-detected if not specified)

• --single-allocation — Bundle all nodes into a single Slurm allocation per scheduler

  By default, creates one Slurm allocation per node (N×1 mode), which allows jobs to start as nodes become available and provides better fault tolerance. By default, creates one Slurm allocation per node (N×1 mode), which allows jobs to start as nodes become available and provides better fault tolerance.

  With this flag, creates one large allocation with all nodes (1×N mode), which requires all nodes to be available simultaneously but uses a single sbatch.

• -u, --user <USER> — User that owns the workflow (defaults to USER environment variable)

• --no-resource-monitoring — Disable resource monitoring (default: enabled with summary granularity and 5s sample rate) With this flag, creates one large allocation with all nodes (1×N mode), which requires all nodes to be available simultaneously but uses a single sbatch.

• -u, --user <USER> — User that owns the workflow (defaults to USER environment variable)

• --no-resource-monitoring — Disable resource monitoring (default: enabled with summary granularity and 5s sample rate)

  Default value: false

• --skip-checks — Skip validation checks (e.g., scheduler node requirements). Use with caution

• --skip-checks — Skip validation checks (e.g., scheduler node requirements). Use with caution

  Default value: false

• --dry-run — Validate the workflow specification without creating it (dry-run mode)

• --dry-run — Validate the workflow specification without creating it (dry-run mode)

torc workflows new

Create a new empty workflow

Usage: torc workflows new [OPTIONS] --name <NAME> --user <USER>

Options:

• -n, --name <NAME> — Name of the workflow
• -d, --description <DESCRIPTION> — Description of the workflow
• -u, --user <USER> — User that owns the workflow (defaults to USER environment variable)
• -n, --name <NAME> — Name of the workflow
• -d, --description <DESCRIPTION> — Description of the workflow
• -u, --user <USER> — User that owns the workflow (defaults to USER environment variable)

torc workflows list

List workflows

Usage: torc workflows list [OPTIONS]

Options:

• -u, --user <USER> — User to filter by (defaults to USER environment variable)

• --all-users — List workflows for all users (overrides --user)

• -l, --limit <LIMIT> — Maximum number of workflows to return

• -u, --user <USER> — User to filter by (defaults to USER environment variable)

• --all-users — List workflows for all users (overrides --user)

• -l, --limit <LIMIT> — Maximum number of workflows to return

  Default value: 10000

• --offset <OFFSET> — Offset for pagination (0-based)

• --offset <OFFSET> — Offset for pagination (0-based)

  Default value: 0

• --sort-by <SORT_BY> — Field to sort by

• --reverse-sort — Reverse sort order

• --archived-only — Show only archived workflows

• --sort-by <SORT_BY> — Field to sort by

• --reverse-sort — Reverse sort order

• --archived-only — Show only archived workflows

  Default value: false

• --include-archived — Include both archived and non-archived workflows

• --include-archived — Include both archived and non-archived workflows

  Default value: false

torc workflows get

Get a specific workflow by ID

Usage: torc workflows get [OPTIONS] [ID]

Arguments:

• <ID> — ID of the workflow to get (optional - will prompt if not provided)
• <ID> — ID of the workflow to get (optional - will prompt if not provided)

Options:

• -u, --user <USER> — User to filter by (defaults to USER environment variable)
• -u, --user <USER> — User to filter by (defaults to USER environment variable)

torc workflows update

Update an existing workflow

Usage: torc workflows update [OPTIONS] [ID]

Arguments:

• <ID> — ID of the workflow to update (optional - will prompt if not provided)
• <ID> — ID of the workflow to update (optional - will prompt if not provided)

Options:

• -n, --name <NAME> — Name of the workflow
• -d, --description <DESCRIPTION> — Description of the workflow
• --owner-user <OWNER_USER> — User that owns the workflow
• -n, --name <NAME> — Name of the workflow
• -d, --description <DESCRIPTION> — Description of the workflow
• --owner-user <OWNER_USER> — User that owns the workflow

torc workflows cancel

Cancel a workflow and all associated Slurm jobs

Usage: torc workflows cancel [WORKFLOW_ID]

Arguments:

• <WORKFLOW_ID> — ID of the workflow to cancel (optional - will prompt if not provided)
• <WORKFLOW_ID> — ID of the workflow to cancel (optional - will prompt if not provided)

torc workflows delete

Delete one or more workflows

Usage: torc workflows delete [OPTIONS] [IDS]...

Arguments:

• <IDS> — IDs of workflows to remove (optional - will prompt if not provided)
• <IDS> — IDs of workflows to remove (optional - will prompt if not provided)

Options:

• --no-prompts — Skip confirmation prompt
• --force — Force deletion even if workflow belongs to a different user
• --no-prompts — Skip confirmation prompt
• --force — Force deletion even if workflow belongs to a different user

torc workflows archive

Archive or unarchive one or more workflows

Usage: torc workflows archive <IS_ARCHIVED> [WORKFLOW_IDS]...

Arguments:

• <IS_ARCHIVED> — Set to true to archive, false to unarchive
• <WORKFLOW_IDS> — IDs of workflows to archive/unarchive (if empty, will prompt for selection)
• <IS_ARCHIVED> — Set to true to archive, false to unarchive
• <WORKFLOW_IDS> — IDs of workflows to archive/unarchive (if empty, will prompt for selection)

torc workflows submit

Submit a workflow: initialize if needed and schedule nodes for on_workflow_start actions This command requires the workflow to have an on_workflow_start action with schedule_nodes Submit a workflow: initialize if needed and schedule nodes for on_workflow_start actions This command requires the workflow to have an on_workflow_start action with schedule_nodes

Usage: torc workflows submit [OPTIONS] [WORKFLOW_ID]

Arguments:

• <WORKFLOW_ID> — ID of the workflow to submit (optional - will prompt if not provided)
• <WORKFLOW_ID> — ID of the workflow to submit (optional - will prompt if not provided)

Options:

• --force — If false, fail the operation if missing data is present (defaults to false)

• --force — If false, fail the operation if missing data is present (defaults to false)

  Default value: false

torc workflows run

Run a workflow locally on the current node

Usage: torc workflows run [OPTIONS] [WORKFLOW_ID]

Arguments:

• <WORKFLOW_ID> — ID of the workflow to run (optional - will prompt if not provided)
• <WORKFLOW_ID> — ID of the workflow to run (optional - will prompt if not provided)

Options:

• -p, --poll-interval <POLL_INTERVAL> — Poll interval in seconds for checking job completion

• -p, --poll-interval <POLL_INTERVAL> — Poll interval in seconds for checking job completion

  Default value: 5.0

• --max-parallel-jobs <MAX_PARALLEL_JOBS> — Maximum number of parallel jobs to run (defaults to available CPUs)

• --output-dir <OUTPUT_DIR> — Output directory for job logs and results

• --max-parallel-jobs <MAX_PARALLEL_JOBS> — Maximum number of parallel jobs to run (defaults to available CPUs)

• --output-dir <OUTPUT_DIR> — Output directory for job logs and results

  Default value: output

torc workflows initialize

Initialize a workflow, including all job statuses

Usage: torc workflows initialize [OPTIONS] [WORKFLOW_ID]

Arguments:

• <WORKFLOW_ID> — ID of the workflow to start (optional - will prompt if not provided)
• <WORKFLOW_ID> — ID of the workflow to start (optional - will prompt if not provided)

Options:

• --force — If false, fail the operation if missing data is present (defaults to false)

• --force — If false, fail the operation if missing data is present (defaults to false)

  Default value: false

• --no-prompts — Skip confirmation prompt

• --dry-run — Perform a dry run without making changes

• --no-prompts — Skip confirmation prompt

• --dry-run — Perform a dry run without making changes

torc workflows reinitialize

Reinitialize a workflow. This will reinitialize all jobs with a status of canceled, submitting, pending, or terminated. Jobs with a status of done will also be reinitialized if an input_file or user_data record has changed Reinitialize a workflow. This will reinitialize all jobs with a status of canceled, submitting, pending, or terminated. Jobs with a status of done will also be reinitialized if an input_file or user_data record has changed

Usage: torc workflows reinitialize [OPTIONS] [WORKFLOW_ID]

Arguments:

• <WORKFLOW_ID> — ID of the workflow to reinitialize (optional - will prompt if not provided)
• <WORKFLOW_ID> — ID of the workflow to reinitialize (optional - will prompt if not provided)

Options:

• --force — If false, fail the operation if missing data is present (defaults to false)

• --force — If false, fail the operation if missing data is present (defaults to false)

  Default value: false

• --dry-run — Perform a dry run without making changes

• --dry-run — Perform a dry run without making changes

torc workflows status

Get workflow status

Usage: torc workflows status [OPTIONS] [WORKFLOW_ID]

Arguments:

• <WORKFLOW_ID> — ID of the workflow to get status for (optional - will prompt if not provided)
• <WORKFLOW_ID> — ID of the workflow to get status for (optional - will prompt if not provided)

Options:

• -u, --user <USER> — User to filter by (defaults to USER environment variable)
• -u, --user <USER> — User to filter by (defaults to USER environment variable)

torc workflows reset-status

Reset workflow and job status

Usage: torc workflows reset-status [OPTIONS] [WORKFLOW_ID]

Arguments:

• <WORKFLOW_ID> — ID of the workflow to reset status for (optional - will prompt if not provided)
• <WORKFLOW_ID> — ID of the workflow to reset status for (optional - will prompt if not provided)

Options:

• --failed-only — Only reset failed jobs

• --failed-only — Only reset failed jobs

  Default value: false

• -r, --reinitialize — Reinitialize the workflow after resetting status

• -r, --reinitialize — Reinitialize the workflow after resetting status

  Default value: false

• --force — Force reset even if there are active jobs (ignores running/pending jobs check)

• --force — Force reset even if there are active jobs (ignores running/pending jobs check)

  Default value: false

• --no-prompts — Skip confirmation prompt

• --no-prompts — Skip confirmation prompt

torc workflows execution-plan

Show the execution plan for a workflow specification or existing workflow

Usage: torc workflows execution-plan <SPEC_OR_ID>

Arguments:

• <SPEC_OR_ID> — Path to specification file OR workflow ID
• <SPEC_OR_ID> — Path to specification file OR workflow ID

torc workflows list-actions

List workflow actions and their statuses (useful for debugging action triggers)

Usage: torc workflows list-actions [OPTIONS] [WORKFLOW_ID]

Arguments:

• <WORKFLOW_ID> — ID of the workflow to show actions for (optional - will prompt if not provided)
• <WORKFLOW_ID> — ID of the workflow to show actions for (optional - will prompt if not provided)

Options:

• -u, --user <USER> — User to filter by when selecting workflow interactively (defaults to USER environment variable)
• -u, --user <USER> — User to filter by when selecting workflow interactively (defaults to USER environment variable)

torc workflows is-complete

Check if a workflow is complete

Usage: torc workflows is-complete [ID]

Arguments:

• <ID> — ID of the workflow to check (optional - will prompt if not provided)
• <ID> — ID of the workflow to check (optional - will prompt if not provided)

torc workflows export

Export a workflow to a portable JSON file

Creates a self-contained export that can be imported into the same or different torc-server instance. All entity IDs are preserved in the export and remapped during import.

Usage: torc workflows export [OPTIONS] [WORKFLOW_ID]

Arguments:

• <WORKFLOW_ID> — ID of the workflow to export (optional - will prompt if not provided)

Options:

• -o, --output <OUTPUT> — Output file path (default: stdout)
• --include-results — Include job results in export
• --include-events — Include events (workflow history) in export

Examples:

# Export workflow to stdout
torc workflows export 123

# Export to a file
torc workflows export 123 -o workflow.json

# Include job results in export
torc workflows export 123 --include-results -o backup.json

# Export with all optional data
torc workflows export 123 --include-results --include-events -o complete.json

torc workflows import

Import a workflow from an exported JSON file

Imports a workflow that was previously exported. All entity IDs are remapped to new IDs assigned by the server. By default, all job statuses are reset to uninitialized for a fresh start.

Usage: torc workflows import [OPTIONS] <FILE>

Arguments:

• <FILE> — Path to the exported workflow JSON file (use '-' for stdin)

Options:

• --name <NAME> — Override the workflow name
• --skip-results — Skip importing results even if present in export
• --skip-events — Skip importing events even if present in export

Examples:

# Import a workflow (resets job statuses by default)
torc workflows import workflow.json

# Import from stdin
cat workflow.json | torc workflows import -

# Import with a different name
torc workflows import workflow.json --name 'my-copy'

# Skip importing results even if present in file
torc workflows import workflow.json --skip-results

torc workflows sync-status

Synchronize job statuses with Slurm (detect and fail orphaned jobs)

This command detects jobs that are stuck in "running" status because their Slurm allocation terminated unexpectedly (e.g., due to timeout, node failure, or admin intervention). It marks these orphaned jobs as failed so the workflow can be recovered or restarted.

Use this when:

• torc recover reports "there are active Slurm allocations" but squeue shows none
• Jobs appear stuck in "running" status after a Slurm allocation ended
• You want to clean up workflow state before running torc recover

Usage: torc workflows sync-status [OPTIONS] [WORKFLOW_ID]

Arguments:

• <WORKFLOW_ID> — ID of the workflow to sync (optional - will prompt if not provided)

Options:

• --dry-run — Preview changes without applying them

Examples:

# Preview what would be cleaned up
torc workflows sync-status 123 --dry-run

# Clean up orphaned jobs
torc workflows sync-status 123

# Get JSON output for scripting
torc -f json workflows sync-status 123

torc compute-nodes

Compute node management commands

Usage: torc compute-nodes <COMMAND>

Subcommands:

• get — Get a specific compute node by ID
• list — List compute nodes for a workflow
• get — Get a specific compute node by ID
• list — List compute nodes for a workflow

torc compute-nodes get

Get a specific compute node by ID

Usage: torc compute-nodes get <ID>

Arguments:

• <ID> — ID of the compute node
• <ID> — ID of the compute node

torc compute-nodes list

List compute nodes for a workflow

Usage: torc compute-nodes list [OPTIONS] [WORKFLOW_ID]

Arguments:

• <WORKFLOW_ID> — List compute nodes for this workflow (optional - will prompt if not provided)
• <WORKFLOW_ID> — List compute nodes for this workflow (optional - will prompt if not provided)

Options:

• -l, --limit <LIMIT> — Maximum number of compute nodes to return

• -l, --limit <LIMIT> — Maximum number of compute nodes to return

  Default value: 10000

• -o, --offset <OFFSET> — Offset for pagination (0-based)

• -o, --offset <OFFSET> — Offset for pagination (0-based)

  Default value: 0

• -s, --sort-by <SORT_BY> — Field to sort by

• -r, --reverse-sort — Reverse sort order

• -s, --sort-by <SORT_BY> — Field to sort by

• -r, --reverse-sort — Reverse sort order

  Default value: false

• --scheduled-compute-node <SCHEDULED_COMPUTE_NODE> — Filter by scheduled compute node ID

• --scheduled-compute-node <SCHEDULED_COMPUTE_NODE> — Filter by scheduled compute node ID

torc files

File management commands

Usage: torc files <COMMAND>

Subcommands:

• create — Create a new file
• list — List files
• get — Get a specific file by ID
• update — Update an existing file
• delete — Delete a file
• list-required-existing — List required existing files for a workflow
• create — Create a new file
• list — List files
• get — Get a specific file by ID
• update — Update an existing file
• delete — Delete a file
• list-required-existing — List required existing files for a workflow

torc files create

Create a new file

Usage: torc files create --name <NAME> --path <PATH> [WORKFLOW_ID]

Arguments:

• <WORKFLOW_ID> — Create the file in this workflow
• <WORKFLOW_ID> — Create the file in this workflow

Options:

• -n, --name <NAME> — Name of the job
• -p, --path <PATH> — Path of the file
• -n, --name <NAME> — Name of the job
• -p, --path <PATH> — Path of the file

torc files list

List files

Usage: torc files list [OPTIONS] [WORKFLOW_ID]

Arguments:

• <WORKFLOW_ID> — List files for this workflow (optional - will prompt if not provided)
• <WORKFLOW_ID> — List files for this workflow (optional - will prompt if not provided)

Options:

• --produced-by-job-id <PRODUCED_BY_JOB_ID> — Filter by job ID that produced the files

• -l, --limit <LIMIT> — Maximum number of files to return

• --produced-by-job-id <PRODUCED_BY_JOB_ID> — Filter by job ID that produced the files

• -l, --limit <LIMIT> — Maximum number of files to return

  Default value: 10000

• --offset <OFFSET> — Offset for pagination (0-based)

• --offset <OFFSET> — Offset for pagination (0-based)

  Default value: 0

• --sort-by <SORT_BY> — Field to sort by

• --reverse-sort — Reverse sort order

• --sort-by <SORT_BY> — Field to sort by

• --reverse-sort — Reverse sort order

torc files get

Get a specific file by ID

Usage: torc files get <ID>

Arguments:

• <ID> — ID of the file to get
• <ID> — ID of the file to get

torc files update

Update an existing file

Usage: torc files update [OPTIONS] <ID>

Arguments:

• <ID> — ID of the file to update
• <ID> — ID of the file to update

Options:

• -n, --name <NAME> — Name of the file
• -p, --path <PATH> — Path of the file
• -n, --name <NAME> — Name of the file
• -p, --path <PATH> — Path of the file

torc files delete

Delete a file

Usage: torc files delete <ID>

Arguments:

• <ID> — ID of the file to remove
• <ID> — ID of the file to remove

torc files list-required-existing

List required existing files for a workflow

Usage: torc files list-required-existing [WORKFLOW_ID]

Arguments:

• <WORKFLOW_ID> — List required existing files for this workflow (optional - will prompt if not provided)
• <WORKFLOW_ID> — List required existing files for this workflow (optional - will prompt if not provided)

torc jobs