Getting Started

Torc is a distributed workflow orchestration system for managing complex computational pipelines with job dependencies, resource requirements, and distributed execution.

Torc uses a client-server architecture where a central server manages workflow state and coordination, while clients create workflows and job runners execute tasks on compute resources.

How You Interact with Torc

Choose the interface that fits your workflow:

• CLI — Primary interface which provides access to all torc functionality
• Dashboard — Web UI (torc-dash) for visual configuration and monitoring
• TUI — Terminal User Interface (torc tui) for visual monitoring in a terminal
• AI Assistants — Use Claude Code or GitHub Copilot to manage workflows through natural language. "Create a workflow with 10 parallel jobs" or "Why did job 5 fail?"
• Spec Files — YAML, KDL, or JSON5 files for version-controlled workflow definitions
• Python/Julia APIs — Libraries for programmatic workflow generation

All interfaces work with the same server — mix and match as needed.

Architecture

flowchart LR
    subgraph you["You"]
        CLI["torc CLI"]
    end

    subgraph server["Server"]
        API["torc-server"]
    end

    subgraph workers["Workers"]
        W1["Job Runner"]
        W2["Job Runner"]
    end

    CLI -->|"create workflow"| API
    CLI -->|"start workers"| W1
    CLI -->|"start workers"| W2
    W1 -->|"claim & report"| API
    W2 -->|"claim & report"| API

    style CLI fill:#4a9eff,color:#fff
    style API fill:#28a745,color:#fff
    style W1 fill:#ffc107,color:#000
    style W2 fill:#ffc107,color:#000

1. You create workflows and start job runners via the CLI
2. Server tracks workflow state and job dependencies
3. Workers poll the server, claim ready jobs, execute them, and report results

Example Files

The repository includes ready-to-run workflow specifications in YAML, JSON5, and KDL formats:

Browse all examples:

• YAML examples
• JSON5 examples
• KDL examples
• Python examples

See the examples README for the complete list.

Choose Your Execution Mode

Torc supports three fundamentally different execution environments. Choose the one that matches your use case:

Local Execution

Best for: Development, testing, small-scale workflows on your workstation or a single server

• Jobs run directly on the machine where you start the job runner
• No scheduler needed — simple setup with torc run
• Resource management via local CPU/memory/GPU tracking
• → Quick Start (Local)

HPC/Slurm

Best for: Large-scale computations on institutional HPC clusters

• Jobs submitted to Slurm scheduler for compute node allocation
• Automatic resource matching to partitions/QOS
• Built-in profiles for common HPC systems
• → Quick Start (HPC/Slurm)

Remote Workers

Best for: Distributed execution across multiple machines you control via SSH

• Jobs distributed to remote workers over SSH
• No HPC scheduler required — you manage the machines
• Flexible heterogeneous resources (mix of CPU/GPU machines)
• → Quick Start (Remote Workers)

All three modes:

• Share the same workflow specification format
• Use the same server API for coordination
• Support the same monitoring tools (CLI, TUI, Dashboard)
• Can be used together (e.g., develop locally, deploy to HPC)

Continue to the Quick Start guide to run your first workflow.