Slurm Overview

This document explains how Torc simplifies running workflows on Slurm-based HPC systems. The key insight is that you don't need to understand Slurm schedulers or workflow actions to run workflows on HPC systems—Torc handles this automatically.

The Simple Approach

Running a workflow on Slurm requires just two things:

Define your jobs with resource requirements
Submit with slurm generate + submit

That's it. Torc will analyze your workflow, generate appropriate Slurm configurations, and submit everything for execution.

⚠️ Important: The slurm generate + submit command uses heuristics to auto-generate Slurm schedulers and workflow actions. For complex workflows with unusual dependency patterns, the generated configuration may not be optimal and could result in suboptimal allocation timing. Always preview the configuration first using torc slurm generate (see Previewing Generated Configuration) before submitting production workflows.

Example Workflow

Here's a complete workflow specification that runs on Slurm:

name: data_analysis_pipeline
description: Analyze experimental data with preprocessing, training, and evaluation

resource_requirements:
  - name: light
    num_cpus: 4
    memory: 8g
    runtime: PT30M

  - name: compute
    num_cpus: 32
    memory: 64g
    runtime: PT2H

  - name: gpu
    num_cpus: 16
    num_gpus: 2
    memory: 128g
    runtime: PT4H

jobs:
  - name: preprocess
    command: python preprocess.py --input data/ --output processed/
    resource_requirements: light

  - name: train_model
    command: python train.py --data processed/ --output model/
    resource_requirements: gpu
    depends_on: [preprocess]

  - name: evaluate
    command: python evaluate.py --model model/ --output results/
    resource_requirements: compute
    depends_on: [train_model]

  - name: generate_report
    command: python report.py --results results/
    resource_requirements: light
    depends_on: [evaluate]

Submitting the Workflow

torc slurm generate writes the augmented spec to stdout, so pipe it straight into torc submit -:

torc slurm generate --account myproject workflow.yaml | torc submit -

Or save the generated spec to a file and submit that file:

torc slurm generate --account myproject -o workflow_with_slurm.yaml workflow.yaml
torc submit workflow_with_slurm.yaml

Torc will:

Detect which HPC system you're on (e.g., NLR Kestrel)
Match each job's requirements to appropriate partitions
Generate Slurm scheduler configurations
Create workflow actions that stage resource allocation based on dependencies
Submit the workflow for execution

How It Works

When you use slurm generate + submit, Torc performs intelligent analysis of your workflow:

1. Per-Job Scheduler Generation

Each job gets its own Slurm scheduler configuration based on its resource requirements. This means:

Jobs are matched to the most appropriate partition
Memory, CPU, and GPU requirements are correctly specified
Walltime is set to the partition's maximum (explained below)

2. Staged Resource Allocation

Torc analyzes job dependencies and creates staged workflow actions. Every scheduler is tied to the jobs it runs via an on_jobs_ready action:

Jobs without dependencies — on_jobs_ready gated on those (root) jobs. They are ready as soon as the workflow is initialized, so their resources are allocated immediately, just as the workflow starts.
Jobs with dependencies — on_jobs_ready gated on those jobs, so resources are allocated only when the jobs become ready to run.

This prevents wasting allocation time on resources that aren't needed yet. For example, in the workflow above:

preprocess resources are allocated at workflow start (it is a root job)
train_model resources are allocated when preprocess completes
evaluate resources are allocated when train_model completes
generate_report resources are allocated when evaluate completes

Tying every scheduler to its jobs (rather than scheduling root jobs with on_workflow_start) also makes re-runs work: if you reset a subset of jobs and reinitialize, only those jobs' actions are re-armed, so a follow-up torc submit re-schedules exactly the jobs being re-run. See Re-running part of a workflow below.

3. Walltime Calculation

By default, Torc sets the walltime to 1.5× your longest job's runtime (capped at the partition's maximum). This provides headroom for jobs that run slightly longer than expected.

You can customize this behavior:

--walltime-strategy max-job-runtime (default): Uses longest job runtime × multiplier
--walltime-strategy max-partition-time: Uses the partition's maximum walltime
--walltime-multiplier 2.0: Change the safety multiplier (default: 1.5)

See Walltime Strategy Options for details.

4. HPC Profile Knowledge

Torc includes built-in knowledge of HPC systems like NLR Kestrel, including:

Available partitions and their resource limits
GPU configurations
Memory and CPU specifications
Special requirements (e.g., minimum node counts for high-bandwidth partitions)

Using an unsupported HPC? Please request built-in support so everyone benefits. You can also create a custom profile for immediate use.

Resource Requirements Specification

Resource requirements are the key to the simplified workflow. Define them once and reference them from jobs:

resource_requirements:
  - name: small
    num_cpus: 4
    num_gpus: 0
    num_nodes: 1
    memory: 8g
    runtime: PT1H

  - name: gpu_training
    num_cpus: 32
    num_gpus: 4
    num_nodes: 1
    memory: 256g
    runtime: PT8H

Fields

Field	Description	Example
`name`	Reference name for jobs	`"compute"`
`num_cpus`	CPU cores required	`32`
`num_gpus`	GPUs required (0 if none)	`2`
`num_nodes`	Nodes required	`1`
`memory`	Memory with unit suffix	`"64g"`, `"512m"`
`runtime`	ISO8601 duration	`"PT2H"`, `"PT30M"`

Runtime Format

Use ISO8601 duration format:

PT30M — 30 minutes
PT2H — 2 hours
PT1H30M — 1 hour 30 minutes
P1D — 1 day
P2DT4H — 2 days 4 hours

Job Dependencies

Define dependencies explicitly or implicitly through file/data relationships:

Explicit Dependencies

jobs:
  - name: step1
    command: ./step1.sh
    resource_requirements: small

  - name: step2
    command: ./step2.sh
    resource_requirements: small
    depends_on: [step1]

  - name: step3
    command: ./step3.sh
    resource_requirements: small
    depends_on: [step1, step2]  # Waits for both

Implicit Dependencies (via Files)

files:
  - name: raw_data
    path: /data/raw.csv
  - name: processed_data
    path: /data/processed.csv

jobs:
  - name: process
    command: python process.py
    input_files: [raw_data]
    output_files: [processed_data]
    resource_requirements: compute

  - name: analyze
    command: python analyze.py
    input_files: [processed_data]  # Creates implicit dependency on 'process'
    resource_requirements: compute

Previewing Generated Configuration

Recommended Practice: Always preview the generated configuration before submitting to Slurm, especially for complex workflows. This allows you to verify that schedulers and actions are appropriate for your workflow structure.

Viewing the Execution Plan

Before generating schedulers, visualize how your workflow will execute in stages:

torc workflows execution-plan workflow.yaml

This shows the execution stages, which jobs run at each stage, and (if schedulers are defined) when Slurm allocations are requested. See Visualizing Workflow Structure for detailed examples.

Generating Slurm Configuration

Preview what Torc will generate:

torc slurm generate --account myproject --profile kestrel workflow.yaml

This outputs the complete workflow with generated schedulers and actions:

Scheduler Grouping Options

By default, Torc creates one scheduler per partition: jobs whose resource requirements would land on the same partition share an allocation. So if three jobs have three different resource requirement definitions (e.g., cpu, memory, mixed) but all fit on the same partition, you get one scheduler instead of three.

The --group-by option controls how jobs are grouped into schedulers:

# Default: one scheduler per partition
torc slurm generate --account myproject workflow.yaml
torc slurm generate --account myproject --group-by partition workflow.yaml
# Result: 1 scheduler (short_scheduler) if all jobs fit on the "short" partition

# One scheduler per resource_requirements name
torc slurm generate --account myproject --group-by resource-requirements workflow.yaml
# Result: 3 schedulers (cpu_scheduler, memory_scheduler, mixed_scheduler)

When to use --group-by partition (default):

Your workflow has resource requirement definitions that all fit on the same partition
You want to minimize Slurm queue overhead by reducing the number of allocations
Jobs have similar characteristics and can share nodes efficiently

When to use --group-by resource-requirements:

Jobs have significantly different resource profiles that benefit from separate allocations
You want fine-grained control over which jobs share resources
You're debugging and want clear separation between job types

When grouping by partition, the scheduler uses the maximum resource values from all grouped requirements (max memory, max CPUs, max runtime, etc.) to ensure all jobs can run.

Walltime Strategy Options

The --walltime-strategy option controls how Torc calculates the walltime for generated schedulers:

# Default: use max job runtime with a safety multiplier (1.5x)
torc slurm generate --account myproject workflow.yaml
torc slurm generate --account myproject --walltime-strategy max-job-runtime workflow.yaml

# Use the partition's maximum allowed walltime
torc slurm generate --account myproject --walltime-strategy max-partition-time workflow.yaml

Walltime strategies:

Strategy	Description
`max-job-runtime`	Uses the longest job's runtime × multiplier (default: 1.5x). Capped at partition max.
`max-partition-time`	Uses the partition's maximum walltime. More conservative but may impact queue scheduling.

Customizing the multiplier:

The --walltime-multiplier option (default: 1.5) provides a safety margin when using max-job-runtime:

# Use 2x the max job runtime for extra buffer
torc slurm generate --account myproject --walltime-multiplier 2.0 workflow.yaml

# Use exact job runtime (no buffer - use with caution)
torc slurm generate --account myproject --walltime-multiplier 1.0 workflow.yaml

When to use max-job-runtime (default):

You want better queue scheduling (shorter walltime requests often get prioritized)
Your job runtime estimates are reasonably accurate
You prefer the Torc runner to exit early rather than holding idle allocations

When to use max-partition-time:

Your job runtimes are highly variable or unpredictable
You consistently underestimate job runtimes
Queue priority is not a concern

name: data_analysis_pipeline
# ... original content ...

jobs:
  - name: preprocess
    command: python preprocess.py --input data/ --output processed/
    resource_requirements: light
    scheduler: preprocess_scheduler

  # ... more jobs ...

slurm_schedulers:
  - name: preprocess_scheduler
    account: myproject
    mem: 8g
    nodes: 1
    walltime: "04:00:00"

  - name: train_model_scheduler
    account: myproject
    mem: 128g
    nodes: 1
    gres: "gpu:2"
    walltime: "04:00:00"

  # ... more schedulers ...

actions:
  - trigger_type: on_jobs_ready
    action_type: schedule_nodes
    jobs: [preprocess]
    scheduler: preprocess_scheduler
    scheduler_type: slurm
    num_allocations: 1

  - trigger_type: on_jobs_ready
    action_type: schedule_nodes
    jobs: [train_model]
    scheduler: train_model_scheduler
    scheduler_type: slurm
    num_allocations: 1

  # ... more actions ...

Every scheduler is gated on the jobs it runs with on_jobs_ready — including root jobs like preprocess, which are ready at workflow start. This is what makes a re-run reschedule only the affected jobs.

Save the output to inspect or modify before submission:

torc slurm generate --account myproject workflow.yaml -o workflow_with_schedulers.yaml

Re-running part of a workflow

Because every scheduler is tied to its jobs with on_jobs_ready, you can re-run a subset of a finished (or partially failed) workflow and have only the affected allocations re-submitted:

# Reset the jobs you want to re-run (downstream jobs reset automatically), then reinitialize:
torc jobs reset-status <id1> <id2> ... --reinit

# Re-submit: only the reset jobs' allocations are scheduled.
torc submit <workflow_id>

How it works: reinitialize re-arms only the actions whose jobs were reset; actions for untouched jobs stay suppressed (so they are not re-scheduled and do not submit duplicate allocations). torc submit then fires every pending schedule_nodes action — which is exactly the reset jobs' actions. Downstream actions whose jobs aren't ready yet are fired later by a running worker, as their dependencies complete.

For example, with separate job classes each on their own partition (see examples/yaml/workflow_actions_multi_class_slurm.yaml), resetting only the GPU and big-memory jobs re-schedules only the GPU and big-memory allocations; the regular-job allocations are left alone.

Important — submit does not right-size the re-run. torc submit re-fires each pending action with the num_allocations from its spec, verbatim. The granularity of a re-run is therefore the granularity of your actions:

With per-stage / per-class actions (as above), resetting a subset re-arms only the matching actions, so submit schedules only those — the common, efficient case.

With a single large fan-in action (e.g. one on_jobs_ready action gating 100k jobs with num_allocations: 500), resetting any subset re-arms that one action, and submit submits its full num_allocations (500) — even if only 100 jobs need to re-run. The jobs still run (the surplus workers find no claimable work and idle out), but it badly over-allocates.

For a right-sized partial re-run of a large fan-in stage, use torc recover instead: it regenerates schedule_nodes actions sized to the jobs that are actually pending (and suppresses the spec's full-size action), so it allocates for the 100 jobs rather than the whole stage.

Note: torc submit is one-shot — it submits the currently-pending allocations and returns. For an unattended multi-stage re-run, follow it with torc watch so that if every worker exits (e.g. Slurm walltime) before a later stage's action fires, the stranded action is picked up. For a guided re-run that also sizes allocations from prior runs, use torc recover.

Tip: This is why auto-generated and recommended specs use on_jobs_ready (tied to jobs) rather than on_workflow_start for schedule_nodes. An on_workflow_start action is kept (not re-armed) on reinitialize and torc submit cannot re-fire it, so a reset root job would have no allocation. on_workflow_start schedule_nodes is still fine for a single allocation that serves the whole workflow, but such an allocation isn't selectively re-run-safe.

Manually scheduling a re-run with `torc slurm schedule-nodes`

If you want to provide the compute yourself instead of letting submit fire the actions — for example "run my 10 reset jobs on 1 node" — use torc slurm schedule-nodes:

torc jobs reset-status <id1> ... <id10> --reinit
torc slurm schedule-nodes -n1 <workflow_id>

A worker started this way still claims the workflow's own pending schedule_nodes actions and submits their allocations. So if the reinit left a coarse action re-armed (e.g. an on_jobs_ready action gating the whole stage), that worker would fire its full num_allocations on top of the one you requested. To prevent surprises, schedule-nodes checks for pending schedule_nodes actions and prompts you to suppress them, proceed (let them fire), or cancel. For non-interactive use:

--suppress-actions — mark the pending actions executed first, so only your -n request is submitted.
--no-prompts — skip the prompt and proceed (let the actions fire); the historical behavior.

(torc recover does the suppression automatically and additionally right-sizes the allocation to the pending jobs, so it is usually the better tool for a partial re-run.)

Choosing a stage's scheduler trigger: upstream vs. the stage's own jobs

For schedule_nodes, what you gate the action on determines how subset re-runs behave:

Gate on the stage's own jobs (on_jobs_ready[<this stage's jobs>]) — resetting any of them re-arms the action, so submit re-schedules them automatically. Best for per-class / per-stage actions where auto re-run at the action's num_allocations is what you want.
Gate on the upstream job whose completion unlocks the stage (on_jobs_complete[<upstream>]) — resetting the stage's own jobs leaves the upstream terminal, so the action stays suppressed and does not re-fire. Best for a large fan-in stage you expect to re-run in subsets: the coarse action won't over-allocate, and you provide the compute with torc slurm schedule-nodes or torc recover. The trade-off is that reinit won't auto-reschedule the reset jobs — which for a coarse stage is what you want anyway.

Torc Server Considerations

The Torc server must be accessible to compute nodes. Options include:

Shared server (Recommended): A team member allocates a dedicated server in the HPC environment
Login node: Suitable for small workflows with few, long-running jobs

For large workflows with many short jobs, a dedicated server prevents overloading login nodes.

Best Practices

1. Focus on Resource Requirements

Spend time accurately defining resource requirements. Torc handles the rest:

resource_requirements:
  # Be specific about what each job type needs
  - name: io_heavy
    num_cpus: 4
    memory: 32g      # High memory for data loading
    runtime: PT1H

  - name: compute_heavy
    num_cpus: 64
    memory: 16g      # Less memory, more CPU
    runtime: PT4H

2. Use Meaningful Names

Name resource requirements by their purpose, not by partition:

# Good - describes the workload
resource_requirements:
  - name: data_preprocessing
  - name: model_training
  - name: inference

# Avoid - ties you to specific infrastructure
resource_requirements:
  - name: short_partition
  - name: gpu_h100

3. Group Similar Jobs

Jobs with similar requirements can share resource requirement definitions:

resource_requirements:
  - name: quick_task
    num_cpus: 2
    memory: 4g
    runtime: PT15M

jobs:
  - name: validate_input
    command: ./validate.sh
    resource_requirements: quick_task

  - name: check_output
    command: ./check.sh
    resource_requirements: quick_task
    depends_on: [main_process]

4. Test Locally First

Validate your workflow logic locally before submitting to HPC:

# Run locally (without Slurm)
torc run workflow.yaml

# Then submit to HPC
torc slurm generate --account myproject workflow.yaml | torc submit -

Limitations and Caveats

The auto-generation in torc slurm generate uses heuristics that work well for common workflow patterns but may not be optimal for all cases:

When Auto-Generation Works Well

Linear pipelines: A → B → C → D
Fan-out patterns: One job unblocks many (e.g., preprocess → 100 work jobs)
Fan-in patterns: Many jobs unblock one (e.g., 100 work jobs → postprocess)
Simple DAGs: Clear dependency structures with distinct resource tiers

When to Use Manual Configuration

Consider using torc slurm generate to preview and manually adjust, or define schedulers manually, when:

Complex dependency graphs: Multiple interleaved dependency patterns
Shared schedulers: You want multiple jobs to share the same Slurm allocation
Custom timing: Specific requirements for when allocations should be requested
Resource optimization: Fine-tuning to minimize allocation waste
Multi-node jobs: Jobs requiring coordination across multiple nodes (see Multi-Node Jobs)

What Could Go Wrong

Without previewing, auto-generation might:

Request allocations too early: Wasting queue time waiting for dependencies
Request allocations too late: Adding latency to job startup
Create suboptimal scheduler groupings: Not sharing allocations when beneficial
Miss optimization opportunities: Not recognizing patterns that could share resources

Dynamic Jobs (`spawn_jobs`)

torc slurm generate analyzes the static workflow specification — the jobs declared at workflow creation. Jobs added at runtime by spawn_jobs (the dynamic-jobs orchestrator pattern) are by definition not visible to it, so the generated schedulers allocate compute capacity only for the originally-declared workload.

For iterative workflows that grow at runtime, pair torc slurm generate with torc watch --auto-schedule. The watch loop detects spawned jobs (rows with origin = 'spawn') and submits additional Slurm allocations to cover them — without it, spawned children sit Ready until the originally-planned allocations happen to have spare capacity.

Best Practice: For production workflows, always run torc slurm generate first, review the output, and submit the reviewed configuration with torc submit.

Advanced: Manual Scheduler Configuration

For advanced users who need fine-grained control, you can define schedulers and actions manually. See Advanced Slurm Configuration for details.

Common reasons for manual configuration:

Non-standard partition requirements
Custom Slurm directives (e.g., --constraint)
Multi-node jobs with specific topology requirements
Reusing allocations across multiple jobs for efficiency

Troubleshooting

"No partition found for job"

Your resource requirements exceed what's available. Check:

Memory doesn't exceed partition limits
Runtime doesn't exceed partition walltime
GPU count is available on GPU partitions

Use torc hpc partitions <profile> to see available resources.

Jobs Not Starting

Ensure the Torc server is accessible from compute nodes:

# From a compute node
curl $TORC_API_URL/health

Wrong Partition Selected

Use torc hpc match to see which partitions match your requirements:

torc hpc match kestrel --cpus 32 --memory 64g --walltime 02:00:00 --gpus 2

Keyboard shortcuts

Torc Documentation