Workflow Specification Reference

This page documents all data models used in workflow specification files. Workflow specs can be written in YAML, JSON, JSON5, or KDL formats.

WorkflowSpec

The top-level container for a complete workflow definition.

Examples with project, metadata, and shared environment

The project and metadata fields are useful for organizing and categorizing workflows. For more detailed guidance on organizing workflows, see Organizing and Managing Workflows.

Use the top-level env map to export environment variables for every job in the workflow. Jobs can also define their own env map to add variables or override workflow-level values with the same key.

YAML example:

name: "ml_training_workflow"
project: "customer-churn-prediction"
metadata: '{"environment":"staging","version":"1.0.0","team":"ml-engineering"}'
description: "Train and evaluate churn prediction model"
env:
  DATA_ROOT: "/scratch/churn"
  LOG_LEVEL: "info"
jobs:
  - name: "preprocess"
    command: "python preprocess.py"
  - name: "train"
    command: "python train.py"
    env:
      LOG_LEVEL: "debug"
    depends_on: ["preprocess"]

JSON example:

{
  "name": "data_pipeline",
  "project": "analytics-platform",
  "metadata": "{\"cost_center\":\"eng-data\",\"priority\":\"high\"}",
  "description": "Daily data processing pipeline",
  "env": {
    "DATA_ROOT": "/scratch/analytics",
    "LOG_LEVEL": "info"
  },
  "jobs": [
    {
      "name": "extract",
      "command": "python extract.py",
      "env": {
        "LOG_LEVEL": "debug"
      }
    }
  ]
}

JobSpec

Defines a single computational task within a workflow.

When both workflow-level and job-level env maps are present, Torc merges them when the job is created and stores the effective environment on the job. Job-level values win on key conflicts. Torc exports these variables before running the job's invocation_script, if any.

FileSpec

Defines input/output file artifacts that establish implicit job dependencies.

UserDataSpec

Arbitrary JSON data that can establish dependencies between jobs.

ResourceRequirementsSpec

Defines compute resource requirements for jobs.

FailureHandlerSpec

Defines error recovery strategies for jobs.

FailureHandlerRuleSpec

A single rule within a failure handler for handling specific exit codes.

SlurmSchedulerSpec

Defines a Slurm HPC job scheduler configuration.

ExecutionConfig

Controls how jobs are executed and terminated. Fields are grouped by which execution mode they apply to. Setting a field that doesn't match the effective mode produces a validation error at workflow creation time.

Shared fields (both modes)

Direct mode fields

These fields only apply when the effective mode is direct. Setting them with mode: slurm produces a validation error. When mode: auto, validation checks the effective mode based on whether Slurm schedulers are present in the spec.

Slurm mode fields

These fields only apply when the effective mode is slurm. Setting them with mode: direct produces a validation error. When mode: auto, validation checks the effective mode based on whether Slurm schedulers are present in the spec.

Worker-per-node Slurm launch field

This field applies only when execution_config.mode: direct is combined with a schedule_nodes action that sets start_one_worker_per_node: true. In that mode, Torc launches one torc-slurm-job-runner per allocated node with an outer srun, and this field is passed to that launcher command.

StdioConfig

Controls how stdout and stderr are captured for job processes.

StdioMode

Per-job overrides can be set via the stdio field on individual JobSpec entries, which takes precedence over the workflow-level setting.

Stdio Examples

Combine stdout and stderr into a single file, and delete it on success:

execution_config:
  stdio:
    mode: combined
    delete_on_success: true

Suppress stdout for most jobs, but keep separate files for a specific job:

execution_config:
  stdio:
    mode: no_stdout

jobs:
  - name: preprocess
    command: python preprocess.py
  - name: train
    command: python train.py
    stdio:
      mode: separate

Execution Modes

Warning: auto will silently select slurm mode when running inside a Slurm allocation. Prefer setting the mode explicitly to avoid unexpected behavior.

Direct Mode Example

execution_config:
  mode: direct
  limit_resources: true
  termination_signal: SIGTERM
  sigterm_lead_seconds: 30
  sigkill_headroom_seconds: 60
  timeout_exit_code: 152
  oom_exit_code: 137

Direct Mode Worker-Per-Node Example

execution_config:
  mode: direct
  srun_mpi: "none"

actions:
  - trigger_type: on_workflow_start
    action_type: schedule_nodes
    scheduler: multi_node
    scheduler_type: slurm
    start_one_worker_per_node: true

This launches the job runners with:

srun --ntasks-per-node=1 --mpi=none torc-slurm-job-runner ...

Slurm Mode Example

execution_config:
  mode: slurm
  srun_termination_signal: "TERM@120"
  sigkill_headroom_seconds: 180
  enable_cpu_bind: false

Termination Timeline (Direct Mode)

With sigkill_headroom_seconds=60 and sigterm_lead_seconds=30:

1. end_time - 90s: Send SIGTERM (or configured termination_signal)
2. end_time - 60s: Send SIGKILL to remaining jobs, set exit code to timeout_exit_code
3. end_time: Job runner exits

Slurm Mode Headroom

In Slurm mode, sigkill_headroom_seconds controls srun --time. The step time limit is set to remaining_time - sigkill_headroom_seconds, allowing the job runner to detect completion before the allocation expires.

SlurmDefaultsSpec

Workflow-level default parameters applied to all Slurm schedulers. This is a map of parameter names to values.

Any valid sbatch long option can be specified (without the leading --), except for parameters managed by torc: partition, nodes, walltime, time, mem, gres, name, job-name.

The account parameter is allowed as a workflow-level default.

Example:

slurm_defaults:
  qos: "high"
  constraint: "cpu"
  mail-user: "user@example.com"
  mail-type: "END,FAIL"

WorkflowActionSpec

Defines conditional actions triggered by workflow or job state changes.

ResourceMonitorConfig

Configuration for resource usage monitoring.

Example with per-job summary monitoring and overall compute-node time-series monitoring:

resource_monitor:
  sample_interval_seconds: 5
  flush_interval_seconds: 300
  jobs:
    enabled: true
    granularity: summary
  compute_node:
    enabled: true
    granularity: time_series
    cpu: true
    memory: true

jobs.granularity controls per-job monitoring. compute_node.granularity independently controls overall compute-node monitoring, so a workflow can store per-job summaries while collecting compute-node time series. Both scopes use the parent sample_interval_seconds; there is not a separate sampling interval per scope. When time-series monitoring is enabled, samples are flushed to SQLite every flush_interval_seconds, which trades lower commit overhead for a bounded amount of crash-exposed buffered data. The current compute-node monitor records CPU and memory; GPU monitoring is expected to use the same nested compute_node configuration in a future release.

For backwards compatibility, top-level enabled and granularity fields are still accepted and apply to job monitoring when jobs is omitted. New workflow specs should use jobs.

JobMonitorConfig

Configuration for per-job CPU and memory monitoring.

ComputeNodeMonitorConfig

Configuration for opt-in overall compute-node CPU and memory monitoring.

MonitorGranularity

Enum specifying the level of detail for resource monitoring.

Job Priority

Use priority when some ready jobs should be claimed before others.

• Higher values are preferred over lower values
• The default is 0
• claim_next_jobs uses a stable tie-breaker for jobs with the same priority
• claim_jobs_based_on_resources prefers GPU jobs first within the same priority
• Priority affects both claim_next_jobs and claim_jobs_based_on_resources

Example:

jobs:
  - name: urgent_step
    command: ./run_urgent.sh
    priority: 100

  - name: background_step
    command: ./run_background.sh
    priority: 10

Parameter Formats

Parameters support several formats for generating multiple jobs or files:

Template substitution in strings:

• Basic: {param_name} - Replace with parameter value
• Formatted integer: {i:03d} - Zero-padded (001, 042, 100)
• Formatted float: {lr:.4f} - Precision (0.0010, 0.1000)

See the Job Parameterization reference for more details.