Resource Monitoring Reference

Technical reference for Torc's resource monitoring system.

Configuration Options

The resource_monitor section has one shared sampling interval and separate nested scopes for jobs and compute nodes:

resource_monitor:
  sample_interval_seconds: 5
  flush_interval_seconds: 300
  jobs:
    enabled: true
    granularity: summary
  compute_node:
    enabled: true
    granularity: time_series

For backwards compatibility, top-level enabled and granularity fields are still accepted and apply to job monitoring when jobs is omitted:

resource_monitor:
  enabled: true
  granularity: time_series
  sample_interval_seconds: 5
  flush_interval_seconds: 300

New workflow specs should use the explicit jobs block.

flush_interval_seconds only affects time-series persistence. Samples are still collected every sample_interval_seconds, but they are buffered in memory and written to SQLite in batches to reduce transaction overhead.

Job Monitoring

resource_monitor.jobs controls per-job CPU and memory monitoring. Summary mode stores peak and average values on job results. Time-series mode also stores per-sample values in a resource metrics database.

Compute Node Monitoring

To opt in to overall compute-node CPU and memory monitoring, add a nested compute_node block. The compute-node monitor supports granularity: "summary" and granularity: "time_series". Summary mode stores peak and average values for the runner lifetime on the compute node record. Time-series mode also stores per-sample values. The current compute-node monitor records CPU and memory; GPU monitoring is reserved for a future extension.

The monitor samples the node the job runner is running on. In a multi-node Slurm allocation with a single runner (start_one_worker_per_node: false), Torc also launches torc monitor-node on every other node of the allocation via srun --overlap, so their CPU and memory are recorded as well. When the runner exits, the script stops the samplers with SIGUSR1 (which srun forwards unchanged, unlike SIGTERM) so they flush their final samples and summary. Each node writes its own resource_metrics_monitor_wf<id>_h<hostname>_r<run>_sl<slurm_job_id>.db under <output_dir>/resource_utilization/; nothing is attached to the compute node record for those nodes. Pass all the files to torc plot-resources to plot the whole allocation; the system timeline draws one CPU and one memory trace per node:

torc plot-resources torc_output/resource_utilization/*.db

torc monitor-node also works outside a workflow: run it with no workflow ID and it samples the local host with -i/--sample-interval-seconds / --flush-interval-seconds (or --summary-only), without contacting the server, writing resource_metrics_monitor_h<hostname>.db.

Granularity Modes

Summary mode ("summary"):

• Stores only peak and average values per job
• Metrics stored in the main database results table
• Minimal storage overhead

Time series mode ("time_series"):

• Stores samples at regular intervals
• Creates separate SQLite database per workflow run
• Database location: <output_dir>/resource_utilization/resource_metrics_<hostname>_<workflow_id>_<run_id>.db

Sample Interval Guidelines

Time Series Database Schema

job_resource_samples Table

job_metadata Table

system_resource_samples Table

This table is always created in the resource metrics database, but rows are only written when resource_monitor.compute_node is enabled with granularity set to "time_series". If compute-node monitoring is disabled or summary-only, the table remains empty.

system_resource_summary Table

This table is always created in the resource metrics database, but a row is only written when compute-node time-series monitoring is enabled. Summary-only compute-node monitoring stores these values on the compute node record instead, leaving this table empty.

Compute Node Summary Fields

When resource_monitor.compute_node.enabled is true, Torc stores overall summary metrics on the compute node record:

These fields are shown by torc compute-nodes get, torc compute-nodes list, the TUI compute nodes view, and the dashboard compute nodes table.

Summary Metrics in Results

When using summary mode, the following fields are added to job results:

check-resource-utilization JSON Output

When using --format json:

{
  "workflow_id": 123,
  "run_id": null,
  "total_results": 10,
  "over_utilization_count": 3,
  "violations": [
    {
      "job_id": 15,
      "job_name": "train_model",
      "resource_type": "Memory",
      "specified": "8.00 GB",
      "peak_used": "10.50 GB",
      "over_utilization": "+31.3%"
    }
  ]
}

Violation Object

correct-resources JSON Output

When using torc -f json workflows correct-resources:

{
  "status": "success",
  "workflow_id": 123,
  "dry_run": false,
  "no_downsize": false,
  "memory_multiplier": 1.2,
  "cpu_multiplier": 1.2,
  "runtime_multiplier": 1.2,
  "resource_requirements_updated": 2,
  "jobs_analyzed": 5,
  "memory_corrections": 1,
  "runtime_corrections": 1,
  "cpu_corrections": 1,
  "downsize_memory_corrections": 2,
  "downsize_runtime_corrections": 2,
  "downsize_cpu_corrections": 0,
  "adjustments": [
    {
      "resource_requirements_id": 10,
      "direction": "upscale",
      "job_ids": [15],
      "job_names": ["train_model"],
      "memory_adjusted": true,
      "original_memory": "8g",
      "new_memory": "13g",
      "max_peak_memory_bytes": 10500000000
    },
    {
      "resource_requirements_id": 11,
      "direction": "downscale",
      "job_ids": [20, 21],
      "job_names": ["preprocess_a", "preprocess_b"],
      "memory_adjusted": true,
      "original_memory": "32g",
      "new_memory": "3g",
      "max_peak_memory_bytes": 2147483648,
      "runtime_adjusted": true,
      "original_runtime": "PT4H",
      "new_runtime": "PT12M"
    }
  ]
}

Top-Level Fields

Adjustment Object

plot-resources Output Files

All plots are self-contained HTML files using Plotly.js with:

• Interactive hover tooltips
• Zoom and pan controls
• Legend toggling
• Export options (PNG, SVG)

Monitored Metrics

Process Tree Tracking

The monitoring system automatically tracks child processes spawned by jobs. When a job creates worker processes (e.g., Python multiprocessing), all descendants are included in the aggregated metrics.

Slurm Accounting Stats

When running inside a Slurm allocation, Torc calls sacct after each job step completes and stores the results in the slurm_stats table. These complement the sysinfo-based metrics above with Slurm-native cgroup measurements.

Fields

Additional identifying fields stored per record: workflow_id, job_id, run_id, attempt_id, slurm_job_id.

Fields are null when:

• The job ran locally (no SLURM_JOB_ID in the environment)
• sacct is not available on the node
• The step was not found in the Slurm accounting database at collection time

Viewing Stats

torc slurm stats <workflow_id>
torc slurm stats <workflow_id> --job-id <job_id>
torc -f json slurm stats <workflow_id>

Performance Characteristics

• Single background monitoring thread regardless of job count
• Typical overhead: <1% CPU even with 1-second sampling
• Uses native OS APIs via the sysinfo crate
• Non-blocking async design