wattameter_logo

wattameter is a Python package for monitoring and recording power usage over time, among other metrics. It enables time series data collection on hardware components such as CPUs and GPUs.

WattAMeter

Current Features

Track power usage for CPU (using RAPL) and GPU (using nvidia-ml-py)
Track GPU utilization and temperature
Periodically log time series data to file
Real-time MQTT publishing for integration with monitoring systems
Customizable logging and output options
Command-line interface for easy usage
Integration with SLURM for HPC environments

Installation

You can install wattameter via pip:

pip install wattameter

Optional extras

Some features are optional and not required for the core runtime. The current optional dependencies are post-processing (postprocessing) and benchmark (benchmark) utilities, and MQTT (mqtt) publishing. Here are a few examples of how to install with optional dependencies:

# Install with post-processing utilities
pip install wattameter[postprocessing]

# Install with benchmark utilities
pip install wattameter[benchmark]

# Install with postprocessing and mqtt
pip install wattameter[postprocessing,mqtt]

Usage

As a Python module

from wattameter import Tracker
from wattameter.readers import NVMLReader

tracker = Tracker(
    reader=NVMLReader((Power,)),
    dt_read=0.1,  # Time interval for reading power data (seconds)
    freq_write=600,  # Frequency (# reads) for writing power data to file
    output="power_log.txt",
)
tracker.start()
# ... your code ...
tracker.stop()

# ... or ...

with Tracker(
    reader=NVMLReader((Power,)),
    dt_read=0.1,
    freq_write=600,
    output="power_log.txt",
) as tracker:
    # ... your code ...

Command-line interface

wattameter --tracker 0.1,nvml-power,rapl --tracker 1.0,nvml-util --suffix test --id 0 --freq-write 600 --log-level info

For MQTT publishing, add MQTT broker configuration:

wattameter \
  --tracker 0.1,nvml-power,rapl \
  --mqtt-broker mqtt.example.com \
  --mqtt-port 1883 \
  --mqtt-username myuser \
  --mqtt-password mypassword \
  --mqtt-topic-prefix "hpc/wattameter"

See MQTT Usage Documentation for detailed information on MQTT publishing.

Option	Short	Default	Description
–tracker		0.1,nvml-power,rapl	Tracker specification: `dt_read,metric1,metric2,...` where `dt_read` is the time interval in seconds between readings. Available metrics: `rapl` (CPU energy), `nvml-energy` (GPU energy), `nvml-power` (GPU power), `nvml-temp` (GPU temperature), `nvml-util` (GPU utilization), `nvml-nvlink` (GPU NVLink throughput). Can be specified multiple times to create multiple trackers with different configurations.
–suffix	-s	None	Suffix for output files
–id	-i	UUID	Identifier for the experiment
–freq-write	-f	3600	Frequency (# reads) for writing data to file
–log-level	-l	warning	Logging level: debug, info, warning, error, critical
–mqtt-broker		None	MQTT broker hostname. If provided, enables real-time publishing to MQTT
–mqtt-port		1883	MQTT broker port
–mqtt-username		None	MQTT authentication username (optional)
–mqtt-password		None	MQTT authentication password (optional)
–mqtt-topic-prefix		wattameter	Prefix for MQTT topics
–mqtt-qos		1	MQTT Quality of Service level (0, 1, or 2)
–help	-h		Show the help message and exit

Command-line interface with SLURM

For usage within SLURM jobs, we recommend using our utility functions start_wattameter and stop_wattameter in slurm.sh. Follow the example examples/slurm.sh, i.e.,

# In a Python environment with wattameter installed,
# load wattameter slurm utilities
WATTAPATH=$(python -c 'import wattameter; import os; print(os.path.dirname(wattameter.__file__))')
source "${WATTAPATH}/utils/slurm.sh"

# Run wattameter on all nodes
start_wattameter

# Input your job commands here
# ...

# Stop wattameter on all nodes
stop_wattameter

All options are the same as the regular command-line interface. The script will automatically handle the output file naming based on the provided SLURM_JOB_ID and node information.

Contributing

Contributions are welcome! Please open issues or submit pull requests at https://github.com/NatLabRockies/WattAMeter/.

Documentation

The API documentation is available at https://NatLabRockies.github.io/WattAMeter/. For specific documentation of the NLR module, visit https://natlabrockies.github.io/HPC/Documentation/Development/Performance_Tools/WattAMeter/.

License

See the LICENSE file for details.

NLR Software Record number: SWR-25-101