Building on HPC systems

Building on HPC systems#

Some HPC systems run operating systems with a glibc that is too old for the prebuilt ONNX Runtime binaries that the Rust ort crate downloads by default (they require glibc >= 2.32; e.g. RHEL 8 ships glibc 2.28). On those systems, importing the compiled extension fails with an error like:

ImportError: .../routee_compass_py.cpython-313-x86_64-linux-gnu.so: undefined symbol: __libc_single_threaded

The fix is to build ONNX Runtime from source on the HPC machine and link against it. The pixi hpc environment automates this.

One-time setup#

pixi run -e hpc build_hpc

This does two things:

  1. Clones and builds ONNX Runtime from source into build/onnxruntime/ (scripts/hpc/build_ort.sh; this takes 30+ minutes the first time).

  2. Builds and installs the python package into the hpc environment with maturin develop --uv --release.

To verify the build imports correctly:

pixi run -e hpc check_hpc

How it works#

The hpc pixi feature's activation sets ORT_LIB_PATH to the local ONNX Runtime build. The ort crate's build script prefers ORT_LIB_PATH over its downloaded binaries, so any build run inside the hpc environment — build_py, build_rust, test, build_hpc_wheel — automatically links the locally-built ONNX Runtime. No special cargo features or flags are needed.

Day-to-day usage on HPC#

Always use the hpc environment. The easiest way is to activate it once per shell session, after which plain commands work as usual:

pixi shell -e hpc
python -c "import nrel.routee.compass"   # works
pytest python/tests/

Or prefix individual commands with -e hpc:

pixi run -e hpc python          # run python with the package importable
pixi run -e hpc test            # run the python test suite
pixi run -e hpc build_rust      # build the rust CLI binaries
pixi run -e hpc build_hpc_wheel # build a distributable wheel (target/wheels/)

Warning

Do not run default-environment pixi commands (e.g. plain pixi run python or pixi install) on an HPC machine with an old glibc. The default environment installs this package as an editable install, and pixi/uv will silently rebuild the extension against the downloaded ONNX Runtime binaries — clobbering the working build and reintroducing the __libc_single_threaded import error.

Knobs#

Environment variables read by the build (defaults set by the hpc feature in pyproject.toml):

  • SKIP_ORT_BUILD=1 — reuse an existing ONNX Runtime build instead of rebuilding it (e.g. SKIP_ORT_BUILD=1 pixi run -e hpc build_hpc).

  • ONNXRUNTIME_TAG — ONNX Runtime git tag to build (default v1.20.1).

  • ORT_BUILD_CONFIG — CMake build config (default RelWithDebInfo).

  • ONNXRUNTIME_DIR — where ONNX Runtime is cloned and built.