Build Options Reference

ExaGOOP is compiled from source using either CMake or GNUmake. Most compile-time options are set as variables in the relevant build file; the table below summarises all user-visible flags. Detailed descriptions follow.

Table 2 Build option summary

Feature

GNUmake variable

CMake variable

GNUmake default

CMake default

Values

Temperature module

USE_TEMP

EXAGOOP_USE_TEMP

TRUE

ON

TRUE/FALSE

Embedded boundary

USE_EB

(hardcoded ON)

FALSE

ON (DIM ≥ 2)

TRUE/FALSE

HDF5 I/O

AMREX_USE_HDF5

EXAGOOP_USE_HDF5

FALSE

OFF

TRUE/FALSE

Floating-point precision

PRECISION

EXAGOOP_PRECISION

DOUBLE

DOUBLE

SINGLE/DOUBLE

Spatial dimensions

DIM

EXAGOOP_DIM

2

3

1, 2, 3

Full profiling

PROFILE

(not exposed)

FALSE

—

TRUE/FALSE

Lightweight profiling

TINY_PROFILE

(not exposed)

FALSE

—

TRUE/FALSE

Debug build

DEBUG

CMAKE_BUILD_TYPE

FALSE

(unset)

TRUE/FALSE

Address/UB sanitizers

FSANITIZER

(not exposed)

FALSE

—

TRUE/FALSE

Note

GNUmake TRUE/FALSE values are translated to 1/0 by the line DEFINES += -DTRUE=1 -DFALSE=0 in Build_Gnumake/GNUmakefile, so #if USE_TEMP guards work correctly for both build paths.

USE_TEMP — Temperature Module

Compiles in the optional coupled heat-transfer solver. See Temperature Module (USE_TEMP) for the full description of the physics and input parameters.

GNUmake

USE_TEMP = TRUE    # enable  (default)
USE_TEMP = FALSE   # disable

Set this in the relevant GNUmakefile (Build_Gnumake/GNUmakefile or a test-specific file) before the include Make.defs line. The build system injects -DUSE_TEMP=1 (enabled) or -DUSE_TEMP=0 (disabled) automatically.

CMake

cmake -DEXAGOOP_USE_TEMP=ON  ..    # enable  (default)
cmake -DEXAGOOP_USE_TEMP=OFF ..    # disable

What it enables

When USE_TEMP=1:

  • Five additional nodal fields are added to the MultiFab (NUM_STATES grows from 20 to 25).

  • Each particle carries five thermal real-data slots (temperature, specific heat, thermal conductivity, heat flux, heat source).

  • The thermal solve sequence (P2G deposition, nodal advance, G2P interpolation) runs every time step after the momentum update.

USE_EB — Embedded Boundaries

Enables AMReX Embedded Boundary (EB / cut-cell) support for domains with complex geometric boundaries.

GNUmake

USE_EB = FALSE   # disable (default)
USE_EB = TRUE    # enable

When TRUE, Build_Gnumake/GNUmakefile additionally includes $(AMREX_HOME)/Src/EB/Make.package.

CMake

EB is enabled by default for DIM ≥ 2 builds via a hardcoded set in CMakeLists.txt; there is no command-line -D switch to disable it without editing the file. The effective preprocessor symbol is set via:

# Inside CMakeLists.txt — not overridable from the command line
set(EXAGOOP_ENABLE_EB ON)

Enforcement rule

EB is silently forced to FALSE/0 in 1D builds regardless of what was requested:

# GNUmake — automatic when DIM=1
ifeq ($(DIM),1)
  USE_EB := FALSE
endif

A warning is printed if USE_EB=TRUE was explicitly set for a 1D build.

What it enables

  • Registers the AMReX EB factory so cut-cell normals, apertures, and volume fractions are available on the background grid.

  • Activates the mpm_eb.cpp routines for EB-aware boundary velocity enforcement.

  • Requires AMReX to be compiled with AMREX_USE_EB (handled automatically by the build system when USE_EB=TRUE).

AMREX_USE_HDF5 — HDF5 Checkpoint/Restart

Links against an HDF5 library and enables HDF5-format checkpoint and restart files through AMReX’s native HDF5 I/O layer.

GNUmake

HDF5 is configured in Build_Gnumake/Make.local so that machine- or environment-specific library paths stay out of the main makefile. A typical Make.local block looks like:

# --- Make.local (machine-specific) ---
HDF5_HOME = /path/to/hdf5          # root of HDF5 installation
HDF5_INC  = $(HDF5_HOME)/include
HDF5_LIB  = $(HDF5_HOME)/lib

USE_HDF5            = TRUE         # signal AMReX to enable HDF5
AMREX_USE_HDF5      = TRUE         # define the preprocessor symbol
AMREX_USE_HDF5_PARALLEL = TRUE     # enable parallel (collective) HDF5

CXXFLAGS += -I$(HDF5_INC)
LDFLAGS  += -L$(HDF5_LIB) -lhdf5 -lhdf5_hl

On systems using Cray modules (e.g. Kestrel at the National Laboratory of the Rockies), the module environment automatically provides the correct paths, so only the USE_HDF5/AMREX_USE_HDF5 flags are needed.

The default in a clean repository is FALSE (HDF5 disabled).

CMake

cmake -DEXAGOOP_USE_HDF5=OFF ..           # disable (default)
cmake -DEXAGOOP_USE_HDF5=ON  ..           # enable serial HDF5
cmake -DEXAGOOP_USE_HDF5_PARALLEL=ON ..   # enable parallel HDF5

These options are forwarded to AMReX as AMReX_HDF5 and AMReX_HDF5_PARALLEL; CMake’s find_package(HDF5) locates the library automatically if HDF5_ROOT or the module environment is set.

What it enables

  • AMReX defines the preprocessor symbol AMREX_USE_HDF5.

  • mpm_init.cpp uses #ifdef AMREX_USE_HDF5 guards to select HDF5 checkpoint reading instead of the native AMReX plot-file format.

  • Parallel HDF5 (AMREX_USE_HDF5_PARALLEL) allows collective I/O across all MPI ranks for large-scale restart files.

PRECISION — Floating-Point Width

Selects whether amrex::Real resolves to float (32-bit) or double (64-bit) throughout the solver.

GNUmake

PRECISION = DOUBLE   # 64-bit (default)
PRECISION = SINGLE   # 32-bit

CMake

cmake -DEXAGOOP_PRECISION=DOUBLE ..   # 64-bit (default)
cmake -DEXAGOOP_PRECISION=SINGLE ..   # 32-bit

Preprocessor symbols injected

Setting

Symbol defined

String constant

DOUBLE

AMREX_USE_DOUBLE

EXAGOOP_PRECISION_STR="double"

SINGLE

AMREX_USE_FLOAT

EXAGOOP_PRECISION_STR="single"

The string constant is printed in the startup banner. Any value other than SINGLE or DOUBLE is a fatal CMake error; the GNUmake path silently applies no precision flags if the value is unrecognised.

DIM — Spatial Dimensions

Selects the number of spatial dimensions compiled into the executable. This is a compile-time constant; a single binary cannot switch between 1D, 2D, and 3D at run time.

GNUmake

DIM = 2   # 2-D build (default in Build_Gnumake/GNUmakefile)
DIM = 3   # 3-D build
DIM = 1   # 1-D build

Test-specific makefiles (e.g. Tests/1D_Heat_Conduction/GNUmakefile) override this to match the problem dimension.

CMake

cmake -DEXAGOOP_DIM=3 ..   # 3-D (default)
cmake -DEXAGOOP_DIM=2 ..   # 2-D
cmake -DEXAGOOP_DIM=1 ..   # 1-D

What it controls

  • AMReX defines AMREX_SPACEDIM to the chosen value; all dimensional arrays (GpuArray, loop bounds, IntVect) are sized at compile time.

  • The executable name includes the dimension suffix, e.g. ExaGOOP2d.gnu.MPI.ex or ExaGOOP3d.gnu.MPI.ex.

  • Setting DIM=1 forces USE_EB=FALSE (AMReX EB is not supported in 1D).

  • mpm.ppc, mpm.prob_lo, and all other spatially-indexed parameters expect exactly SPACEDIM entries in the input file. Per-face momentum BC keys (mpm.bc_xlo_mom, mpm.bc_ylo_mom, etc.) exist for each of the 2 * SPACEDIM domain faces.

PROFILE — Full AMReX Profiling

Enables AMReX’s detailed timer-based profiler, which wraps every BL_PROFILE region and produces a bl_prof/ output directory containing call-graph timing data.

GNUmake

PROFILE = FALSE   # disable (default)
PROFILE = TRUE    # enable

Related flags (all default FALSE):

COMM_PROFILE  = TRUE   # also profile MPI communication
TRACE_PROFILE = TRUE   # record full call traces
MEM_PROFILE   = TRUE   # track memory usage

CMake

Full profiling is an AMReX-layer flag and is not exposed as an -DEXAGOOP_… option in CMakeLists.txt. Pass it directly to the AMReX sub-build or enable it inside CMakeLists.txt before the add_subdirectory(amrex) call.

What it enables

  • AMReX defines AMREX_PROFILING (and optionally AMREX_COMM_PROFILING, AMREX_TRACE_PROFILING).

  • A .PROF suffix is appended to the GNUmake executable name.

  • PROFILE=TRUE and TINY_PROFILE=TRUE are mutually exclusive; AMReX will error if both are set.

TINY_PROFILE — Lightweight Profiling

Enables AMReX’s low-overhead profiler, which accumulates per-region wall-clock totals and prints a summary table at the end of the run. Use this as a first-pass performance diagnostic when full PROFILE overhead is undesirable.

GNUmake

TINY_PROFILE = FALSE   # disable (default)
TINY_PROFILE = TRUE    # enable (only when PROFILE = FALSE)

CMake

Not exposed as an ExaGOOP CMake option; configure at the AMReX level if needed.

What it enables

  • AMReX defines AMREX_TINY_PROFILING.

  • A .TPROF suffix is appended to the GNUmake executable name.

  • Mutually exclusive with PROFILE=TRUE.

DEBUG — Debug Build

Switches from an optimised release build to a debug build: disables most optimisations, enables full debug symbols, activates AMReX runtime assertions, and adds array-bounds checking.

GNUmake

DEBUG = FALSE   # optimised release (default)
DEBUG = TRUE    # debug

When TRUE, AMReX’s gnu.mak (or equivalent compiler file) substitutes -g -O0 -ggdb -ftrapv for the release flags -g1 -O3, and defines AMREX_DEBUG.

CMake

cmake -DCMAKE_BUILD_TYPE=Release ..   # optimised (recommended default)
cmake -DCMAKE_BUILD_TYPE=Debug   ..   # debug

The CMake path defines EXAGOOP_BUILD_TYPE="Debug" or "Release" in the startup banner via add_compile_definitions. Standard -DNDEBUG suppression applies for Release builds.

What it enables

  • AMReX array-bounds and precondition assertions are active.

  • A .DEBUG suffix is appended to the GNUmake executable name.

  • Incompatible with GPU profiling runs due to severe performance degradation.

FSANITIZER — Address and Undefined-Behaviour Sanitizers

Instruments the binary with LLVM/GCC runtime sanitizers to detect memory errors, undefined behaviour, and pointer misuse at run time.

GNUmake

FSANITIZER = FALSE   # disable (default)
FSANITIZER = TRUE    # enable AddressSanitizer + UBSan

A related flag enables thread-safety checking independently:

THREAD_SANITIZER = FALSE   # disable (default)
THREAD_SANITIZER = TRUE    # enable ThreadSanitizer

FSANITIZER and THREAD_SANITIZER should not be set TRUE simultaneously — the two sanitizer runtimes conflict.

CMake

Not exposed as an ExaGOOP CMake option. Pass the sanitizer flags manually via CMAKE_CXX_FLAGS if needed:

cmake -DCMAKE_CXX_FLAGS="-fsanitize=address,undefined" ..

Compiler flags injected (via AMReX’s gnu.mak )

-fsanitize=address
-fsanitize=undefined
-fsanitize=pointer-compare
-fsanitize=pointer-subtract
-fsanitize=builtin
-fsanitize=pointer-overflow

What it enables

  • AddressSanitizer catches heap/stack buffer overflows, use-after-free, and use-after-return.

  • UndefinedBehaviorSanitizer detects signed-integer overflow, null pointer dereferences, misaligned accesses, and invalid enum values.

  • Pointer sanitizers flag illegal pointer arithmetic.

  • Requires a runtime library (libasan, libubsan); the executable is significantly slower and uses more memory than a release build.

Note

Sanitizers are incompatible with GPU backends (USE_CUDA, USE_HIP). Use them on CPU-only debug builds only.