Benchpark Modifiers
In Benchpark, a modifier follows the Ramble Modifier and is an
abstract object that can be applied to a large set of reproducible specifications.
Modifiers are intended to encapsulate reusable patterns that perform a specific
configuration of an experiment. This may include injecting performance analysis or
setting up system resources.
Affinity: Threads, GPUs
We are using (with permission) the following implementation of Linux thread and GPU affinity checks. The following checks are possible:
affinity.mpi: for testing thread affinity of each rank in an MPI jobaffinity.rocm: for testing AMD GPU affinity of each rank an MPI jobaffinity.cuda: for testing NVIDIA GPU affinity of each rank in an MPI job
To use the Affinity modifier:
When you initialize your experiment, add
affinity=ontoexperiment init.A small, separate run will execute in your allocation to record the information.
The Affinity modifier will output a text file in the experiment directory which will look like this for
benchpark experiment init --dest=saxpy system-name saxpy+openmp affinity=onon 8 ranks, 2 threads/proc, 1 node:
affinity.mpi.out
affinity test for 8 MPI ranks
rank 0 @ dane1514
thread 0 on cores [112]
thread 1 on cores [119]
rank 1 @ dane1514
thread 0 on cores [126]
thread 1 on cores [133]
rank 2 @ dane1514
thread 0 on cores [140]
thread 1 on cores [147]
rank 3 @ dane1514
thread 0 on cores [154]
thread 1 on cores [161]
rank 4 @ dane1514
thread 0 on cores [168]
thread 1 on cores [175]
rank 5 @ dane1514
thread 0 on cores [182]
thread 1 on cores [189]
rank 6 @ dane1514
thread 0 on cores [196]
thread 1 on cores [203]
rank 7 @ dane1514
thread 0 on cores [210]
thread 1 on cores [217]
If also running with the caliper modifier, affinity information will be included
in the Caliper metadata.
Caliper: Profiling
Caliper is an instrumentation and performance profiling library. We have implemented a Caliper modifier to enable profiling of Caliper-instrumented benchmarks in Benchpark.
To turn on profiling with Caliper, add caliper=<caliper_variant> to the experiment
init setup step:
benchpark experiment init --dest=</path/to/experiment_root> </path/to/system> <benchmark> caliper=<caliper_variant>
Valid values for <caliper_variant> are found in the Caliper Variant column of
the table below. Benchpark will link the experiment to Caliper, and inject appropriate
Caliper configuration at runtime. After the experiments in the workspace have completed
running, a .cali file is created which contains the collected performance metrics.
Caliper Variant |
Where Applicable |
Metrics Collected |
|---|---|---|
time |
Platform-independent |
- Min/Max/Avg time/rank: Minimum/Maximum/Average time (in seconds) across
all ranks
- Total time: Aggregated time (in seconds) for all ranks
|
mpi |
Platform-independent |
- Same as basic caliper modifier above
- Profiles MPI functions
|
cuda |
NVIDIA GPUs |
- CUDA API functions
- GPU time
|
rocm |
AMD GPUs |
- HIP API functions
- GPU time
|
topdown-counters-all |
x86 Intel CPUs |
- Raw counter values for Intel top-down analysis (all levels)
|
topdown-counters-toplevel |
x86 Intel CPUs |
- Raw counter values for Intel top-down analysis (top level)
|
topdown-all |
x86 Intel CPUs |
- Top-down analysis for Intel CPUs (all levels)
|
topdown-toplevel |
x86 Intel CPUs |
- Top-down analysis for Intel CPUs (top level)
|
Caliper services can also be enabled independently of the main caliper mode by
adding caliper_services=<service> to the experiment spec:
benchpark experiment init --dest=</path/to/experiment_root> </path/to/system> <benchmark> caliper_services=<service>
Available services are:
nvtxfor CUDA systemsrocprofilerfor ROCm systems
An experiment must inherit from the Caliper experiment class to make use of the Caliper functionality. Most existing experiments should already do this, but if adding to a new experiment, it is as simple as adding it to the class definition signature. For example:
class Amg2023(Experiment, Caliper):
Allocation: Resources
Given:
an experiment that requests resources (nodes, cpus, gpus, etc.), and
a specification of the resources available on the system (cores_per_node, gpus_per_node, etc.),
the Allocation Modifier generates the appropriate scheduler request for these
resources (how many nodes are required to run a given experiment, etc.).
Resource |
Available on the System |
Requested for the Experiment |
|---|---|---|
Total Nodes |
(opt) sys_nodes |
n_nodes |
Total MPI Ranks |
n_ranks |
|
CPU cores per node |
sys_cores_per_node |
(opt) n_cores_per_node |
GPUs per node |
sys_gpus_per_node |
(opt) n_gpus_per_node |
Memory per node |
sys_mem_per_node_GB |
(opt) n_mem_per_node |
The experiment is required to specify:
n_ranks it requires
n_gpus (if using GPUs)
If the experiment does not specify n_nodes, the modifier will compute the number of
nodes to allocate to provide the n_ranks and/or n_gpus required for the
experiment.
The system is required to specify:
sys_cores_per_node
sys_gpus_per_node (if it has GPUs)
sys_mem_per_node_GB
The modifier checks the resources requested by the experiment, computes the values for the unspecified variables, and checks that the request does not exceed the resources available on the system.
The resource allocation modifier is used by default in your experiment. However, it will only calculate values if you have not specified them yourself.
If you do not specify values, it will assign the default values as listed below.
Variable |
Default Value |
|---|---|
n_nodes |
(n_ranks / sys_cores_per_node) OR (n_gpus / sys_gpus_per_node) whichever is greater |
n_ranks |
(n_nodes * n_ranks_per_node) OR (n_gpus) |
n_gpus |
0 |
n_threads_per_proc |
1 |
Hwloc: Hardware Map
The hwloc modifier enables using hwloc in Benchpark to record the hierarchical map of key computing elements on the given system, such as: NUMA memory nodes, shared caches, processor sockets, processor cores, and processor threads.
To use the hwloc modifier, add hwloc=on to the experiment init setup step. This
modifier is disabled by default (hwloc=none).:
benchpark experiment init --dest=</path/to/experiment_root> </path/to/system> <benchmark> caliper=<caliper_variant> hwloc=on
The hwloc modifier will output the hardware information in a flattened JSON file in the experiment directory.
If also running with the caliper modifier, hwloc information will be included in
the Caliper metadata.
GitHash: Version Metadata
The githash modifier records version metadata for the Benchpark repository, the
pinned Benchpark dependencies, and the application package plus any dependencies. The
package metadata describes the spec installed in the experiment’s Spack environment.
The githash modifier is enabled by default (githash=on). You can specify
githash=none to disable usage.
benchpark experiment init --dest=</path/to/experiment_root> </path/to/system> <benchmark> caliper=<caliper_variant> githash=on
The githash modifier writes this information to githash_metadata.json in the
experiment directory.
If also running with the caliper modifier, Benchpark adds githash_metadata.json
to the Caliper metadata.
Compiler Memtrace: HIP Memory Access
The compiler_memtrace modifier uses memtrace-llvm to catalog and trace memory operations in HIP
kernels. Enable it on a ROCm experiment:
benchpark experiment init --dest=raja-memtrace /path/to/system \
raja-perf+rocm compiler_memtrace=trace n_resources=1 process_problem_size=1024
The modifier requires package_manager=spack, one amdgpu_target, and a Clang
compiler with LLVM 19 or newer development files. The benchmark and memtrace-llvm
must use the same C++ compiler and HIP installation. The package builds against that
compiler’s LLVM, and installs the pass, HIP runtime, device headers, Python reader, and
memtrace command. Version 0.1.0 is pinned to commit
e229827a58479ba87a1efd0b1255a8ad6eb911d5 from PR #8.
Setup checks that the Python reader imports before building the benchmark. Count and
trace runs repeat this check before launching. The NumPy dependency uses Spack’s builtin
py-numpy recipe. With MKL, reader processes preload the interface, threading, and
core libraries before importing NumPy. This handles Meson builds in which lld omits the
latter two libraries from NumPy’s direct dependencies. Benchmark processes run without
these reader preloads. For existing environments that used Benchpark’s NumPy override,
setup selects the builtin recipe through Spack’s concrete-spec update API while
preserving unrelated package selections.
The modes are:
none(default): disable this modifier.baseline: build with the pass and write static catalogs without device instrumentation.count: build and run a count executable.trace: build count and trace executables, run count first, then run trace with the same command-line arguments and the recorded counts.
Setup creates separate Spack environments and installation prefixes for the instrumented
builds. Compiler flags identify the mode and tracing options in the Spack spec.
Dependencies are installed without the tracing environment; only the benchmark root is
instrumented. Catalogs and builds.json are under
{env_path}/compiler-memtrace/<configuration>/. The latter records executable
prefixes, reader paths and preloads, and collection options. Count and trace must use
the same sources and inputs. Applications that modify input files during execution must
restore those inputs between runs or use separate experiment workflows.
For CMake packages, setup adds MemTrace compile and link options to targets in the
staged benchmark source tree. CMake’s compiler checks run without these options,
including the host-only HIP check that cannot link device definitions. The package
install preserves the prepared stage with --dont-restage. Setup requires a catalog
containing benchmark memory operations for the selected mode and GPU target;
compiler-probe catalogs do not satisfy this check.
Each execution creates a fresh directory beneath the experiment run directory:
compiler_memtrace.run.<unique suffix>/
count/rank-0/
trace/rank-0/
Each rank writes its own manifest, catalogs, and binary files. Count and trace use separate MPI launches. Rank identification supports Flux, PMI, PMIx, Open MPI, and Slurm. A multi-rank run fails if its rank cannot be identified. The output filesystem must be visible to every rank.
Collection fails if no launches are recorded, a catalog or data file is missing, the
trace is invalid, or the count and trace launch sets differ. Successful validation
writes validated.json in each rank directory. Count output goes to
compiler_memtrace.<executable>.count.out; trace output uses the application’s normal
log. To inspect one rank’s output with the package loaded:
memtrace summary /path/to/trace/rank-0
memtrace check /path/to/trace/rank-0
memtrace csv /path/to/trace/rank-0 accesses.csv
These Ramble variables can be supplied through experiment init or in the generated
YAML:
Variable |
Default |
Meaning |
|---|---|---|
|
|
Space-separated executable name patterns. Change this for applications whose benchmark executable has a different Ramble name. |
|
|
Colon-separated AMDGPU address spaces. Add |
|
|
Set to |
|
empty |
Path to an occupancy file containing a mangled kernel name and waves per execution unit on each line. Use occupancy from an uninstrumented build. |
|
|
Set to |
Selected Ramble executables must each contain one direct benchmark command. Shell
pipelines, compound commands, and launcher wrappers inside that command are unsupported;
use Ramble’s mpi_command for the launcher. The executable must reside in the
benchmark package prefix or its Spack view. Use separate experiments for
compiler_attribution and compiler_memtrace.
The upstream runtime traces hipLaunchKernel outside stream capture. Other launch
APIs, device functions that remain uninlined, and device memory intrinsics are outside
its trace coverage. One launch’s trace must fit in GPU memory, at 528 bytes per wave
record, plus optional timing data. Use small workloads when establishing a collection
configuration. Instrumented launches are serialized; their execution times are
unsuitable for baseline performance comparisons.