For the Impatient

You need git and Python 3.8+:

git clone https://github.com/LLNL/benchpark.git
cd benchpark
. setup-env.sh
benchpark --version

python3 -m venv my-env
. my-env/bin/activate

pip install -r requirements.txt

Set up a Workspace

benchpark system init --dest=</output/path/to/system_def_dir/mySystemName> <SystemName> compiler=<Compiler>
benchpark experiment init <mySystemName> <Benchmark> +/~<Boolean Variant> <String Variant>=<value>
benchpark setup <mySystemName>/<Benchmark> </output/path/to/workspace>

where:

• <Benchmark>: amg2023 | saxpy | etc. (predefined choices in Benchmarks and Experiments)

• <SystemName>: Cts | Tioga | etc. (predefined systems in System Specifications)

Experiment variables can be overridden with name=value assignments, even when the experiment does not declare them as variants. For example:

benchpark experiment init tuolumne raja-perf+rocm n_nodes=384 n_resources=1536 n_cores_per_rank=21

Declared variant names keep their existing validation. Other assignments override experiment and helper variable defaults, or add variables when absent. These values are applied before scaling, so scaling starts from the supplied values. Integers and floating-point values are converted to numbers; expressions and other strings are preserved. Comma-separated values define a list, and shell quotes preserve values containing spaces. For grouped variables, override the individual dimensions, such as npx=4 npy=2 npz=1.

benchpark setup will output instructions to follow:

. <experiments_root>/setup.sh

Build an Experiment

cd <experiments_root>/<System>/<Benchmark>/workspace
ramble --workspace-dir . workspace setup

Run the Experiment

To run all of the experiments in the workspace:

ramble --workspace-dir . on

To run a single experiment in the workspace, invoke the execute_experiment script for the specific experiment (e.g., $workspace/experiments/amg2023/problem1/amg2023_cuda11.8.0_problem1_1_8_2_2_2_10_10_10/execute_experiment).

Experiment pass/fail

Once the experiments completed running, the command:

ramble --workspace-dir . workspace analyze

can be used to analyze figures of merit and evaluate success/failure of the experiments. Ramble generates a file with summary of the results in $workspace.

If the benchmark you are running is instrumented with Caliper, you can use the Caliper modifier (see Benchpark Modifiers) to collect detailed measurements you can later analyze with Thicket.