Benchpark Lists

The easiest way to get started with Benchpark is to run already specified experiments on already specified systems, or to modify one that is similar. You can search through the existing experiments and benchmarks with the below commands.

Search for available system and experiment specifications in Benchpark.

Searching for specifications in Benchpark

Command	Description	Listing in the docs
benchpark list <category>	Lists available objects. Choose one of benchmarks, experiments, systems, or modifiers.
benchpark list systems	Lists all systems specified in Benchpark	System Specifications
benchpark list benchmarks	Lists all benchmarks specified in Benchpark	Benchmarks and Experiments
benchpark list experiments	Lists all experiments specified in Benchpark	Benchmarks and Experiments
benchpark list modifiers	Lists all modifiers specified in Benchpark	Benchpark Modifiers
benchpark tags	Lists all tags specified in Benchpark
benchpark tags -a application	Lists all tags specified for a given application in Benchpark
benchpark tags -t tag	Lists all experiments in Benchpark with a given tag
benchpark info system <system>	Lists all information about a given system
benchpark info experiment <experiment>	Lists all information about a given experiment
benchpark bootstrap	Manually trigger bootstrapping or update the bootstrap

Benchpark workflow and utility commands

Command	Description
benchpark system init <system-spec>	Initializes a system configuration from a system spec.
benchpark system id <system-dir>	Prints the system ID for an initialized system directory.
benchpark system external <system-spec>	Checks packages found with spack external find against a Benchpark system definition.
benchpark experiment init <system-dir> <experiment-spec>	Initializes an experiment for a generated system directory.
benchpark setup <experiment> <experiments-root>	Sets up an experiment and prepares it to build and run.
benchpark redo <system-dir> <experiments-root>	Re-instantiates all experiments under a system directory.
benchpark aggregate –dest <out-dir> <workspace> [workspace …]	Aggregates multiple experiment workspaces into one submission script.
benchpark configure	Writes Benchpark environment configuration such as the bootstrap location.
benchpark mirror create <workspace> <dest-dir>	Copies a Benchpark workspace and its resources into a mirror directory.
benchpark show-build dump <workspace> <dest-dir>	Dumps logs and resources needed to inspect how Spack built a benchmark.
benchpark unit-test	Runs Benchpark unit tests.
benchpark audit	Looks for problems in system and experiment repositories.
benchpark analyze	Performs pre-defined analysis on Caliper performance data after ramble on.

Benchpark also has a help menu:

$ benchpark --help

usage: main.py [-h] [-V] {tags,system,experiment,setup,unit-test,audit,mirror,info,show-build,list,bootstrap,analyze,redo,aggregate,configure} ...

Benchpark
options:
  -h, --help            show this help message and exit
  -V, --version         show version number and exit
Subcommands:
  {tags,system,experiment,setup,unit-test,audit,mirror,info,show-build,list,bootstrap,analyze,redo,aggregate,configure}
    tags                Tags in Benchpark experiments
    system              Initialize a system config
    experiment          Interact with experiments
    setup               Set up an experiment and prepare it to build/run
    unit-test           Run benchpark unit tests
    audit               Look for problems in System/Experiment repos
    mirror              Copy a benchpark workspace
    info                Get information about Systems and Experiments
    show-build          Show how spack built a benchmark
    list                List experiments, systems, benchmarks, and modifiers
    bootstrap           Bootstrap benchpark or update an existing bootstrap
    analyze             Perform pre-defined analysis on the performance data (caliper files) after 'ramble on'
    redo                Re-instantiate system and all associated experiments
    aggregate           Aggregate multiple experiments (even across workspaces) into the same submission script
    configure           Configure options relating to the Benchpark environment

The benchpark list command is used to search the available benchmarks, systems, modifiers, and experiments in Benchpark.

usage: main.py list [-h] {benchmarks,experiments,systems,modifiers} ...

positional arguments:
  {benchmarks,experiments,systems,modifiers}
                        List available experiments, systems, and modifiers

options:
  -h, --help            show this help message and exit

Benchmarks:
    ad
    amg2023
    babelstream
    branson
    commbench
    genesis
    gpcnet
    gromacs
    hpcg
...

Systems - SYSTEM_DEFINITION CLUSTER/INSTANCE
    aws-pcluster instance_type=[c6g.xlarge|c4.xlarge|hpc7a.48xlarge|hpc6a.48xlarge]
    aws-tutorial instance_type=[c7i.48xlarge|c7i.metal-48xl|c7i.24xlarge|c7i.metal-24xl|c7i.12xlarge]
    csc-lumi
    cscs-daint
    cscs-eiger
    fluxtainer
    generic-x86
    jsc-juwels
    lanl-rocinante cluster=[rocinante|tycho|crossroads]
...

Modifiers:
    affinity
    	(all benchmarks)
    allocation
    	(all benchmarks)
    caliper
    	amg2023
    	babelstream
    	branson
    	hpcg
    	hpl
    	kripke
    	laghos
    	py-scaffold
    	quicksilver
    	qws
    	raja-perf
    	remhos
    	stream
    hwloc
    	(all benchmarks)

Experiments - BENCHMARK+PROGRAMMING_MODEL+SCALING
    ad+[mpi]
    amg2023+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    babelstream+[openmp|cuda|rocm]
    branson+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    commbench+[cuda|rocm]
    genesis+[openmp|mpi]
    gpcnet+[mpi]
    gromacs+[openmp|cuda|rocm|mpi]
    hpcg+[openmp|mpi]+[strong|weak]
...

Additionally, this command can be used to search for experiments with one or more programming models:

$ benchpark list experiments --experiment openmp rocm

Experiments - BENCHMARK+PROGRAMMING_MODEL+SCALING
    amg2023+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    babelstream+[openmp|cuda|rocm]
    branson+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    commbench+[cuda|rocm]
    genesis+[openmp|mpi]
    gromacs+[openmp|cuda|rocm|mpi]
    hpcg+[openmp|mpi]+[strong|weak]
    hpl+[openmp|mpi]+[strong|weak]
    kripke+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    laghos+[cuda|rocm|mpi]+[strong|weak|throughput]
    lammps+[openmp|cuda|rocm|mpi]+[strong]
    osu-micro-benchmarks+[cuda|rocm|mpi]
    py-scaffold+[cuda|rocm]+[strong|weak]
    quicksilver+[openmp|mpi]+[strong|weak]
    qws+[openmp|mpi]+[strong|weak|throughput]
    raja-perf+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    remhos+[cuda|rocm|mpi]+[strong|weak|throughput]
    salmon-tddft+[openmp|mpi]
    sparta-snl+[openmp|cuda|rocm|mpi]

Or search which experiments have the Caliper modifier (see Benchpark Modifiers) available:

$ benchpark list modifiers

Modifiers:
    affinity
    	(all benchmarks)
    allocation
    	(all benchmarks)
    caliper
    	amg2023
    	babelstream
    	branson
    	hpcg
    	hpl
    	kripke
    	laghos
    	py-scaffold
    	quicksilver
    	qws
    	raja-perf
    	remhos
    	stream
    hwloc
    	(all benchmarks)

Now that you know the existing benchmarks and systems, you can determine your necessary workflow in Benchpark Workflow.