Benchpark Commands

The easiest way to get started is to run existing experiments one existing systems, or to modify one that is similar. You can search through the existing experiments and benchmarks with the below commands.

Search for available system and experiment specifications in Benchpark.

Searching for specifications in Benchpark

Command	Description	Listing in the docs
benchpark list	Lists all benchmarks, systems, and experiments specified in Benchpark
benchpark list systems	Lists all system specified in Benchpark	System Specifications
benchmark list benchmarks	Lists all benchmarks specified in Benchpark	Benchmarks and Experiments
benchmark list experiments	Lists all experiments specified in Benchpark	Benchmarks and Experiments
benchmark list modifiers	Lists all modifiers specified in Benchpark	Benchpark Modifiers
benchpark tags workspace	Lists all tags specified in Benchpark
benchpark tags -a application workspace	Lists all tags specified for a given application in Benchpark
benchpark tags -t tag workspace	Lists all experiments in Benchpark with a given tag
benchpark info system <system>	Lists all information about a given system
benchpark info experiment <experiment>	Lists all information about a given experiment

Benchpark also has a help menu:

$ benchpark --help

usage: main.py [-h] [-V] {tags,system,experiment,setup,unit-test,audit,info,list} ...

Benchpark

options:
  -h, --help            show this help message and exit
  -V, --version         show version number and exit

Subcommands:
  {tags,system,experiment,setup,unit-test,audit,info,list}
    tags                Tags in Benchpark experiments
    system              Initialize a system config
    experiment          Interact with experiments
    setup               Set up an experiment and prepare it to build/run
    unit-test           Run benchpark unit tests
    audit               Look for problems in System/Experiment repos
    info                Get information about Systems and Experiments
    list                List experiments, systems, benchmarks, and modifiers

benchpark list

usage: main.py list [-h] {benchmarks,experiments,systems,modifiers} ...

positional arguments:
  {benchmarks,experiments,systems,modifiers}
                        List available experiments, systems, and modifiers

options:
  -h, --help            show this help message and exit

The benchpark list command is used to search the available benchmarks, systems, modifiers, and experiments in Benchpark.

Benchmarks:
    ad
    amg2023
    babelstream
    genesis
    gpcnet
    gromacs
    hpcg
    hpl
    ior
...

Systems:
    aws-pcluster/c4.xlarge
    aws-pcluster/c6g.xlarge
    aws-pcluster/hpc7a.48xlarge
    aws-pcluster/hpc6a.48xlarge
    csc-lumi/lumi
    cscs-daint/daint
    cscs-eiger/eiger
    generic-x86
    jsc-juwels/juwels
...

Modifiers:
    allocation
    caliper

Experiments:
    ad+single_node
    amg2023+single_node
    amg2023+openmp
    amg2023+cuda
    amg2023+rocm
    amg2023+strong
    amg2023+weak
    amg2023+throughput
    babelstream+single_node
...

Additionally, this command can be used to search for experiments with one or more programming models:

$ benchpark list experiments --experiment openmp rocm

Experiments:
    amg2023+openmp
    amg2023+rocm
    babelstream+rocm
    babelstream+openmp
    genesis+openmp
    gromacs+openmp
    gromacs+rocm
    hpcg+openmp
    hpl+openmp
    kripke+openmp
    kripke+rocm
    laghos+rocm
    lammps+openmp
    lammps+rocm
    osu-micro-benchmarks+rocm
    quicksilver+openmp
    qws+openmp
    raja-perf+rocm
    raja-perf+openmp
    remhos+rocm
    salmon-tddft+openmp
    saxpy+openmp
    saxpy+rocm

Or search which experiments have the Caliper modifier (see Benchpark Modifiers) available:

$ benchpark list modifiers --name caliper --experiments

Modifiers:
    caliper
    	amg2023
    	babelstream
    	hpcg
    	hpl
    	kripke
    	laghos
    	raja-perf
    	remhos
    	saxpy
    	stream

Now that you know the existing benchmarks and systems, you can determine your necessary workflow in Benchpark Workflow.