Benchpark Lists

The easiest way to get started with Benchpark is to run already specified experiments on already specified systems, or to modify one that is similar. You can search through the existing experiments and benchmarks with the below commands.

Search for available system and experiment specifications in Benchpark.

Searching for specifications in Benchpark

Command

Description

Listing in the docs

benchpark list

Lists all benchmarks, systems, and experiments specified in Benchpark

benchpark list systems

Lists all system specified in Benchpark

System Specifications

benchmark list benchmarks

Lists all benchmarks specified in Benchpark

Benchmarks and Experiments

benchmark list experiments

Lists all experiments specified in Benchpark

Benchmarks and Experiments

benchmark list modifiers

Lists all modifiers specified in Benchpark

Benchpark Modifiers

benchpark tags workspace

Lists all tags specified in Benchpark

benchpark tags -a application workspace

Lists all tags specified for a given application in Benchpark

benchpark tags -t tag workspace

Lists all experiments in Benchpark with a given tag

benchpark info system <system>

Lists all information about a given system

benchpark info experiment <experiment>

Lists all information about a given experiment

benchpark bootstrap

Manually trigger bootstrapping or update the bootstrap

Benchpark also has a help menu:

$ benchpark --help
usage: main.py [-h] [-V] {tags,system,experiment,setup,unit-test,audit,mirror,info,show-build,list,bootstrap,analyze} ...

Benchpark

options:
  -h, --help            show this help message and exit
  -V, --version         show version number and exit

Subcommands:
  {tags,system,experiment,setup,unit-test,audit,mirror,info,show-build,list,bootstrap,analyze}
    tags                Tags in Benchpark experiments
    system              Initialize a system config
    experiment          Interact with experiments
    setup               Set up an experiment and prepare it to build/run
    unit-test           Run benchpark unit tests
    audit               Look for problems in System/Experiment repos
    mirror              Copy a benchpark workspace
    info                Get information about Systems and Experiments
    show-build          Show how spack built a benchmark
    list                List experiments, systems, benchmarks, and modifiers
    bootstrap           Bootstrap benchpark or update an existing bootstrap
    analyze             Perform pre-defined analysis on the performance data (caliper files) after 'ramble on'

The benchpark list command is used to search the available benchmarks, systems, modifiers, and experiments in Benchpark.

Seaborn not found, so skipping imports of plotting in thicket.stats
To enable this plotting, install seaborn or thicket[plotting]
usage: main.py list [-h] {benchmarks,experiments,systems,modifiers} ...

positional arguments:
  {benchmarks,experiments,systems,modifiers}
                        List available experiments, systems, and modifiers

options:
  -h, --help            show this help message and exit
Seaborn not found, so skipping imports of plotting in thicket.stats
To enable this plotting, install seaborn or thicket[plotting]
Benchmarks:
    ad
    amg2023
    babelstream
    genesis
    gpcnet
    gromacs
    hpcg
...
Seaborn not found, so skipping imports of plotting in thicket.stats
To enable this plotting, install seaborn or thicket[plotting]
Systems - SYSTEM_DEFINITION CLUSTER/INSTANCE
    aws-pcluster instance_type=[c6g.xlarge|c4.xlarge|hpc7a.48xlarge|hpc6a.48xlarge]
    csc-lumi
    cscs-daint
    cscs-eiger
    generic-x86
    jsc-juwels
    lanl-venado cluster=[grace-hopper|grace-grace]
...
Seaborn not found, so skipping imports of plotting in thicket.stats
To enable this plotting, install seaborn or thicket[plotting]
Modifiers:
    affinity
    	(all benchmarks)
    allocation
    	(all benchmarks)
    caliper
    	amg2023
    	babelstream
    	hpcg
    	hpl
    	kripke
    	laghos
    	quicksilver
    	qws
    	raja-perf
    	remhos
    	saxpy
    	stream
    hwloc
    	(all benchmarks)
Seaborn not found, so skipping imports of plotting in thicket.stats
To enable this plotting, install seaborn or thicket[plotting]
Experiments - BENCHMARK+PROGRAMMING_MODEL+SCALING
    ad+[mpi]
    amg2023+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    babelstream+[openmp|cuda|rocm]
    genesis+[openmp|mpi]
    gpcnet+[mpi]
    gromacs+[openmp|cuda|rocm|mpi]
    hpcg+[openmp|mpi]+[strong|weak]
...

Additionally, this command can be used to search for experiments with one or more programming models:

$ benchpark list experiments --experiment openmp rocm
Seaborn not found, so skipping imports of plotting in thicket.stats
To enable this plotting, install seaborn or thicket[plotting]
Experiments - BENCHMARK+PROGRAMMING_MODEL+SCALING
    amg2023+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    babelstream+[openmp|cuda|rocm]
    genesis+[openmp|mpi]
    gromacs+[openmp|cuda|rocm|mpi]
    hpcg+[openmp|mpi]+[strong|weak]
    hpl+[openmp|mpi]+[strong|weak]
    kripke+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    laghos+[cuda|rocm|mpi]+[strong]
    lammps+[openmp|cuda|rocm|mpi]+[strong]
    osu-micro-benchmarks+[cuda|rocm|mpi]
    quicksilver+[openmp|mpi]+[strong|weak]
    qws+[openmp|mpi]
    raja-perf+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    remhos+[cuda|rocm|mpi]+[strong]
    salmon-tddft+[openmp|mpi]
    saxpy+[openmp|cuda|rocm|mpi]

Or search which experiments have the Caliper modifier (see Benchpark Modifiers) available:

$ benchpark list modifiers
Seaborn not found, so skipping imports of plotting in thicket.stats
To enable this plotting, install seaborn or thicket[plotting]
Modifiers:
    affinity
    	(all benchmarks)
    allocation
    	(all benchmarks)
    caliper
    	amg2023
    	babelstream
    	hpcg
    	hpl
    	kripke
    	laghos
    	quicksilver
    	qws
    	raja-perf
    	remhos
    	saxpy
    	stream
    hwloc
    	(all benchmarks)

Now that you know the existing benchmarks and systems, you can determine your necessary workflow in Benchpark Workflow.