Adding an Experiment

Glossary terms: Benchmark, Package, Application, Spec, Variant, Experiment, System, Modifier, Scaling Study

Create either missing prerequisite before adding the experiment:

Missing a Spack package?

Missing a Ramble application?

This guide is intended for developers who want to add the ability to run a benchmark with Benchpark. Once a Spack package.py and Ramble application.py describe how to build and run the benchmark, the next step is creating its experiment.py. This guide demonstrates that process for HPL.

Create the Experiment Definition

This example defines the HPL experiment used throughout the Beginner Tutorial. Replace experiments/hpl/experiment.py with the below definition:

 1# Copyright 2023 Lawrence Livermore National Security, LLC and other
 2# Benchpark Project Developers. See the top-level COPYRIGHT file for details.
 3#
 4# SPDX-License-Identifier: Apache-2.0
 5
 6from benchpark.experiment import Experiment
 7# Step 1 adds the programming-model import here.
 8from benchpark.caliper import Caliper
 9from benchpark.directives import variant, maintainers
10from benchpark.scaling import Scaling, ScalingMode
11
12
13class Hpl(
14    Experiment,
15    # Step 1 adds the supported programming models here.
16    Caliper,
17    Scaling(ScalingMode.Strong, ScalingMode.Weak),
18):
19    variant(
20        "workload",
21        default="standard",
22        description="Which ramble workload to execute.",
23    )
24
25    # Step 2 adds the version variant here.
26
27    maintainers("daboehme")
28
29    def compute_applications_section(self):
30
31        if self.spec.satisfies("exec_mode=test"):
32            self.add_experiment_variable("n_ranks", 1, True)
33            self.add_experiment_variable("Ns", 1000, True)
34            self.add_experiment_variable("N-Grids", 1, False)
35            self.add_experiment_variable("Ps", "{n_ranks}", True)
36            self.add_experiment_variable("Qs", "1", False)
37            self.add_experiment_variable("N-Ns", 1, False)
38            self.add_experiment_variable("N-NBs", 1, False)
39            self.add_experiment_variable("NBs", 128, False)
40
41        # Step 3 adds the exec_mode=perf else block here.
42
43        # Step 4 adds the required-variable definitions here.
44
45        self.register_scaling_config(
46            {
47                ScalingMode.Strong: {
48                    "n_ranks": lambda var, itr, dim, scaling_factor: (
49                        var.val(dim) * scaling_factor
50                    ),
51                    "Ns": lambda var, itr, dim, scaling_factor: var.val(dim),
52                },
53                ScalingMode.Weak: {
54                    "n_ranks": lambda var, itr, dim, scaling_factor: (
55                        var.val(dim) * scaling_factor
56                    ),
57                    "Ns": lambda var, itr, dim, scaling_factor: (
58                        var.val(dim) * scaling_factor
59                    ),
60                },
61            }
62        )
63
64    def compute_package_section(self):
65        self.add_package_spec(self.name, [f"hpl{self.determine_version()}"])

1. Add Programming Models

Programming models specify how the source code parallelizes the benchmark. Replace the first Step 1 placeholder with the programming-model import:

from benchpark.programming_model import ProgrammingModel, ProgrammingModelType

Replace the second Step 1 placeholder in the Hpl class bases with the supported MPI and OpenMP programming models:

ProgrammingModel(ProgrammingModelType.Mpionly, ProgrammingModelType.Openmp),

2. Add the Version Variant

The version variant will enable us to build and run the source code versions specified in the benchmark’s Spack package. For HPL, 2.3-caliper, 2.3 and 2.2 are defined in repos/spack_repo/benchpark/packages/hpl/package.py. The latest value is a Benchpark keyword that selects the newest version defined by the Spack package.

Replace the Step 2 placeholder with the complete version variant:

variant(
    "version",
    default="2.3-caliper",
    values=("latest", "2.3-caliper", "2.3", "2.2"),
    description="Which benchmark version to use.",
)

3. Add a Performance Mode

The exec_mode=test experiment definition provides small values for the problem size and number of ranks, suitable for short CI runs. Replace the Step 3 placeholder with the following else branch to define a set of larger experiment variables for collecting performance data:

# Must be exec_mode=perf if not test mode.
else:
    self.add_experiment_variable("n_ranks", 32, True)
    self.add_experiment_variable("Ns", 100000, True)
    self.add_experiment_variable("N-Grids", 1, False)
    self.add_experiment_variable("Ps", "({n_ranks} // 8)", True)
    self.add_experiment_variable("Qs", "8", False)
    self.add_experiment_variable("N-Ns", 1, False)
    self.add_experiment_variable("N-NBs", 1, False)
    self.add_experiment_variable("NBs", 128, False)

4. Set Required Variables

We record the problem sizes for each experiment, and the resources the experiment requires. Replace the Step 4 placeholder with the following call, which defines the number of ranks, the HPL problem size per rank, and the total HPL problem size:

# Set the variables required by the experiment
self.set_required_variables(
    n_resources="{n_ranks}",
    process_problem_size="{Ns}/{n_ranks}",
    total_problem_size="{Ns}",
)

For the complete experiment authoring reference, see Adding an Experiment.

Next Step

I want to add a scaling configuration to this experiment.

I want to build and run the new experiment on my existing system.