Perform a Parameter Sweep
Glossary terms: Benchmark, Experiment, System
Initialize the AWS tutorial system:
benchpark]$ benchpark system init --dest=hpc-tutorial fluxtainer-aws
Then, initialize the HPL experiment:
benchpark]$ benchpark experiment init --dest=hpl hpc-tutorial hpl +strong caliper=time,mpi
Create the experiment workspace:
benchpark]$ benchpark setup hpc-tutorial/hpl wkp
The setup.sh script generated by benchpark setup activates Benchpark’s Spack and
Ramble environments.
benchpark]$ . wkp/setup.sh
The wkp directory contains ramble, spack, spack-packages, and
hpc-tutorial/hpl/workspace sub-directories. The ramble and spack directories
contain the source for those respective tools. The spack-packages directory contains
a Spack packages repository. The hpc-tutorial/hpl/workspace directory contains the
Ramble workspace for the experiments in this tutorial.
At this point, move into the workspace directory to continue with the remaining steps:
benchpark]$ cd wkp/hpc-tutorial/hpl/workspace
Within this workspace, we can see what experiments we have configured:
workspace]$ ramble --workspace-dir . workspace info
This should print the following output:
Workspace: workspace
Location: /home/fluxuser/benchpark/wkp/hpc-tutorial/hpl/workspace
Experiments:
Application: hpl
Workload: standard
Experiment 1: hpl.standard.hpl_standard_test_mpi_caliper_time_mpi_strong_scaling_1_1000_1
Experiment 2: hpl.standard.hpl_standard_test_mpi_caliper_time_mpi_strong_scaling_2_1000_2
Experiment 3: hpl.standard.hpl_standard_test_mpi_caliper_time_mpi_strong_scaling_4_1000_4
Experiment 4: hpl.standard.hpl_standard_test_mpi_caliper_time_mpi_strong_scaling_8_1000_8
However, we would like to change these experiments to allow us to explore what happens when the block size in HPL changes.
Before editing the configuration file, we will inspect the HPL application definition,
and see which variables exist. The workload we are using is called standard.
workspace]$ ramble info hpl --attrs workloads -v -p standard
This will print the information about the standard workload for HPL, but the portion of interest for this tutorial is:
NBs:
Description: NB values
Default: 1 2 3 4
The NB parameter in HPL defines the size of the blocks used in the matrix solve. Now
that we know the variable name is NBs, we can edit the configuration file. To do
this, open the configs/ramble.yaml file in a text editor.
workspace]$ nano configs/ramble.yaml
Within this file, you should see the following content:
variables:
N-Grids: 1
N-NBs: 1
N-Ns: 1
NBs: 128
Ns:
- 1000
- 1000
- 1000
- 1000
Ps: '{n_ranks}'
Qs: '1'
n_ranks:
- 1
- 2
- 4
- 8
n_resources: '{n_ranks}'
process_problem_size: '{Ns}/{n_ranks}'
total_problem_size: '{Ns}'
The are the variable definitions for the experiments in this workspace. Here you can see
that Ns and n_ranks are both vector variables (meaning they have multiple
values). Ramble will implicly zip these together, since they are the same length. This
means the first values of each are grouped together into the first experiment, the
second together, and so on.
You can see there is a definition for NBs already, that sets it to 128. We want
to parameterize this, and make sure the block size is swept over all of our rank sizes
as well. Replace the NBs definition with the following:
NBs:
- 64
- 100
- 128
- 150
- 256
Now that we have defined additional NBs values, we need to tell Ramble how to
construct experiments from this definition. Ramble requires that each experiment has a
unique name (to avoid aliasing issues). The current experiment name template in this
file looks like the following:
hpl_standard_test_mpi_caliper_time_mpi_strong_scaling_{n_ranks}_{Ns}_{Ps}:
Add the value of NBs to this, as in the following:
hpl_standard_test_mpi_caliper_time_mpi_strong_scaling_{n_ranks}_{Ns}_{Ps}_{NBs}:
Finally, we need to tell Ramble that we want experiments to be generated by computing
the cross-product of NBs with the other (already zipped) variables from our existing
experiments. To do this, we need to add a matrix definition within the experiment
template block. In the experiment block, there is already a definition:
matrix: []
Replace this with:
matrix:
- NBs
This tells Ramble to consume the NBs variable as a dimension of a matrixed set of
experiments. Since we already have the other variables implicly zipped, Ramble will
implicitly add those as the final dimension of the matrix. It is also possible to make
this definition explicit for the sake of readability. This would look like the
following:
matrix:
- NBs
- ranks_config
zips:
ranks_config
- n_ranks
- Ns
This would requiring removing the zips: {} definition as well. However it is not
necessary in this tutorial. Now, set up the experiments:
workspace]$ ramble --workspace-dir . workspace setup
At this point, Ramble has used Spack to install HPL, and configure the experiment directories for you. We’ll explore the experiment directories later, but to see the output from spack you can open the setup log for the first experiment:
workspace]$ nano logs/setup.latest/hpl.standard.hpl_standard_test_mpi_caliper_time_mpi_strong_scaling_1_-1_1.out
If you explore this file, you should see output related to spack concretize and
spack install. For more information on what these content mean, please refer to
Spack’s documentation. At the bottom of the file, you can see all of the phases Ramble
executed to set up these experiments, along with timing data for each of these.
Now, run the experiments:
workspace]$ ramble --workspace-dir . on
This command submits the batch scripts (e.g., submission scripts) generated in ramble
workspace to the system’s resource manager (which is specified in the files generated
by benchpark system init). For the AWS infrastructure used in this tutorial, the
resource manager is LLNL’s Flux resource manager.
If the above command is successful, you should see something like:
==> Streaming details to log:
==> /home/fluxuser/benchpark/wkp/hpc-tutorial/hpl/workspace/logs/execute.2026-07-09_18.14.08.out
==> Executing 4 out of 4 experiments:
==> Log files for experiments are stored in: /home/fluxuser/benchpark/wkp/hpc-tutorial/hpl/workspace/logs/execute.2026-07-09_18.14.08
==> Running executors...
ƒV54uD5o5
ƒV57fKkEK
ƒV5ANUS6s
ƒV5D498h5
The final lines printed by the ramble on command are the job IDs produced by the
system resource manager. You can use these IDs to track the progress of the jobs in your
experiment. For example, with Flux, you can see job status by running:
flux jobs -a
This command will produce an output like:
Note
If you are running on our AWS infrastructure, it should take roughly 1 minute for all jobs to finish running. Additionally, only one job will run at a time under our infrastructure because each user only has 1 node. If you are running on an HPC system, expect the jobs to complete faster.
After the jobs have drained out of the queue, we can analyze them using:
workspace]$ ramble --workspace-dir . workspace analyze
This creates a file in the root of the workspace named results.latest.txt. Opening
this file, we should see the following:
Experiment hpl.standard.hpl_standard_test_mpi_caliper_time_mpi_strong_scaling_1_-1_1 figures of merit:
Status = FAILED
And in fact, all of the experiments have failed. To find out why, we can open one of the
experiment log files. Ramble creates independent directories for each experiment within
the experiments subdirectory of the workspace. Navigate into one of them using:
workspace]$ cd experiments/hpl/standard/hpl_standard_test_mpi_caliper_time_mpi_strong_scaling_1_-1_1/
Within this directory, you should see the following files:
execute_experiment - The script that is run for the experiment
HPL.dat - The input for HPL
hpl_standard_test_mpi_caliper_time_mpi_strong_scaling_1_-1_1.out - The redirected output from the experiment's execution
Along with several other files. If we open the
hpl_standard_test_mpi_caliper_time_mpi_strong_scaling_1_-1_1.out file, we should see
the following:
HPL ERROR from process # 0, on line 343 of function HPL_pdinfo:
>>> Value of N less than 0 <<<
HPL ERROR from process # 0, on line 621 of function HPL_pdinfo:
>>> Illegal input in file HPL.dat. Exiting ... <<<
flux-job: task(s) exited with exit code 1
This means our HPL input was invalid, causing the run to prematurely exit. This all
stems from our addition of Ns=-1 on the benchpark experiment init line. If we
hadn’t added this line, benchpark would have set the value to be Ns=1000 by default.
To fix this, we will navigate back into the workspace root:
hpl_standard_test_mpi_caliper_time_mpi_strong_scaling_1_-1_1]$ cd ../../../../
Here, open the workspace config file again:
workspace]$ nano configs/ramble.yaml
In the top portion of the file, you should see a group of variables that are defined.
Some are scalars, some are variable references, some are even lists (i.e. n_ranks).
Ramble uses these when generating the individual experiments within this workspace. Find
the variable definition that looks like:
Ns:
- -1
- -1
- -1
- -1
and change all of the -1 to 1000. Then save and exit the file. There are
multiple lines with -1 to allow each value of n_ranks below to have a different
Ns value. Now, delete the old experiment directories to ensure no stale caliper
files exist:
workspace]$ rm -rf experiments/
Then set up the workspace again:
workspace]$ ramble --workspace-dir . workspace setup
And re-submit the jobs:
At this point, set up the workspace again:
workspace]$ ramble --workspace-dir . on
Once the jobs are all finished, we can re-analyze the workspace:
workspace]$ ramble --workspace-dir . workspace analyze
And we can examine the contents of the results.latest.txt file:
workspace]$ nano results.latest.txt
This file should contain the GFLOPs rating for the HPL experiments that you ran, in addition to information about the software stack, and variants used.
Each experiment directory (i.e., subdirectories of experiments/hpl/standard) will
now contain a Caliper output file (i.e., a .cali file) containing performance data
for the job. For individual experiment instances, rerunning experiments, and combining
workspaces into one allocation, see Run an Experiment.