This page contains answers to frequently asked questions about Benchpark. If you have questions that are not answered here, feel free to ask on #benchpark-support channel in Spack Slack or GitHub Discussions. If you’ve learned the answer to a question that you think should be here, please consider contributing to this page.

Frequently Asked Questions

Benchmark not yet in Spack/Ramble

By default, Benchpark uses benchmark definitions (package.py and application.py) provided by the upstream Spack and Ramble repositories. Override either definition by adding it to the corresponding Benchpark repository:

benchpark/
└── repos/
    ├── ramble_applications/
    │   └── <benchmark-name>/
    │       └── application.py
    └── spack_repo/
        └── benchpark/
            └── packages/
                └── <benchmark-name>/
                    └── package.py

These Benchpark repositories can be used as staging areas for a benchmark’s build and run definitions so that you can test them with the experiment definition on a given system. Once the experiment is working and merged into Benchpark’s develop branch, contributors are encouraged to upstream package.py to Spack and application.py to Ramble. After those pull requests are merged, submit a Benchpark pull request that removes the duplicated definitions.

Spack/Ramble versions in Benchpark

Benchpark depends on the build functionality provided in Spack, the Spack packages provided in Spack-packages, and run functionality provided in Ramble. To allow for testing before pulling in updates in Spack and Ramble, Benchpark clones and uses the versions of Spack and Ramble specified in checkout-versions.yaml.

The user might notice this delay in Spack and Ramble versions in two ways:

The latest functionality of Spack and/or Ramble may not yet be available in Benchpark.
The latest package.py specifications available in Spack and application.py specifications available in Ramble may not be available in Benchpark.

At your own risk, you may go into the cloned Spack, Spack-packages, and Ramble directories and perform a git pull to grab those updates (these are located in your home directory (~) or an alternate location you indicated when bootstrapping Benchpark). This may be useful for package updates (i.e. Spack-packages), but perilous for functionality updates (i.e. Spack or Ramble).

Alternatively, the user may temporarily copy the needed packages into benchpark/repo, and remove them when Benchpark updates to the next version of Spack/Ramble.

What to rerun after edits

I made changes. What should I rerun?
What I changed	Commands to rerun
configs	`ramble --workspace-dir . workspace setup`
benchmark’s package.py	`ramble --workspace-dir . workspace setup`
dependency of package.py	`ramble --workspace-dir . workspace setup`
experiment parameters	delete `workspace/experiments`
wish to rerun experiments	delete `workspace/experiments`

How to move the Benchpark bootstrap location from the home directory

Remove your current bootstrap in home (~) and then use: benchpark configure to set your preferred location. Keep in mind that Spack, Spack-packages, and Ramble will be cloned to this location, so if you are using slow storage this may take awhile.

rm -rf ~/.benchpark
benchpark configure --bootstrap-location /my/path
benchpark bootstrap

benchpark will save this location, and you can optionally set a new bootstrap location for each clone of benchpark, or reuse the same location.