2.3. RAJA Performance Suite
The RAJA Performance Suite is a collection of loop-based computational kernels representing important algorithm patterns found in HPC applications. The RAJA Performance Suite Benchmark uses a subset of kernels in the RAJA Performance Suite.
All benchmark source code and run scripts are included in the RAJAPerf-Benchmark GitHub repo The latest version is:
Version
v2026.10.0, which includesVersion
v2026.07.0of RAJA Performance Suite which in turn includesVersion
v2026.07.1of RAJA
In addition to a local copy of the RAJAPerf-Benchmark repo, the following items are required
to build the benchmark:
CMake to configure a build
A C++20 compliant compiler to build the code
An MPI library installation to link against
2.3.1. Purpose
The RAJA Performance Suite is a companion project to RAJA, which is a library of software abstractions used by C++ application developers to write portable, performant, single-source code. The main purpose of the RAJA Performance Suite is to analyze performance of computational kernels implemented using RAJA and common parallel programming models. Each kernel in the Suite appears in multiple variants that employ programming models, such as OpenMP and CUDA. Variants include implementations that use RAJA and those that don’t; non-RAJA variants are referred to as Base variants.
The RAJA Performance Suite Benchmark is a single node performance benchmark. All applications that use RAJA follow the MPI + X parallel application paradigm. That is, MPI is used for coarse-grained, distributed memory parallelism and X, which is RAJA in this case, supports fine-grained parallelism within each MPI rank. Typically, we run the Benchmark with multiple MPI ranks. When this is done, the same code executes on each rank and synchronization and communication are performed only to collect execution timing information on rank zero for reporting purposes. The Benchmark run scripts launch multiple MPI ranks, one for each GPU on a node for each architecture so that the node is fully subscribed when the Benchmark is run.
2.3.2. Problems
The RAJA Performance Suite Benchmark consists of a subset of RAJA Performance Suite kernels. The Benchmark kernels are partitioned into two Tiers.
Descriptions of Tier 1 kernels, how they are built, run, and assessed for the Benchmark are provided below.
Similar information for Tier 2 kernels can be found in RAJA Performance Suite Tier 2 Kernels.
Note
In the RAJA Performance Suite repository, each kernel is implemented in
multiple files. Source files include <kernel-name>.cpp, which contains code that is
independent of kernel variants, and multiple <kernel-name>-<back-end>.cpp files,
each of which contains variants that exercise the named programming model back-end. A header file
<kernel-name>.hpp contains a reference implementation of the kernel,
which is a C-style sequential version. The reference is similar to the base sequential variant
implemented in the Suite, which is used to generate the kernel reference checksum against which
we compare results of all other variants for correctness validation.
When the benchmark kernels are run following the process encoded in the run scripts, run time data is output in files that are subsequently processed to generate a Figure of Merit (FOM) for each kernel, which is described in Figure of Merit.
2.3.2.1. Tier 1 kernels
The following list describes the Tier 1 kernels and the RAJA features they exercise.
In the RAJA Performance Suite repo, they are located in RAJAPerf/src sub-directories noted below (i.e., RAJAPerf/src/<subdir>/<kernel>):
stream/TRIAD common stream TRIAD kernel used to set an upper bound on realizable memory bandwidth (RAJA::forall API)
apps/DIFFUSION3DPA element-wise action of a 3D finite element volume diffusion operator via partial assembly and sum factorization (nested loops, GPU shared memory, RAJA::launch API)
apps/EDGE3D stiffness matrix assembly for a 3D MHD calculation (single loop with included function call, RAJA::forall API)
apps/ENERGY internal energy calculation from an explicit hydrodynamics algorithm; (multiple single-loop operations in sequence, conditional logic for correctness checks and cutoffs, RAJA::forall API)
apps/FEMSWEEP finite element implementation of linear sweep algorithm used in radiation transport, with a register-heavy LU solver (nested loops, RAJA::launch API)
apps/INTSC_HEXRECT intersection between a 24-sided hexahedron and a rectangular solid, including volume and moment calculations (single loop, RAJA::forall API)
apps/MASS3DEA element assembly of a 3D finite element mass matrix (nested loops, GPU shared memory, RAJA::launch API)
apps/MASS3DPA_ATOMIC action of a 3D finite element mass matrix on elements with shared DOFs via partial assembly and sum factorization (nested loops, GPU shared memory, RAJA::launch API)
apps/MASSVEC3DPA element-wise action of a 3D finite element mass matrix via partial assembly and sum factorization on a block vector (nested loops, GPU shared memory, RAJA::launch API)
apps/NODAL_ACCUMULATION_3D on a 3D structured hexahedral mesh, sum a contribution from each hex vertex (nodal value) to its centroid (zonal value) (single loop, data access via indirection array, 8-way atomic contention, RAJA::forall API)
apps/VOL3D on a 3D structured hexahedral mesh (faces are not necessarily planes), compute volume of each zone (hex) (single loop, data access via indirection array, RAJA::forall API)
2.3.3. Figure of Merit
The Figure of Merit (FOM) for each kernel and variant is determined by the problem size at which the kernel variant saturates resources on a single compute node. We run each kernel and variant over a sequence of problem sizes to determine a saturation point where the computational throughput curve becomes flat, with zero derivative, and beyond which running a larger problem size yields no further increase in compute rate. Then, the FOM consists of 3 numerical values:
the saturation problem size (GB)
the compute rate (GFLOP/sec) achieved at the saturation problem size
the memory bandwidth (GB/sec) achieved at the saturation problem size
In the Benchmark, the problem size refers to the amount of memory allocated for the requested problem to run. The notion of problem size can depend on what one is trying to measure. For our perspective on this topic, please see the Problem Size section of the RAJA Performance Suite User Guide.
Computational throughput may be visualized on a plot where compute rate
(vertical axis) is plotted as a function of problem size (horizontal axis).
Ideally, such a curve is monotonically increasing and transitions to a flat,
horizontal line. Then, the saturation point is the problem size at which the derivative
of the throughput curve becomes zero. In reality, a measured throughput curve may be
non-monotonic or not have a strictly zero derivative for all points beyond some problem
size. Therefore, we apply a simple median based smoothing algorithm to the computed throughput
data points and use simple heuristics to estimate the saturation point based on a smoothed
approximation of the throughput curve. The details of our approach are documented in the Python script
process_data.py that lives in the scripts directory of the
RAJAPerf-Benchmark GitHub repo
for the baseline set we are interested in. We describe the use of the scripts to run the kernels and to
process the output data to generate the FOM in Baseline on El Capitan and
Baseline on NVIDIA H100.
Important
The scripts and their invocations described in Running show how we determine the FOM values. Other methods to generate the FOMs for the kernels are acceptable.
In any case, the saturation point for all Tier 1 kernels should be estimated based on compute rate data.
2.3.4. Source code modifications
Please see Run Rules Synopsis for general guidance on allowed Benchmark modifications. For the RAJA Performance Suite Benchmark, we restrict source code modifications as follows:
Source code changes to the RAJA Performance Suite kernels and to RAJA may be proposed for improved performance.
RAJA may not be removed from RAJA kernel variants in the Suite or replaced with any other execution abstractions.
The non-RAJA (i.e., Base) kernel variants in the Suite are provided to show how each kernel may be implemented directly using the associated programming model without using RAJA. Apart from a few exceptions, the RAJA and non-RAJA variants for each kernel and back-end should perform the same operations.
2.3.5. Building
2.3.5.1. Getting the code
All non-system software needed to compile and run the Benchmark is contained in the
RAJAPerf-Benchmark GitHub repo
repository and its Git submodules. The v2026.10.0 version of that repo
was used to generate the results presented in this document.
To clone the repo:
$ git clone --recursive git@github.com:llnl/RAJAPerf-Benchmark.git
This will make a copy of the repo in a local directory and put you on the main
branch of the repo, which is the default branch. To get the version used to
generate the results presented below, execute the following commands:
$ cd path/to/RAJAPerf-Benchmark
$ git checkout v2026.10.0
$ git submodule update --init --recursive
This will ensure that you have the proper version of the Benchmark code, including the RAJAPerf and RAJA submodules, in your local directory.
2.3.5.2. Configuration and compilation
The RAJA Performance Suite uses a CMake-based system to configure the code for compilation. When building the code, RAJA and the RAJA Performance Suite are built together with the same configuration which is specified at the RAJA Performance Suite level. Generically, the process for specifying a configuration and generating a build space involves creating a build directory and running CMake in it with the desired options:
$ cd path/to/RAJAPerf
$ mkdir my-build
$ cd my-build
$ cmake <cmake args> ..
$ make -j (or make -j <N> to build with a specified number of cores)
For convenience and informational purposes, we maintain configuration scripts
in RAJAPerf/scripts subdirectories for various builds. For example, the
RAJAPerf/scripts/lc-builds directory contains scripts that we use to
generate build configurations for machines in the Livermore Computing (LC)
Center at Lawrence Livermore National Laboratory. These scripts are run from
the top-level RAJAPerf directory. Each script creates a descriptively-named build
space directory and runs CMake to generate a build space in it. Executing a
configuration script with no arguments will produce a message indicating
which arguments are required.
These scripts were used to configure the Benchmark code for compilation to generate the Benchmark results as described in this document.
2.3.5.3. AMD MI300A architecture
The following commands were used to configure and build the code to generate the AMD MI300A results presented in Baseline on El Capitan:
$ cd path/to/RAJAPerf
$ ./scripts/lc-builds/toss4_cray-mpich_amdclang.sh 9.0.1 6.4.3 gfx942
$ cd build_lc_toss4-cray-mpich-9.0.1-amdclang-6.4.3-gfx942
$ make -j
Specifically, we built the code with version 9.0.1 of the Cray MPICH MPI library and the AMD clang compiler with ROCm version 6.4.3 targeting GPU compute architecture gfx942. This was done on an ASC ATS-4 platform (El Capitan architecture) at LLNL.
Note
While newer ROCm compilers are available, version 6.4.3 was used by all LLNL codes using RAJA for production builds at the time we generated the baseline results, which is why we use that version here.
2.3.5.4. NVIDIA H100 architecture
The following commands were used to configure and build the code to generate the NVIDIA H100 results presented in Baseline on NVIDIA H100:
$ cd path/to/RAJAPerf
$ ./scripts/lc-builds/toss4_mvapich2_nvcc_gcc.sh 2.3.7 13.1.1 90 10.3.1
$ cd build_lc_toss4-mvapich2-2.3.7-nvcc-13.1.1-90-gcc-10.3.1
$ make -j
Specifically, we built the code with version 2.3.7 of the MVAPICH2 MPI library, version 13.1.1 of the nvcc CUDA compiler targeting GPU compute architecture sm_90, and version 10.3.1 of the GNU compiler for host code. This was done on a Livermore Computing platform with 4th generation Intel Sapphire Rapids CPUs and NVIDIA H100 GPUs.
Note
These compiler and MPI versions were the system defaults on the the LC platform we used when we generated the baseline results.
2.3.6. Running
When the RAJA Performance Suite code is built using the steps above, the
executable will be generated in the bin subdirectory of the build space.
Important
In Baseline on El Capitan and Baseline on NVIDIA H100, we provide the exact commands we used to run the code and generate the baseline results for the Benchmark.
To learn about other ways in which the RAJA Performance Suite code can be run, if needed, please consult the user documentation in the RAJA Performance Suite GitHub repo.
2.3.7. Validation
When each kernel variant is run, it produces a checksum value based on kernel output, such as a data array whose entries are computed by the kernel. In general, such a checksum value depends on the kernel problem size; thus, we compute the checksum each time a kernel variant runs. Result validation compares the checksum of the kernel variant run and a reference variant of the same kernel. For the Benchmark, the reference variant is the sequential (CPU) variant. Thus, the Benchmark run scripts execute the sequential variants although we we are only interested in the FOM for the Benchmark (CUDA and HIP) variants.
Checksum tolerance is specific to each kernel and is set in the kernel class constructor
based on how sensitive kernel results are to order of operation differences when run in parallel
or some other behavior of the kernel. Whether the checksum for a kernel is within its expected tolerance
is reported as checksum PASSED or FAILED in the checksum output files.
Important
All Tier 1 kernels have the normal checksum tolerance values for the Suite by default, which is 1e-10, except for the FEMSWEEP kernel which has a loose tolerance value of 5e-6. The reason for the loose tolerance is that the implementation of the FEMSWEEP kernel uses an LU factorization without pivoting in the linear solver, unlike the application it came from, and the kernel data is initialized in a way that may cause a high degree of variation in the output.
2.3.8. Memory Usage
For the RAJA Performance Suite Benchmark, we run each kernel over a sequence of problem sizes to generate a throughput curve and, based on that, estimate a saturation point. The memory usage for each entry in the sequence is roughly the same for each kernel. However, there is no significant meaning to take away from this since the memory used by kernels like those in the Suite will be determined by the application context in which they are used.
2.3.9. Baseline on El Capitan
We are interested in single-node throughput with this Benchmark and the FOM
described above. To generate throughput curves and
estimate saturation points, we use a bash shell script to run the code
and a Python script to process the data. These scripts are available in the
RAJAPerf-Benchmark GitHub repo
in the RAJAPerf-Benchmark/RPB-v2026.10.0/scripts directory.
Important
In the following sections, we present detailed results,
including FOM tables and throughput plots for the Tier 1
kernels described above. Data and plot files are included
in the baseline_data directory of this benchmark
repository for reference.
2.3.9.1. AMD MI300A throughput results (Tier 1 kernels)
For the MI300A architecture, we present two sets of throughput results. One is
run in SPX mode where we use 4 MPI ranks on a node, one for each MI300A APU,
thus treating each APU as a single GPU. The other is run in CPX mode where
we run with 24 MPI ranks on a node, six for each MI300A APU, treating each APU as
6 GPUs (i.e., one GPU = 1 XCD).
2.3.9.1.1. SPX mode (Tier 1)
For SPX mode (1 MPI rank per APU on a node), we chose the problem size range such that the smallest problem allocates ~100,000 bytes of memory and the largest problem allocates ~400MB of memory. The largest size is about 1.5 times the MALL (Memory Attached Last-Level cache) size on the MI300A. The MALL is 256MB (256 * 1024 * 1024 = 268,435,456 bytes).
Note that we run a different problem size range for two of the kernels,
FEMSWEEP and MASS3DEA, because they don’t clearly saturate.
For those kernels, the smallest problem allocates ~3.2MB of memory and the
largest problem allocates ~600MB memory, which is more than twice the size of the MALL.
After building the code as described in AMD MI300A architecture, we
ran the Tier 1 kernels in SPX mode using the following commands:
$ cd path/to/RAJAPerf
$ cd build_lc_toss4-cray-mpich-9.0.1-amdclang-6.4.3-gfx942
$ cp ../../RPB-v2026.10.0/scripts/run_tier_mi300a.sh .
$ ./run_tier_mi300a.sh spx tier1
This generates a directory named RPBenchmark_MI300A_tier1-SPX containing
output files for each kernel.
Then, we processed the output to generate FOM and throughput plots by running a Python script we provide:
$ cd path/to/RAJAPerf
$ cd build_lc_toss4-cray-mpich-9.0.1-amdclang-6.4.3-gfx942
$ cp ../../RPB-v2026.10.0/scripts/process_data.py .
$ python3 ./process_data.py --root-dir ./RPBenchmark_MI300A_tier1-SPX --output-dir ./RPBenchmark_MI300A_tier1-SPX/Output --exclude-plot-variant Base_Seq
This generates throughput curve files for Base_HIP and RAJA_HIP
variants of each kernel and summarizes the FOM (described in
Figure of Merit) in a CSV file. We include the files generated by the
process_data.py script in this repo in the directory
./docs/13_rajaperf/baseline_data/RPBenchmark_MI300A_tier1-SPX.
Kernel |
Sat Problem Size |
Sat GFLOP/s |
Sat B/W (GiB per sec.) |
|---|---|---|---|
Stream_TRIAD-Base_HIP-block_256 |
533334.0 |
291.187 |
3254.27 |
Stream_TRIAD-RAJA_HIP-block_256 |
533334.0 |
290.782 |
3249.74 |
Apps_DIFFUSION3DPA-Base_HIP-block_64 |
197262.0 |
3254.31 |
1832.39 |
Apps_DIFFUSION3DPA-RAJA_HIP-block_64 |
197262.0 |
3333.71 |
1877.1 |
Apps_EDGE3D-Base_HIP-block_256 |
343000.0 |
28379.6 |
64.6823 |
Apps_EDGE3D-RAJA_HIP-block_256 |
343000.0 |
28455.1 |
64.8545 |
Apps_ENERGY-Base_HIP-block_256 |
1250000.0 |
839.545 |
3070.68 |
Apps_ENERGY-RAJA_HIP-block_256 |
1250000.0 |
840.9 |
3075.64 |
Apps_FEMSWEEP-Base_HIP-block_64 |
839808.0 |
1148.1 |
221.08 |
Apps_FEMSWEEP-Base_HIP-unroll_64 |
839808.0 |
1117.79 |
215.243 |
Apps_FEMSWEEP-RAJA_HIP-block_64 |
1152000.0 |
948.786 |
183.294 |
Apps_FEMSWEEP-RAJA_HIP-unroll_64 |
1152000.0 |
945.893 |
182.735 |
Apps_INTSC_HEXRECT-Base_HIP-block_64 |
551368.0 |
854.299 |
10.4371 |
Apps_INTSC_HEXRECT-RAJA_HIP-block_64 |
287496.0 |
840.507 |
10.2808 |
Apps_MASS3DEA-Base_HIP-compile_time_block_stride_loop_64 |
48521216.0 |
22838.8 |
200.406 |
Apps_MASS3DEA-RAJA_HIP-cached_block_stride_loop_64 |
48521216.0 |
27107.8 |
237.866 |
Apps_MASS3DEA-RAJA_HIP-compile_time_block_stride_loop_64 |
18198528.0 |
5003.62 |
43.9058 |
Apps_MASS3DPA_ATOMIC-Base_HIP-block_64 |
132651.0 |
2307.93 |
1072.71 |
Apps_MASS3DPA_ATOMIC-RAJA_HIP-block_64 |
132651.0 |
2315.99 |
1076.46 |
Apps_MASSVEC3DPA-Base_HIP-direct_64 |
1146960.0 |
3108.81 |
948.321 |
Apps_MASSVEC3DPA-Base_HIP-runtime_block_stride_loop_64 |
1146960.0 |
2394.83 |
730.527 |
Apps_MASSVEC3DPA-RAJA_HIP-cached_block_stride_loop_64 |
1146960.0 |
2481.33 |
756.914 |
Apps_MASSVEC3DPA-RAJA_HIP-direct_64 |
1146960.0 |
3105.23 |
947.228 |
Apps_MASSVEC3DPA-RAJA_HIP-runtime_block_stride_loop_64 |
573480.0 |
1530.99 |
467.018 |
Apps_NODAL_ACCUMULATION_3D-Base_HIP-block_256 |
1000000.0 |
205.504 |
690.81 |
Apps_NODAL_ACCUMULATION_3D-RAJA_HIP-block_256 |
1000000.0 |
205.757 |
691.658 |
Apps_VOL3D-Base_HIP-block_256 |
343000.0 |
2969.24 |
1242.01 |
Apps_VOL3D-RAJA_HIP-block_256 |
343000.0 |
3046.28 |
1274.24 |
2.3.9.1.2. CPX mode (Tier 1)
For CPX mode (run with 6 MPI ranks per APU on a node), we choose the smallest problem to allocate ~50,000 bytes of memory and the largest problem to allocate ~75MB of memory, which is slightly less than 1/3 the MALL size.
Similar to SPX mode, we ran a different problem size range for the FEMSWEEP
and MASS3DEA kernels because they don’t clearly saturate. For those, we chose
the smallest problem to allocate ~1.6MB of memory and the largest problem to
allocate ~200MB memory, which is a little less than the MALL size.
Similar to the SPX mode description above, we ran the Tier 1 kernels in
CPX mode using the following commands:
$ cd path/to/RAJAPerf
$ cd build_lc_toss4-cray-mpich-9.0.1-amdclang-6.4.3-gfx942
$ cp ../../RPB-v2026.10.0/scripts/run_tier_mi300a.sh .
$ ./run_tier_mi300a.sh cpx tier1
This generates a directory named RPBenchmark_MI300A_tier1-CPX containing
output files for each kernel.
Then, we processed the data to generate FOM and throughput plots by running a Python script we provide:
$ cd path/to/RAJAPerf
$ cp ../../RPB-v2026.10.0/scripts/process_data.py .
$ python3 ./process_data.py --root-dir ./RPBenchmark_MI300A_tier1-CPX --output-dir ./RPBenchmark_MI300A_tier1-CPX/Output --exclude-plot-variant Base_Seq
This generates throughput curve files for Base_HIP and RAJA_HIP
variants of each kernel and summarizes the FOM (described in
Figure of Merit) in a CSV file. We include these files in this repo in the
directory ./docs/13_rajaperf/baseline_data/RPBenchmark_MI300A_tier1-CPX.
Kernel |
Sat Problem Size |
Sat GFLOP/s |
Sat B/W (GiB per sec.) |
|---|---|---|---|
Stream_TRIAD-Base_HIP-block_256 |
66667.0 |
70.883 |
792.179 |
Stream_TRIAD-RAJA_HIP-block_256 |
33334.0 |
47.0912 |
526.285 |
Apps_DIFFUSION3DPA-Base_HIP-block_64 |
24678.0 |
612.534 |
344.914 |
Apps_DIFFUSION3DPA-RAJA_HIP-block_64 |
24678.0 |
615.458 |
346.56 |
Apps_EDGE3D-Base_HIP-block_256 |
35937.0 |
4501.14 |
10.3756 |
Apps_EDGE3D-RAJA_HIP-block_256 |
35937.0 |
4532.44 |
10.4478 |
Apps_ENERGY-Base_HIP-block_256 |
53334.0 |
219.701 |
803.568 |
Apps_ENERGY-RAJA_HIP-block_256 |
53334.0 |
235.409 |
861.022 |
Apps_FEMSWEEP-Base_HIP-block_64 |
589824.0 |
259.161 |
49.701 |
Apps_FEMSWEEP-Base_HIP-unroll_64 |
589824.0 |
234.711 |
45.0119 |
Apps_FEMSWEEP-RAJA_HIP-block_64 |
589824.0 |
256.915 |
49.2702 |
Apps_FEMSWEEP-RAJA_HIP-unroll_64 |
589824.0 |
262.535 |
50.348 |
Apps_INTSC_HEXRECT-Base_HIP-block_64 |
74088.0 |
141.338 |
1.73495 |
Apps_INTSC_HEXRECT-RAJA_HIP-block_64 |
74088.0 |
141.972 |
1.74273 |
Apps_MASS3DEA-Base_HIP-compile_time_block_stride_loop_64 |
6213632.0 |
4902.68 |
43.0202 |
Apps_MASS3DEA-RAJA_HIP-cached_block_stride_loop_64 |
9101312.0 |
4918.8 |
43.1616 |
Apps_MASS3DEA-RAJA_HIP-compile_time_block_stride_loop_64 |
3108864.0 |
876.714 |
7.69305 |
Apps_MASS3DPA_ATOMIC-Base_HIP-block_64 |
19683.0 |
408.771 |
192.298 |
Apps_MASS3DPA_ATOMIC-RAJA_HIP-block_64 |
19683.0 |
395.107 |
185.871 |
Apps_MASSVEC3DPA-Base_HIP-direct_64 |
143370.0 |
570.096 |
173.908 |
Apps_MASSVEC3DPA-Base_HIP-runtime_block_stride_loop_64 |
143370.0 |
427.57 |
130.431 |
Apps_MASSVEC3DPA-RAJA_HIP-cached_block_stride_loop_64 |
143370.0 |
459.754 |
140.248 |
Apps_MASSVEC3DPA-RAJA_HIP-direct_64 |
143370.0 |
569.174 |
173.627 |
Apps_MASSVEC3DPA-RAJA_HIP-runtime_block_stride_loop_64 |
71685.0 |
305.513 |
93.1999 |
Apps_NODAL_ACCUMULATION_3D-Base_HIP-block_256 |
117649.0 |
35.4271 |
120.977 |
Apps_NODAL_ACCUMULATION_3D-RAJA_HIP-block_256 |
117649.0 |
35.6658 |
121.792 |
Apps_VOL3D-Base_HIP-block_256 |
79507.0 |
626.699 |
263.825 |
Apps_VOL3D-RAJA_HIP-block_256 |
35937.0 |
614.464 |
259.95 |
2.3.9.1.3. AMD MI300A throughput plots (Tier 1)
This section contains throughput plots for each Tier 1 kernel on the MI300A architecture in SPX mode (left column) and CPX mode (right column). Each plot has multiple curves with GFLOP/sec (compute rate) plotted as a function of problem size (allocated bytes). Each plot includes:
Throughput curves for Base and RAJA variant(s) of the kernels. Solid line segments connect the dots, where the dots are actual GFLOP rates output by the run.
Smoothed versions of the throughput curves, plotted as dashed lines, which are computed using a median based smoothing algorithm.
Stars indicating approximate saturation points based on the smoothed curves and computed using simple heuristics. The legend contains the (x, y) values for the saturation points.
Most plots contain two variants, with the Base non-RAJA variant in blue and RAJA variant in orange. In each case, the throughput curves and saturation points are close, which indicates that the RAJA variants perform as well as the non-RAJA variants that are written directly in HIP with no RAJA abstractions. Some kernel plots contain additional curves that show results of additional tunings. These curves are included to show how kernel execution choices, such as RAJA execution policies, can have a significant impact on run time performance.
Tier 1 Kernels: MI300A Node Throughput (SPX Mode) |
Tier 1 Kernels: MI300A Node Throughput (CPX Mode) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2.3.10. Baseline on NVIDIA H100
For NVIDIA GPUs, we are interested in the same measure of single-node throughput and FOM
as described above for AMD MI300A. To generate throughput curves and
estimate saturation points, we use a bash shell script to run the code
and a Python script to process the data. These scripts are available in the
RAJAPerf-Benchmark GitHub repo
in the RAJAPerf-Benchmark/RPB-v2026.10.0/scripts directory.
2.3.10.1. NVIDIA H100 throughput results (Tier 1 kernels)
For the NVIDIA H100 architecture, we present one set of throughput results, where we ran with 4 MPI ranks on a node – one for each H100 GPU.
Similar to our methodology for the MI300A architecture, we ran each kernel and variant over a range of problem sizes to expose the saturation point. Specifically, we chose the smallest problem to allocate ~50,000 bytes of memory and the largest problem to allocate ~150MB of memory, which is about 3 times the L2 cache size on the H100 GPU. The L2 cache is 50MB (50 * 1024 * 1024 = 52,428,800 bytes).
As for MI300A, we ran a different problem size range for the FEMSWEEP and MASS3DEA
kernels because they don’t clearly saturate. For those, we chose the smallest
problem to allocate ~1.6MB of memory and the largest problem to allocate ~300MB of memory,
which is about 6 times the L2 cache size.
After building the code as described in NVIDIA H100 architecture, we
ran the Tier 1 kernels using the following commands:
$ cd path/to/RAJAPerf
$ cd build_lc_toss4-mvapich2-2.3.7-nvcc-13.1.1-90-gcc-10.3.1
$ cp ../../RPB-v2026.10.0/scripts/run_tier_h100.sh .
$ ./run_tier_h100.sh tier1
This generates a directory named RPBenchmark_H100_tier1 containing
results files for each kernel run.
Then, we processed the data to generate FOM and throughput plots by running a Python script we provide:
$ cd path/to/RAJAPerf
$ cd build_lc_toss4-mvapich2-2.3.7-nvcc-13.1.1-90-gcc-10.3.1
$ cp ../../RPB-v2026.10.0/scripts/process_data.py .
$ python3 ./process_data.py --root-dir ./RPBenchmark_H100_tier1 --output-dir ./RPBenchmark_H100_tier1/Output --exclude-plot-variant Base_Seq
This generates throughput curve files for Base_CUDA and RAJA_CUDA
variants of each kernel and summarizes the FOM (described in
Figure of Merit) in a CSV file. We include
the files generated by the process_data.py script in this repo in the
directory ./docs/13_rajaperf/baseline_data/RPBenchmark_H100_tier1.
Kernel |
Sat Problem Size |
Sat GFLOP/s |
Sat B/W (GiB per sec.) |
|---|---|---|---|
Stream_TRIAD-Base_CUDA-block_256 |
533334.0 |
299.852 |
3351.11 |
Stream_TRIAD-RAJA_CUDA-block_256 |
533334.0 |
299.664 |
3349.0 |
Apps_DIFFUSION3DPA-Base_CUDA-block_64 |
197262.0 |
2913.9 |
1640.72 |
Apps_DIFFUSION3DPA-RAJA_CUDA-block_64 |
197262.0 |
2945.62 |
1658.58 |
Apps_EDGE3D-Base_CUDA-block_256 |
166375.0 |
1524.99 |
3.48548 |
Apps_EDGE3D-RAJA_CUDA-block_256 |
166375.0 |
1522.27 |
3.47925 |
Apps_ENERGY-Base_CUDA-block_256 |
625000.0 |
806.613 |
2950.23 |
Apps_ENERGY-RAJA_CUDA-block_256 |
426667.0 |
864.38 |
3161.52 |
Apps_FEMSWEEP-Base_CUDA-block_64 |
1152000.0 |
263.067 |
50.8212 |
Apps_FEMSWEEP-Base_CUDA-unroll_64 |
1152000.0 |
716.81 |
138.479 |
Apps_FEMSWEEP-RAJA_CUDA-block_64 |
1152000.0 |
252.307 |
48.7426 |
Apps_FEMSWEEP-RAJA_CUDA-unroll_64 |
1533312.0 |
592.42 |
114.751 |
Apps_FEMSWEEP-RAJA_CUDA-unroll_64_min_blocks_per_sm_5 |
1152000.0 |
610.165 |
117.876 |
Apps_INTSC_HEXRECT-Base_CUDA-block_64 |
551368.0 |
1554.05 |
18.986 |
Apps_INTSC_HEXRECT-RAJA_CUDA-block_64 |
551368.0 |
1297.06 |
15.8463 |
Apps_MASS3DEA-Base_CUDA-compile_time_block_stride_loop_64 |
18198528.0 |
24910.1 |
218.581 |
Apps_MASS3DEA-RAJA_CUDA-cached_block_stride_loop_64 |
18198528.0 |
22976.6 |
201.615 |
Apps_MASS3DEA-RAJA_CUDA-compile_time_block_stride_loop_64 |
18198528.0 |
25593.7 |
224.58 |
Apps_MASS3DPA_ATOMIC-Base_CUDA-block_64 |
250047.0 |
1718.17 |
796.607 |
Apps_MASS3DPA_ATOMIC-RAJA_CUDA-block_64 |
250047.0 |
1719.29 |
797.128 |
Apps_MASSVEC3DPA-Base_CUDA-direct_64 |
1146960.0 |
2399.45 |
731.937 |
Apps_MASSVEC3DPA-Base_CUDA-runtime_block_stride_loop_64 |
1146960.0 |
1647.15 |
502.451 |
Apps_MASSVEC3DPA-RAJA_CUDA-cached_block_stride_loop_64 |
1146960.0 |
1613.76 |
492.266 |
Apps_MASSVEC3DPA-RAJA_CUDA-direct_64 |
1146960.0 |
2387.93 |
728.42 |
Apps_MASSVEC3DPA-RAJA_CUDA-runtime_block_stride_loop_64 |
1146960.0 |
1617.17 |
493.307 |
Apps_NODAL_ACCUMULATION_3D-Base_CUDA-block_256 |
493039.0 |
246.065 |
830.478 |
Apps_NODAL_ACCUMULATION_3D-RAJA_CUDA-block_256 |
493039.0 |
245.91 |
829.957 |
Apps_VOL3D-Base_CUDA-block_256 |
704969.0 |
4885.28 |
2038.97 |
Apps_VOL3D-RAJA_CUDA-block_256 |
343000.0 |
4679.07 |
1957.22 |
2.3.10.1.1. NVIDIA H100 throughput plots (Tier 1)
This section contains throughput plots for each Tier 1 kernel run on the H100 architecture. The plots contain information similar to what is described in AMD MI300A throughput plots (Tier 1).
Tier 1 Kernels H100 Node Throughput |
|
|
|
|
|
|
|
|
|
|
|
2.3.11. Performance Projection
For the RAJA Performance Suite Benchmark, we want to know the projected or measured FOM, as defined in Figure of Merit. Specifically, for each kernel executed on a single node in a manner that node resources are subscribed fully (as described above), the FOM consists of three numerical values:
the saturation problem size (GB)
the compute rate (GFLOP/sec) achieved at the saturation problem size
the memory bandwidth (GB/sec) achieved at the saturation problem size
The scripts and their invocations described in Running show how we determined the FOM values. Other methods to estimate/generate the FOM values are acceptable as long as they are quantitative and explained.
2.3.12. References
The GitHub repositories are the primary references for RAJA and the RAJA Performance Suite:
Other helpful references include:
Olga Pearce, Jason Burmark, Rich Hornung, Befikir Bogale, Ian Lumsden, Michael McKinsey, Dewi Yokelson, David Boehme, Stephanie Brink, Michela Taufer, Tom Scogland, “RAJA Performance Suite: Performance Portability Analysis with Caliper and Thicket”, in 2024 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC) at the International Conference on High Performance Computing, Network, Storage, and Analysis (SC-W 2024). https://dl.acm.org/doi/pdf/10.1109/SCW63240.2024.00162
Beckingsale, J. Burmark, R. Hornung, H. Jones, W. Killian, A. J. Kunen, O. Pearce, P. Robinson, B. S. Ryujin, T. R. W. Scogland, “RAJA: Portable Performance for Large-Scale Scientific Applications”, 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC). https://conferences.computer.org/sc19w/2019/#!/toc/14
Arturo Vargas, Thomas M. Stitt, Kenneth Weiss, Vladimir Z. Tomov, Jean-Sylvain Camier, Tzanio Kolev, Robert N. Rieben, “Matrix-free Approaches for GPU Acceleration of a High-order Finite Element Hydrodynamic Application using MFEM, Umpire, and RAJA”, International Journal of High Performance Computing Applications. 36(4):492-509 (2022). https://journals.sagepub.com/doi/10.1177/10943420221100262