2.3. RAJA Performance Suite
The RAJA Performance Suite is a collection of loop-based computational kernels that represent important algorithm patterns found in HPC applications. It is a companion project to RAJA, which is a library of software abstractions used by C++ application developers to write portable, performant, single-source code. Each kernel in the Suite appears in multiple variants that employ common parallel programming models, such as OpenMP and CUDA. Variants include implementations that use RAJA and those that don’t; non-RAJA variants are referred to as Base variants.
The main purpose of the RAJA Performance Suite is to study performance of computational kernels implemented using RAJA and common parallel programming models. Comparison of RAJA and non-RAJA variants helps to improve RAJA implementations and to understand impacts that C++ abstractions have on compilers’ abilities to optimize. The Suite is an important collaboration tool used by the RAJA team to work with vendors and other collaborators to resolve performance issues observed in production applications that use RAJA.
Important things to know about the RAJA Performance Suite Benchmark are:
The RAJA Performance Suite Benchmark exercises a small subset of kernels in the RAJA Performance Suite. The subset is described in the Problems section.
All code needed to generate the RAJA Performance Suite Benchmark results described in this document is included in the RAJAPerf-Benchmark GitHub repo. The discussion herein refers to the contents of that repo. Instructions for building the Benchmark code and running it to generate the figures of merit (FOMs) can be found in the Building and Running sections.
More information about RAJA and the RAJA Performance Suite is available in their documentation linked to their respective GitHub repos:
Important
The latest version of the RAJA Performance Suite Benchmark uses:
Version
v2026.07.0of the RAJA Performance Suite, which in turn usesVersion
v2026.07.1of RAJAAll code needed for the latest version of the Benchmark is available in the version
v2026.10.0version of the RAJAPerf-Benchmark GitHub repoDetails are provided in the Building section of this documentation, along with build and run instructions including compilers and versions.
2.3.1. Purpose
The purpose of the RAJA Performance Suite Benchmark is for vendors and other collaborators to generate the FOMs, described in Figure of Merit, for a subset of the RAJA Performance Suite Benchmark kernel subset. The Benchmark kernels have been identified by RAJA developers as representing important computations in production applications using RAJA at Lawrence Livermore National Laboratory.
2.3.2. Characteristics
The RAJA Performance Suite Benchmark is a single node performance benchmark. All applications that use RAJA follow the MPI + X parallel application paradigm. That is, MPI is used for coarse-grained, distributed memory parallelism and X, which is RAJA in this case, supports fine-grained parallelism within each MPI rank. Typically, we run the Benchmark with multiple MPI ranks. When this is done, the same code executes on each rank and synchronization and communication are only done to collect and send execution timing information to rank zero for reporting purposes. The included run scripts launch multiple MPI ranks, one for each GPU on a node for each architecture so that the node is fully subscribed when the Benchmark is run.
While the RAJA Performance Suite can be run in myriad ways via command-line options and arguments, the RAJA Performance Suite Benchmark is run in a specific fashion to generate the results reported below. The RAJAPerf-Benchmark GitHub repo contains everything needed to build and run the RAJA Performance Suite Benchmark to generate the FOM. This includes the RAJA Performance Suite and RAJA codes in Git submodules and scripts to build, run, and generate throughput plots and FOM data. Versions of the RAJA Performance Suite and RAJA dependencies are pinned to each version of the RAJA Performance Suite Benchmark repo. Thus, it is important to ensure that the desired Benchmark version is checked out and that the submodules are consistent with that. Building the RAJA Performance Suite Benchmark code requires:
CMake to configure a build
A C++20 compliant compiler to build the code
An MPI library installation to link the executable against
Instructions for getting the RAJA Performance Suite Benchmark code, building it, and running it are described in the Building and Running sections.
2.3.2.1. Problems
The RAJA Performance Suite Benchmark consists of a subset of RAJA Performance Suite kernels that represent key computational patterns found in LLNL HPC applications. The Benchmark kernels are partitioned into two tier levels described below. A basic description of each kernel is provided along with RAJA features it exercises summarized in parentheses.
Note
In the RAJA Performance Suite repository, each kernel is implemented in
multiple files. Source files include <kernel-name>.cpp (common code
independent of kernel variant) and <kernel-name>-<back-end>.cpp
(variants that exercise a programming model back-end). The header file
<kernel-name>.hpp contains a reference implementation of the kernel,
which is a C-style sequential version of the kernel. This simple non-parallel
version is used to generate the reference checksum against which to compare
parallel variants for correctness.
The RAJA Performance Suite Benchmark kernels in each tier level are described below.
2.3.2.1.1. Tier 1 kernels
Tier 1 kernels are most important. They are located in the
RAJAPerf/src/apps sub-directory:
DIFFUSION3DPA element-wise action of a 3D finite element volume diffusion operator via partial assembly and sum factorization (nested loops, GPU shared memory, RAJA::launch API)
EDGE3D stiffness matrix assembly for a 3D MHD calculation (single loop with included function call, RAJA::forall API)
ENERGY internal energy calculation from an explicit hydrodynamics algorithm; (multiple single-loop operations in sequence, conditional logic for correctness checks and cutoffs, RAJA::forall API)
FEMSWEEP finite element implementation of linear sweep algorithm used in radiation transport, with a register-heavy LU solver (nested loops, RAJA::launch API)
INTSC_HEXRECT intersection between a 24-sided hexahedron and a rectangular solid, including volume and moment calculations (single loop, RAJA::forall API)
MASS3DEA element assembly of a 3D finite element mass matrix (nested loops, GPU shared memory, RAJA::launch API)
MASS3DPA_ATOMIC action of a 3D finite element mass matrix on elements with shared DOFs via partial assembly and sum factorization (nested loops, GPU shared memory, RAJA::launch API)
MASSVEC3DPA element-wise action of a 3D finite element mass matrix via partial assembly and sum factorization on a block vector (nested loops, GPU shared memory, RAJA::launch API)
NODAL_ACCUMULATION_3D on a 3D structured hexahedral mesh, sum a contribution from each hex vertex (nodal value) to its centroid (zonal value) (single loop, data access via indirection array, 8-way atomic contention, RAJA::forall API)
VOL3D on a 3D structured hexahedral mesh (faces are not necessarily planes), compute volume of each zone (hex) (single loop, data access via indirection array, RAJA::forall API)
2.3.2.1.2. Tier 2 kernels
Tier 2 kernels are also important, but less so than the Tier 1
kernels listed above. Tier 2 kernels are located in RAJAPerf/src sub-directories
noted below (i.e., RAJAPerf/src/<subdir>/<kernel>):
apps/CONVECTION3DPA element-wise action of a 3D finite element volume convection operator via partial assembly and sum factorization (nested loops, GPU shared memory, RAJA::launch API)
apps/DEL_DOT_VEC_2D divergence of a vector field at a set of points on a mesh (single loop, data access via indirection array, RAJA::forall API)
apps/INTSC_HEXHEX intersection between two 24-sided hexahedra, including volume and moment calculations (multiple single-loop operations in sequence, RAJA::forall API)
apps/LTIMES one step of the source-iteration technique for solving the steady-state linear Boltzmann equation, multi-dimensional matrix product (nested loops, RAJA::kernel API)
apps/MASS3DPA element-wise action of a 3D finite element mass matrix via partial assembly and sum factorization (nested loops, GPU shared memory, RAJA::launch API)
apps/MATVEC_3D_STENCIL matrix-vector product based on a 3D mesh stencil (single loop, data access via indirection array, RAJA::forall API)
basic/MULTI_REDUCE multiple reductions in a kernel, where number of reductions is set at run time (single loop, irregular atomic contention, RAJA::forall API)
basic/REDUCE_STRUCT multiple reductions in a kernel, where number of reductions (6) is known at compile time (single loop, multiple reductions, RAJA::forall API)
basic/INDEXLIST_3LOOP construction of set of indices used in other kernel executions (single loops, vendor scan implementations, RAJA::forall API)
comm/HALO_PACKING_FUSED packing and unpacking MPI message buffers for point-to-point distributed memory halo data exchange for mesh-based codes (overhead of launching many small kernels, GPU variants use RAJA::Workgroup concepts to execute multiple kernels with one launch)
2.3.2.2. Figure of Merit
Each kernel variant has a Figure of Merit (FOM) that is determined by the problem size at which the kernel variant saturates resources on a single compute node. The saturation point is the problem size at which a computational throughput curve becomes flat, with zero derivative, and beyond which running a larger problem size yields no further increase in compute rate. Specifically, the FOM consists of 3 numerical values:
the saturation problem size (GB)
the compute rate (GFLOP/sec) at the saturation problem size
the memory bandwidth (GB/sec) at the saturation problem size
Important
The scripts and their invocations described in Running show how we determine the FOM values. Others are free to generate the values by other means if they prefer.
Computational throughput may be visualized on a plot where compute rate
(vertical axis) is plotted as a function of problem size (horizontal axis).
Ideally, such a curve is monotonically increasing and transitions to a flat,
horizontal line. Then, the saturation point is the problem size at which the derivative
of the throughput curve becomes zero. In reality, a measured throughput curve may be
non-monotonic or not have a strictly zero derivative for all points beyond some problem
size. Therefore, we apply a simple median based smoothing algorithm to the throughput
curve data and use simple heuristics to estimate the saturation point based on the smoothed
throughput curve. The details of our approach are documented in the Python script
process_data.py that lives in the scripts directory of the
RAJAPerf-Benchmark GitHub repo
for the baseline set we are interested in. We describe the use of the script
in Benchmark Results.
Note
In the Benchmark, problem size is computed for each kernel based on the memory allocated for the requested problem size to run. The notion of problem size is subjective and depends on what one is trying to measure. For our perspective on this topic, please see the Problem Size section of the RAJA Performance Suite User Guide.
When the Suite is run, problem size, compute rate, memory bandwidth, result checksums, etc. are reported in various files in an output directory. We provide a Python script to traverse the contents of an output directory and generate summary files, throughput plots, and FOM information. Script usage is described in Running.
Important
For RAJA Performance Suite Benchmark execution on any multicore architecture,, MPI must be used to run to ensure that all resources on a compute node are being exercised so as to avoid misrepresentation of kernel and node performance. The scripts provided to run the Benchmark illustrate how this is done on the architectures for which we have provided baseline results. Specific commands used to invoke the scripts are shown in Running.
2.3.3. Source code modifications
Please see Run Rules Synopsis for general guidance on allowed Benchmark modifications. For the RAJA Performance Suite Benchmark, we restrict source code modifications as follows:
Source code changes to the RAJA Performance Suite kernels and to RAJA may be proposed for improved performance.
RAJA may not be removed from RAJA kernel variants in the Suite or replaced with any other execution abstractions.
The non-RAJA kernel variants in the Suite are provided to show how each kernel may be implemented directly using each programming model without using RAJA. Apart from a few instances, the RAJA and non-RAJA variants for each kernel and back-end should perform the same operations.
2.3.4. Building
2.3.4.1. Getting the code
All non-system software needed to compile and run the Benchmark is contained in the
RAJAPerf-Benchmark GitHub repo
repository and its Git submodules. The v2026.10.0 version of that repo
was used to generate the baseline data described in the Benchmark Results section.
To clone the repo:
$ git clone --recursive git@github.com:llnl/RAJAPerf-Benchmark.git
This will clone the repo into a local directory and put you on the main
branch of the Benchmark repo, which is the default branch. To get a local copy
of the version used to generate the baselines in this document, execute the following
commands:
$ cd path/to/RAJAPerf-Benchmark
$ git checkout v2026.10.0
$ git submodule update --init --recursive
This will assure that you have the proper version of the Benchmark code, including the RAJAPerf and RAJA submodules, in your local directory.
2.3.4.2. Configuration and compilation
The RAJA Performance Suite uses a CMake-based system to configure the code for compilation. When building the code, RAJA and the RAJA Performance Suite are built together with the same configuration which is specified at the RAJA Performance Suite level. Generically, the process for specifying a configuration and generating a build space involves creating a build directory and running CMake in it with the proper options. For example:
$ cd path/to/RAJAPerf
$ mkdir my-build
$ cd my-build
$ cmake <cmake args> ..
$ make -j (or make -j <N> to build with a specified number of cores)
For convenience and informational purposes, we maintain configuration scripts
in RAJAPerf/scripts subdirectories for various builds. For example, the
RAJAPerf/scripts/lc-builds directory contains scripts that we use to
generate build configurations for machines in the Livermore Computing (LC)
Center at Lawrence Livermore National Laboratory. These scripts are run from
the top-level RAJAPerf directory. Each script creates a descriptively-named build
space directory and runs CMake to generate a build space appropriate for the
platform and compiler(s) indicated by the script name and arguments passed to
it. Executing a script with no arguments will print a message indicating
which arguments are required.
These scripts were used to configure the Benchmark code for compilation to generate the Benchmark baseline results described below.
2.3.4.3. MI300A architecture
To configure and build the code to generate baseline data on a system with AMD MI300A processors (the ATS-4 (El Capitan) architecture) discussed in Benchmark Results, we ran the following commands:
$ cd path/to/RAJAPerf
$ ./scripts/lc-builds/toss4_cray-mpich_amdclang.sh 9.0.1 6.4.3 gfx942
$ cd build_lc_toss4-cray-mpich-9.0.1-amdclang-6.4.3-gfx942
$ make -j
That is, we configured and compiled the code using version 9.0.1 of the Cray MPICH MPI library and the AMD clang compiler with ROCm version 6.4.3 targeting GPU compute architecture gfx942.
Note
While newer ROCm compilers are available, version 6.4.3 was used by all LLNL codes using RAJA for production builds when we generated the baseline results. Thus, we use that version of the compiler here.
2.3.4.4. H100 architecture
To configure and build the code to generate baseline data on a system with NVIDIA H100 processors discussed in Benchmark Results, we ran the following commands:
$ cd path/to/RAJAPerf
$ ./scripts/lc-builds/toss4_mvapich2_nvcc_gcc.sh 2.3.7 13.1.1 90 10.3.1
$ cd build_lc_toss4-mvapich2-2.3.7-nvcc-13.1.1-90-gcc-10.3.1
$ make -j
That is, we configured and compiled the code using version 2.3.7 of the MVAPICH2 MPI library, version 13.1.1 of the nvcc compiler for CUDA targeting GPU compute architecture sm_90, and version 10.3.1 of the GNU compiler for compiling host code.
Note
The compiler and MPI versions we used were the system defaults on the NVIDIA GPU based machine we had access to at LLNL when we generated the baseline results.
2.3.5. Running
When the RAJA Performance Suite code is built using the procedure above, the executable
will be generated in the bin subdirectory of the build space.
To see full details about all the code execution options, use the help option:
$ cd path/to/RAJAPerf
$ cd my-build
$ ls bin
raja-perf.exe
$ ./bin/raja-perf.exe --help (or -h)
This will print all available command-line arguments along with options and defaults. All arguments are optional. If none are specified, the suite will run all Suite kernels in their default configurations for the variants that are available based on the way the code was compiled.
Important
In Benchmark Results, we provide the exact commands we used to run the code and generate the baseline results for the Benchmark.
2.3.6. Validation
Each kernel variant that is run produces a checksum value based on kernel output, such as a data array whose entries are computed by the kernel. Since a checksum value generally depends on the kernel problem size, the checksum is computed at run time after kernel execution is complete. Results validation compares the checksum of the kernel variant run and a reference variant of the same kernel. For the Benchmark, the reference variant is the sequential (CPU) variant of each kernel. Thus, the Benchmark run scripts execute the baseline sequential variant in addition to the Benchmark variants we are interested in to validate kernel results.
Each kernel is annotated in the source code as to whether variant checksums
are expected to match the reference checksum exactly, or to be within some tolerance
due to order of operation or other differences when run in parallel. Whether the
checksum for a kernel is within its expected tolerance is reported as checksum
PASSED or FAILED in the checksum output files.
2.3.7. Benchmark Results
As stated earlier, we are interested in single-node throughput
with this Benchmark. To generate throughput curves and estimate
saturation points, we use a bash shell script to run the code on each
platform and a Python script to process the data to construct throughput
plots, estimate saturation points, and make CSV files for tables of results.
These scripts are available in the
RAJAPerf-Benchmark GitHub repo
in the RAJAPerf-Benchmark/RPB-v2026.10.0/scripts directory.
Important
In the following sections, we present detailed results, including FOM tables and throughput plots for the Tier 1 kernels described above. For completeness, we also include a brief summary of results for Tier 2 kernels in less detail. Data files containing results for all Benchmark kernels are included in this LLNL Benchmarks repository for reference.
2.3.7.1. AMD MI300A throughput results (Tier 1 kernels)
For the MI300A architecture, we present two sets of throughput results. One is
run in SPX mode where we use 4 MPI ranks on a node, one for each MI300A APU,
thus treating each APU as a single GPU. The other is run in CPX mode where
we run with 24 MPI ranks on a node, six for each MI300A APU, treating each APU as
6 GPUs (i.e., one GPU = 1 XCD). For each mode, we run each kernel over a sequence
of problem sizes so that the saturation point is evident on its associated throughput curve.
2.3.7.1.1. SPX mode (Tier 1)
For SPX mode (1 MPI rank per APU on a node), we choose the smallest problem to use ~100,000 bytes of allocated memory and the largest problem to use ~400MB of allocated memory, which is about 1.5 times the MALL (Memory Attached Last-Level cache) size on the MI300A. The MALL is 256MB (256 * 1024 * 1024 = 268,435,456 bytes).
Note that for two of the kernels, FEMSWEEP and MASS3DEA, we ran a
different problem size range because these kernels don’t clearly saturate.
For them, we chose the smallest problem to use ~3.2MB of allocated
memory and the largest problem to use ~600MB memory, which is over twice as
big as the MALL.
After building the code as described in MI300A architecture, we
ran the Tier 1 kernels in SPX mode as follows:
$ cd path/to/RAJAPerf
$ cd build_lc_toss4-cray-mpich-9.0.1-amdclang-6.4.3-gfx942
$ cp ../../RPB-v2026.10.0/scripts/run_tier_mi300a.sh .
$ ./run_tier_mi300a.sh spx tier1
This generates a directory named RPBenchmark_MI300A_tier1-SPX containing
output files for each kernel run over its range of problem sizes.
Then, we processed the data to generate more concise baseline results by running a Python script we provide:
$ cd path/to/RAJAPerf
$ cd build_lc_toss4-cray-mpich-9.0.1-amdclang-6.4.3-gfx942
$ cp ../../RPB-v2026.10.0/scripts/process_data.py .
$ python3 ./process_data.py --root-dir ./RPBenchmark_MI300A_tier1-SPX --output-dir ./RPBenchmark_MI300A_tier1-SPX/Output --exclude-plot-variant Base_Seq
This generates throughput curve files for Base_HIP and RAJA_HIP
variants of each kernel and summarizes the FOM (described in
Figure of Merit) in a CSV file. We include the files generated by the
process_data.py script in this repo in the directory
./docs/13_rajaperf/baseline_data/RPBenchmark_MI300A_tier1-SPX.
Kernel |
Sat Problem Size |
Sat GFLOP/s |
Sat B/W (GiB per sec.) |
|---|---|---|---|
Apps_DIFFUSION3DPA-Base_HIP-block_64 |
197262.0 |
3254.31 |
1832.39 |
Apps_DIFFUSION3DPA-Base_Seq-default |
789048.0 |
2.17152 |
1.2227 |
Apps_DIFFUSION3DPA-RAJA_HIP-block_64 |
197262.0 |
3333.71 |
1877.1 |
Apps_EDGE3D-Base_HIP-block_256 |
343000.0 |
28379.6 |
64.6823 |
Apps_EDGE3D-Base_Seq-default |
1331.0 |
8.41524 |
0.0201652 |
Apps_EDGE3D-RAJA_HIP-block_256 |
343000.0 |
28455.1 |
64.8545 |
Apps_ENERGY-Base_HIP-block_256 |
1250000.0 |
839.545 |
3070.68 |
Apps_ENERGY-Base_Seq-default |
834.0 |
11.7132 |
42.8418 |
Apps_ENERGY-RAJA_HIP-block_256 |
1250000.0 |
840.9 |
3075.64 |
Apps_FEMSWEEP-Base_HIP-block_64 |
839808.0 |
1148.1 |
221.08 |
Apps_FEMSWEEP-Base_HIP-unroll_64 |
839808.0 |
1117.79 |
215.243 |
Apps_FEMSWEEP-Base_Seq-default |
31104.0 |
3.25879 |
0.585225 |
Apps_FEMSWEEP-Base_Seq-unroll |
31104.0 |
4.66669 |
0.838061 |
Apps_FEMSWEEP-RAJA_HIP-block_64 |
1152000.0 |
948.786 |
183.294 |
Apps_FEMSWEEP-RAJA_HIP-unroll_64 |
1152000.0 |
945.893 |
182.735 |
Apps_INTSC_HEXRECT-Base_HIP-block_64 |
551368.0 |
854.299 |
10.4371 |
Apps_INTSC_HEXRECT-Base_Seq-default |
2744.0 |
4.18339 |
0.0524653 |
Apps_INTSC_HEXRECT-RAJA_HIP-block_64 |
287496.0 |
840.507 |
10.2808 |
Apps_MASS3DEA-Base_HIP-compile_time_block_stride_loop_64 |
48521216.0 |
22838.8 |
200.406 |
Apps_MASS3DEA-Base_Seq-default |
389120.0 |
10.0806 |
0.0884598 |
Apps_MASS3DEA-RAJA_HIP-cached_block_stride_loop_64 |
48521216.0 |
27107.8 |
237.866 |
Apps_MASS3DEA-RAJA_HIP-compile_time_block_stride_loop_64 |
18198528.0 |
5003.62 |
43.9058 |
Apps_MASS3DPA_ATOMIC-Base_HIP-block_64 |
132651.0 |
2307.93 |
1072.71 |
Apps_MASS3DPA_ATOMIC-Base_Seq-default |
1331.0 |
9.93768 |
4.87952 |
Apps_MASS3DPA_ATOMIC-RAJA_HIP-block_64 |
132651.0 |
2315.99 |
1076.46 |
Apps_MASSVEC3DPA-Base_HIP-direct_64 |
1146960.0 |
3108.81 |
948.321 |
Apps_MASSVEC3DPA-Base_HIP-runtime_block_stride_loop_64 |
1146960.0 |
2394.83 |
730.527 |
Apps_MASSVEC3DPA-Base_Seq-default |
4536.0 |
10.7929 |
3.2954 |
Apps_MASSVEC3DPA-RAJA_HIP-cached_block_stride_loop_64 |
1146960.0 |
2481.33 |
756.914 |
Apps_MASSVEC3DPA-RAJA_HIP-direct_64 |
1146960.0 |
3105.23 |
947.228 |
Apps_MASSVEC3DPA-RAJA_HIP-runtime_block_stride_loop_64 |
573480.0 |
1530.99 |
467.018 |
Apps_NODAL_ACCUMULATION_3D-Base_HIP-block_256 |
1000000.0 |
205.504 |
690.81 |
Apps_NODAL_ACCUMULATION_3D-Base_Seq-default |
2744.0 |
1.3962 |
5.15493 |
Apps_NODAL_ACCUMULATION_3D-RAJA_HIP-block_256 |
1000000.0 |
205.757 |
691.658 |
Apps_VOL3D-Base_HIP-block_256 |
343000.0 |
2969.24 |
1242.01 |
Apps_VOL3D-Base_Seq-default |
1331.0 |
16.3028 |
7.16966 |
Apps_VOL3D-RAJA_HIP-block_256 |
343000.0 |
3046.28 |
1274.24 |
2.3.7.1.2. SPX mode (Tier 2)
The process for generating results for the Tier 2 kernels is essentially
the same as for the Tier 1 kernels just described. The only differences are
that the command to run the kernels passes the tier2 argument (instead of
tier1):
$ ./run_tier_mi300a.sh spx tier2
and the results directories have tier1 replaced by tier2 in their names.
For the Tier2 case, note that the two kernels INDEXLIST_3LOOP and HALO_PACKING_FUSED
do not perform any floating point operations; i.e., they have zero GFLOP/sec rates.
They represent recurring computational patterns in our applications that are important despite not
doing floating point operations. Thus, we consider bandwidth as the appropriate metric to consider.
We include the baseline data files for Tier 2 kernels run on the MI300A architecture in
SPX mode in this repo in the directory ./docs/13_rajaperf/baseline_data/RPBenchmark_MI300A_tier2-SPX.
Kernel |
Sat Problem Size |
Sat GFLOP/s |
Sat B/W (GiB per sec.) |
|---|---|---|---|
Apps_CONVECTION3DPA-Base_HIP-block_64 |
351216.0 |
2170.44 |
1198.68 |
Apps_CONVECTION3DPA-Base_Seq-default |
1377.0 |
13.9032 |
7.69814 |
Apps_CONVECTION3DPA-RAJA_HIP-block_64 |
351216.0 |
2147.23 |
1185.86 |
Apps_DEL_DOT_VEC_2D-Base_HIP-block_256 |
1060900.0 |
2148.15 |
1780.63 |
Apps_DEL_DOT_VEC_2D-Base_Seq-default |
1849.0 |
8.2547 |
7.04795 |
Apps_DEL_DOT_VEC_2D-RAJA_HIP-block_256 |
264196.0 |
2025.11 |
1680.83 |
Apps_INTSC_HEXHEX-Base_HIP-block_64 |
125.0 |
21.7452 |
0.468793 |
Apps_INTSC_HEXHEX-Base_Seq-default |
27.0 |
4.67961 |
0.100885 |
Apps_INTSC_HEXHEX-RAJA_HIP-block_64 |
6859.0 |
88.4545 |
1.90694 |
Apps_LTIMES-Base_HIP-block_m_25 |
309696.0 |
350.901 |
488.049 |
Apps_LTIMES-Base_HIP-block_zgm_256 |
619392.0 |
645.836 |
898.235 |
Apps_LTIMES-Base_Seq-default |
2496.0 |
5.33075 |
7.4616 |
Apps_LTIMES-RAJA_HIP-kernel_m_25 |
309696.0 |
341.375 |
474.8 |
Apps_LTIMES-RAJA_HIP-kernel_zgm_256 |
619392.0 |
528.943 |
735.659 |
Apps_LTIMES-RAJA_HIP-launch_m_25 |
309696.0 |
342.979 |
477.032 |
Apps_LTIMES-RAJA_HIP-launch_zgm_256 |
619392.0 |
645.634 |
897.955 |
Apps_MASS3DPA-Base_HIP-block_64 |
2133336.0 |
2318.07 |
2657.07 |
Apps_MASS3DPA-Base_Seq-default |
4168.0 |
17.9066 |
20.5352 |
Apps_MASS3DPA-RAJA_HIP-block_64 |
2133336.0 |
2243.58 |
2571.69 |
Apps_MATVEC_3D_STENCIL-Base_HIP-block_256 |
157464.0 |
822.697 |
2036.54 |
Apps_MATVEC_3D_STENCIL-Base_Seq-default |
216.0 |
4.8756 |
15.3453 |
Apps_MATVEC_3D_STENCIL-RAJA_HIP-block_256 |
157464.0 |
844.116 |
2089.56 |
Basic_INDEXLIST_3LOOP-Base_HIP-block_256 |
160000.0 |
0.0 |
282.205 |
Basic_INDEXLIST_3LOOP-Base_Seq-default |
160000.0 |
0.0 |
10.9491 |
Basic_INDEXLIST_3LOOP-RAJA_HIP-block_256 |
160000.0 |
0.0 |
280.934 |
Basic_MULTI_REDUCE-Base_HIP-atomic_direct_256 |
6399995.0 |
173.959 |
2592.19 |
Basic_MULTI_REDUCE-Base_HIP-atomic_occgs_256 |
9374995.0 |
195.342 |
2910.82 |
Basic_MULTI_REDUCE-Base_Seq-default |
6245.0 |
0.315099 |
4.70286 |
Basic_MULTI_REDUCE-RAJA_HIP-atomic_direct_256 |
6399995.0 |
163.844 |
2441.47 |
Basic_MULTI_REDUCE-RAJA_HIP-atomic_occgs_256 |
9374995.0 |
191.009 |
2846.26 |
Basic_REDUCE_STRUCT-Base_HIP-blkatm_direct_256 |
100000.0 |
2.81939 |
21.0058 |
Basic_REDUCE_STRUCT-Base_HIP-blkatm_occgs_256 |
18750000.0 |
153.387 |
1142.83 |
Basic_REDUCE_STRUCT-Base_Seq-cascade |
6250.0 |
1.24456 |
9.27125 |
Basic_REDUCE_STRUCT-Base_Seq-default |
6250.0 |
2.40357 |
17.9051 |
Basic_REDUCE_STRUCT-Base_Seq-kahan |
6250.0 |
0.614229 |
4.57563 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkatm_direct_256 |
6400000.0 |
53.7546 |
400.503 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkatm_occgs_256 |
12800000.0 |
307.278 |
2289.4 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_direct_256 |
6400000.0 |
48.7602 |
363.292 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_direct_new_256 |
200000.0 |
2.80331 |
20.8862 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_occgs_256 |
9375000.0 |
258.939 |
1929.24 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_occgs_new_256 |
25000000.0 |
118.711 |
884.463 |
Comm_HALO_PACKING_FUSED-Base_HIP-direct_1024 |
91125.0 |
0.0 |
52.784 |
Comm_HALO_PACKING_FUSED-Base_Seq-direct |
91125.0 |
0.0 |
31.051 |
Comm_HALO_PACKING_FUSED-RAJA_HIP-direct_1024 |
91125.0 |
0.0 |
46.7484 |
Comm_HALO_PACKING_FUSED-RAJA_HIP-funcptr_1024 |
91125.0 |
0.0 |
47.6385 |
Comm_HALO_PACKING_FUSED-RAJA_HIP-virtfunc_1024 |
91125.0 |
0.0 |
45.9727 |
2.3.7.1.3. CPX mode (Tier 1)
For CPX mode (run with 6 MPI ranks per APU on a node), we choose the smallest problem to use ~50,000 bytes of allocated memory and the largest problem to use ~75MB of allocated memory, which is slightly less than 1/3 the MALL size.
Similar to SPX mode, we ran a different problem size range for the FEMSWEEP
and MASS3DEA kernels because they don’t clearly saturate. For those, we chose
the smallest problem to use ~1.6MB of allocated memory and the largest problem to
use ~200MB memory, which is a little less than the MALL size.
Similar to the SPX mode description above, we run the Tier 1 kernels in
CPX mode as follows:
$ cd path/to/RAJAPerf
$ cd build_lc_toss4-cray-mpich-9.0.1-amdclang-6.4.3-gfx942
$ $ cp ../../RPB-v2026.10.0/scripts/run_tier_mi300a.sh .
$ ./run_tier_mi300a.sh cpx tier1
This generates a directory named RPBenchmark_MI300A_tier1-CPX, which
contains all the results files for each kernel run over its range of problem
sizes.
Then, we process the data for reporting the results here in a concise form by running a Python script we provide:
$ cd path/to/RAJAPerf
$ cp ../../RPB-v2026.10.0/scripts/process_data.py .
$ python3 ./process_data.py --root-dir ./RPBenchmark_MI300A_tier1-CPX --output-dir ./RPBenchmark_MI300A_tier1-CPX/Output --exclude-plot-variant Base_Seq
This generates throughput curve files for Base_HIP and RAJA_HIP
variants of each kernel and summarizes the FOM (described in
Figure of Merit) in a CSV file. These files are included in this repo in the
directory ./docs/13_rajaperf/baseline_data/RPBenchmark_MI300A_tier1-CPX.
Kernel |
Sat Problem Size |
Sat GFLOP/s |
Sat B/W (GiB per sec.) |
|---|---|---|---|
Apps_DIFFUSION3DPA-Base_HIP-block_64 |
24678.0 |
612.534 |
344.914 |
Apps_DIFFUSION3DPA-Base_Seq-default |
394524.0 |
3.23378 |
1.82082 |
Apps_DIFFUSION3DPA-RAJA_HIP-block_64 |
24678.0 |
615.458 |
346.56 |
Apps_EDGE3D-Base_HIP-block_256 |
35937.0 |
4501.14 |
10.3756 |
Apps_EDGE3D-Base_Seq-default |
512.0 |
8.38987 |
0.0204979 |
Apps_EDGE3D-RAJA_HIP-block_256 |
35937.0 |
4532.44 |
10.4478 |
Apps_ENERGY-Base_HIP-block_256 |
53334.0 |
219.701 |
803.568 |
Apps_ENERGY-Base_Seq-default |
417.0 |
11.9252 |
43.6169 |
Apps_ENERGY-RAJA_HIP-block_256 |
53334.0 |
235.409 |
861.022 |
Apps_FEMSWEEP-Base_HIP-block_64 |
589824.0 |
259.161 |
49.701 |
Apps_FEMSWEEP-Base_HIP-unroll_64 |
589824.0 |
234.711 |
45.0119 |
Apps_FEMSWEEP-Base_Seq-default |
31104.0 |
2.82937 |
0.508108 |
Apps_FEMSWEEP-Base_Seq-unroll |
31104.0 |
4.20581 |
0.755293 |
Apps_FEMSWEEP-RAJA_HIP-block_64 |
589824.0 |
256.915 |
49.2702 |
Apps_FEMSWEEP-RAJA_HIP-unroll_64 |
589824.0 |
262.535 |
50.348 |
Apps_INTSC_HEXRECT-Base_HIP-block_64 |
74088.0 |
141.338 |
1.73495 |
Apps_INTSC_HEXRECT-Base_Seq-default |
1728.0 |
4.07062 |
0.0513518 |
Apps_INTSC_HEXRECT-RAJA_HIP-block_64 |
74088.0 |
141.972 |
1.74273 |
Apps_MASS3DEA-Base_HIP-compile_time_block_stride_loop_64 |
6213632.0 |
4902.68 |
43.0202 |
Apps_MASS3DEA-Base_Seq-default |
196608.0 |
9.9702 |
0.0874952 |
Apps_MASS3DEA-RAJA_HIP-cached_block_stride_loop_64 |
9101312.0 |
4918.8 |
43.1616 |
Apps_MASS3DEA-RAJA_HIP-compile_time_block_stride_loop_64 |
3108864.0 |
876.714 |
7.69305 |
Apps_MASS3DPA_ATOMIC-Base_HIP-block_64 |
19683.0 |
408.771 |
192.298 |
Apps_MASS3DPA_ATOMIC-Base_Seq-default |
729.0 |
9.75866 |
4.88195 |
Apps_MASS3DPA_ATOMIC-RAJA_HIP-block_64 |
19683.0 |
395.107 |
185.871 |
Apps_MASSVEC3DPA-Base_HIP-direct_64 |
143370.0 |
570.096 |
173.908 |
Apps_MASSVEC3DPA-Base_HIP-runtime_block_stride_loop_64 |
143370.0 |
427.57 |
130.431 |
Apps_MASSVEC3DPA-Base_Seq-default |
2268.0 |
10.8968 |
3.3303 |
Apps_MASSVEC3DPA-RAJA_HIP-cached_block_stride_loop_64 |
143370.0 |
459.754 |
140.248 |
Apps_MASSVEC3DPA-RAJA_HIP-direct_64 |
143370.0 |
569.174 |
173.627 |
Apps_MASSVEC3DPA-RAJA_HIP-runtime_block_stride_loop_64 |
71685.0 |
305.513 |
93.1999 |
Apps_NODAL_ACCUMULATION_3D-Base_HIP-block_256 |
117649.0 |
35.4271 |
120.977 |
Apps_NODAL_ACCUMULATION_3D-Base_Seq-default |
1000.0 |
1.39555 |
5.386 |
Apps_NODAL_ACCUMULATION_3D-RAJA_HIP-block_256 |
117649.0 |
35.6658 |
121.792 |
Apps_VOL3D-Base_HIP-block_256 |
79507.0 |
626.699 |
263.825 |
Apps_VOL3D-Base_Seq-default |
512.0 |
15.981 |
7.16574 |
Apps_VOL3D-RAJA_HIP-block_256 |
35937.0 |
614.464 |
259.95 |
2.3.7.1.4. CPX mode (Tier 2)
Similar to SPX mode, the process for generating results for the Tier 2 kernels in
CPX mode is essentially the same as for the Tier 1 kernels just described. The only
differences are that the command to run the kernels passes the tier2 argument (instead of
tier1):
$ ./run_tier_mi300a.sh cpx tier2
and the results directories have tier1 replaced by tier2 in their names.
As noted earlier, two of the kernels INDEXLIST_3LOOP and HALO_PACKING_FUSED do not
perform any floating point operations. So, we consider the bandwidth
as the appropriate metric to consider. We include the baseline data files for Tier 2
kernels run on the MI300A architecture in CPX mode in this repo in the
directory ./docs/13_rajaperf/baseline_data/RPBenchmark_MI300A_tier2-CPX.
Kernel |
Sat Problem Size |
Sat GFLOP/s |
Sat B/W (GiB per sec.) |
|---|---|---|---|
Apps_CONVECTION3DPA-Base_HIP-block_64 |
21951.0 |
358.86 |
198.22 |
Apps_CONVECTION3DPA-Base_Seq-default |
702.0 |
13.4031 |
7.43967 |
Apps_CONVECTION3DPA-RAJA_HIP-block_64 |
21951.0 |
350.698 |
193.711 |
Apps_DEL_DOT_VEC_2D-Base_HIP-block_256 |
32400.0 |
459.404 |
383.139 |
Apps_DEL_DOT_VEC_2D-Base_Seq-default |
841.0 |
7.82698 |
6.78255 |
Apps_DEL_DOT_VEC_2D-RAJA_HIP-block_256 |
32400.0 |
466.122 |
388.742 |
Apps_INTSC_HEXHEX-Base_HIP-block_64 |
216.0 |
3.47517 |
0.074919 |
Apps_INTSC_HEXHEX-Base_Seq-default |
64.0 |
4.38675 |
0.0945714 |
Apps_INTSC_HEXHEX-RAJA_HIP-block_64 |
512.0 |
13.0664 |
0.28169 |
Apps_LTIMES-Base_HIP-block_m_25 |
19392.0 |
64.9539 |
90.4112 |
Apps_LTIMES-Base_HIP-block_zgm_256 |
19392.0 |
107.632 |
149.816 |
Apps_LTIMES-Base_Seq-default |
1344.0 |
5.23483 |
7.36753 |
Apps_LTIMES-RAJA_HIP-kernel_m_25 |
19392.0 |
62.1527 |
86.512 |
Apps_LTIMES-RAJA_HIP-kernel_zgm_256 |
19392.0 |
91.7377 |
127.692 |
Apps_LTIMES-RAJA_HIP-launch_m_25 |
19392.0 |
62.3861 |
86.837 |
Apps_LTIMES-RAJA_HIP-launch_zgm_256 |
19392.0 |
109.717 |
152.718 |
Apps_MASS3DPA-Base_HIP-block_64 |
66664.0 |
333.37 |
382.135 |
Apps_MASS3DPA-Base_Seq-default |
2088.0 |
17.2167 |
19.7534 |
Apps_MASS3DPA-RAJA_HIP-block_64 |
133336.0 |
385.233 |
441.577 |
Apps_MATVEC_3D_STENCIL-Base_HIP-block_256 |
17576.0 |
172.271 |
442.234 |
Apps_MATVEC_3D_STENCIL-Base_Seq-default |
64.0 |
4.396 |
15.488 |
Apps_MATVEC_3D_STENCIL-RAJA_HIP-block_256 |
17576.0 |
183.41 |
470.829 |
Basic_INDEXLIST_3LOOP-Base_HIP-block_256 |
80000.0 |
0.0 |
144.438 |
Basic_INDEXLIST_3LOOP-Base_Seq-default |
80000.0 |
0.0 |
14.2619 |
Basic_INDEXLIST_3LOOP-RAJA_HIP-block_256 |
80000.0 |
0.0 |
141.908 |
Basic_MULTI_REDUCE-Base_HIP-atomic_direct_256 |
799995.0 |
27.1483 |
404.546 |
Basic_MULTI_REDUCE-Base_HIP-atomic_occgs_256 |
1599995.0 |
32.0872 |
478.14 |
Basic_MULTI_REDUCE-Base_Seq-default |
3120.0 |
0.319179 |
4.77138 |
Basic_MULTI_REDUCE-RAJA_HIP-atomic_direct_256 |
799995.0 |
25.6092 |
381.611 |
Basic_MULTI_REDUCE-RAJA_HIP-atomic_occgs_256 |
1599995.0 |
30.565 |
455.456 |
Basic_REDUCE_STRUCT-Base_HIP-blkatm_direct_256 |
100000.0 |
1.0459 |
7.79252 |
Basic_REDUCE_STRUCT-Base_HIP-blkatm_occgs_256 |
4687500.0 |
36.3019 |
270.47 |
Basic_REDUCE_STRUCT-Base_Seq-cascade |
3125.0 |
1.22113 |
9.09523 |
Basic_REDUCE_STRUCT-Base_Seq-default |
3125.0 |
2.25344 |
16.7841 |
Basic_REDUCE_STRUCT-Base_Seq-kahan |
3125.0 |
0.606479 |
4.51718 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkatm_direct_256 |
800000.0 |
8.57419 |
63.8826 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkatm_occgs_256 |
1600000.0 |
47.6719 |
355.183 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_direct_256 |
800000.0 |
7.61954 |
56.7699 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_direct_new_256 |
800000.0 |
5.60441 |
41.756 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_occgs_256 |
1600000.0 |
42.8746 |
319.44 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_occgs_new_256 |
1600000.0 |
41.344 |
308.037 |
Comm_HALO_PACKING_FUSED-Base_HIP-direct_1024 |
42875.0 |
0.0 |
35.331 |
Comm_HALO_PACKING_FUSED-Base_Seq-direct |
42875.0 |
0.0 |
33.6208 |
Comm_HALO_PACKING_FUSED-RAJA_HIP-direct_1024 |
42875.0 |
0.0 |
29.8946 |
Comm_HALO_PACKING_FUSED-RAJA_HIP-funcptr_1024 |
42875.0 |
0.0 |
30.895 |
Comm_HALO_PACKING_FUSED-RAJA_HIP-virtfunc_1024 |
42875.0 |
0.0 |
29.309 |
2.3.7.1.5. AMD MI300A throughput plots (Tier 1)
This section contains throughput plots for each Tier 1 kernel run as described above on the MI300A architecture in SPX mode (left column) and CPX mode (right column). Each plot has multiple curves with GFLOP/sec (compute rate) plotted as a function of problem size (allocated bytes). The legend in each plot indicates the curves shown. Each plot includes:
Throughput curves for Base and RAJA variant(s) of the kernels. Solid line segments connect the dots, where the dots are actual GFLOP rates determined from the kernel being run at a given problem size.
Smoothed versions of the throughput curves (dashed lines), which are constructed from the dots.
Stars indicate approximate saturation points based on the smoothed curves and computed using simple heuristics. The legend contains the (x, y) values for the saturation points.
Most plots contain two variants, with the Base non-RAJA variant in blue and RAJA variant in orange. In each case, the throughput curves and saturation points are close, which indicates that the RAJA variants perform as well as the non-RAJA variants that are written directly in HIP with no RAJA abstractions. Two kernels (MASS3DEA, MASSVEC3DPA) contain additional curves that show more variants. These additional curves are included to show how kernel execution choices, RAJA execution policies in particular, can have a significant impact on performance.
Tier 1 Kernels: MI300A Node Throughput (SPX Mode) |
Tier 1 Kernels: MI300A Node Throughput (CPX Mode) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2.3.7.2. NVIDIA H100 throughput results (Tier 1 kernels)
For the H100 architecture, we present one set of throughput results, where we ran with
4 MPI ranks on a node – one for each H100 GPU. We ran each Tier 1
kernel over a sequence of problem sizes such that the saturation point is
evident on its associated throughput curve.
We choose the smallest problem to use ~50,000 bytes of allocated memory and the largest problem to use ~150MB of allocated memory, which is about 3 times the L2 cache size on the H100 GPU. The L2 cache is 50MB (50 * 1024 * 1024 = 52428800 bytes).
Again, we ran a different problem size range for the FEMSWEEP and MASS3DEA
kernels because they kernels don’t clearly saturate. For them, we chose the smallest
problem to use ~1.6MB of allocated memory and the largest problem to use ~300MB memory,
which is about 6 times the L2 cache size.
After building the code as described in H100 architecture, we
run the Tier 1 kernels as follows:
$ cd path/to/RAJAPerf
$ cd build_lc_toss4-mvapich2-2.3.7-nvcc-13.1.1-90-gcc-10.3.1
$ cp ../../RPB-v2026.10.0/scripts/run_tier_h100.sh .
$ ./run_tier_h100.sh tier1
This generates a directory named RPBenchmark_H100_tier1, which contains
the results files for each kernel run over its range of problem sizes.
Then, we process the data for reporting the results here in a concise form by running the Python script we provide:
$ cd path/to/RAJAPerf
$ cd build_lc_toss4-mvapich2-2.3.7-nvcc-13.1.1-90-gcc-10.3.1
$ cp ../../RPB-v2026.10.0/scripts/process_data.py .
$ python3 ./process_data.py --root-dir ./RPBenchmark_H100_tier1 --output-dir ./RPBenchmark_H100_tier1/Output --exclude-plot-variant Base_Seq
This generates throughput curve files for Base_CUDA and RAJA_CUDA
variants of each kernel and summarizes the FOM (described in
Figure of Merit) in a CSV file. We include
the files generated by the process_data.py script in this repo in the
directory ./docs/13_rajaperf/baseline_data/RPBenchmark_H100_tier1.
Kernel |
Sat Problem Size |
Sat GFLOP/s |
Sat B/W (GiB per sec.) |
|---|---|---|---|
Apps_DIFFUSION3DPA-Base_CUDA-block_64 |
197262.0 |
2951.3 |
1661.78 |
Apps_DIFFUSION3DPA-Base_OpenMP-default |
49329.0 |
294.975 |
166.094 |
Apps_DIFFUSION3DPA-Base_Seq-default |
405.0 |
15.9577 |
9.0161 |
Apps_DIFFUSION3DPA-RAJA_CUDA-block_64 |
197262.0 |
2942.08 |
1656.59 |
Apps_DIFFUSION3DPA-RAJA_OpenMP-default |
98631.0 |
302.886 |
170.546 |
Apps_EDGE3D-Base_CUDA-block_256 |
166375.0 |
1522.51 |
3.47979 |
Apps_EDGE3D-Base_OpenMP-default |
512.0 |
176.458 |
0.431117 |
Apps_EDGE3D-Base_Seq-default |
512.0 |
9.31918 |
0.0227684 |
Apps_EDGE3D-RAJA_CUDA-block_256 |
166375.0 |
1520.36 |
3.47489 |
Apps_EDGE3D-RAJA_OpenMP-default |
512.0 |
176.367 |
0.430895 |
Apps_ENERGY-Base_CUDA-block_256 |
625000.0 |
808.567 |
2957.38 |
Apps_ENERGY-Base_OpenMP-default |
106667.0 |
237.985 |
870.442 |
Apps_ENERGY-Base_Seq-default |
417.0 |
13.6841 |
50.0504 |
Apps_ENERGY-RAJA_CUDA-block_256 |
625000.0 |
806.032 |
2948.11 |
Apps_ENERGY-RAJA_OpenMP-default |
106667.0 |
234.827 |
858.895 |
Apps_FEMSWEEP-Base_CUDA-block_64 |
1152000.0 |
262.807 |
50.771 |
Apps_FEMSWEEP-Base_CUDA-unroll_64 |
1152000.0 |
716.364 |
138.393 |
Apps_FEMSWEEP-Base_OpenMP-default |
31104.0 |
68.1662 |
12.2415 |
Apps_FEMSWEEP-Base_OpenMP-unroll |
31104.0 |
117.188 |
21.045 |
Apps_FEMSWEEP-Base_Seq-default |
31104.0 |
3.41317 |
0.612948 |
Apps_FEMSWEEP-Base_Seq-unroll |
31104.0 |
5.7872 |
1.03928 |
Apps_FEMSWEEP-RAJA_CUDA-block_64 |
1152000.0 |
251.72 |
48.6292 |
Apps_FEMSWEEP-RAJA_CUDA-unroll_64 |
1533312.0 |
595.04 |
115.258 |
Apps_FEMSWEEP-RAJA_CUDA-unroll_64_min_blocks_per_sm_5 |
1152000.0 |
608.452 |
117.545 |
Apps_FEMSWEEP-RAJA_OpenMP-default |
31104.0 |
69.1868 |
12.4248 |
Apps_FEMSWEEP-RAJA_OpenMP-unroll |
31104.0 |
101.031 |
18.1435 |
Apps_INTSC_HEXRECT-Base_CUDA-block_64 |
551368.0 |
1555.05 |
18.9982 |
Apps_INTSC_HEXRECT-Base_OpenMP-default |
39304.0 |
85.895 |
1.05686 |
Apps_INTSC_HEXRECT-Base_Seq-default |
1728.0 |
4.09666 |
0.0516803 |
Apps_INTSC_HEXRECT-RAJA_CUDA-block_64 |
551368.0 |
1299.64 |
15.8779 |
Apps_INTSC_HEXRECT-RAJA_OpenMP-default |
39304.0 |
85.904 |
1.05697 |
Apps_MASS3DEA-Base_CUDA-compile_time_block_stride_loop_64 |
18198528.0 |
25194.2 |
221.075 |
Apps_MASS3DEA-Base_OpenMP-default |
778240.0 |
446.227 |
3.91565 |
Apps_MASS3DEA-Base_Seq-default |
196608.0 |
22.0242 |
0.193277 |
Apps_MASS3DEA-RAJA_CUDA-cached_block_stride_loop_64 |
9101312.0 |
20980.5 |
184.101 |
Apps_MASS3DEA-RAJA_CUDA-compile_time_block_stride_loop_64 |
18198528.0 |
25725.6 |
225.738 |
Apps_MASS3DEA-RAJA_OpenMP-default |
389120.0 |
391.398 |
3.43461 |
Apps_MASS3DPA_ATOMIC-Base_CUDA-block_64 |
250047.0 |
1722.05 |
798.408 |
Apps_MASS3DPA_ATOMIC-Base_OpenMP-default |
35937.0 |
127.065 |
59.4919 |
Apps_MASS3DPA_ATOMIC-Base_Seq-default |
729.0 |
11.2842 |
5.64513 |
Apps_MASS3DPA_ATOMIC-RAJA_CUDA-block_64 |
250047.0 |
1722.85 |
798.778 |
Apps_MASS3DPA_ATOMIC-RAJA_OpenMP-default |
68921.0 |
135.6 |
63.23 |
Apps_MASSVEC3DPA-Base_CUDA-direct_64 |
1146960.0 |
2404.13 |
733.362 |
Apps_MASSVEC3DPA-Base_CUDA-runtime_block_stride_loop_64 |
1146960.0 |
1644.29 |
501.579 |
Apps_MASSVEC3DPA-Base_OpenMP-default |
143370.0 |
242.989 |
74.1241 |
Apps_MASSVEC3DPA-Base_Seq-default |
2268.0 |
12.986 |
3.9688 |
Apps_MASSVEC3DPA-RAJA_CUDA-cached_block_stride_loop_64 |
573480.0 |
1492.54 |
455.29 |
Apps_MASSVEC3DPA-RAJA_CUDA-direct_64 |
1146960.0 |
2392.88 |
729.93 |
Apps_MASSVEC3DPA-RAJA_CUDA-runtime_block_stride_loop_64 |
573480.0 |
1493.59 |
455.611 |
Apps_MASSVEC3DPA-RAJA_OpenMP-default |
143370.0 |
243.616 |
74.3153 |
Apps_NODAL_ACCUMULATION_3D-Base_CUDA-block_256 |
493039.0 |
246.25 |
831.105 |
Apps_NODAL_ACCUMULATION_3D-Base_OpenMP-default |
54872.0 |
4.54475 |
15.6592 |
Apps_NODAL_ACCUMULATION_3D-Base_Seq-default |
1000.0 |
4.38626 |
16.9284 |
Apps_NODAL_ACCUMULATION_3D-RAJA_CUDA-block_256 |
493039.0 |
245.992 |
830.233 |
Apps_NODAL_ACCUMULATION_3D-RAJA_OpenMP-default |
54872.0 |
4.50015 |
15.5055 |
Apps_VOL3D-Base_CUDA-block_256 |
704969.0 |
5010.48 |
2091.23 |
Apps_VOL3D-Base_OpenMP-default |
79507.0 |
186.304 |
78.4296 |
Apps_VOL3D-Base_Seq-default |
512.0 |
10.4241 |
4.67405 |
Apps_VOL3D-RAJA_CUDA-block_256 |
343000.0 |
4699.88 |
1965.93 |
Apps_VOL3D-RAJA_OpenMP-default |
79507.0 |
174.992 |
73.6677 |
2.3.7.2.1. H100 (Tier 2)
The process for generating results for the Tier 2 kernels is essentially
the same as for the Tier 1 kernels just described. The only differences are that
the command to run the kernels passes the tier2 argument (instead of tier1):
$ ./run_tier_h100.sh tier2
and the results directories have tier1 replaced by tier2 in their names.
For the Tier2 case, the INDEXLIST_3LOOP and HALO_PACKING_FUSED kernels
do not perform any floating point operations, so they have zero GFLOP/sec rates.
Thus, we consider the bandwidth as the appropriate metric to consider for them.
We include the baseline data files for Tier 2 kernels run on the H100 architecture
in this repo in the directory ./docs/13_rajaperf/baseline_data/RPBenchmark_H100_tier2-SPX.
Kernel |
Sat Problem Size |
Sat GFLOP/s |
Sat B/W (GiB per sec.) |
|---|---|---|---|
Apps_CONVECTION3DPA-Base_CUDA-block_64 |
351216.0 |
2374.57 |
1311.41 |
Apps_CONVECTION3DPA-Base_OpenMP-default |
87804.0 |
317.9 |
175.574 |
Apps_CONVECTION3DPA-Base_Seq-default |
702.0 |
17.2454 |
9.57239 |
Apps_CONVECTION3DPA-RAJA_CUDA-block_64 |
351216.0 |
2358.3 |
1302.43 |
Apps_CONVECTION3DPA-RAJA_OpenMP-default |
175608.0 |
287.746 |
158.917 |
Apps_DEL_DOT_VEC_2D-Base_CUDA-block_256 |
264196.0 |
2853.07 |
2368.03 |
Apps_DEL_DOT_VEC_2D-Base_OpenMP-default |
131044.0 |
200.921 |
166.945 |
Apps_DEL_DOT_VEC_2D-Base_Seq-default |
841.0 |
10.3641 |
8.98111 |
Apps_DEL_DOT_VEC_2D-RAJA_CUDA-block_256 |
264196.0 |
2874.57 |
2385.87 |
Apps_DEL_DOT_VEC_2D-RAJA_OpenMP-default |
32400.0 |
122.205 |
101.918 |
Apps_INTSC_HEXHEX-Base_CUDA-block_64 |
3375.0 |
1818.13 |
39.196 |
Apps_INTSC_HEXHEX-Base_OpenMP-default |
1000.0 |
81.3847 |
1.75452 |
Apps_INTSC_HEXHEX-Base_Seq-default |
27.0 |
3.82093 |
0.0823732 |
Apps_INTSC_HEXHEX-RAJA_CUDA-block_64 |
3375.0 |
1823.28 |
39.307 |
Apps_INTSC_HEXHEX-RAJA_OpenMP-default |
1000.0 |
80.0336 |
1.7254 |
Apps_LTIMES-Base_CUDA-block_m_25 |
309696.0 |
441.728 |
614.376 |
Apps_LTIMES-Base_CUDA-block_zgm_256 |
309696.0 |
660.505 |
918.662 |
Apps_LTIMES-Base_OpenMP-default |
77568.0 |
65.106 |
90.5666 |
Apps_LTIMES-Base_Seq-default |
1344.0 |
3.72949 |
5.24891 |
Apps_LTIMES-RAJA_CUDA-kernel_m_25 |
309696.0 |
439.801 |
611.695 |
Apps_LTIMES-RAJA_CUDA-kernel_zgm_256 |
309696.0 |
649.847 |
903.838 |
Apps_LTIMES-RAJA_CUDA-launch_m_25 |
309696.0 |
439.38 |
611.11 |
Apps_LTIMES-RAJA_CUDA-launch_zgm_256 |
309696.0 |
659.295 |
916.979 |
Apps_LTIMES-RAJA_OpenMP-kernel |
77568.0 |
35.2679 |
49.0599 |
Apps_LTIMES-RAJA_OpenMP-launch |
154944.0 |
70.6043 |
98.2049 |
Apps_MASS3DPA-Base_CUDA-block_64 |
2133336.0 |
1734.08 |
1987.68 |
Apps_MASS3DPA-Base_OpenMP-default |
33336.0 |
90.9215 |
104.224 |
Apps_MASS3DPA-Base_Seq-default |
2088.0 |
11.1872 |
12.8355 |
Apps_MASS3DPA-RAJA_CUDA-block_64 |
2133336.0 |
1730.35 |
1983.4 |
Apps_MASS3DPA-RAJA_OpenMP-default |
266664.0 |
123.738 |
141.835 |
Apps_MATVEC_3D_STENCIL-Base_CUDA-block_256 |
157464.0 |
1068.11 |
2644.03 |
Apps_MATVEC_3D_STENCIL-Base_OpenMP-default |
8000.0 |
38.4003 |
100.603 |
Apps_MATVEC_3D_STENCIL-Base_Seq-default |
64.0 |
2.85231 |
10.0493 |
Apps_MATVEC_3D_STENCIL-RAJA_CUDA-block_256 |
157464.0 |
993.509 |
2459.37 |
Apps_MATVEC_3D_STENCIL-RAJA_OpenMP-default |
8000.0 |
40.1918 |
105.297 |
Basic_INDEXLIST_3LOOP-Base_CUDA-block_256 |
80000.0 |
0.0 |
186.912 |
Basic_INDEXLIST_3LOOP-Base_OpenMP-default |
80000.0 |
0.0 |
135.439 |
Basic_INDEXLIST_3LOOP-Base_Seq-default |
80000.0 |
0.0 |
8.34432 |
Basic_INDEXLIST_3LOOP-RAJA_CUDA-block_256 |
80000.0 |
0.0 |
183.786 |
Basic_INDEXLIST_3LOOP-RAJA_OpenMP-default |
80000.0 |
0.0 |
98.7727 |
Basic_MULTI_REDUCE-Base_CUDA-atomic_direct_256 |
18749995.0 |
16.758 |
249.714 |
Basic_MULTI_REDUCE-Base_CUDA-atomic_occgs_256 |
15624995.0 |
13.1408 |
195.813 |
Basic_MULTI_REDUCE-Base_OpenMP-default |
99995.0 |
0.0378265 |
0.563715 |
Basic_MULTI_REDUCE-Base_Seq-default |
99995.0 |
0.467263 |
6.96346 |
Basic_MULTI_REDUCE-RAJA_CUDA-atomic_direct_256 |
18749995.0 |
16.6319 |
247.835 |
Basic_MULTI_REDUCE-RAJA_CUDA-atomic_occgs_256 |
18749995.0 |
15.7036 |
234.002 |
Basic_MULTI_REDUCE-RAJA_OpenMP-default |
99995.0 |
3.46015 |
51.5655 |
Basic_REDUCE_STRUCT-Base_CUDA-blkatm_direct_256 |
12500000.0 |
40.8347 |
304.242 |
Basic_REDUCE_STRUCT-Base_CUDA-blkatm_occgs_256 |
18750000.0 |
50.3368 |
375.039 |
Basic_REDUCE_STRUCT-Base_OpenMP-default |
800000.0 |
30.2817 |
225.616 |
Basic_REDUCE_STRUCT-Base_Seq-cascade |
100000.0 |
0.844658 |
6.29313 |
Basic_REDUCE_STRUCT-Base_Seq-default |
100000.0 |
1.88811 |
14.0674 |
Basic_REDUCE_STRUCT-Base_Seq-kahan |
100000.0 |
0.944241 |
7.03508 |
Basic_REDUCE_STRUCT-RAJA_CUDA-blkatm_direct_256 |
15625000.0 |
25.397 |
189.222 |
Basic_REDUCE_STRUCT-RAJA_CUDA-blkatm_occgs_256 |
18750000.0 |
31.1127 |
231.807 |
Basic_REDUCE_STRUCT-RAJA_CUDA-blkdev_direct_256 |
800000.0 |
24.5375 |
182.819 |
Basic_REDUCE_STRUCT-RAJA_CUDA-blkdev_direct_new_256 |
3200000.0 |
33.194 |
247.314 |
Basic_REDUCE_STRUCT-RAJA_CUDA-blkdev_occgs_256 |
12500000.0 |
273.221 |
2035.66 |
Basic_REDUCE_STRUCT-RAJA_CUDA-blkdev_occgs_new_256 |
12500000.0 |
267.272 |
1991.33 |
Basic_REDUCE_STRUCT-RAJA_OpenMP-default |
200000.0 |
5.91619 |
44.0788 |
Basic_REDUCE_STRUCT-RAJA_OpenMP-new |
800000.0 |
28.8138 |
214.679 |
Comm_HALO_PACKING_FUSED-Base_CUDA-direct_1024 |
42875.0 |
0.0 |
28.3313 |
Comm_HALO_PACKING_FUSED-Base_OpenMP-direct |
42875.0 |
0.0 |
13.3719 |
Comm_HALO_PACKING_FUSED-Base_Seq-direct |
42875.0 |
0.0 |
32.0479 |
Comm_HALO_PACKING_FUSED-RAJA_CUDA-direct_1024 |
42875.0 |
0.0 |
27.0527 |
Comm_HALO_PACKING_FUSED-RAJA_CUDA-funcptr_1024 |
42875.0 |
0.0 |
24.998 |
Comm_HALO_PACKING_FUSED-RAJA_CUDA-virtfunc_1024 |
42875.0 |
0.0 |
24.7011 |
Comm_HALO_PACKING_FUSED-RAJA_OpenMP-direct |
42875.0 |
0.0 |
0.922968 |
Comm_HALO_PACKING_FUSED-RAJA_OpenMP-funcptr |
42875.0 |
0.0 |
0.921585 |
Comm_HALO_PACKING_FUSED-RAJA_OpenMP-virtfunc |
42875.0 |
0.0 |
0.866138 |
2.3.7.2.2. NVIDIA H100 throughput plots (Tier 1)
The following table contains throughput plots for each kernel run as described above for the H100 architecture. Each plot has multiple curves where GFLOP/sec (compute rate) is plotted as a function of problem size (allocated bytes). The legend in each plot indicates the curves shown. Each plot includes:
Throughput curves for Base and RAJA variant(s) of the kernel (solid line segments connecting the dots, where the dots are actual GFLOP rates determined from the kernel being run at a given problem size).
Smoothed versions of the throughput curves (dashed lines), which are constructed from the dots.
Stars that indicate approximate saturation points based on the smoothed curves and computed using simple heuristics. The legend contains the (x, y) values for the saturation points.
Most plots contain two variants, with the Base non-RAJA variant in blue and RAJA variant in orange. In these cases, the throughput and saturation are close, which indicates that the RAJA variants perform as well as the non-RAJA variants that are written directly in CUDA with no RAJA abstractions. Two kernels (MASS3DEA, MASSVEC3DPA) contain additional curves that show more variants. These additional curves were included to show how kernel execution choices, RAJA execution policies in particular, can have a noticeable impact on performance.
Tier 1 Kernels H100 Node Throughput |
|
|
|
|
|
|
|
|
|
|
2.3.8. Memory Usage
For the RAJA Performance Suite Benchmark, we run each kernel over a sequence of problem sizes to generate a throughput curve and, based on that, estimate a saturation point. The memory usage for each entry in the sequence is roughly the same for each kernel. However, there is no significant meaning to take away from this since the memory usage of kernels like those in the Suite will be determined by the application context in which they are used.
2.3.9. References
The GitHub repositories are the primary references for RAJA and the RAJA Performance Suite:
Other helpful references include:
Olga Pearce, Jason Burmark, Rich Hornung, Befikir Bogale, Ian Lumsden, Michael McKinsey, Dewi Yokelson, David Boehme, Stephanie Brink, Michela Taufer, Tom Scogland, “RAJA Performance Suite: Performance Portability Analysis with Caliper and Thicket”, in 2024 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC) at the International Conference on High Performance Computing, Network, Storage, and Analysis (SC-W 2024). https://dl.acm.org/doi/pdf/10.1109/SCW63240.2024.00162
Beckingsale, J. Burmark, R. Hornung, H. Jones, W. Killian, A. J. Kunen, O. Pearce, P. Robinson, B. S. Ryujin, T. R. W. Scogland, “RAJA: Portable Performance for Large-Scale Scientific Applications”, 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC). https://conferences.computer.org/sc19w/2019/#!/toc/14
Arturo Vargas, Thomas M. Stitt, Kenneth Weiss, Vladimir Z. Tomov, Jean-Sylvain Camier, Tzanio Kolev, Robert N. Rieben, “Matrix-free Approaches for GPU Acceleration of a High-order Finite Element Hydrodynamic Application using MFEM, Umpire, and RAJA”, International Journal of High Performance Computing Applications. 36(4):492-509 (2022). https://journals.sagepub.com/doi/10.1177/10943420221100262