3.3. RAJA Performance Suite Tier 2 Kernels
The main RAJA Performance Suite Benchmark documentation provides general information that applies to all kernels in the Benchmark, both Tier 1 and Tier2, such as purpose and characteristics, FOM, how to build and run, etc. This section provides information specific to the Tier 2 kernels in the RAJA Performance Suite Benchmark.
3.3.1. Problems
The following list describes the Tier 2 kernels and the RAJA features they exercise.
In the RAJA Performance Suite repo, they are located in RAJAPerf/src sub-directories noted below (i.e., RAJAPerf/src/<subdir>/<kernel>):
apps/CONVECTION3DPA element-wise action of a 3D finite element volume convection operator via partial assembly and sum factorization (nested loops, GPU shared memory, RAJA::launch API)
apps/DEL_DOT_VEC_2D divergence of a vector field at a set of points on a mesh (single loop, data access via indirection array, RAJA::forall API)
apps/INTSC_HEXHEX intersection between two 24-sided hexahedra, including volume and moment calculations (multiple single-loop operations in sequence, RAJA::forall API)
apps/LTIMES one step of the source-iteration technique for solving the steady-state linear Boltzmann equation, multi-dimensional matrix product (nested loops, RAJA::kernel API)
apps/MASS3DPA element-wise action of a 3D finite element mass matrix via partial assembly and sum factorization (nested loops, GPU shared memory, RAJA::launch API)
apps/MATVEC_3D_STENCIL matrix-vector product based on a 3D mesh stencil (single loop, data access via indirection array, RAJA::forall API)
basic/MULTI_REDUCE multiple reductions in a kernel, where number of reductions is set at run time (single loop, irregular atomic contention, RAJA::forall API)
basic/REDUCE_STRUCT multiple reductions in a kernel, where number of reductions (6) is known at compile time (single loop, multiple reductions, RAJA::forall API)
basic/INDEXLIST_3LOOP construction of set of indices used in other kernel executions (single loops, vendor scan implementations, RAJA::forall API)
comm/HALO_PACKING_FUSED packing and unpacking MPI message buffers for point-to-point distributed memory halo data exchange for mesh-based codes (overhead of launching many small kernels, GPU variants use RAJA::Workgroup concepts to execute multiple kernels with one launch)
3.3.2. Figure of Merit
The FOM for each kernel in the RAJA Performance Suite Benchmark is described in Figure of Merit. Please refer to that section for FOM definition and how it is determined.
For Tier 2 kernels, the saturation point is estimated based on compute rate data,
as it was for Tier 1 kernels, except for the kernels INDEXLIST_3LOOP and HALO_PACKING_FUSED.
Those kernels represent important recurring computational patterns in HPC applications. However, they
perform no floating point operations, so we estimate the saturation point for them based on memory
3.3.3. Baseline on El Capitan
The process of running the Tier 2 kernels and generating the FOM for them is essentially the same as what is described in Baseline on El Capitan for the Tier 1 kernels. Here, we summarize the process as it applies to the Tier 2 kernels.
3.3.3.1. AMD MI300A throughput results (Tier 2 kernels)
For the MI300A architecture, we present two sets of throughput results. One is
run in SPX mode where we use 4 MPI ranks on a node, one for each MI300A APU,
thus treating each APU as a single GPU. The other is run in CPX mode where
we run with 24 MPI ranks on a node, six for each MI300A APU, treating each APU as
6 GPUs (i.e., one GPU = 1 XCD).
3.3.3.1.1. SPX mode (Tier 2)
For SPX mode (1 MPI rank per APU on a node), we chose we chose the problem size range such that the smallest problem allocates ~100,000 bytes of memory and the largest problem allocates ~400MB of memory. The largest size is about 1.5 times the MALL (Memory Attached Last-Level cache) size on the MI300A. The MALL is 256MB (256 * 1024 * 1024 = 268,435,456 bytes).
Because the Tier 2 kernels, INDEXLIST_3LOOP and HALO_PACKING_FUSED,
do not perform any floating point operations, they have zero GFLOP/sec rates.
So, we consider bandwidth as an appropriate metric for them.
After building the code as described in AMD MI300A architecture, we
ran the Tier 2 kernels in SPX mode using the following commands:
$ cd path/to/RAJAPerf
$ cd build_lc_toss4-cray-mpich-9.0.1-amdclang-6.4.3-gfx942
$ cp ../../RPB-v2026.10.0/scripts/run_tier_mi300a.sh .
$ ./run_tier_mi300a.sh spx tier2
This generates a directory named RPBenchmark_MI300A_tier2-SPX containing
output files for each kernel.
Then, we processed the data to generate FOM and throughput plots by running a Python script we provide:
$ cd path/to/RAJAPerf
$ cd build_lc_toss4-cray-mpich-9.0.1-amdclang-6.4.3-gfx942
$ cp ../../RPB-v2026.10.0/scripts/process_data.py .
$ python3 ./process_data.py --root-dir ./RPBenchmark_MI300A_tier2-SPX --output-dir ./RPBenchmark_MI300A_tier2-SPX/Output --exclude-plot-variant Base_Seq
This generates throughput curve files for Base_HIP and RAJA_HIP
variants of each kernel and summarizes the FOM (described in
Figure of Merit) in a CSV file. We include the files generated by the
process_data.py script in this repo in the directory
./docs/13_rajaperf/baseline_data/RPBenchmark_MI300A_tier2-SPX.
Kernel |
Sat Problem Size |
Sat GFLOP/s |
Sat B/W (GiB per sec.) |
|---|---|---|---|
Apps_CONVECTION3DPA-Base_HIP-block_64 |
351216.0 |
2170.44 |
1198.68 |
Apps_CONVECTION3DPA-RAJA_HIP-block_64 |
351216.0 |
2147.23 |
1185.86 |
Apps_DEL_DOT_VEC_2D-Base_HIP-block_256 |
1060900.0 |
2148.15 |
1780.63 |
Apps_DEL_DOT_VEC_2D-RAJA_HIP-block_256 |
264196.0 |
2025.11 |
1680.83 |
Apps_INTSC_HEXHEX-Base_HIP-block_64 |
125.0 |
21.7452 |
0.468793 |
Apps_INTSC_HEXHEX-RAJA_HIP-block_64 |
6859.0 |
88.4545 |
1.90694 |
Apps_LTIMES-Base_HIP-block_m_25 |
309696.0 |
350.901 |
488.049 |
Apps_LTIMES-Base_HIP-block_zgm_256 |
619392.0 |
645.836 |
898.235 |
Apps_LTIMES-RAJA_HIP-kernel_m_25 |
309696.0 |
341.375 |
474.8 |
Apps_LTIMES-RAJA_HIP-kernel_zgm_256 |
619392.0 |
528.943 |
735.659 |
Apps_LTIMES-RAJA_HIP-launch_m_25 |
309696.0 |
342.979 |
477.032 |
Apps_LTIMES-RAJA_HIP-launch_zgm_256 |
619392.0 |
645.634 |
897.955 |
Apps_MASS3DPA-Base_HIP-block_64 |
2133336.0 |
2318.07 |
2657.07 |
Apps_MASS3DPA-RAJA_HIP-block_64 |
2133336.0 |
2243.58 |
2571.69 |
Apps_MATVEC_3D_STENCIL-Base_HIP-block_256 |
157464.0 |
822.697 |
2036.54 |
Apps_MATVEC_3D_STENCIL-RAJA_HIP-block_256 |
157464.0 |
844.116 |
2089.56 |
Basic_INDEXLIST_3LOOP-Base_HIP-block_256 |
10240000.0 |
0.0 |
2414.32 |
Basic_INDEXLIST_3LOOP-RAJA_HIP-block_256 |
10240000.0 |
0.0 |
2383.95 |
Basic_MULTI_REDUCE-Base_HIP-atomic_direct_256 |
6399995.0 |
173.959 |
2592.19 |
Basic_MULTI_REDUCE-Base_HIP-atomic_occgs_256 |
9374995.0 |
195.342 |
2910.82 |
Basic_MULTI_REDUCE-RAJA_HIP-atomic_direct_256 |
6399995.0 |
163.844 |
2441.47 |
Basic_MULTI_REDUCE-RAJA_HIP-atomic_occgs_256 |
9374995.0 |
191.009 |
2846.26 |
Basic_REDUCE_STRUCT-Base_HIP-blkatm_direct_256 |
100000.0 |
2.81939 |
21.0058 |
Basic_REDUCE_STRUCT-Base_HIP-blkatm_occgs_256 |
18750000.0 |
153.387 |
1142.83 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkatm_direct_256 |
6400000.0 |
53.7546 |
400.503 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkatm_occgs_256 |
12800000.0 |
307.278 |
2289.4 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_direct_256 |
6400000.0 |
48.7602 |
363.292 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_direct_new_256 |
200000.0 |
2.80331 |
20.8862 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_occgs_256 |
9375000.0 |
258.939 |
1929.24 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_occgs_new_256 |
25000000.0 |
118.711 |
884.463 |
Comm_HALO_PACKING_FUSED-Base_HIP-direct_1024 |
15438249.0 |
0.0 |
766.329 |
Comm_HALO_PACKING_FUSED-RAJA_HIP-direct_1024 |
15438249.0 |
0.0 |
703.465 |
Comm_HALO_PACKING_FUSED-RAJA_HIP-funcptr_1024 |
19465109.0 |
0.0 |
795.311 |
Comm_HALO_PACKING_FUSED-RAJA_HIP-virtfunc_1024 |
19465109.0 |
0.0 |
770.866 |
3.3.3.1.2. CPX mode (Tier 2)
For CPX mode (run with 6 MPI ranks per APU on a node), we choose the smallest problem to allocate ~50,000 bytes of memory and the largest problem to allocate ~75MB of memory, which is slightly less than 1/3 the MALL size.
As noted earlier, two of the kernels INDEXLIST_3LOOP and HALO_PACKING_FUSED do not
perform any floating point operations. So, we consider the bandwidth
as the appropriate metric for them.
Similar to the SPX mode description above, we run the Tier 2 kernels in
CPX mode using the following commands:
$ cd path/to/RAJAPerf
$ cd build_lc_toss4-cray-mpich-9.0.1-amdclang-6.4.3-gfx942
$ cp ../../RPB-v2026.10.0/scripts/run_tier_mi300a.sh .
$ ./run_tier_mi300a.sh cpx tier2
This generates a directory named RPBenchmark_MI300A_tier2-CPX containing
output files for each kernel.
Then, we processed the data to generate FOM and throughput plots by running a Python script we provide:
$ cd path/to/RAJAPerf
$ cp ../../RPB-v2026.10.0/scripts/process_data.py .
$ python3 ./process_data.py --root-dir ./RPBenchmark_MI300A_tier2-CPX --output-dir ./RPBenchmark_MI300A_tier2-CPX/Output --exclude-plot-variant Base_Seq
This generates throughput curve files for Base_HIP and RAJA_HIP
variants of each kernel and summarizes the FOM (described in
Figure of Merit) in a CSV file. We include these files in this repo in the
directory ./docs/13_rajaperf/baseline_data/RPBenchmark_MI300A_tier2-CPX.
Kernel |
Sat Problem Size |
Sat GFLOP/s |
Sat B/W (GiB per sec.) |
|---|---|---|---|
Apps_CONVECTION3DPA-Base_HIP-block_64 |
21951.0 |
358.86 |
198.22 |
Apps_CONVECTION3DPA-RAJA_HIP-block_64 |
21951.0 |
350.698 |
193.711 |
Apps_DEL_DOT_VEC_2D-Base_HIP-block_256 |
32400.0 |
459.404 |
383.139 |
Apps_DEL_DOT_VEC_2D-RAJA_HIP-block_256 |
32400.0 |
466.122 |
388.742 |
Apps_INTSC_HEXHEX-Base_HIP-block_64 |
216.0 |
3.47517 |
0.074919 |
Apps_INTSC_HEXHEX-RAJA_HIP-block_64 |
512.0 |
13.0664 |
0.28169 |
Apps_LTIMES-Base_HIP-block_m_25 |
19392.0 |
64.9539 |
90.4112 |
Apps_LTIMES-Base_HIP-block_zgm_256 |
19392.0 |
107.632 |
149.816 |
Apps_LTIMES-RAJA_HIP-kernel_m_25 |
19392.0 |
62.1527 |
86.512 |
Apps_LTIMES-RAJA_HIP-kernel_zgm_256 |
19392.0 |
91.7377 |
127.692 |
Apps_LTIMES-RAJA_HIP-launch_m_25 |
19392.0 |
62.3861 |
86.837 |
Apps_LTIMES-RAJA_HIP-launch_zgm_256 |
19392.0 |
109.717 |
152.718 |
Apps_MASS3DPA-Base_HIP-block_64 |
66664.0 |
333.37 |
382.135 |
Apps_MASS3DPA-RAJA_HIP-block_64 |
133336.0 |
385.233 |
441.577 |
Apps_MATVEC_3D_STENCIL-Base_HIP-block_256 |
17576.0 |
172.271 |
442.234 |
Apps_MATVEC_3D_STENCIL-RAJA_HIP-block_256 |
17576.0 |
183.41 |
470.829 |
Basic_INDEXLIST_3LOOP-Base_HIP-block_256 |
640000.0 |
0.0 |
379.523 |
Basic_INDEXLIST_3LOOP-RAJA_HIP-block_256 |
640000.0 |
0.0 |
379.361 |
Basic_MULTI_REDUCE-Base_HIP-atomic_direct_256 |
799995.0 |
27.1483 |
404.546 |
Basic_MULTI_REDUCE-Base_HIP-atomic_occgs_256 |
1599995.0 |
32.0872 |
478.14 |
Basic_MULTI_REDUCE-RAJA_HIP-atomic_direct_256 |
799995.0 |
25.6092 |
381.611 |
Basic_MULTI_REDUCE-RAJA_HIP-atomic_occgs_256 |
1599995.0 |
30.565 |
455.456 |
Basic_REDUCE_STRUCT-Base_HIP-blkatm_direct_256 |
100000.0 |
1.0459 |
7.79252 |
Basic_REDUCE_STRUCT-Base_HIP-blkatm_occgs_256 |
4687500.0 |
36.3019 |
270.47 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkatm_direct_256 |
800000.0 |
8.57419 |
63.8826 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkatm_occgs_256 |
1600000.0 |
47.6719 |
355.183 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_direct_256 |
800000.0 |
7.61954 |
56.7699 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_direct_new_256 |
800000.0 |
5.60441 |
41.756 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_occgs_256 |
1600000.0 |
42.8746 |
319.44 |
Basic_REDUCE_STRUCT-RAJA_HIP-blkdev_occgs_new_256 |
1600000.0 |
41.344 |
308.037 |
Comm_HALO_PACKING_FUSED-Base_HIP-direct_1024 |
1860867.0 |
0.0 |
198.541 |
Comm_HALO_PACKING_FUSED-RAJA_HIP-direct_1024 |
1860867.0 |
0.0 |
183.357 |
Comm_HALO_PACKING_FUSED-RAJA_HIP-funcptr_1024 |
1860867.0 |
0.0 |
177.145 |
Comm_HALO_PACKING_FUSED-RAJA_HIP-virtfunc_1024 |
1860867.0 |
0.0 |
166.03 |
3.3.3.1.3. AMD MI300A throughput plots (Tier 2)
This section contains throughput plots for each Tier 2 kernel run as described above on the MI300A architecture in SPX mode (left column) and CPX mode (right column). Each plot has multiple curves with GFLOP/sec (compute rate), or memory bandwidth (GB/sec) for kernels with zero FLOPs, plotted as a function of problem size (allocated bytes). The legend in each plot indicates the curves shown.
Tier 2 Kernels: MI300A Node Throughput (SPX Mode) |
Tier 2 Kernels: MI300A Node Throughput (CPX Mode) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3.3.4. Baseline on NVIDIA H100
For NVIDIA GPUs, we are interested in the same measure of single-node throughput and FOM as described above for AMD MI300A.
3.3.4.1. NVIDIA H100 throughput results (Tier 2 kernels)
For the NVIDIA H100 architecture, we present one set of throughput results, where we ran with 4 MPI ranks on a node – one for each H100 GPU.
Similar to our methodology for the MI300A architecture, we ran each kernel and variant over a range of problem sizes to expose the saturation point. Specifically, we chose the smallest problem to allocate ~50,000 bytes of memory and the largest problem to allocate ~150MB of memory, which is about 3 times the L2 cache size on the H100 GPU. The L2 cache is 50MB (50 * 1024 * 1024 = 52,428,800 bytes).
After building the code as described in NVIDIA H100 architecture, we
run the Tier 2 kernels using the following commands:
$ cd path/to/RAJAPerf
$ cd build_lc_toss4-mvapich2-2.3.7-nvcc-13.1.1-90-gcc-10.3.1
$ cp ../../RPB-v2026.10.0/scripts/run_tier_h100.sh .
$ ./run_tier_h100.sh tier2
This generates a directory named RPBenchmark_H100_tier2 containing
results files for each kernel run.
Then, we processed the data to generate FOM and throughput plots by running a Python script we provide:
$ cd path/to/RAJAPerf
$ cd build_lc_toss4-mvapich2-2.3.7-nvcc-13.1.1-90-gcc-10.3.1
$ cp ../../RPB-v2026.10.0/scripts/process_data.py .
$ python3 ./process_data.py --root-dir ./RPBenchmark_H100_tier2 --output-dir ./RPBenchmark_H100_tier2/Output --exclude-plot-variant Base_Seq
This generates throughput curve files for Base_CUDA and RAJA_CUDA
variants of each kernel and summarizes the FOM (described in
Figure of Merit) in a CSV file. We include
the files generated by the process_data.py script in this repo in the
directory ./docs/13_rajaperf/baseline_data/RPBenchmark_H100_tier2.
Since the Tier2 kernels, INDEXLIST_3LOOP and HALO_PACKING_FUSED,
do not perform any floating point operations, they have zero GFLOP/sec rates.
We consider bandwidth as an appropriate metric for them.
We include the baseline data files for Tier 2 kernels run on the H100 architecture
in this repo in the directory ./docs/13_rajaperf/baseline_data/RPBenchmark_H100_tier2-SPX.
Kernel |
Sat Problem Size |
Sat GFLOP/s |
Sat B/W (GiB per sec.) |
|---|---|---|---|
Apps_CONVECTION3DPA-Base_CUDA-block_64 |
351216.0 |
2361.29 |
1304.08 |
Apps_CONVECTION3DPA-RAJA_CUDA-block_64 |
351216.0 |
2355.89 |
1301.1 |
Apps_DEL_DOT_VEC_2D-Base_CUDA-block_256 |
264196.0 |
2796.32 |
2320.92 |
Apps_DEL_DOT_VEC_2D-RAJA_CUDA-block_256 |
264196.0 |
2812.87 |
2334.66 |
Apps_INTSC_HEXHEX-Base_CUDA-block_64 |
3375.0 |
1817.06 |
39.1729 |
Apps_INTSC_HEXHEX-RAJA_CUDA-block_64 |
3375.0 |
1825.38 |
39.3522 |
Apps_LTIMES-Base_CUDA-block_m_25 |
309696.0 |
442.761 |
615.813 |
Apps_LTIMES-Base_CUDA-block_zgm_256 |
309696.0 |
662.663 |
921.664 |
Apps_LTIMES-RAJA_CUDA-kernel_m_25 |
309696.0 |
440.941 |
613.281 |
Apps_LTIMES-RAJA_CUDA-kernel_zgm_256 |
309696.0 |
653.44 |
908.836 |
Apps_LTIMES-RAJA_CUDA-launch_m_25 |
309696.0 |
440.682 |
612.921 |
Apps_LTIMES-RAJA_CUDA-launch_zgm_256 |
309696.0 |
663.125 |
922.305 |
Apps_MASS3DPA-Base_CUDA-block_64 |
1066664.0 |
1663.91 |
1907.25 |
Apps_MASS3DPA-RAJA_CUDA-block_64 |
1066664.0 |
1646.11 |
1886.85 |
Apps_MATVEC_3D_STENCIL-Base_CUDA-block_256 |
79507.0 |
779.224 |
1945.92 |
Apps_MATVEC_3D_STENCIL-RAJA_CUDA-block_256 |
157464.0 |
952.889 |
2358.82 |
Basic_INDEXLIST_3LOOP-Base_CUDA-block_256 |
2560000.0 |
0.0 |
2079.2 |
Basic_INDEXLIST_3LOOP-RAJA_CUDA-block_256 |
2560000.0 |
0.0 |
2144.01 |
Basic_MULTI_REDUCE-Base_CUDA-atomic_direct_256 |
18749995.0 |
16.728 |
249.267 |
Basic_MULTI_REDUCE-Base_CUDA-atomic_occgs_256 |
18749995.0 |
15.7544 |
234.759 |
Basic_MULTI_REDUCE-RAJA_CUDA-atomic_direct_256 |
18749995.0 |
16.7078 |
248.966 |
Basic_MULTI_REDUCE-RAJA_CUDA-atomic_occgs_256 |
18749995.0 |
15.5719 |
232.039 |
Basic_REDUCE_STRUCT-Base_CUDA-blkatm_direct_256 |
12500000.0 |
40.8389 |
304.273 |
Basic_REDUCE_STRUCT-Base_CUDA-blkatm_occgs_256 |
12500000.0 |
35.633 |
265.487 |
Basic_REDUCE_STRUCT-RAJA_CUDA-blkatm_direct_256 |
15625000.0 |
25.4149 |
189.356 |
Basic_REDUCE_STRUCT-RAJA_CUDA-blkatm_occgs_256 |
15625000.0 |
26.1203 |
194.612 |
Basic_REDUCE_STRUCT-RAJA_CUDA-blkdev_direct_256 |
800000.0 |
24.5622 |
183.003 |
Basic_REDUCE_STRUCT-RAJA_CUDA-blkdev_direct_new_256 |
3200000.0 |
33.1641 |
247.092 |
Basic_REDUCE_STRUCT-RAJA_CUDA-blkdev_occgs_256 |
12500000.0 |
274.847 |
2047.77 |
Basic_REDUCE_STRUCT-RAJA_CUDA-blkdev_occgs_new_256 |
12500000.0 |
266.783 |
1987.69 |
Comm_HALO_PACKING_FUSED-Base_CUDA-direct_1024 |
884736.0 |
0.0 |
78.8916 |
Comm_HALO_PACKING_FUSED-RAJA_CUDA-direct_1024 |
884736.0 |
0.0 |
77.6849 |
Comm_HALO_PACKING_FUSED-RAJA_CUDA-funcptr_1024 |
884736.0 |
0.0 |
73.1703 |
Comm_HALO_PACKING_FUSED-RAJA_CUDA-virtfunc_1024 |
884736.0 |
0.0 |
71.92 |
3.3.4.1.1. NVIDIA H100 throughput plots (Tier 2)
This section contains throughput plots for each Tier 2 kernel run as described above on the H100 architecture. Each plot has multiple curves with GFLOP/sec (compute rate), or memory bandwidth (GB/sec) for kernels with zero FLOPs, plotted as a function of problem size (allocated bytes). The legend in each plot indicates the curves shown.
Tier 2 Kernels H100 Node Throughput |
|
|
|
|
|
|
|
|
|
|