2.2. Laghos

2.2.1. Purpose

Laghos (LAGrangian High-Order Solver) [Laghos] is a miniapp that solves the time-dependent Euler equations of compressible gas dynamics in a moving Lagrangian frame using unstructured high-order finite element spatial discretization and explicit high-order time-stepping. It is available at https://github.com/CEED/Laghos .

2.2.2. Characteristics

2.2.2.1. Problems

The test problems are the Sedov shock (problem 1 in Laghos) in 3D. The test problems should be run with a conforming mesh. Linear, quadratic, and cubic orders are of interest with the following priorities:

2.2.2.1.1. Priority 1 problems

1. 3D Linear This problem uses a kinematic order of 1, and a thermodynamic order of 0 (Q1Q0).

2. 3D Quadratic This problem uses a kinematic order of 2, and a thermodynamic order of 1 (Q2Q1).

2.2.2.1.2. Priority 2 problems

3. 3D Cubic This problem uses a kinematic order of 3, and a thermodynamics order of 2 (Q3Q2).

The problem sizes and partitioning scheme for both problems can be set by the user from the command line. The kinematic order is set using the -ok command-line option while the thermodynamic order is set using -ot. In all cases the default quadrature integration order should be used. See Running for how to run each of the different problems.

2.2.2.2. Figure of Merit

Each time step in Laghos contains 3 major distinct computations:

1. The inversion of the global kinematic mass matrix (CG H1).

2. The force operator evaluation from degrees of freedom to quadrature points (Forces).

3. The physics kernel in quadrature points (UpdateQuadData).

Laghos is instrumented to report the total execution times and rates, in terms of millions of degrees of freedom per second (megadofs), for each of these computational phases. (The time for inversion of the local thermodynamic mass matrices (CG L2) is also reported, but that takes a small part of the overall computation.) Rates are averaged over all RK stages taken and for the purposes of benchmarking are configured to attempt 250 RK4 timesteps.

Laghos also reports the total rate for these major kernels, which is the Figure of Merit (FOM) for benchmarking purposes.

2.2.3. Source code modifications

Please see Run Rules Synopsis for general guidance on allowed modifications.

For Laghos we define the following restrictions on source code modifications:

• Laghos must use MFEM and Hypre as the solver library, available at https://github.com/mfem/mfem and https://github.com/hypre-space/hypre respectively. Hypre must be built with HYPRE_ENABLE_MIXEDINT=ON. The final validated results must match or exceed the results shown in Validation.

• Laghos and MFEM must be built using RAJA, available at https://github.com/llnl/raja . Depending on system configuration RAJA can be built with “CUDA” or “HIP” device backends.

• The listed command line options for Laghos shown in Running must be used without modifications. Adding the appropriate options to enable GPU execution or GPU-aware MPI is ok. Users may configure MPI to launch Laghos as desired. Example: on El Capitan in order to use GPU-aware MPI the MPICH_GPU_SUPPORT_ENABLED environment variable must be set.

• Hypre/MFEM/Laghos must be built with Umpire (https://github.com/LLNL/Umpire). The host memory allocator may be changed to any of the ones available in mfem. The device memory allocators must use Umpire pool(s), though the allocation strategy may be changed; for example: separate Umpire pools for Hypre and MFEM, using managed memory pools, etc.

• LAGHOS_DEVICE_SYNC in laghos_solver.cpp must not be changed to get an accurate FOM.

• Code related to validating the Sedov solution must not be changed. These include sedov_sol.hpp, sedov_sol.cpp, bisect.hpp, adaptive_quad.hpp, and err_order in laghos.cpp. The Sedov solution must be computed using double precision regardless of the precision Laghos simulations are computed in.

2.2.4. Building

Prerequisites:

• CMake 3.24.0+

• C compiler

• C++20 compiler

• MPI

These instructions install all dependencies to a user-defined $INSTALLDIR using a user-defined $CC C compiler, $CXX C++-17 compiler, $NVCC CUDA compiler (for CUDA acceleration), and $HIPCC HIP compiler (for HIP acceleration). Both nvcc and clang are supported as the CUDA compiler.

The minimum required version of each library are:

• Umpire: v2026.07.2

• RAJA: v2026.07.1

• Hypre: v3.2.0

• MFEM: v4.10

• Laghos: v3.2

To review the available Spack variants for Laghos, use

spack info laghos

When Laghos is built using Spack, all the required libraries are built automatically. When building without Spack, follow the linked build instructions for Laghos’s required libraries: Umpire, RAJA, Hypre, and MFEM. Additionally, Adiak and Caliper are required when Caliper instrumentation is enabled.

Example of a GPU build:

With Spack:

spack install laghos@3.2+rocm

or

spack install laghos@3.2+cuda

To enable Caliper, use the variant +caliper

Without Spack:

git clone --branch v3.2 --recurse-submodules https://github.com/CEED/Laghos.git
cd Laghos
make -j \
  MFEM_DIR="${MFEM_DIR}" \
  CONFIG_MK="${MFEM_DIR}/share/mfem/config.mk" \
  TEST_MK="${MFEM_DIR}/share/mfem/test.mk" \
  LAGHOS_USE_CALIPER=ON \
  CALIPER_DIR="${CALIPER_DIR}" ADIAK_DIR="${ADIAK_DIR}"

2.2.5. Running

Benchmark runs should aim to maximize the FOM by choosing an appropriate -epm (element per mpi rank) value. To ensure domains are well shaped and minimize the impact of MPI communications it is recommended to set -epm to either a power of 2, or something of the form b*2^a, where b is small and a is large. Note that MFEM/Laghos currently does not support more than 32-bit local indexing so avoid choosing epm values which can result in more than 2^31-1 quadrature points per rank. The number of rank local quadrature points can be computed by multiplying the epm by the number of quadrature points per zone (tabulated below).

Table 2.2 Quadrature points per zone.

Order	QPts per zone
Q1Q0	8
Q2Q1	64
Q3Q2	216

Example run commands using -epm values to achieve 16 million quadrature points per rank are given below.

# 3D Q1Q0
laghos -dim 3 -p 1 -ok 1 -ot 0 -oq -1 -pa -no-nc -ms 250 -tf 100000 --mem --fom -epm 524288 -d raja-gpu
# 3D Q2Q1
laghos -dim 3 -p 1 -ok 2 -ot 1 -oq -1 -pa -no-nc -ms 250 -tf 100000 --mem --fom -epm 65536 -d raja-gpu
# 3D Q3Q2
laghos -dim 3 -p 1 -ok 3 -ot 2 -oq -1 -pa -no-nc -ms 250 -tf 100000 --mem --fom -epm 19418 -d raja-gpu

Available Laghos options for configuring the GPU compute and MPI:

• -dev for specifying which GPU to run on for a multi-GPU system (if not restricted by the job scheduler first).

• -gam for GPU-aware MPI

• -dev-pool-size for specifying an initial Umpire device memory pool size.

Any MPI scheduler options are allowed.

2.2.6. Validation

Code correctness is validated by using the following tests and comparing the outputted Energy diff, and Density L2 error. These quantities must be less than or equal to the following values (CPU and GPU cases should give a similar output):

# can be any power of 2 up to 64: [1,2,4,8,16,32,64]
nprocs_per_side=1
nranks=$((nprocs_per_side*nprocs_per_side*nprocs_per_side))
epm=$((262144/nranks))
# MPI run command can be modified as long as $nranks total ranks are used
mpirun -n $nranks laghos -dim 3 -p 1 -ok 1 -ot 0 -oq -1 -pa -no-nc -tf 0.6 -err -epm $epm
mpirun -n $nranks laghos -dim 3 -p 1 -ok 1 -ot 0 -oq -1 -pa -no-nc -tf 0.6 -err -d raja-gpu -epm $epm
Energy  diff: 7.61e-05
Density L2 error: 1.95e-01
# MPI run command can be modified as long as $nranks total ranks are used
mpirun -n $nranks laghos -dim 3 -p 1 -ok 2 -ot 1 -oq -1 -pa -no-nc -tf 0.6 -err -epm $epm
mpirun -n $nranks laghos -dim 3 -p 1 -ok 2 -ot 1 -oq -1 -pa -no-nc -tf 0.6 -err -d raja-gpu -epm $epm
Energy  diff: 3.46e-06
Density L2 error: 1.28e-01
# MPI run command can be modified as long as $nranks total ranks are used
mpirun -n $nranks laghos -dim 3 -p 1 -ok 3 -ot 2 -oq -1 -pa -no-nc -tf 0.6 -err -epm $epm
mpirun -n $nranks laghos -dim 3 -p 1 -ok 3 -ot 2 -oq -1 -pa -no-nc -tf 0.6 -err -d raja-gpu -epm $epm
Energy  diff: 8.82e-06
Density L2 error: 1.03e-01

The Density L2 error for other resolutions is shown in the following plot (for an N*N*N domain).

Fig. 2.1 Density L2 error for an NxNxN zone domain

2.2.7. Memory Usage

Total memory usage scales roughly proportional to the total number of DOFs. Both CPU and GPU memory usage can be outputted using the --mem option.

This will output the memory usage as (max rank CPU mem)/(total CPU mem) MB, (max rank GPU mem)/(total GPU mem) MB, where max rank CPU mem and max rank GPU mem are the maximum CPU and GPU memory usage of any single MPI rank respectively, while total CPU mem and total GPU mem are the total amount of CPU and GPU memory used by all ranks.

Sample CPU and GPU memory usage on a single El Capitan node are shown below.

Fig. 2.2 CPU memory use on El Capitan with 4 ranks on a single node

Fig. 2.3 GPU memory use on El Capitan with 4 ranks on a single node

Fig. 2.4 GPU memory use on El Capitan with 4 ranks on a single node per DOF

2.2.8. Baseline on El Capitan

Weak scaling performance of Laghos on El Capitan (AMD Epyc + AMD MI300A) for each order is provided as follows.

2.2.8.1. Linear (Q1Q0)

Nodes	GPUs	FOM (megadofs x time steps / second)	FOM/GPU
1	4	6112.468	1528.117
2	8	12254.224	1531.778
4	16	23914.144	1494.634
8	32	48594.592	1518.581
16	64	94537.728	1477.152
32	128	188198.784	1470.303
64	256	358770.944	1401.449
128	512	742596.608	1450.384
192	768	1104036.864	1437.548
256	1024	1447717.888	1413.787
384	1536	2145162.240	1396.590

Timings were captured using Caliper and are presented below.

2.2.8.2. Quadratic (Q2Q1)

Nodes	GPUs	FOM (megadofs x time steps / second)	FOM/GPU
1	4	11534.976	2883.744
2	8	23342.104	2917.763
4	16	42123.856	2632.741
8	32	89491.968	2796.624
16	64	181810.368	2840.787
32	128	364787.200	2849.900
64	256	714403.584	2790.639
128	512	1427020.800	2787.150
192	768	2142893.568	2790.226
256	1024	2852945.920	2786.080
384	1536	3909196.800	2545.050

Timings were captured using Caliper and are presented below.

2.2.8.3. Cubic (Q3Q2)

Nodes	GPUs	FOM (megadofs x time steps / second)	FOM/GPU
1	4	13965.396	3491.349
2	8	27201.888	3400.236
4	16	54138.960	3383.685
8	32	106450.304	3326.572
16	64	213633.216	3338.019
32	128	425203.968	3321.906
64	256	826212.608	3227.393
128	512	1678204.416	3277.743
192	768	2463618.048	3207.836
256	1024	3321225.216	3243.384
384	1536	4523349.504	2944.889

Timings were captured using Caliper and are presented below.

2.2.9. Performance Projection

The last row of each table in the above baselines captures the performance on a single Scalable Unit (SU) of El Capitan (384 nodes, 1536 GPUs).

To project Laghos performance for a Scalable Unit (SU), scale the number of MPI ranks to match the number of GPUs in the SU. There can be one or more MPI rank per GPU as long as each GPU has the same number of ranks. All other configuration options, including -epm, must remain unchanged.

Any -epm value which uses at least 10% of the memory available on each node of the SU may be used. See Running for recommendations for selecting an appropriate -epm value.

2.2.10. References

[Laghos]

22. Dobrev, Tz. Kolev and R. Rieben ‘High-order curvilinear finite element methods for Lagrangian hydrodynamics’, SIAM Journal on Scientific Computing, (34) 2012, pp. B606–B641. https://doi.org/10.1137/120864672