2.3.1. MPI: OMB

https://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-7.2.tar.gz

2.3.1.1. Purpose

The OSU Microbenchmarks (OMB) are widely used to measure and evaluate the performance of MPI operations for point-to-point, multi-pair, collective, and one-sided communications.

2.3.1.2. Characteristics

Official site: OSUMB

2.3.1.2.1. Problem

The OSU benchmarks are a suite of microbenchmarks designed to measure network characteristics on HPC systems.

2.3.1.3. Source code modifications

Please see Run Rules Synopsis for general guidance on allowed modifications.

2.3.1.4. Building

On GPU enabled systems add these flags to the following configure lines:

--enable-cuda
--with-cuda-include=/path/to/cuda/include
--with-cuda-libpath=/path/to/cuda/lib

Build and install the benchmarks.

./configure --prefix=$INSTALL_DIR
make -j
make -j install

Before configuring make sure your CXX and CC environment variables are set to an MPI compiler or wrapper. On most systems this will look like:

export CC=mpicc CXX=mpicxx

2.3.1.4.1. Special Considerations for Cray Systems

HPE-Cray systems may use custom wrappers:

export CC=cc CXX=CC

If using mpicc and mpicxx, note that you may need to add -lmpi_gtl_hsa or -lmpi_gtl_cuda at link time in order to get GPU-Aware Cray MPICH. If building the OSU benchmarks with Spack (recommended), then there is a convenience wrapper package located in the Phloem repository.

2.3.1.5. Running

Required: Project the output of osu_init for the total number of Host CPU Cores in your SU. You may use a low-noise configuration, but you must document that configuration and use it for ALL BENCHMARKS. For example, if you have 96 physical cores, but you have sequestered 12 of them for noisy system processes, then you would run with 84 processes per node:

srun -N 384 -n 32256 osu_init

Not Required: You are encouraged to use the OSU micro-benchmarks to compare against the output of Phloem as follows:

Latency and Bidirectional Bandwidth:

srun -N 1 -n 2 osu_latency
srun -N 1 -n 2 osu_latency D D
srun -N 1 -n 2 osu_bibw
srun -N 1 -n 2 osu_latency D D
srun -N 2 -n 2 -w host1,host2 osu_latency
srun -N 2 -n 2 -w host1,host2 osu_latency D D
srun -N 2 -n 2 -w host1,host2 osu_bibw
srun -N 2 -n 2 -w host1,host2 osu_latency D D
srun -N 2 -n 2 -w host1,host9001 osu_latency
srun -N 2 -n 2 -w host1,host9001 osu_latency D D
srun -N 2 -n 2 -w host1,host9001 osu_bibw
srun -N 2 -n 2 -w host1,host9001 osu_latency D D

Barrier

srun -N 384 -n 32256 osu_barrier
srun -N 384 -n 1536 osu_barrier -d openacc

Broadcast

srun -N 384 -n 32256 osu_bcast
srun -N 384 -n 1536 osu_bcast -d openacc

Allreduce

srun -N 384 -n 32256 osu_allreduce
srun -N 384 -n 1536 osu_allreduce -d openacc

Reduce

srun -N 384 -n 32256 osu_reduce
srun -N 384 -n 1536 osu_reduce -d openacc