com

LLNL MPI Communication Benchmark - Point-to-point tests

Manual section:: 1
Manual group:: Benchmarking Tools
Version:: UCRL-CODE-2001-028
Date:: 2026

SYNOPSIS

com [options] [tests types]

DESCRIPTION

com is a comprehensive MPI point-to-point communication benchmark developed at Lawrence Livermore National Laboratory (LLNL). It measures bandwidth and latency for various communication patterns between MPI ranks.

The benchmark supports multiple test types including unidirectional and bidirectional communication, both blocking and non-blocking, as well as pure latency measurements.

OPTIONS

-a

Print individual test resuts.

Default: Off

-b bytes

Message start size.

Default: 0

-c integer

Samples

Default: 3

-d file path

Task/rank target list source file. The file shall have 2 columns, representing the pairing of the communicating ranks.

-e bytes

Message stop size.

Default: 32 MiB

-f file path

Process count source file.

-h

Print use information.

-i

Print process pairs for each measurement. NOTE: Not for use at scale.

Default: Off

-k

Print individual rank times. NOTE: Not for use at scale.

Default: Off

-l

Print hostname information. NOTE: Not for use at scale.

Default: Off

-m file path

Message size source file: A single-column file of integers representing the message sizes to test in bytes.

-n

Use barrier within measurement.

Default: Off

-o

Number of operations between measurements.

Default: 0

-p [c|b]

Allocate processes: c(yclic) or b(lock)

Default: Block

-q

Print test names.

Default: TODO

-r

Partner processes with nearby rank.

Default: Off

-s directory

A directory containing files representing rank pair mappings.

-t integer

Processes per SMP

Default: TODO

-v

Verify message data.

Default: Off

-w

List of full test names.

-x

Calculate bandwidth by volume/longest task time.

Default: Off

TEST TYPES

The benchmark implements five distinct communication patterns:

UNIDIR - Unidirectional Blocking

One-way communication using blocking MPI_Send and MPI_Recv.

Sender posts MPI_Send
Receiver posts MPI_Recv
Measures unidirectional bandwidth
Blocking operations ensure completion before proceeding

UNIDIRNB - Unidirectional Non-Blocking

One-way communication using non-blocking MPI_Isend and MPI_Irecv.

Sender posts MPI_Isend
Receiver posts MPI_Irecv
Uses MPI_Wait or MPI_Test for completion
Allows potential overlap with computation

BIDIR - Bidirectional Blocking

Simultaneous two-way communication using blocking operations.

Both ranks send and receive simultaneously
Uses MPI_Sendrecv or paired MPI_Send/MPI_Recv
Measures aggregate bidirectional bandwidth
Tests full-duplex network capability

BIDIRNB - Bidirectional Non-Blocking

Simultaneous two-way communication using non-blocking operations.

Both ranks post MPI_Isend and MPI_Irecv
Completion via MPI_Waitall
Measures bidirectional bandwidth with overlap potential
Tests network’s ability to handle concurrent operations

LATEN - Latency Test

Ping-pong latency measurement.

Measures round-trip time for small messages
Typically uses zero-byte or small messages
Reports half round-trip time as latency
Critical for understanding communication overhead

ALGORITHM

Test Execution Flow

For each test type and message size:

Initialization: Allocate buffers and setup communicators
Warmup: Execute several iterations to warm caches
Timing: Measure specified number of iterations
Aggregation: Collect results from all ranks
Reporting: Calculate and print bandwidth/latency statistics

Bandwidth Calculation

bandwidth (MB/s) = (message_size × iterations × send_factor) / (time × 1048576)

Where:

send_factor = 1 for unidirectional, 2 for bidirectional
time is measured in seconds
Result is in MiB/s (1 MiB = 1048576 bytes)

Latency Calculation

latency (μs) = (round_trip_time / 2) × 1000000

USAGE

Basic Execution

Run all tests with default parameters:

srun -N 2 -n 2 com

Run specific test type:

srun -N 2 -n 2 com --test BIDIR

Run with custom message sizes:

srun -N 2 -n 2 com --min-size 1024 --max-size 1048576

Rank Placement

Two ranks on different nodes (recommended for inter-node testing):

# Slurm
srun -N 2 -n 2 --ntasks-per-node=1 com

# OpenMPI
mpirun -n 1 -host node1 : -n 1 -host node2 com

Two ranks on same node (intra-node testing):

# Slurm
srun -N 1 -n 2 --ntasks-per-node=2 com

# OpenMPI
mpirun -n 2 -host node1 com

OPTIONS

Test Selection

–test TYPE: Run specific test type: UNIDIR, UNIDIRNB, BIDIR, BIDIRNB, LATEN
–list-tests: Display available test types and exit

Message Size Control

–min-size bytes: Minimum message size to test. Default: 0
–max-size bytes: Maximum message size to test. Default: 4194304 (4 MiB)
–message-list size1,size2,…: Comma-separated list of specific message sizes to test

Iteration Control

–iterations N

Number of iterations per test. Default: 1000

Larger values improve timing accuracy but increase runtime

–warmup N

Number of warmup iterations. Default: 10

Output Control

–verbose: Enable verbose output with detailed timing information
–quiet: Suppress progress messages, only show results
–output file: Write results to specified file instead of stdout

Environment Information

–print-env: Display MPI environment variables and system information
–help: Display usage information and exit

EXAMPLES

Basic Bandwidth Test

Measure bidirectional bandwidth between two nodes:

srun -N 2 -n 2 --ntasks-per-node=1 com --test BIDIR

Latency Measurement

Measure latency with small messages:

mpirun -n 2 com --test LATEN --max-size 1024

Message Size Sweep

Test specific message sizes:

mpirun -n 2 com --message-list 1,4,16,64,256,1024,4096

Compare Blocking vs Non-Blocking

Run both unidirectional tests:

mpirun -n 2 com --test UNIDIR > blocking.txt
mpirun -n 2 com --test UNIDIRNB > nonblocking.txt

Full Test Suite

Run all tests with extended message sizes:

srun -N 2 -n 2 com --min-size 1 --max-size 16777216 --iterations 10000

OUTPUT

Header Information

LLNL MPI Communication Benchmark
UCRL-CODE-2001-028

Test Configuration:
  Ranks: 2
  Test Type: BIDIR
  Iterations: 1000
  Message Size Range: 0 - 4194304 bytes

Test Results

Test: Bidirectional Blocking (BIDIR)

Size(B)    Bandwidth(MB/s)    Latency(μs)    Iterations
-------    ---------------    -----------    ----------
      0              0.00           1.23          1000
      1             12.34           2.45          1000
      2             24.56           3.21          1000
      4             48.23           4.12          1000
   1024           1234.56          10.23          1000
   4096           4567.89          15.67          1000
1048576          12345.67         100.45          1000
4194304          23456.78         200.12          1000

Maximum Bandwidth: 23456.78 MB/s at 4194304 bytes

Column Descriptions

Size(B): Message size in bytes
Bandwidth(MB/s): Measured bandwidth in MiB/s (1 MiB = 1048576 bytes)
Latency(μs): Message latency in microseconds (for latency tests)
Iterations: Number of iterations performed for this message size
Maximum Bandwidth: Peak bandwidth achieved and corresponding message size

INTERPRETATION

Performance Metrics

Bandwidth

Measures data transfer rate
Higher is better
Typically increases with message size
Plateaus at network hardware limit

Latency

Measures communication overhead
Lower is better
Relatively constant for small messages
Increases with message size

Communication Patterns

Unidirectional vs Bidirectional

Unidirectional: Tests one-way bandwidth
Bidirectional: Tests full-duplex capability
Bidirectional should approach 2× unidirectional on full-duplex networks

Blocking vs Non-Blocking

Blocking: Simpler, guaranteed completion
Non-blocking: Potential for overlap, may show higher bandwidth
Difference indicates overlap capability

Scaling Characteristics

Small Messages (< 1 KB)

Latency-dominated
Performance limited by software overhead
MPI implementation efficiency critical

Medium Messages (1 KB - 64 KB)

Transition region
Protocol switching may occur
Both latency and bandwidth matter

Large Messages (> 64 KB)

Bandwidth-dominated
Performance approaches hardware limits
Network hardware capability critical

Common Patterns

Good Performance

Smooth bandwidth increase with message size
Bidirectional ≈ 2× unidirectional
Low latency for small messages
Bandwidth approaches theoretical maximum

Performance Issues

Sudden drops at specific message sizes (protocol changes)
Bidirectional << 2× unidirectional (half-duplex or contention)
High latency (software overhead or network issues)
Bandwidth plateau well below hardware capability

TYPICAL USE CASES

Network Characterization

Understand network performance characteristics:

# Test all patterns
for test in UNIDIR UNIDIRNB BIDIR BIDIRNB LATEN; do
    srun -N 2 -n 2 com --test $test > ${test}.txt
done

Acceptance Testing

Verify network meets specifications:

srun -N 2 -n 2 com --test BIDIR --min-size 1048576 --max-size 4194304

Compare MPI Implementations

Test different MPI libraries:

# OpenMPI
module load openmpi
mpirun -n 2 com > openmpi_results.txt

# MPICH
module load mpich
mpirun -n 2 com > mpich_results.txt

Regression Testing

Detect performance changes:

# Baseline
srun -N 2 -n 2 com > baseline.txt

# After changes
srun -N 2 -n 2 com > new.txt

# Compare
diff baseline.txt new.txt

Topology Analysis

Test different node pairs:

# Adjacent nodes
srun --nodelist=node[1-2] -n 2 com > adjacent.txt

# Distant nodes
srun --nodelist=node1,node100 -n 2 com > distant.txt

LIMITATIONS

Two-rank only: Only tests communication between two ranks
Point-to-point only: Does not test collective operations
No multi-pair: Cannot saturate network with multiple concurrent pairs
Fixed patterns: Limited to predefined communication patterns
No GPU support: CPU memory only (no GPU-aware MPI testing)

DIAGNOSTICS

Common Issues

“Error: Requires exactly 2 MPI ranks”

Must run with exactly 2 ranks. Adjust -n parameter.

Very low bandwidth

Check rank placement (same node vs different nodes)
Verify network connectivity
Check for system interference

High variance in results

Increase iteration count
Reduce system load
Check for network contention

Unexpected latency spikes

System interference (check dmesg)
Network congestion
NUMA effects

Performance Debugging

Compare intra-node vs inter-node:

# Intra-node (should be very fast)
srun -N 1 -n 2 com --test BIDIR

# Inter-node (network speed)
srun -N 2 -n 2 com --test BIDIR

Check for protocol switching:

# Look for sudden bandwidth changes
com --message-list 1,2,4,8,16,32,64,128,256,512,1024,2048,4096,8192

Verify full-duplex operation:

# Bidirectional should be ~2× unidirectional
com --test UNIDIR > uni.txt
com --test BIDIR > bi.txt

ENVIRONMENT

MPI Implementation Variables

OMPI_*, MPICH_*, I_MPI_*: MPI implementation-specific tuning parameters
OMPI_MCA_btl: OpenMPI transport layer selection
MPICH_ASYNC_PROGRESS: Enable asynchronous progress (MPICH)

System Variables

OMP_NUM_THREADS: Should be set to 1 for pure MPI benchmarks
CUDA_VISIBLE_DEVICES: Control GPU visibility (if GPU-aware MPI)

AUTHORS

Developed at Lawrence Livermore National Laboratory (LLNL) by Chris Chambreau and Nathan Hanford.

COPYRIGHT

Produced at Lawrence Livermore National Laboratory (LLNL)

This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.

UCRL-CODE-2001-028

LICENSE

This work was prepared as an account of work sponsored by an agency of the United States government.

This software is licensed under the MIT License.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

See the LICENSE file in the source distribution for full license text.

BUGS

Report bugs to: https://github.com/llnl/phloem/issues

NOTES

Verbose Output at Scale

Verbose output is only for debugging rank placement at smaller scales. Running verbose output at system scale will generally overwhelm stdout redirection, which is generally handled by the workload manager.

com

LLNL MPI Communication Benchmark - Point-to-point tests

SYNOPSIS

DESCRIPTION

OPTIONS

TEST TYPES

UNIDIR - Unidirectional Blocking

UNIDIRNB - Unidirectional Non-Blocking

BIDIR - Bidirectional Blocking

BIDIRNB - Bidirectional Non-Blocking

LATEN - Latency Test

ALGORITHM

Test Execution Flow

Bandwidth Calculation

Latency Calculation

USAGE

Basic Execution

Rank Placement

OPTIONS

Test Selection

Message Size Control

Iteration Control

Output Control

Environment Information

EXAMPLES

Basic Bandwidth Test

Latency Measurement

Message Size Sweep

Compare Blocking vs Non-Blocking

Full Test Suite

OUTPUT

Header Information

Test Results

Column Descriptions

INTERPRETATION

Performance Metrics

Communication Patterns

Scaling Characteristics

Common Patterns

TYPICAL USE CASES

Network Characterization

Acceptance Testing

Compare MPI Implementations

Regression Testing

Topology Analysis

LIMITATIONS

DIAGNOSTICS

Common Issues

Performance Debugging

ENVIRONMENT

MPI Implementation Variables

System Variables

SEE ALSO

AUTHORS

COPYRIGHT

LICENSE

BUGS

NOTES