Pre-Defined Analysis for Scaling Studies

The benchpark analyze command can be used to generate pre-defined charts for analysis of scaling studies using Caliper and Thicket. We generate Caliper performance data using the Caliper modifier caliper=time,mpi (Benchpark Modifiers). We use the Thicket performance analysis library to compose and visualize the Caliper performance data. Thicket is embedded into the benchpark analyze command, leveraging metadata to generate the scaling study figures.

After running, ramble on, run the benchpark analyze command on the ramble workspace directory to generate:

A calltree of the regions that were executed in the benchmark.
A scaling study figure, which can be configured with command-line arguments.
A csv of the datapoints plotted on the figure.

Note

This command required optional packages to be installed, which can be achieved with pip install .[analyze] (assuming you are in the benchpark directory)

How to Run

$ benchpark analyze --workspace-dir RAMBLE_WORKSPACE_DIR <optional_arguments>

Available Arguments

Seaborn not found, so skipping imports of plotting in thicket.stats
To enable this plotting, install seaborn or thicket[plotting]
usage: main.py analyze [-h] --workspace-dir RAMBLE_WORKSPACE_DIR
                       [--calltree-unification {intersection,union}]
                       [--chart-type {raw,percentage}]
                       [--xaxis-parameter PARAM [PARAM ...]]
                       [--yaxis-metric YAXIS_METRIC]
                       [--filter-regions-name-prefix PREFIX]
                       [--top-n-regions N] [--group-regions-name] [--no-mpi]
                       [--chart-title CHART_TITLE]
                       [--chart-xlabel CHART_XLABEL]
                       [--chart-ylabel CHART_YLABEL]
                       [--chart-figsize CHART_FIGSIZE [CHART_FIGSIZE ...]]
                       [--chart-fontsize CHART_FONTSIZE]
                       [--chart-yaxis-limits YMIN YMAX]
                       [--file-name-match FILE_NAME_MATCH]

options:
  -h, --help            show this help message and exit
  --workspace-dir RAMBLE_WORKSPACE_DIR
                        Directory of ramble workspace.
  --calltree-unification {intersection,union}
                        Type of unification operation to perform the Caliper
                        calltrees.
  --chart-type {raw,percentage}
                        Specify processing on the metric. 'raw' does nothing,
                        'percentage' shows the metric values as a percentage
                        relative to the total summation of all regions.
  --xaxis-parameter PARAM [PARAM ...]
                        One or more parameters from the metadata that are
                        varied during the experiment (values will become the
                        x-axis).
  --yaxis-metric YAXIS_METRIC
                        Performance metric to be visualized on the y-axis.
  --filter-regions-name-prefix PREFIX
                        Filter for region names starting with PREFIX to be
                        included in the chart.
  --top-n-regions N     Filters only top N largest metric entries to be
                        included in chart (based on the first profile).
  --group-regions-name  Whether to combine regions (sum of metric) with the
                        same name.
  --no-mpi              Hide MPI regions in the tree.
  --chart-title CHART_TITLE
                        Title of the output chart.
  --chart-xlabel CHART_XLABEL
                        X Label of chart.
  --chart-ylabel CHART_YLABEL
                        Y Label of chart.
  --chart-figsize CHART_FIGSIZE [CHART_FIGSIZE ...]
                        Size of the output chart (xdim, ydim). Ex: --chart-
                        figsize 12 6
  --chart-fontsize CHART_FONTSIZE
                        Font size of the output chart.
  --chart-yaxis-limits YMIN YMAX
                        Set both y-axis limits: --chart-yaxis-limits YMIN YMAX
  --file-name-match FILE_NAME_MATCH
                        Set optional cali file name to match. Useful if
                        multiple caliper files are generated per experiment
                        (e.g. RAJAPerf)

Analysis of Strong, Weak, and Throughput Scaling of Kripke

Kripke Calltree

 main
├─  Generate
│  ├─  MPI_Allreduce
│  ├─  MPI_Comm_split
│  └─  MPI_Scan
├─  MPI_Allreduce
├─  MPI_Bcast
├─  MPI_Comm_dup
├─  MPI_Comm_free
├─  MPI_Comm_split
├─  MPI_Finalize
├─  MPI_Finalized
├─  MPI_Gather
├─  MPI_Get_library_version
├─  MPI_Initialized
└─  Solve
  └─  solve
      ├─  LPlusTimes
      ├─  LTimes
      ├─  Population
      │  └─  MPI_Allreduce
      ├─  Scattering
      ├─  Source
      └─  SweepSolver
        ├─  MPI_Irecv
        ├─  MPI_Isend
        ├─  MPI_Testany
        ├─  MPI_Waitall
        └─  SweepSubdomain

Strong

Generate the Strong dataset:

$ benchpark experiment init --dest=kripke/cuda/strong kripke+cuda+strong~single_node caliper=time,mpi
$ benchpark system init --dest=lassen llnl-sierra
$ benchpark setup kripke/cuda/strong lassen/ wkp
// Follow instructions for running Ramble ...

Run pre-defined analysis:

$ benchpark analyze --workspace-dir wkp/kripke/cuda/strong/lassen/workspace/

The default configuration will visualize a stacked area chart of:

The exclusive average time per rank Avg time/rank (exc) collected by the caliper modifier caliper=time.
Number of nodes and number of MPI ranks on the x-axis, which are the parameters we vary in the experiment (for strong scaling).
Constant information will be contained in the title, such as benchmark, programming model, benchmark version, cluster, etc.
The legend will contain the regions and maximum calls out of all of the ranks (Calls/rank (max)).

Weak

Generate the Weak dataset:

$ benchpark experiment init --dest=kripke/cuda/weak kripke+cuda+weak~single_node caliper=time,mpi
$ benchpark setup kripke/cuda/weak lassen/ wkp
// Follow instructions for running Ramble ...

Run pre-defined analysis:

$ benchpark analyze --workspace-dir wkp/kripke/cuda/weak/lassen/workspace/

Throughput

Generate the Throughput dataset:

$ benchpark experiment init --dest=kripke/cuda/throughput kripke+cuda+throughput~single_node caliper=time,mpi
$ benchpark setup kripke/cuda/throughput lassen/ wkp
// Follow instructions for running Ramble ...

Run pre-defined analysis:

$ benchpark analyze --workspace-dir wkp/kripke/cuda/throughput/lassen/workspace/

_images/kripke_cuda_throughput_raw_exc.png

Percentage

$ benchpark analyze --workspace-dir wkp/kripke/cuda/strong/lassen/workspace/ --chart-type percentage

The --chart-type percentage option, visualizes the y-axis metric, relative to the summation of the metric for all regions. This is useful in this example to visualize what percentage of the time each region is taking.

_images/kripke_cuda_strong_percentage_exc.png

Inclusive Metrics

$ benchpark analyze --workspace-dir wkp/kripke/cuda/strong/lassen/workspace/ --yaxis-metric "Avg time/rank"

We can also visualize any inclusive metrics by selecting them as the yaxis_metric. Here we use Avg time/rank instead of Avg time/rank (exc). The main node is automatically removed from the figure, because this information is redundant for the inclusive metric.

Arguments for Region Filtering

$ benchpark analyze --workspace-dir wkp/kripke/cuda/strong/lassen/workspace/ --group-regions-name --top-n-regions 10

An efficient way to filter out smaller regions quickly, is to use the --group-regions-name and --top-n-regions parameters. --group-regions-name computes the sum of the metric for all of the regions with the same name, so multiple regions are shown as a single region. --top-n-regions filters the data to only show the n regions with the highest values for the given metric (based on the first profile). We can also add the --no-mpi argument to filter out all MPI_* regions.

_images/kripke_cuda_strong_raw_exc-2.png