Analyze a Scaling Study

The benchpark analyze command can be used to generate pre-defined charts for analysis of scaling studies using Caliper and Thicket. The study must use the Caliper modifier caliper=time,mpi (Benchpark Modifiers) to collect the performance data. benchpark analyze uses the Thicket performance analysis library to compose and visualize the Caliper performance data.

benchpark analyze will generate:

A calltree of the regions that were executed in the benchmark.
A scaling study figure, which can be configured with command-line arguments.
A csv of the datapoints plotted on the figure.

benchpark analyze requires additional Python packages, which can be installed by running pip install .[analyze] in the benchpark directory.

To run benchpark analyze:

$ benchpark analyze --workspace-dir RAMBLE_WORKSPACE_DIR <optional_arguments>

benchpark analyze has many options, explained in the help menu for the command.

Seaborn not found, so skipping imports of plotting in thicket.stats
To enable this plotting, install seaborn or thicket[plotting]
usage: main.py analyze [-h] --workspace-dir RAMBLE_WORKSPACE_DIR
                       [--calltree-unification {intersection,union}]
                       [--chart-type {raw,percentage}]
                       [--xaxis-parameter PARAM [PARAM ...]]
                       [--yaxis-metric YAXIS_METRIC]
                       [--filter-regions-name-prefix PREFIX]
                       [--top-n-regions N] [--group-regions-name] [--no-mpi]
                       [--chart-title CHART_TITLE]
                       [--chart-xlabel CHART_XLABEL]
                       [--chart-ylabel CHART_YLABEL]
                       [--chart-figsize CHART_FIGSIZE [CHART_FIGSIZE ...]]
                       [--chart-fontsize CHART_FONTSIZE]
                       [--chart-yaxis-limits YMIN YMAX]
                       [--file-name-match FILE_NAME_MATCH]
                       [--output-filename OUTPUT_FILENAME]
                       [--tree-metric TREE_METRIC]

options:
  -h, --help            show this help message and exit
  --workspace-dir RAMBLE_WORKSPACE_DIR
                        Directory of ramble workspace.
  --calltree-unification {intersection,union}
                        Type of unification operation to perform the Caliper
                        calltrees.
  --chart-type {raw,percentage}
                        Specify processing on the metric. 'raw' does nothing,
                        'percentage' shows the metric values as a percentage
                        relative to the total summation of all regions.
  --xaxis-parameter PARAM [PARAM ...]
                        One or more parameters from the metadata that are
                        varied during the experiment (values will become the
                        x-axis).
  --yaxis-metric YAXIS_METRIC
                        Performance metric to be visualized on the y-axis.
  --filter-regions-name-prefix PREFIX
                        Filter for region names starting with PREFIX to be
                        included in the chart.
  --top-n-regions N     Filters only top N largest metric entries to be
                        included in chart (based on the first profile).
  --group-regions-name  Whether to combine regions (sum of metric) with the
                        same name.
  --no-mpi              Hide MPI regions in the tree.
  --chart-title CHART_TITLE
                        Title of the output chart.
  --chart-xlabel CHART_XLABEL
                        X Label of chart.
  --chart-ylabel CHART_YLABEL
                        Y Label of chart.
  --chart-figsize CHART_FIGSIZE [CHART_FIGSIZE ...]
                        Size of the output chart (xdim, ydim). Ex: --chart-
                        figsize 12 6
  --chart-fontsize CHART_FONTSIZE
                        Font size of the output chart.
  --chart-yaxis-limits YMIN YMAX
                        Set both y-axis limits: --chart-yaxis-limits YMIN YMAX
  --file-name-match FILE_NAME_MATCH
                        Set optional cali file name to match. Useful if
                        multiple caliper files are generated per experiment
                        (e.g. RAJAPerf)
  --output-filename OUTPUT_FILENAME
                        Configure the output file names (the default value is
                        already unique to the workspace).
  --tree-metric TREE_METRIC
                        Metric to show on the tree output

Calltree

We will use one of the experiments available in Benchpark, Kripke, to demonstrate the output of the benchpark analyze command.

The output includes the Calltree of the benchmark:

 main
├─  Generate
│  ├─  MPI_Allreduce
│  ├─  MPI_Comm_split
│  └─  MPI_Scan
├─  MPI_Allreduce
├─  MPI_Bcast
├─  MPI_Comm_dup
├─  MPI_Comm_free
├─  MPI_Comm_split
├─  MPI_Finalize
├─  MPI_Finalized
├─  MPI_Gather
├─  MPI_Get_library_version
├─  MPI_Initialized
└─  Solve
  └─  solve
      ├─  LPlusTimes
      ├─  LTimes
      ├─  Population
      │  └─  MPI_Allreduce
      ├─  Scattering
      ├─  Source
      └─  SweepSolver
        ├─  MPI_Irecv
        ├─  MPI_Isend
        ├─  MPI_Testany
        ├─  MPI_Waitall
        └─  SweepSubdomain

Strong Scaling

To generate a strong scaling dataset, you would need to run the strong scaling scaling study of Kripke:

$ benchpark system init --dest=lassen llnl-sierra
$ benchpark experiment init --dest=kripke/cuda/strong kripke+cuda+strong~single_node caliper=time,mpi --system=lassen
$ benchpark setup kripke/cuda/strong lassen/ wkp
// Follow instructions for running Ramble ...

Run benchpark analyze:

$ benchpark analyze --workspace-dir wkp/kripke/cuda/strong/lassen/workspace/

The default configuration will visualize a stacked area chart of:

The exclusive average time per rank Avg time/rank (exc) collected by the caliper modifier caliper=time.
Number of nodes and number of MPI ranks on the x-axis, which are the parameters we vary in the experiment (for strong scaling).
Constant information will be contained in the title, such as benchmark, programming model, benchmark version, cluster, etc.
The legend will contain the regions and maximum calls out of all of the ranks (Calls/rank (max)).

Weak Scaling

To generate the weak scaling dataset:

$ benchpark experiment init --dest=kripke/cuda/weak kripke+cuda+weak~single_node caliper=time,mpi
$ benchpark setup kripke/cuda/weak lassen/ wkp
// Follow instructions for running Ramble ...

Run benchpark analyze:

$ benchpark analyze --workspace-dir wkp/kripke/cuda/weak/lassen/workspace/

Throughput

To generate the throughput dataset:

$ benchpark experiment init --dest=kripke/cuda/throughput kripke+cuda+throughput~single_node caliper=time,mpi
$ benchpark setup kripke/cuda/throughput lassen/ wkp
// Follow instructions for running Ramble ...

Run benchpark analyze:

$ benchpark analyze --workspace-dir wkp/kripke/cuda/throughput/lassen/workspace/

_images/kripke_cuda_throughput_raw_exc.png

Percentage

$ benchpark analyze --workspace-dir wkp/kripke/cuda/strong/lassen/workspace/ --chart-type percentage

The --chart-type percentage option, visualizes the y-axis metric, relative to the summation of the metric for all regions. This is useful in this example to visualize what percentage of the time each region is taking.

_images/kripke_cuda_strong_percentage_exc.png

Inclusive Metrics

$ benchpark analyze --workspace-dir wkp/kripke/cuda/strong/lassen/workspace/ --yaxis-metric "Avg time/rank"

We can also visualize any inclusive metrics by selecting them as the yaxis_metric. Here we use Avg time/rank instead of Avg time/rank (exc). The main node is automatically removed from the figure, because this information is redundant for the inclusive metric.

Region Filtering

$ benchpark analyze --workspace-dir wkp/kripke/cuda/strong/lassen/workspace/ --group-regions-name --top-n-regions 10

An efficient way to filter out smaller regions quickly, is to use the --group-regions-name and --top-n-regions parameters. --group-regions-name computes the sum of the metric for all of the regions with the same name, so multiple regions are shown as a single region. --top-n-regions filters the data to only show the n regions with the highest values for the given metric (based on the first profile). We can also add the --no-mpi argument to filter out all MPI_* regions.

_images/kripke_cuda_strong_raw_exc-2.png