Analyze a Scaling Study

Glossary terms: Benchmark, Experiment, System, Modifier, Scaling Study

The benchpark analyze command can be used to generate pre-defined charts for analysis of scaling studies using Caliper and Thicket. The study must use the Caliper modifier caliper=time,mpi (Benchpark Modifiers) to collect the performance data. benchpark analyze uses the Thicket performance analysis library to compose and visualize the Caliper performance data.

benchpark analyze will generate:

  1. A calltree of the regions that were executed in the benchmark.

  2. A scaling study figure, which can be configured with command-line arguments.

  3. A csv of the datapoints plotted on the figure.

benchpark analyze requires additional Python packages, which can be installed by running pip install .[analyze] in the benchpark directory.

To run benchpark analyze:

$ benchpark analyze --workspace-dir RAMBLE_WORKSPACE_DIR <optional_arguments>

benchpark analyze has many options, explained in the help menu for the command.

usage: benchpark analyze [-h] --workspace-dir RAMBLE_WORKSPACE_DIR
                         [--calltree-unification {intersection,union}]
                         [--chart-type {raw,percentage}]
                         [--xaxis-parameter PARAM [PARAM ...]]
                         [--yaxis-metric YAXIS_METRIC]
                         [--filter-regions-byname PREFIX [PREFIX ...]]
                         [--query-regions-byname REGION [REGION ...]]
                         [--top-n-regions N] [--group-regions-name] [--no-mpi]
                         [--normalize-by COLUMN] [--chart-title CHART_TITLE]
                         [--chart-xlabel CHART_XLABEL]
                         [--chart-ylabel CHART_YLABEL]
                         [--chart-figsize CHART_FIGSIZE [CHART_FIGSIZE ...]]
                         [--chart-fontsize CHART_FONTSIZE]
                         [--chart-yaxis-limits YMIN YMAX]
                         [--file-name-match FILE_NAME_MATCH]
                         [--output-filename OUTPUT_FILENAME]
                         [--tree-metric TREE_METRIC]
                         [--chart-kind {area,line,bar,scatter}]
                         [--no-update-inc-cols] [--yaxis-log]
                         [--disable-legend]
                         [--exclude-regions EXCLUDE_REGIONS [EXCLUDE_REGIONS ...]]
                         [--dry-run] [--archive-output ARCHIVE_OUTPUT]
                         [{clean,archive}]

positional arguments:
  {clean,archive}       Optional trailing action to manage the workspace:
                        'clean' to remove contents, 'archive' to create a
                        tar.gz of the workspace. If omitted, performs
                        analysis.

options:
  -h, --help            show this help message and exit
  --workspace-dir RAMBLE_WORKSPACE_DIR
                        Directory Caliper files. Files will be found
                        recursively.
  --calltree-unification {intersection,union}
                        Type of unification operation to perform the Caliper
                        calltrees.
  --chart-type {raw,percentage}
                        Specify processing on the metric. 'raw' does nothing,
                        'percentage' shows the metric values as a percentage
                        relative to the total summation of all regions.
  --xaxis-parameter PARAM [PARAM ...]
                        One or more parameters from the metadata that are
                        varied during the experiment (values will become the
                        x-axis).
  --yaxis-metric YAXIS_METRIC
                        Performance metric to be visualized on the y-axis.
  --filter-regions-byname PREFIX [PREFIX ...]
                        Filter for region names starting with one or more
                        PREFIX values.
  --query-regions-byname REGION [REGION ...]
                        Query for one or more regions REGION. Includes
                        children of region.
  --top-n-regions N     Filters only top N largest metric entries to be
                        included in chart (based on the first profile).
  --group-regions-name  Whether to combine regions (sum of metric) with the
                        same name.
  --no-mpi              Hide MPI regions in the tree.
  --normalize-by COLUMN
                        Optionally normalize the y-axis column by this column.
  --chart-title CHART_TITLE
                        Title of the output chart.
  --chart-xlabel CHART_XLABEL
                        X Label of chart.
  --chart-ylabel CHART_YLABEL
                        Y Label of chart.
  --chart-figsize CHART_FIGSIZE [CHART_FIGSIZE ...]
                        Size of the output chart (xdim, ydim). Ex: --chart-
                        figsize 12 6
  --chart-fontsize CHART_FONTSIZE
                        Font size of the output chart.
  --chart-yaxis-limits YMIN YMAX
                        Set both y-axis limits: --chart-yaxis-limits YMIN YMAX
  --file-name-match FILE_NAME_MATCH
                        Set optional cali file name to match. Useful if
                        multiple caliper files are generated per experiment
                        (e.g. RAJAPerf)
  --output-filename OUTPUT_FILENAME
                        Configure the output file names (the default value is
                        already unique to the workspace).
  --tree-metric TREE_METRIC
                        Metric to show on the tree output
  --chart-kind {area,line,bar,scatter}
                        Type of chart to generate
  --no-update-inc-cols  Don't call Thicket.update_inclusive_columns() which
                        can take a while.
  --yaxis-log           Change yaxis to log base 2.
  --disable-legend      Turn off the legend on the figure
  --exclude-regions EXCLUDE_REGIONS [EXCLUDE_REGIONS ...]
                        One or more patterns to exclude based on region name
  --dry-run             With 'clean', show items that would be removed without
                        deleting.
  --archive-output ARCHIVE_OUTPUT
                        With 'archive', path for the .tar.gz (defaults to
                        CWD/<workspace>-<timestamp>.tar.gz)

Calltree

We will use one of the experiments available in Benchpark, Kripke, to demonstrate the output of the benchpark analyze command.

The output includes the Calltree of the benchmark:

 main
├─  Generate
│  ├─  MPI_Allreduce
│  ├─  MPI_Comm_split
│  └─  MPI_Scan
├─  MPI_Allreduce
├─  MPI_Bcast
├─  MPI_Comm_dup
├─  MPI_Comm_free
├─  MPI_Comm_split
├─  MPI_Finalize
├─  MPI_Finalized
├─  MPI_Gather
├─  MPI_Get_library_version
├─  MPI_Initialized
└─  Solve
  └─  solve
      ├─  LPlusTimes
      ├─  LTimes
      ├─  Population
      │  └─  MPI_Allreduce
      ├─  Scattering
      ├─  Source
      └─  SweepSolver
        ├─  MPI_Irecv
        ├─  MPI_Isend
        ├─  MPI_Testany
        ├─  MPI_Waitall
        └─  SweepSubdomain

Strong Scaling

To generate a strong scaling dataset, you would need to run the strong scaling scaling study of Kripke:

$ benchpark system init --dest=lassen llnl-sierra
$ benchpark experiment init --dest=kripke/cuda/strong lassen kripke+cuda+strong caliper=time,mpi
$ benchpark setup lassen/kripke/cuda/strong wkp
// Follow instructions for running Ramble ...

Run benchpark analyze:

$ benchpark analyze --workspace-dir wkp/kripke/cuda/strong/lassen/workspace/
_images/kripke_cuda_strong_raw_exc.png

The default configuration will visualize a stacked area chart of:

  • The exclusive average time per rank Avg time/rank (exc) collected by the caliper modifier caliper=time.

  • Number of nodes and number of MPI ranks on the x-axis, which are the parameters we vary in the experiment (for strong scaling).

  • Constant information will be contained in the title, such as benchmark, programming model, benchmark version, cluster, etc.

  • The legend will contain the regions and maximum calls out of all of the ranks (Calls/rank (max)).

Weak Scaling

To generate the weak scaling dataset:

$ benchpark experiment init --dest=kripke/cuda/weak lassen kripke+cuda+weak caliper=time,mpi
$ benchpark setup lassen/kripke/cuda/weak wkp
// Follow instructions for running Ramble ...

Run benchpark analyze:

$ benchpark analyze --workspace-dir wkp/kripke/cuda/weak/lassen/workspace/
_images/kripke_cuda_weak_raw_exc.png

Throughput

To generate the throughput dataset:

$ benchpark experiment init --dest=kripke/cuda/throughput lassen kripke+cuda+throughput caliper=time,mpi
$ benchpark setup lassen/kripke/cuda/throughput wkp
// Follow instructions for running Ramble ...

Run benchpark analyze:

$ benchpark analyze --workspace-dir wkp/kripke/cuda/throughput/lassen/workspace/
_images/kripke_cuda_throughput_raw_exc.png

Percentage

$ benchpark analyze --workspace-dir wkp/kripke/cuda/strong/lassen/workspace/ --chart-type percentage

The --chart-type percentage option, visualizes the y-axis metric, relative to the summation of the metric for all regions. This is useful in this example to visualize what percentage of the time each region is taking.

_images/kripke_cuda_strong_percentage_exc.png

Inclusive Metrics

$ benchpark analyze --workspace-dir wkp/kripke/cuda/strong/lassen/workspace/ --yaxis-metric "Avg time/rank"

We can also visualize any inclusive metrics by selecting them as the yaxis_metric. Here we use Avg time/rank instead of Avg time/rank (exc). The main node is automatically removed from the figure, because this information is redundant for the inclusive metric.

_images/kripke_cuda_strong_raw_inc.png

Region Filtering

$ benchpark analyze --workspace-dir wkp/kripke/cuda/strong/lassen/workspace/ --group-regions-name --top-n-regions 10

An efficient way to filter out smaller regions quickly, is to use the --group-regions-name and --top-n-regions parameters. --group-regions-name computes the sum of the metric for all of the regions with the same name, so multiple regions are shown as a single region. --top-n-regions filters the data to only show the n regions with the highest values for the given metric (based on the first profile). We can also add the --no-mpi argument to filter out all MPI_* regions.

_images/kripke_cuda_strong_raw_exc-2.png

Visualize Data From Multiple Workspaces

Data from multiple clusters will end up in separate Ramble workspaces. Simply point at the Benchpark workspace instead of the Ramble workspace to include multiple Ramble workspaces in your analysis. This example uses the line chart functionality to visualize a single node memory bandwidth study. Other options are bar and scatter.

Note

The area chart will not work for data from multiple Ramble workspaces.

$ benchpark analyze --workspace-dir wkp/ --query-regions-byname Stream_TRIAD --chart-kind line --file-name-match Base_Seq-default --yaxis-metric 'Memory Bandwidth (GB/s)' --chart-yaxis-limits 8 2048 --chart-figsize 12 7 --yaxis-log --no-mpi
_images/raja-perf_mpi_strong_raw_exc.png

Visualize a Metadata Column

benchpark analyze is not limited to performance data columns. Provide the name of a metadata column to visualize that instead. This is useful for metrics like FOM’s, which only have one value per profile.

$ benchpark analyze --workspace-dir problem1/ --yaxis-metric Final-FOM --chart-kind line --disable-legend
_images/amg2023_rocm_weak_raw_exc.png

Choose Your Next Step