Canned Analysis for Scaling Studies

The benchpark analyze command can be used to generate pre-configured charts for analysis of scaling studies using Caliper and Thicket. We use Thicket to help compose and visualize Caliper performance data collected from running our experiment with the Caliper modifier. After running, ramble on, run the benchpark analyze command on the ramble workspace directory.

Note

This command required optional packages to be installed, which can be achieved with pip install .[analyze] (assuming you are in the benchpark directory)

How to Run

$ benchpark analyze --workspace-dir RAMBLE_WORKSPACE_DIR <optional_arguments>

Available Arguments

Seaborn not found, so skipping imports of plotting in thicket.stats
To enable this plotting, install seaborn or thicket[plotting]
usage: main.py analyze [-h] --workspace-dir WORKSPACE_DIR
                       [--chart-type {percentage_time,time}]
                       [--x_axis-unique-metadata X_AXIS_UNIQUE_METADATA]
                       [--y-axis-metric Y_AXIS_METRIC]
                       [--filter-nodes-name-prefix FILTER_NODES_NAME_PREFIX]
                       [--group-nodes-name GROUP_NODES_NAME]
                       [--top-n-functions TOP_N_FUNCTIONS]
                       [--chart-title CHART_TITLE]
                       [--chart-xlabel CHART_XLABEL]
                       [--chart-ylabel CHART_YLABEL]
                       [--chart-figsize CHART_FIGSIZE [CHART_FIGSIZE ...]]
                       [--chart-fontsize CHART_FONTSIZE] [--no-mpi]

options:
  -h, --help            show this help message and exit
  --workspace-dir WORKSPACE_DIR
                        Directory of ramble workspace.
  --chart-type {percentage_time,time}
                        Specify type of output chart.
  --x_axis-unique-metadata X_AXIS_UNIQUE_METADATA
                        Parameter that is varied during the experiment.
  --y-axis-metric Y_AXIS_METRIC
                        Metric to be visualized.
  --filter-nodes-name-prefix FILTER_NODES_NAME_PREFIX
                        Optional: Filters only entries with prefix to be
                        included in chart.
  --group-nodes-name GROUP_NODES_NAME
                        Optional: Specify if nodes with the same name are
                        combined or not.
  --top-n-functions TOP_N_FUNCTIONS
                        Optional: Filters only top n longest time entries to
                        be included in chart.
  --chart-title CHART_TITLE
                        Optional: Title of the output chart.
  --chart-xlabel CHART_XLABEL
                        Optional: X Label of chart.
  --chart-ylabel CHART_YLABEL
                        Optional: Y Label of chart.
  --chart-figsize CHART_FIGSIZE [CHART_FIGSIZE ...]
                        Optional: Size of the output chart (xdim, ydim). Ex:
                        --chart-figsize 12 6
  --chart-fontsize CHART_FONTSIZE
                        Optional: Font size of the output chart.
  --no-mpi              Hide MPI regions in the tree.

Analysis of Strong, Weak, and Throughput Scaling of Kripke

Kripke Calltree

 main
├─  Generate
│  ├─  MPI_Allreduce
│  ├─  MPI_Comm_split
│  └─  MPI_Scan
├─  MPI_Allreduce
├─  MPI_Bcast
├─  MPI_Comm_dup
├─  MPI_Comm_free
├─  MPI_Comm_split
├─  MPI_Finalize
├─  MPI_Finalized
├─  MPI_Gather
├─  MPI_Get_library_version
├─  MPI_Initialized
└─  Solve
  └─  solve
      ├─  LPlusTimes
      ├─  LTimes
      ├─  Population
      │  └─  MPI_Allreduce
      ├─  Scattering
      ├─  Source
      └─  SweepSolver
        ├─  MPI_Irecv
        ├─  MPI_Isend
        ├─  MPI_Testany
        ├─  MPI_Waitall
        └─  SweepSubdomain

Strong

Generate the Strong dataset:

$ benchpark experiment init --dest=kripke/cuda/strong kripke+cuda+strong~single_node caliper=time,mpi
$ benchpark system init --dest=lassen llnl-sierra
$ benchpark setup kripke/cuda/strong lassen/ wkp
// Follow instructions for running Ramble ...

Run canned analysis:

$ benchpark analyze --workspace-dir wkp/kripke/cuda/strong/lassen/workspace/ --chart-type "percentage_time" --top-n-functions 10

_images/kripke_cuda_strong_percentage_time_exc.png

$ benchpark analyze --workspace-dir wkp/kripke/cuda/strong/lassen/workspace/ --chart-type "time" --top-n-functions 10

Weak

Generate the Weak dataset:

$ benchpark experiment init --dest=kripke/cuda/weak kripke+cuda+weak~single_node caliper=time,mpi
$ benchpark setup kripke/cuda/weak lassen/ wkp
// Follow instructions for running Ramble ...

Run canned analysis:

$ benchpark analyze --workspace-dir wkp/kripke/cuda/weak/lassen/workspace/ --chart-type "percentage_time" --top-n-functions 10

_images/kripke_cuda_weak_percentage_time_exc.png

$ benchpark analyze --workspace-dir wkp/kripke/cuda/weak/lassen/workspace/ --chart-type "time" --top-n-functions 10

Throughput

Generate the Throughput dataset:

$ benchpark experiment init --dest=kripke/cuda/throughput kripke+cuda+throughput~single_node caliper=time,mpi
$ benchpark setup kripke/cuda/throughput lassen/ wkp
// Follow instructions for running Ramble ...

Run canned analysis:

$ benchpark analyze --workspace-dir wkp/kripke/cuda/throughput/lassen/workspace/ --chart-type "percentage_time" --top-n-functions 10

_images/kripke_cuda_throughput_percentage_time_exc.png

$ benchpark analyze --workspace-dir wkp/kripke/cuda/throughput/lassen/workspace/ --chart-type "time" --top-n-functions 10

_images/kripke_cuda_throughput_time_exc.png

Inclusive Metrics

$ benchpark analyze --workspace-dir wkp/kripke/cuda/strong/lassen/workspace/ --chart-type "time" --y-axis-metric "Avg time/rank" --top-n-functions 10

We can also visualize any inclusive metrics by selecting them as the y_axis_metric. Here we use Avg time/rank instead of Avg time/rank (exc). The main node is automatically removed from the figure, because this information is redundant for the inclusive metric.