Readers API

Readers are responsible for loading and parsing Caliper performance files.

CaliReader

class CaliReader(path='', pool_size=10, inclusive_strings=None)

Primary reader for Caliper files with parallel processing support.

Parameters:

path (str or list[str]) – Path to Caliper files. Can be: - A directory path (loads all .cali files recursively) - A single file path - A list of directory/file paths
pool_size (int) – Number of parallel processes for loading files (default: 10)
inclusive_strings (list[str] or None) – Custom metric names to use instead of defaults. Must provide 4 strings: [min, max, avg, sum]

import treescape as ts

# Load from a directory
reader = ts.CaliReader("/path/to/cali/files")

# Load from multiple sources
reader = ts.CaliReader([
    "/path/to/dir1",
    "/path/to/dir2",
    "/path/to/file.cali"
])

# Custom metrics
reader = ts.CaliReader(
    path="/path/to/files",
    pool_size=8,
    inclusive_strings=[
        "min#inclusive#sum#my.timer",
        "max#inclusive#sum#my.timer",
        "avg#inclusive#sum#my.timer",
        "sum#inclusive#sum#my.timer"
    ]
)

Attributes:

inclusive_strings

List of metric names used for min, max, avg, and sum.

Default:

"min#inclusive#sum#time.duration"
"max#inclusive#sum#time.duration"
"avg#inclusive#sum#time.duration"
"sum#inclusive#sum#time.duration"

meta_globals: Dictionary mapping metadata keys to their types.

Methods:

get_entire()

Get the complete dataset including nodes, children map, and metadata.

Returns:: Dictionary with keys: - nodes: Performance data for each node - childrenMap: Parent-child relationships - meta_globals: Metadata type information
Return type:: dict

reader = ts.CaliReader("/path/to/files")
data = reader.get_entire()

print(data["nodes"].keys())        # All function names
print(data["childrenMap"]["main"]) # Children of main
print(data["meta_globals"])        # Metadata types

__iter__()

Iterate over drill levels (node names) in the dataset.

Returns:: Iterator yielding (node_name, data) tuples
Return type:: iterator

reader = ts.CaliReader("/path/to/files")

for node_name, data in reader:
    print(f"{node_name}: {data['xaxis']}")

ThicketReader

class ThicketReader(th_ens, profiles, xaxis)

Reader that uses the Thicket library for data processing.

Parameters:

th_ens (thicket.Thicket) – Thicket ensemble object
profiles (list[str]) – List of Caliper file paths
xaxis (str) – Metadata key to use as the x-axis

import treescape as ts
import thicket as tt

# Load with Thicket
profiles = ["/path/to/file1.cali", "/path/to/file2.cali"]
th_ens = tt.Thicket.from_caliperreader(profiles)

# Create reader
reader = ts.ThicketReader(
    th_ens=th_ens,
    profiles=profiles,
    xaxis="launchdate"
)

Methods:

get_entire()

Get the complete dataset in TreeScape format.

Returns:: Dictionary with nodes, childrenMap, parentMap, and meta_globals
Return type:: dict

get_entire_for_xaxis(xaxis_value)

Get data for a specific x-axis value.

Parameters:: xaxis_value (any) – The x-axis value to filter by
Returns:: List of node data dictionaries
Return type:: list[dict]

reader = ts.ThicketReader(th_ens, profiles, "launchdate")

# Get data for a specific date
data = reader.get_entire_for_xaxis(1609459200)

get_all_xaxis()

Get all unique x-axis values in the dataset.

Returns:: List of x-axis values
Return type:: list

get_all_xaxis_meta()

Get metadata for all runs.

Returns:: List of metadata dictionaries
Return type:: list[dict]

__iter__()

Iterate over x-axis values and their data.

Returns:: Iterator yielding (xaxis_value, metadata, node_data) tuples
Return type:: iterator

for xaxis_val, metadata, node_data in reader:
    print(f"X-axis: {xaxis_val}")
    print(f"Metadata: {metadata}")
    print(f"Nodes: {[n['name'] for n in node_data]}")

Reader Base Class

class Reader

Abstract base class for all readers.

All reader implementations must implement the get_entire() method.

get_entire(xaxis_name)

Abstract method to retrieve the complete dataset.

Parameters:: xaxis_name (str) – Name of the x-axis metadata field
Returns:: Complete dataset dictionary
Return type:: dict

Helper Class: TH_ens

class TH_ens

Helper class for managing Thicket ensemble creation.

get_th_ens(cali_files)

Create or retrieve a cached Thicket ensemble.

Parameters:: cali_files (str or list[str]) – Path(s) to Caliper files
Returns:: Tuple of (thicket_ensemble, profile_list)
Return type:: tuple

from treescape import TH_ens

th = TH_ens()
th_ens, profiles = th.get_th_ens("/path/to/cali/files")

Performance Considerations

CaliReader Performance:

pool_size: Set to match your CPU core count for optimal parallel loading
Multiprocessing: Files are loaded in parallel, significantly faster for large datasets
Per-file indexing: Maintains separate indices for efficient data access
Memory usage: Scales with number and size of Caliper files

ThicketReader Performance:

DataFrame operations: Uses pandas for efficient data manipulation
Vectorized aggregation: ~100x faster than row-by-row iteration
Memory overhead: Higher than CaliReader due to Thicket’s DataFrame structure
Best for: Integration with existing Thicket workflows

Choosing a Reader

Use CaliReader when:

You have many Caliper files (100s to 1000s)
You need maximum performance
You’re starting a new project
You don’t need Thicket’s DataFrame features

Use ThicketReader when:

You’re already using Thicket in your workflow
You need Thicket’s data manipulation features
You’re migrating from existing Thicket code
You want to leverage Thicket’s filtering/aggregation

Examples

Loading from multiple sources:

import treescape as ts

reader = ts.CaliReader([
    "/nightly/2024-01-01/",
    "/nightly/2024-01-02/",
    "/nightly/2024-01-03/specific_test.cali"
])

print(f"Loaded {len(reader.xy_idx_by_drill_level)} nodes")

Custom metrics:

reader = ts.CaliReader(
    path="/path/to/data",
    inclusive_strings=[
        "min#inclusive#sum#custom.metric",
        "max#inclusive#sum#custom.metric",
        "avg#inclusive#sum#custom.metric",
        "sum#inclusive#sum#custom.metric"
    ]
)

Iterating over data:

reader = ts.CaliReader("/path/to/data")

for node_name, node_data in reader:
    print(f"Node: {node_name}")
    print(f"  X-axis values: {node_data['xaxis']}")
    print(f"  Y-data points: {len(node_data['ydata'])}")

Using ThicketReader:

import thicket as tt
import treescape as ts

# Load with Thicket
profiles = ["/path/to/file1.cali", "/path/to/file2.cali"]
th_ens = tt.Thicket.from_caliperreader(profiles)

# Create reader
reader = ts.ThicketReader(th_ens, profiles, "problem_size")

# Get all problem sizes
sizes = reader.get_all_xaxis()
print(f"Problem sizes: {sorted(set(sizes))}")