Annotation API reference¶

Caliper provides source-code annotation APIs to mark and name code regions or features of interest. The high-level annotation API provides easy access to pre-defined annotation types. The low-level annotation API lets users create their own context attributes to capture custom code features.

High-level Annotations¶

Caliper provides pre-defined annotation types for functions, loops, and user-defined code regions, and a set of C and C++ macros to mark and name those entities.

Function Annotations¶

Function annotations export the name of the annotated function, which is taken from the __FUNC__ compiler macro. The annotations should be placed within a function of interest, ideally near the top. In C, the function begin and all exit points must be marked. In C++, only the begin has to be marked. Caliper exports the hierarchy of annotated functions in the function attribute.

C++ example:

#include <caliper/cali.h>

int foo(int r)
{
  CALI_CXX_MARK_FUNCTION; // Exports "function=foo"

  if (r > 0)
    return r-1;

  return r;
  // In C++, the function annotation is closed automatically
}

C example:

#include <caliper/cali.h>

int foo(int r)
{
  CALI_MARK_FUNCTION;       /* Exports "function=foo"          */

  if (r > 0) {
    CALI_MARK_FUNCTION_END; /* ALL exit points must be marked! */
    return r-1;
  }

  CALI_MARK_FUNCTION_END;   /* ALL exit points must be marked! */
  return r;
}

Loop Annotations¶

Loop annotations mark and name loops. The annotations should be placed immediately outside a loop of interest. Both begin and end of the loop have to be marked. Optionally, the loop iterations themselves can be marked and the iteration number exported by placing loop iteration annotations inside the loop body. Caliper exports the loop name in the loop attribute. Loop iterations are exported in per-loop attributes named iteration#name, where name is the user-provided loop name given to the loop annotation.

C++ example:

#include <caliper/cali.h>

CALI_CXX_MARK_LOOP_BEGIN(myloop_id, "myloop"); // exports loop=myloop
for (int i = 0; i < ITER; ++i) {
  CALI_CXX_MARK_LOOP_ITERATION(myloop_id, i);  // exports iteration#myloop=<i>

  // In C++, the iteration annotation is closed automatically
}
CALI_CXX_MARK_LOOP_END(myloop_id);

C example:

#include <caliper/cali.h>

CALI_MARK_LOOP_BEGIN(myloop_id, "myloop"); // exports loop=myloop
for (int i = 0; i < ITER; ++i) {
  CALI_MARK_ITERATION_BEGIN(myloop_id, i); // exports iteration#myloop=<i>

  if (test(i) == 0) {
    CALI_MARK_ITERATION_END(myloop_id);    // must mark ALL iteration exit points
    break;
  }

  CALI_MARK_ITERATION_END(myloop_id);      // must mark ALL iteration exit points
}
CALI_MARK_LOOP_END(myloop_id);

Code-region Annotations¶

Code-region annotations mark and name user-defined source code regions. Caliper exports the region names in the annotation attribute. Annotated code regions must be properly nested (see Nesting).

Example:

#include <caliper/cali.h>

CALI_MARK_BEGIN("Important code"); // exports annotation="Important code"
// ... important code
CALI_MARK_END("Important code");

Nesting¶

Annotated source-code regions of any of the pre-defined context attributes (function, loop, and annotation) can be nested within each other. Caliper preserves this nesting information. Users must ensure that the nesting is correct. That is, annotated code regions have to be enclosed completely within each other; they cannot partially overlap. Example:

#include <caliper/cali.h>

int foo()
{
  CALI_CXX_MARK_FUNCTION;

  CALI_MARK_BEGIN("outer");
  CALI_MARK_BEGIN("inner"); // The hierarchy is now "foo/outer/inner"

#if 0
  CALI_MARK_END("outer");   // Error! Must end "inner" before "outer"!
  CALI_MARK_END("inner");
#endif

  CALI_MARK_END("inner");
  CALI_MARK_END("outer");   // Correct nesting
}

To annotate arbitrary features or regions that can overlap with others, use the low-level annotation API and create a user-defined context attribute.

Low-level Annotation API¶

The “low-level” annotation API lets users create and export additional context attributes with user-defined names to capture custom code features or annotate arbitrary, non-nested code regions.

Attribute keys¶

Context attributes are the basic element in Caliper’s key:value data model. The high-level annotation API uses pre-defined attribute keys, but users can create their own. Attribute keys have a unique name, and store the attribute’s data type as well as optional property flags. Property flags control how the Caliper runtime system handles the associated attributes.

enum cali_attr_properties¶

Attribute property flags.

These flags control how the caliper runtime system handles the associated attributes. Flags can be combined with a bitwise OR (however, the scope flags are mutually exclusive).

Values:

enumerator CALI_ATTR_DEFAULT¶: Default value.

enumerator CALI_ATTR_ASVALUE¶

Store directly as key:value pair, not in the context tree.

Entries with this property will be not be put into the context tree, but stored directly as key:value pairs on the blackboard and in snapshot records. ASVALUE attributes cannot be nested. Only applicable to scalar data types.

enumerator CALI_ATTR_NOMERGE¶

Create a separate context tree root node for this attribute.

Useful for attributes that form overlapping hierarchies separate from the main region stack.

enumerator CALI_ATTR_SCOPE_PROCESS¶: Process-scope attribute. Shared between all threads.

enumerator CALI_ATTR_SCOPE_THREAD¶: Thread-scope attribute.

enumerator CALI_ATTR_SCOPE_TASK¶: Task-scope attribute. Currently unused.

enumerator CALI_ATTR_SKIP_EVENTS¶: Skip event callbacks for blackboard updates with this attribute.

enumerator CALI_ATTR_HIDDEN¶: Do not include this attribute in snapshots.

enumerator CALI_ATTR_NESTED¶

Begin/end calls are properly aligned with the call stack.

Indicates that begin/end calls for this attribute are correctly nested with the call stack and other NESTED attributes. That is, an active region of a NESTED attribute does not partially overlap function calls or other NESTED attribute regions.

enumerator CALI_ATTR_GLOBAL¶

A metadata attribute describing global information for a measurement run.

Global attributes represent metadata associated with an application run (e.g., application executable name and version, start date and time, and so on). They may be written in a separate metadata section in some output formats. For distributed programs (e.g. MPI), global attributes should have the same value on each process.

enumerator CALI_ATTR_UNALIGNED¶

This attribute is not aligned with stacked begin/end regions.

Entries with this property may still be merged into a single context tree branch, but one that is separate from the properly nested region branch. Stack nesting checks are skipped.

enumerator CALI_ATTR_AGGREGATABLE¶

This attribute is aggregatable (i.e., a metric).

Attributes with this flag will automatically be aggregated by the aggregate service. This flag replaces the previous class.aggregatable meta-attribute.

Attribute keys can be created with cali_create_attribute():

cali_id_t cali_create_attribute(const char *name, cali_attr_type type, int properties)¶

Create an attribute key.

Parameters:

name – Name of the attribute
type – Type of the attribute
properties – Attribute properties

Returns:

Attribute id

C annotation API¶

The C annotation API provides the cali_begin_*[byname], cali_end[_byname], and cali_set_*_[byname] family of functions. Their behavior is as follows:

begin: Marks the begin of a region with the given attribute/value. The new value will be nested under already open regions of the same attribute.
set: Sets or overwrites the top-most value for the given attribute.
end: Closes the top-most open region for the given attribute.

The byname variants refer to attribute keys by their name. If no attribute key with the given name exists, it will be created with default properties. The basic variants take an attribute ID, e.g. from cali_create_attribute().

Example:

#include <caliper/cali.h>

/* Exports CustomAttribute="My great example" */
cali_begin_string_byname("CustomAttribute", "My great example")

/* Creates attribute key "myvar" with ASVALUE storage property */
cali_id_t myvar_attr =
  cali_create_attribute("myvar", CALI_TYPE_INT, CALI_ATTR_ASVALUE);

/* Exports myvar=42 */
cali_set_int(myvar_attr, 42);

/* Closes CustomAttribute="My great example" */
cali_end_byname("CustomAttribute");

C++ annotation API¶

The C++ annotation API is implemented in the class cali::Annotation.

Data tracking API¶

Caliper also supports tracking allocated data. Doing so provides advanced data-centric attributes, such as recording allocation events and determining the containers for memory addresses provided by services like libpfm. To take advantage of annotated memory allocations, the alloc service must be enabled at runtime.

Memory allocation annotations are similar to code region annotations, we can define labels for allocations using macros, which will use the variable name of the given pointer as label for the memory allocations. We can either label 1-dimensional ranges of bytes using CALI_DATATRACKER_TRACK or multi-dimensional ranges of specified element sizes using CALI_DATATRACKER_TRACK_DIMENSIONAL. The following example shows both:

void do_work(size_t M, size_t W, size_t N)
{
    double *arrayA = (double*)malloc(N);
    CALI_DATATRACKER_TRACK(arrayA, N);

    double *matA =
         (double*)malloc(sizeof(double)*M*W);

    size_t num_dimensions = 2;
    size_t A_dims[] = {M,W};
    CALI_DATATRACKER_TRACK_DIMENSIONAL(
                matA,
                sizeof(double),
                A_dims,
                num_dimensions);
        ...

    CALI_DATATRACKER_FREE(arrayA);
    CALI_DATATRACKER_FREE(matA);
}

API Reference¶

group AnnotationAPI

The user-facing source-code annotation API.

Low-level source-code annotation API

void cali_begin_region(const char *name)¶

Begin nested region name.

Begins nested region name using the built-in annotation attribute. Equivalent to the macro CALI_MARK_REGION_BEGIN.

See also

cali_end_region()

void cali_end_region(const char *name)¶

End nested region name.

Ends nested region name using the built-in annotation attribute. Prints an error if name does not match the currently open region. Equivalent to the macro CALI_MARK_REGION_END.

See also

cali_begin_region()

void cali_begin(cali_id_t attr)¶: Begin region where the value for attr is true on the blackboard.

void cali_begin_double(cali_id_t attr, double val)¶

Begin region val for attribute attr on the blackboard.

The region may be nested.

void cali_begin_int(cali_id_t attr, int val)¶

Begin region val for attribute attr on the blackboard.

The region may be nested.

void cali_begin_string(cali_id_t attr, const char *val)¶

Begin region val for attribute attr on the blackboard.

The region may be nested.

void cali_end(cali_id_t attr)¶: End innermost open region for attr on the blackboard.

void cali_set(cali_id_t attr, const void *value, size_t size)¶: Set value for attribute attr to val on the blackboard.

void cali_set_double(cali_id_t attr, double val)¶: Set value for attribute attr to val on the blackboard.

void cali_set_int(cali_id_t attr, int val)¶: Set value for attribute attr to val on the blackboard.

void cali_set_string(cali_id_t attr, const char *val)¶: Set value for attribute attr to val on the blackboard.

void cali_begin_byname(const char *attr_name)¶

Begin region where the value for the attribute named attr_name is set to true on the blackboard.

Deprecated:: Use cali_begin_region()

void cali_begin_double_byname(const char *attr_name, double val)¶

Begin region val for attribute named attr_name on the blackboard.

The region may be nested.

void cali_begin_int_byname(const char *attr_name, int val)¶

Begin region val for attribute named attr_name on the blackboard.

The region may be nested.

void cali_begin_string_byname(const char *attr_name, const char *val)¶

Begin region val for attribute named attr_name on the blackboard.

The region may be nested.

void cali_set_double_byname(const char *attr_name, double val)¶: Set value for attribute named attr_name to val on the blackboard.

void cali_set_int_byname(const char *attr_name, int val)¶: Set value for attribute named attr_name to val on the blackboard.

void cali_set_string_byname(const char *attr_name, const char *val)¶: Set value for attribute named attr_name to val on the blackboard.

void cali_end_byname(const char *attr_name)¶: End innermost open region for attribute named attr_name on the blackboard.

void cali_set_global_double_byname(const char *attr_name, double val)¶: Set a global attribute with name attr_name to val.

void cali_set_global_int_byname(const char *attr_name, int val)¶: Set a global attribute with name attr_name to val.

void cali_set_global_string_byname(const char *attr_name, const char *val)¶: Set a global attribute with name attr_name to val.

void cali_set_global_uint_byname(const char *attr_name, uint64_t val)¶: Set a global attribute with name attr_name to val.

Defines

CALI_DATATRACKER_TRACK(ptr, size)¶

Label and track the given memory region.

Labels the memory region of size size bytes pointed to by ptr with the variable name of ptr. The memory region can then be tracked and resolved with the alloc service.

CALI_DATATRACKER_TRACK_DIMENSIONAL(ptr, elem_size, dimensions, num_dimensions)¶

Label and track a multi-dimensional array.

Labels a multi-dimensional array with with the variable name of ptr. The array can then be tracked and resolved with the alloc service.

Parameters:

elem_size – Size of the array elements in bytes
dimension – Array of type size_t[] with the sizes of each dimension. Must have num_dimensions entries.
num_dimensions – The number of dimensions

CALI_DATATRACKER_UNTRACK(ptr)¶: Stop tracking the memory region pointed to by ptr.

CALI_CXX_MARK_FUNCTION¶

C++ macro to mark a function.

Mark begin and end of a function. Should be placed at the top of the function, and will automatically “close” the function at any return point. Will export the annotated function by name in the pre-defined function attribute. Only available in C++.

CALI_CXX_MARK_SCOPE(name)¶

C++ macro marking a scoped region.

Mark begin and end of a C++ scope. Should be placed at the top of the scope, and will automatically “close” the function at any return point. Will export the annotated function by name in the pre-defined annotation attribute. Only available in C++.

CALI_CXX_MARK_LOOP_BEGIN(loop_id, name)¶

Mark loop in C++.

Mark begin of a loop. Will export the user-provided loop name in the pre-defined loop attribute. This macro should be placed before the loop of interest, and CALI_MARK_LOOP_END should be placed after the loop.

Parameters:

loop_id – A loop identifier. Needed to refer to the loop from the iteration and end annotations.
name – Name of the loop.

CALI_CXX_MARK_LOOP_END(loop_id)¶

Mark loop end in C++.

Mark end of a loop. Will export the user-provided loop name in the pre-defined loop attribute.

This macro should be placed after the loop of interest. Users must ensure proper begin/end matching: If the surrounding function can be exited from within the loop (e.g., from a return statement within the loop), an exit marker must be placed there as well.

Parameters:

loop_id – The loop identifier given to CALI_MARK_LOOP_BEGIN.

CALI_CXX_MARK_LOOP_ITERATION(loop_id, iter)¶

C++ macro for a loop iteration.

Create a C++ annotation for a loop iteration. The loop must be marked with CALI_CXX_MARK_LOOP_BEGIN and CALI_CXX_MARK_LOOP_END. This will export the loop’s iteration count given in iter in an attribute named iteration#name, where name is the loop name given in CALI_CXX_MARK_LOOP_BEGIN. The macro should be placed at the beginning of the loop block. The annotation will be closed automatically. Example:

CALI_CXX_MARK_LOOP_BEGIN(mainloop_id, "mainloop");
for (int i = 0; i < ITER; ++i) {
  CALI_CXX_MARK_LOOP_ITERATION(mainloop_id, i);
  // ...
}
CALI_CXX_MARK_LOOP_END(mainloop_id);

Parameters:

loop_id – The loop identifier given to CALI_CXX_MARK_LOOP_BEGIN
iter – The iteration number. Must be convertible to int.

CALI_MARK_FUNCTION_BEGIN¶

Mark begin of a function.

Exports the annotated function’s name in the pre=defined function attribute. A CALI_MARK_FUNCTION_END marker must be placed at all function exit points. For C++, we recommend using CALI_CXX_MARK_FUNCTION instead.

CALI_MARK_FUNCTION_END¶

Mark end of a function.

Must be placed at all exit points of a function marked with CALI_MARK_FUNCTION_BEGIN.

See also

CALI_MARK_FUNCTION_BEGIN

CALI_MARK_LOOP_BEGIN(loop_id, name)¶

Mark a loop.

Mark begin of a loop. Will export the user-provided loop name in the pre-defined loop attribute. This macro should be placed before the loop of interest, and CALI_MARK_LOOP_END should be placed after the loop.

Parameters:

loop_id – A loop identifier. Needed to refer to the loop from the iteration and end annotations.
name – Name of the loop.

CALI_MARK_LOOP_END(loop_id)¶

Mark a loop.

Mark end of a loop. Will export the user-provided loop name in the pre-defined loop attribute.

This macro should be placed after the loop of interest. Users must ensure proper begin/end matching: If the surrounding function can be exited from within the loop (e.g., from a return statement within the loop), an exit marker must be placed there as well.

Parameters:

loop_id – The loop identifier given to CALI_MARK_LOOP_BEGIN.

CALI_MARK_ITERATION_BEGIN(loop_id, iter)¶

Mark begin of a loop iteration.

This annotation should be placed at the top inside of the loop body. The loop must be annotated with CALI_MARK_LOOP_BEGIN. The iteration number will be exported in the attribute iteration::name, where name is the loop name given to CALI_MARK_LOOP_BEGIN. In C++, we recommend using CALI_CXX_MARK_LOOP_ITERATION.

Parameters:

loop_id – Loop identifier, must match the identifier given to CALI_MARK_LOOP_BEGIN.
iter – Current iteration number. This macro argument must be convertible to int.

CALI_MARK_ITERATION_END(loop_id)¶

Mark end of a loop iteration.

This annotation should be placed at the end inside of the loop body. If an iteration can be left prematurely (e.g., from a continue, break, or return statement), an end marker must be placed there as well.

See also

CALI_MARK_ITERATION_BEGIN

Parameters:

loop_id – Loop identifier given in CALI_MARK_LOOP_BEGIN.

CALI_WRAP_STATEMENT(name, statement)¶

Wrap Caliper annotations around a C/C++ statement.

The wrapped statement will be annotated with the given name in the statement attribute. Example

double res;
/* Wrap the sqrt() call */
CALI_WRAP_STATEMENT( "sqrt", res = sqrt(49) );

Parameters:

name – The user-defined region name. Must be convertible into a const char*.
statement – C/C++ statement(s) that should be wrapped. The statements must complete within the wrapped region, that is, they cannot branch out of the macro (e.g. with goto, continue, break, or return).

CALI_MARK_BEGIN(name)¶

Mark begin of a user-defined code region.

This annotation should be placed before a code region of interest. The user-provided region name will be exported in the pre-defined annotation attribute.

Users must ensure proper nesting: Each CALI_MARK_BEGIN must be matched by a corresponding CALI_MARK_END in the correct order. Regions may be nested within another, but they cannot overlap partially.

See also

CALI_MARK_END

Parameters:

name – The region name. Must be convertible to const char*.

CALI_MARK_END(name)¶

Mark end of a user-defined code region.

This annotation should be placed after a code region of interest that has been annotated with CALI_MARK_BEGIN.

See also

CALI_MARK_BEGIN

Parameters:

name – The region name given to CALI_MARK_BEGIN. The macro will check if the name matches, and report an error if it doesn’t.

Functions

void *cali_datatracker_allocate(const char *label, size_t size)¶

Allocate and track the resulting allocation in Caliper.

Note

Tracking information will be automatically removed if the allocation is freed

Parameters:

label – The unique label with which to track the allocation requested
size – The total size of the allocation requested

Returns:

A new allocation of size size

void *cali_datatracker_allocate_dimensional(const char *label, size_t elem_size, const size_t *dimensions, size_t num_dimensions)¶

Allocate and track the resulting allocation, with dimensions, in Caliper.

Note

Tracking information will be automatically removed if the allocation is freed

Parameters:

label – The unique label with which to track the allocation requested
elem_size – The size of an individual element in the allocation requested
dimensions – An array of the dimensions of the allocation requested
num_dimensions – The number of dimensions in dimensions

Returns:

A new allocation of size product(dimensions)*elem_size

void cali_datatracker_free(void *ptr)¶

Free and untrack an allocation in Caliper.

Parameters:: ptr – The pointer to the beginning of the allocation to free/untrack

void cali_datatracker_track(const void *ptr, const char *label, size_t size)¶

Track an existing allocation in Caliper.

Note

This will track the entire allocation pointed to by ptr. To track a subset of a larger allocation, use cali_datatracker_track_dimensional.

Parameters:

ptr – The pointer to the beginning of the allocation to track
label – The unique label with which to track the allocation
size – Size of the allocation

void cali_datatracker_track_dimensional(const void *ptr, const char *label, size_t elem_size, const size_t *dimensions, size_t num_dimensions)¶

Track an existing allocation in Caliper.

Parameters:

ptr – The pointer to the beginning of the allocation to track
label – The unique label with which to track the allocation
elem_size – The size of an individual element in the allocation
dimensions – An array of the dimensions of the allocation
num_dimensions – The number of dimensions in dimensions

void cali_datatracker_untrack(const void *ptr)¶

Untrack a previously tracked allocation in Caliper.

Parameters:: ptr – The pointer to the beginning of the allocation to untrack

Variables

cali_id_t cali_loop_attr_id¶

cali_id_t cali_annotation_attr_id¶

class Function¶: #include <Annotation.h>

Pre-defined function annotation class.

class ScopeAnnotation¶: #include <Annotation.h>

Pre-defined region annotation class. Region begins and ends with object construction and destruction.

class Loop¶

#include <Annotation.h>

Pre-defined loop annotation class, with optional iteration attribute.

class Iteration¶

class Annotation¶

#include <Annotation.h>

Instrumentation interface to add and manipulate context attributes.

The Annotation class is the primary source-code instrumentation interface for Caliper. Annotation objects provide access to named Caliper context attributes. If the referenced attribute does not exist yet, it will be created automatically.

Example:

cali::Annotation phase_ann("myprogram.phase");

phase_ann.begin("Initialization");
  // ...
phase_ann.end();

This example creates an annotation object for the myprogram.phase attribute, and uses the begin()/end() methods to mark a section of code where that attribute is set to “Initialization”.

Note

Access to the underlying named context attribute through Annotation objects is not exclusive: multiple Annotation objects can reference and update the same context attribute.

begin() overloads

Annotation &begin(int data)¶

Begin name=data region for the associated context attribute.