Annotation API reference

Caliper provides source-code annotation APIs to mark and name code regions or features of interest. The high-level annotation API provides easy access to pre-defined annotation types. The low-level annotation API lets users create their own context attributes to capture custom code features.

High-level Annotations

Caliper provides pre-defined annotation types for functions, loops, and user-defined code regions, and a set of C and C++ macros to mark and name those entities.

Function Annotations

Function annotations export the name of the annotated function, which is taken from the __FUNC__ compiler macro. The annotations should be placed within a function of interest, ideally near the top. In C, the function begin and all exit points must be marked. In C++, only the begin has to be marked. Caliper exports the hierarchy of annotated functions in the function attribute.

C++ example:

#include <caliper/cali.h>

int foo(int r)
{
  CALI_CXX_MARK_FUNCTION; // Exports "function=foo"

  if (r > 0)
    return r-1;

  return r;
  // In C++, the function annotation is closed automatically
}

C example:

#include <caliper/cali.h>

int foo(int r)
{
  CALI_MARK_FUNCTION;       /* Exports "function=foo"          */

  if (r > 0) {
    CALI_MARK_FUNCTION_END; /* ALL exit points must be marked! */
    return r-1;
  }

  CALI_MARK_FUNCTION_END;   /* ALL exit points must be marked! */
  return r;
}

Loop Annotations

Loop annotations mark and name loops. The annotations should be placed immediately outside a loop of interest. Both begin and end of the loop have to be marked. Optionally, the loop iterations themselves can be marked and the iteration number exported by placing loop iteration annotations inside the loop body. Caliper exports the loop name in the loop attribute. Loop iterations are exported in per-loop attributes named iteration#name, where name is the user-provided loop name given to the loop annotation.

C++ example:

#include <caliper/cali.h>

CALI_CXX_MARK_LOOP_BEGIN(myloop_id, "myloop"); // exports loop=myloop
for (int i = 0; i < ITER; ++i) {
  CALI_CXX_MARK_LOOP_ITERATION(myloop_id, i);  // exports iteration#myloop=<i>

  // In C++, the iteration annotation is closed automatically
}
CALI_CXX_MARK_LOOP_END(myloop_id);

C example:

#include <caliper/cali.h>

CALI_MARK_LOOP_BEGIN(myloop_id, "myloop"); // exports loop=myloop
for (int i = 0; i < ITER; ++i) {
  CALI_MARK_ITERATION_BEGIN(myloop_id, i); // exports iteration#myloop=<i>

  if (test(i) == 0) {
    CALI_MARK_ITERATION_END(myloop_id);    // must mark ALL iteration exit points
    break;
  }

  CALI_MARK_ITERATION_END(myloop_id);      // must mark ALL iteration exit points
}
CALI_MARK_LOOP_END(myloop_id);

Code-region Annotations

Code-region annotations mark and name user-defined source code regions. Caliper exports the region names in the annotation attribute. Annotated code regions must be properly nested (see Nesting).

Example:

#include <caliper/cali.h>

CALI_MARK_BEGIN("Important code"); // exports annotation="Important code"
// ... important code
CALI_MARK_END("Important code");

Nesting

Annotated source-code regions of any of the pre-defined context attributes (function, loop, and annotation) can be nested within each other. Caliper preserves this nesting information. Users must ensure that the nesting is correct. That is, annotated code regions have to be enclosed completely within each other; they cannot partially overlap. Example:

#include <caliper/cali.h>

int foo()
{
  CALI_CXX_MARK_FUNCTION;

  CALI_MARK_BEGIN("outer");
  CALI_MARK_BEGIN("inner"); // The hierarchy is now "foo/outer/inner"

#if 0
  CALI_MARK_END("outer");   // Error! Must end "inner" before "outer"!
  CALI_MARK_END("inner");
#endif

  CALI_MARK_END("inner");
  CALI_MARK_END("outer");   // Correct nesting
}

To annotate arbitrary features or regions that can overlap with others, use the low-level annotation API and create a user-defined context attribute.

Low-level Annotation API

The “low-level” annotation API lets users create and export additional context attributes with user-defined names to capture custom code features or annotate arbitrary, non-nested code regions.

Attribute keys

Context attributes are the basic element in Caliper’s key:value data model. The high-level annotation API uses pre-defined attribute keys, but users can create their own. Attribute keys have a unique name, and store the attribute’s data type as well as optional property flags. Property flags control how the Caliper runtime system handles the associated attributes.

enum cali_attr_properties

Attribute property flags.

These flags control how the caliper runtime system handles the associated attributes. Flags can be combined with a bitwise OR (however, the scope flags are mutually exclusive).

Values:

CALI_ATTR_DEFAULT = 0

Default value.

CALI_ATTR_ASVALUE = 1

Store directly as key:value pair, not in the context tree.

Entries with this property will be not be put into the context tree, but stored directly as key:value pairs on the blackboard and in snapshot records. ASVALUE attributes cannot be nested. Only applicable to scalar data types.

CALI_ATTR_NOMERGE = 2

Create a separate context tree root node for this attribute.

Useful for attributes that form overlapping hierarchies separate from the main region stack.

CALI_ATTR_SCOPE_PROCESS = 12

Process-scope attribute. Shared between all threads.

CALI_ATTR_SCOPE_THREAD = 20

Thread-scope attribute.

CALI_ATTR_SCOPE_TASK = 24

Task-scope attribute. Currently unused.

CALI_ATTR_SKIP_EVENTS = 64

Skip event callbacks for blackboard updates with this attribute.

CALI_ATTR_HIDDEN = 128

Do not include this attribute in snapshots.

CALI_ATTR_NESTED = 256

Begin/end calls are properly aligned with the call stack.

Indicates that begin/end calls for this attribute are correctly nested with the call stack and other NESTED attributes. That is, an active region of a NESTED attribute does not partially overlap function calls or other NESTED attribute regions.

CALI_ATTR_GLOBAL = 512

A metadata attribute describing global information for a measurement run.

Global attributes represent metadata associated with an application run (e.g., application executable name and version, start date and time, and so on). They may be written in a separate metadata section in some output formats. For distributed programs (e.g. MPI), global attributes should have the same value on each process.

CALI_ATTR_UNALIGNED = 1024

This attribute is not aligned with stacked begin/end regions.

Entries with this property may still be merged into a single context tree branch, but one that is separate from the properly nested region branch. Stack nesting checks are skipped.

Attribute keys can be created with cali_create_attribute():

cali_id_t cali_create_attribute(const char *name, cali_attr_type type, int properties)

Create an attribute key.

Return

Attribute id

Parameters
  • name: Name of the attribute

  • type: Type of the attribute

  • properties: Attribute properties

C annotation API

The C annotation API provides the cali_begin_*[byname], cali_end[_byname], and cali_set_*_[byname] family of functions. Their behavior is as follows:

begin

Marks the begin of a region with the given attribute/value. The new value will be nested under already open regions of the same attribute.

set

Sets or overwrites the top-most value for the given attribute.

end

Closes the top-most open region for the given attribute.

The byname variants refer to attribute keys by their name. If no attribute key with the given name exists, it will be created with default properties. The basic variants take an attribute ID, e.g. from cali_create_attribute().

Example:

#include <caliper/cali.h>

/* Exports CustomAttribute="My great example" */
cali_begin_string_byname("CustomAttribute", "My great example")

/* Creates attribute key "myvar" with ASVALUE storage property */
cali_id_t myvar_attr =
  cali_create_attribute("myvar", CALI_TYPE_INT, CALI_ATTR_ASVALUE);

/* Exports myvar=42 */
cali_set_int(myvar_attr, 42);

/* Closes CustomAttribute="My great example" */
cali_end_byname("CustomAttribute");

C++ annotation API

The C++ annotation API is implemented in the class cali::Annotation.

Data tracking API

Caliper also supports tracking allocated data. Doing so provides advanced data-centric attributes, such as recording allocation events and determining the containers for memory addresses provided by services like libpfm. To take advantage of annotated memory allocations, the alloc service must be enabled at runtime.

Memory allocation annotations are similar to code region annotations, we can define labels for allocations using macros, which will use the variable name of the given pointer as label for the memory allocations. We can either label 1-dimensional ranges of bytes using CALI_DATATRACKER_TRACK or multi-dimensional ranges of specified element sizes using CALI_DATATRACKER_TRACK_DIMENSIONAL. The following example shows both:

void do_work(size_t M, size_t W, size_t N)
{
    double *arrayA = (double*)malloc(N);
    CALI_DATATRACKER_TRACK(arrayA, N);

    double *matA =
         (double*)malloc(sizeof(double)*M*W);

    size_t num_dimensions = 2;
    size_t A_dims[] = {M,W};
    CALI_DATATRACKER_TRACK_DIMENSIONAL(
                matA,
                sizeof(double),
                A_dims,
                num_dimensions);
        ...

    CALI_DATATRACKER_FREE(arrayA);
    CALI_DATATRACKER_FREE(matA);
}

API Reference

group AnnotationAPI

The user-facing source-code annotation API.

Low-level source-code annotation API

void cali_begin_region(const char *name)

Begin nested region name.

Begins nested region name using the built-in annotation attribute. Equivalent to the macro CALI_MARK_REGION_BEGIN.

See

cali_end_region()

void cali_end_region(const char *name)

End nested region name.

Ends nested region name using the built-in annotation attribute. Prints an error if name does not match the currently open region. Equivalent to the macro CALI_MARK_REGION_END.

See

cali_begin_region()

void cali_begin(cali_id_t attr)

Begin region where the value for attr is true on the blackboard.

void cali_begin_double(cali_id_t attr, double val)

Begin region val for attribute attr on the blackboard.

The region may be nested.

void cali_begin_int(cali_id_t attr, int val)

Begin region val for attribute attr on the blackboard.

The region may be nested.

void cali_begin_string(cali_id_t attr, const char *val)

Begin region val for attribute attr on the blackboard.

The region may be nested.

void cali_end(cali_id_t attr)

End innermost open region for attr on the blackboard.

void cali_safe_end_string(cali_id_t attr, const char *val)

Remove innermost value for attribute attr from the blackboard.

Creates a mismatch warning if the current value does not match val. This function is primarily used by the high-level annotation API.

Parameters
  • attr: Attribute ID

  • val: Expected value

void cali_set(cali_id_t attr, const void *value, size_t size)

Set value for attribute attr to val on the blackboard.

void cali_set_double(cali_id_t attr, double val)

Set value for attribute attr to val on the blackboard.

void cali_set_int(cali_id_t attr, int val)

Set value for attribute attr to val on the blackboard.

void cali_set_string(cali_id_t attr, const char *val)

Set value for attribute attr to val on the blackboard.

void cali_begin_byname(const char *attr_name)

Begin region where the value for the attribute named attr_name is set to true on the blackboard.

void cali_begin_double_byname(const char *attr_name, double val)

Begin region val for attribute named attr_name on the blackboard.

The region may be nested.

void cali_begin_int_byname(const char *attr_name, int val)

Begin region val for attribute named attr_name on the blackboard.

The region may be nested.

void cali_begin_string_byname(const char *attr_name, const char *val)

Begin region val for attribute named attr_name on the blackboard.

The region may be nested.

void cali_set_double_byname(const char *attr_name, double val)

Set value for attribute named attr_name to val on the blackboard.

void cali_set_int_byname(const char *attr_name, int val)

Set value for attribute named attr_name to val on the blackboard.

void cali_set_string_byname(const char *attr_name, const char *val)

Set value for attribute named attr_name to val on the blackboard.

void cali_end_byname(const char *attr_name)

End innermost open region for attribute named attr_name on the blackboard.

void cali_set_global_double_byname(const char *attr_name, double val)

Set a global attribute with name attr_name to val.

void cali_set_global_int_byname(const char *attr_name, int val)

Set a global attribute with name attr_name to val.

void cali_set_global_string_byname(const char *attr_name, const char *val)

Set a global attribute with name attr_name to val.

void cali_set_global_uint_byname(const char *attr_name, uint64_t val)

Set a global attribute with name attr_name to val.

Defines

CALI_DATATRACKER_TRACK(ptr, size)

Label and track the given memory region.

Labels the memory region of size size bytes pointed to by ptr with the variable name of ptr. The memory region can then be tracked and resolved with the alloc service.

CALI_DATATRACKER_TRACK_DIMENSIONAL(ptr, elem_size, dimensions, num_dimensions)

Label and track a multi-dimensional array.

Labels a multi-dimensional array with with the variable name of ptr. The array can then be tracked and resolved with the alloc service.

Parameters
  • elem_size: Size of the array elements in bytes

  • dimension: Array of type size_t[] with the sizes of each dimension. Must have num_dimensions entries.

  • num_dimensions: The number of dimensions

CALI_DATATRACKER_UNTRACK(ptr)

Stop tracking the memory region pointed to by ptr.

CALI_CXX_MARK_FUNCTION

C++ macro to mark a function.

Mark begin and end of a function. Should be placed at the top of the function, and will automatically “close” the function at any return point. Will export the annotated function by name in the pre-defined function attribute. Only available in C++.

CALI_CXX_MARK_SCOPE(name)

C++ macro marking a scoped region.

Mark begin and end of a C++ scope. Should be placed at the top of the scope, and will automatically “close” the function at any return point. Will export the annotated function by name in the pre-defined annotation attribute. Only available in C++.

CALI_CXX_MARK_LOOP_BEGIN(loop_id, name)

Mark loop in C++.

Mark begin of a loop. Will export the user-provided loop name in the pre-defined loop attribute. This macro should be placed before the loop of interest, and CALI_MARK_LOOP_END should be placed after the loop.

Parameters
  • loop_id: A loop identifier. Needed to refer to the loop from the iteration and end annotations.

  • name: Name of the loop.

CALI_CXX_MARK_LOOP_END(loop_id)

Mark loop end in C++.

Mark end of a loop. Will export the user-provided loop name in the pre-defined loop attribute.

This macro should be placed after the loop of interest. Users must ensure proper begin/end matching: If the surrounding function can be exited from within the loop (e.g., from a return statement within the loop), an exit marker must be placed there as well.

Parameters

CALI_CXX_MARK_LOOP_ITERATION(loop_id, iter)

C++ macro for a loop iteration.

Create a C++ annotation for a loop iteration. The loop must be marked with CALI_CXX_MARK_LOOP_BEGIN and CALI_CXX_MARK_LOOP_END. This will export the loop’s iteration count given in iter in an attribute named iteration#name, where name is the loop name given in CALI_CXX_MARK_LOOP_BEGIN. The macro should be placed at the beginning of the loop block. The annotation will be closed automatically. Example:

CALI_CXX_MARK_LOOP_BEGIN(mainloop_id, "mainloop");
for (int i = 0; i < ITER; ++i) {
  CALI_CXX_MARK_LOOP_ITERATION(mainloop_id, i);
  // ...
}
CALI_CXX_MARK_LOOP_END(mainloop_id);

Parameters
  • loop_id: The loop identifier given to CALI_CXX_MARK_LOOP_BEGIN

  • iter: The iteration number. Must be convertible to int.

CALI_MARK_FUNCTION_BEGIN

Mark begin of a function.

Exports the annotated function’s name in the pre=defined function attribute. A CALI_MARK_FUNCTION_END marker must be placed at all function exit points. For C++, we recommend using CALI_CXX_MARK_FUNCTION instead.

See

CALI_MARK_FUNCTION_END, CALI_CXX_MARK_FUNCTION

CALI_MARK_FUNCTION_END

Mark end of a function.

Must be placed at all exit points of a function marked with CALI_MARK_FUNCTION_BEGIN.

See

CALI_MARK_FUNCTION_BEGIN

CALI_MARK_LOOP_BEGIN(loop_id, name)

Mark a loop.

Mark begin of a loop. Will export the user-provided loop name in the pre-defined loop attribute. This macro should be placed before the loop of interest, and CALI_MARK_LOOP_END should be placed after the loop.

Parameters
  • loop_id: A loop identifier. Needed to refer to the loop from the iteration and end annotations.

  • name: Name of the loop.

CALI_MARK_LOOP_END(loop_id)

Mark a loop.

Mark end of a loop. Will export the user-provided loop name in the pre-defined loop attribute.

This macro should be placed after the loop of interest. Users must ensure proper begin/end matching: If the surrounding function can be exited from within the loop (e.g., from a return statement within the loop), an exit marker must be placed there as well.

Parameters

CALI_MARK_ITERATION_BEGIN(loop_id, iter)

Mark begin of a loop iteration.

This annotation should be placed at the top inside of the loop body. The loop must be annotated with CALI_MARK_LOOP_BEGIN. The iteration number will be exported in the attribute iteration::name, where name is the loop name given to CALI_MARK_LOOP_BEGIN. In C++, we recommend using CALI_CXX_MARK_LOOP_ITERATION.

See

CALI_MARK_ITERATION_END, CALI_MARK_LOOP_BEGIN, CALI_CXX_MARK_LOOP_ITERATION

Parameters
  • loop_id: Loop identifier, must match the identifier given to CALI_MARK_LOOP_BEGIN.

  • iter: Current iteration number. This macro argument must be convertible to int.

CALI_MARK_ITERATION_END(loop_id)

Mark end of a loop iteration.

This annotation should be placed at the end inside of the loop body. If an iteration can be left prematurely (e.g., from a continue, break, or return statement), an end marker must be placed there as well.

See

CALI_MARK_ITERATION_BEGIN

Parameters

CALI_WRAP_STATEMENT(name, statement)

Wrap Caliper annotations around a C/C++ statement.

The wrapped statement will be annotated with the given name in the statement attribute. Example

double res;
/* Wrap the sqrt() call */
CALI_WRAP_STATEMENT( "sqrt", res = sqrt(49) );

Parameters
  • name: The user-defined region name. Must be convertible into a const char*.

  • statement: C/C++ statement(s) that should be wrapped. The statements must complete within the wrapped region, that is, they cannot branch out of the macro (e.g. with goto, continue, break, or return).

CALI_MARK_BEGIN(name)

Mark begin of a user-defined code region.

This annotation should be placed before a code region of interest. The user-provided region name will be exported in the pre-defined annotation attribute.

Users must ensure proper nesting: Each CALI_MARK_BEGIN must be matched by a corresponding CALI_MARK_END in the correct order. Regions may be nested within another, but they cannot overlap partially.

See

CALI_MARK_END

Parameters
  • name: The region name. Must be convertible to const char*.

CALI_MARK_END(name)

Mark end of a user-defined code region.

This annotation should be placed after a code region of interest that has been annotated with CALI_MARK_BEGIN.

See

CALI_MARK_BEGIN

Parameters
  • name: The region name given to CALI_MARK_BEGIN. The macro will check if the name matches, and report an error if it doesn’t.

Functions

void *cali_datatracker_allocate(const char *label, size_t size)

Allocate and track the resulting allocation in Caliper.

Note

Tracking information will be automatically removed if the allocation is freed

Return

A new allocation of size size

Parameters
  • label: The unique label with which to track the allocation requested

  • size: The total size of the allocation requested

void *cali_datatracker_allocate_dimensional(const char *label, size_t elem_size, const size_t *dimensions, size_t num_dimensions)

Allocate and track the resulting allocation, with dimensions, in Caliper.

Note

Tracking information will be automatically removed if the allocation is freed

Return

A new allocation of size product(dimensions)*elem_size

Parameters
  • label: The unique label with which to track the allocation requested

  • elem_size: The size of an individual element in the allocation requested

  • dimensions: An array of the dimensions of the allocation requested

  • num_dimensions: The number of dimensions in dimensions

void cali_datatracker_free(void *ptr)

Free and untrack an allocation in Caliper.

Parameters
  • ptr: The pointer to the beginning of the allocation to free/untrack

void cali_datatracker_track(const void *ptr, const char *label, size_t size)

Track an existing allocation in Caliper.

Note

This will track the entire allocation pointed to by ptr. To track a subset of a larger allocation, use cali_datatracker_track_dimensional.

Parameters
  • ptr: The pointer to the beginning of the allocation to track

  • label: The unique label with which to track the allocation

  • size: Size of the allocation

void cali_datatracker_track_dimensional(const void *ptr, const char *label, size_t elem_size, const size_t *dimensions, size_t num_dimensions)

Track an existing allocation in Caliper.

Parameters
  • ptr: The pointer to the beginning of the allocation to track

  • label: The unique label with which to track the allocation

  • elem_size: The size of an individual element in the allocation

  • dimensions: An array of the dimensions of the allocation

  • num_dimensions: The number of dimensions in dimensions

void cali_datatracker_untrack(const void *ptr)

Untrack a previously tracked allocation in Caliper.

Parameters
  • ptr: The pointer to the beginning of the allocation to untrack

Variables

cali_id_t cali_function_attr_id
cali_id_t cali_loop_attr_id
cali_id_t cali_statement_attr_id
cali_id_t cali_annotation_attr_id
class Function
#include <Annotation.h>

Pre-defined function annotation class.

class ScopeAnnotation
#include <Annotation.h>

Pre-defined region annotation class. Region begins and ends with object construction and destruction.

class Loop
#include <Annotation.h>

Pre-defined loop annotation class, with optional iteration attribute.

class Annotation
#include <Annotation.h>

Instrumentation interface to add and manipulate context attributes.

The Annotation class is the primary source-code instrumentation interface for Caliper. Annotation objects provide access to named Caliper context attributes. If the referenced attribute does not exist yet, it will be created automatically.

Example:

cali::Annotation phase_ann("myprogram.phase");

phase_ann.begin("Initialization");
  // ...
phase_ann.end();
This example creates an annotation object for the myprogram.phase attribute, and uses the begin()/end() methods to mark a section of code where that attribute is set to “Initialization”.

Note

Access to the underlying named context attribute through Annotation objects is not exclusive: multiple Annotation objects can reference and update the same context attribute.

begin() overloads

Annotation &begin(int data)

Begin name=data region for the associated context attribute.

Marks begin of the name=data region, where name is the attribute name given in cali::Annotation::Annotation(). The new value will be nested under already open regions for the name context attribute.

Annotation &begin(double data)

Begin name=data region for the associated context attribute.

Marks begin of the name=data region, where name is the attribute name given in cali::Annotation::Annotation(). The new value will be nested under already open regions for the name context attribute.

Annotation &begin(const char *data)

Begin name=data region for the associated context attribute.

Marks begin of the name=data region, where name is the attribute name given in cali::Annotation::Annotation(). The new value will be nested under already open regions for the name context attribute.

Annotation &begin(const Variant &data)

Begin name=data region for the associated context attribute.

Marks begin of the name=data region, where name is the attribute name given in cali::Annotation::Annotation(). The new value will be nested under already open regions for the name context attribute.

set() overloads

Annotation &set(int data)

Set name=data for the associated context attribute.

Exports name=data, where name is the attribute name given in cali::Annotation::Annotation(). The top-most prior open value for the name context attribute, if any, will be overwritten.

Annotation &set(double data)

Set name=data for the associated context attribute.

Exports name=data, where name is the attribute name given in cali::Annotation::Annotation(). The top-most prior open value for the name context attribute, if any, will be overwritten.

Annotation &set(const char *data)

Set name=data for the associated context attribute.

Exports name=data, where name is the attribute name given in cali::Annotation::Annotation(). The top-most prior open value for the name context attribute, if any, will be overwritten.

Annotation &set(const Variant &data)

Set name=data for the associated context attribute.

Exports name=data, where name is the attribute name given in cali::Annotation::Annotation(). The top-most prior open value for the name context attribute, if any, will be overwritten.

Public Functions

Annotation(const char *name, int opt = 0)

Creates an annotation object to manipulate the context attribute with the given name.

Parameters
  • name: The attribute name

  • opt: Attribute flags. Bitwise OR combination of cali_attr_properties values.

Annotation(const char *name, const MetadataListType &metadata, int opt = 0)

Creates an annotation object to manipulate the context attribute with the given name.

Parameters
  • name: The attribute name

  • opt: Attribute flags. Bitwise OR combination of cali_attr_properties values.

  • metadata: a map of

void end()

Close top-most open region for the associated context attribute.

class Guard
#include <Annotation.h>

Scope guard to automatically close an annotation at the end of the C++ scope.

Example:

int var = 42;
while (condition) {
  cali::Annotation::Guard
    g( cali::Annotation("myvar").set(var) );
  // Sets "myvar=<var>" and automatically closes it at the end of the loop
}