C++ Instrumentation with OTeL

Often times, when practitioners think of generating spans/traces, we think of languages like Python, Javascript, Java, etc. Even GoLang has become indexed in the minds of the observability community as a defacto language. However, C++ is still a mainstay in many systems, and according to the TIOBE Index , is #2 (ranked behind Python).

Luckily, OTeL has a robust C++ SDK. An opinionated way to instrument is to be minimalistic on library components and rely on a host collector to access the wealth of processors and exporters. With that in mind, one can simply use the OTLP GRPC exporter to send to a local collector, creating a decoupled architecture with minimal potential impact.

A particularly valuable use case for the C++ OTeL SDK is tracing, judiciously, intensive computing jobs. We can treat spans as “structured logs” in this case and methodically add them in sections. Here is an example “epidemic simulation” :

#include <iostream>
#include <vector>
#include <random>
#include <memory>
#include <thread>
#include <chrono>
#include <algorithm>  // for std::count

// OpenTelemetry headers
#include "opentelemetry/exporters/otlp/otlp_grpc_exporter_factory.h"
#include "opentelemetry/sdk/trace/simple_processor_factory.h"
#include "opentelemetry/sdk/trace/tracer_provider_factory.h"
#include "opentelemetry/trace/provider.h"

namespace trace     = opentelemetry::trace;
namespace trace_sdk = opentelemetry::sdk::trace;
namespace otlp      = opentelemetry::exporter::otlp;

namespace
{
opentelemetry::exporter::otlp::OtlpGrpcExporterOptions opts;
std::shared_ptr<opentelemetry::sdk::trace::TracerProvider> provider;

void InitTracer()
{
    // Create OTLP exporter instance
    auto exporter  = otlp::OtlpGrpcExporterFactory::Create(opts);
    auto processor = trace_sdk::SimpleSpanProcessorFactory::Create(std::move(exporter));
    provider       = trace_sdk::TracerProviderFactory::Create(std::move(processor));

    // Set the global trace provider
    std::shared_ptr<opentelemetry::trace::TracerProvider> api_provider = provider;
    trace::Provider::SetTracerProvider(api_provider);
}

void CleanupTracer()
{
    // We call ForceFlush to prevent canceling running exports, but it's optional.
    if (provider)
    {
        provider->ForceFlush();
    }

    provider.reset();
    std::shared_ptr<opentelemetry::trace::TracerProvider> none;
    trace::Provider::SetTracerProvider(none);
}
}  // namespace

int main()
{
    opts.endpoint = "localhost:4317";  // Default OTLP gRPC endpoint
    opts.use_ssl_credentials = false;  // Disable SSL/TLS for local communication

    // Initialize the tracer
    InitTracer();

    auto tracer = trace::Provider::GetTracerProvider()->GetTracer("epidemic_simulation");

    // Start the main simulation span
    auto simulation_span = tracer->StartSpan("Simulation");
    auto scoped_simulation = trace::Scope(simulation_span);

    // Simulation parameters
    const int population_size = 1000;
    const int initial_infected = 10;
    const double infection_rate = 0.05;
    const double recovery_rate = 0.01;
    const double mortality_rate = 0.005;

    // States: false = susceptible, true = infected, 2 = recovered, 3 = dead
    std::vector<int> population(population_size, 0);
    std::fill(population.begin(), population.begin() + initial_infected, 1);

    std::default_random_engine generator;
    std::uniform_real_distribution<double> infection_dist(0.0, 1.0);
    std::uniform_real_distribution<double> recovery_dist(0.0, 1.0);
    std::uniform_real_distribution<double> mortality_dist(0.0, 1.0);

    int day = 1;
    while (true)  // Infinite loop, change condition to stop based on your needs
    {
        auto day_span = tracer->StartSpan("Day " + std::to_string(day));
        auto scoped_day = trace::Scope(day_span);

        int new_infections = 0;
        int recoveries = 0;
        int deaths = 0;

        // Copy of the population to avoid modifying while iterating
        std::vector<int> new_population = population;

        for (int i = 0; i < population_size; ++i)
        {
            if (population[i] == 1)  // Infected
            {
                // Chance to recover or die
                if (recovery_dist(generator) < recovery_rate)
                {
                    new_population[i] = 2;  // Recovered
                    recoveries++;
                }
                else if (mortality_dist(generator) < mortality_rate)
                {
                    new_population[i] = 3;  // Dead
                    deaths++;
                }
                else
                {
                    // Try to infect others
                    for (int j = 0; j < population_size; ++j)
                    {
                        if (population[j] == 0 && infection_dist(generator) < infection_rate)
                        {
                            new_population[j] = 1;  // Newly infected
                            new_infections++;
                        }
                    }
                }
            }
        }

        population = new_population;

        int total_infected = std::count(population.begin(), population.end(), 1);
        int total_recovered = std::count(population.begin(), population.end(), 2);
        int total_dead = std::count(population.begin(), population.end(), 3);

        // Log the day's results
        std::cout << "Day " << day << ": " << new_infections << " new infections, " 
                  << recoveries << " recoveries, " << deaths << " deaths.\n";

        // Add events and attributes to the span
        day_span->AddEvent("Day summary", {
            {"new_infections", new_infections},
            {"recoveries", recoveries},
            {"deaths", deaths},
        });
        day_span->SetAttribute("day_number", day);
        day_span->SetAttribute("total_infected", total_infected);
        day_span->SetAttribute("total_recovered", total_recovered);
        day_span->SetAttribute("total_dead", total_dead);

        day_span->End();
        day++;

        // Sleep to slow down the simulation
        std::this_thread::sleep_for(std::chrono::seconds(1));  // Sleep for 1 second between days
    }

    simulation_span->End();

    // Clean up and flush tracer
    CleanupTracer();

    return 0;
}

Key points to note:

1. InitTracer : where we create our tracer instance and pass options (such as OTLP endpoint)

2. tracer->StartSpan declarations to create spans

3. AddEvent and SetAttribute to add crucial metadata to the spans. This can be valuable to store counts, facets for later processing in storage.

4. Ending and Cleaning up to flush

Leveraging spans as a rich data structure provides a versatile way to instrument C++ applications and get a detailed inspection of behavior and performance.