Omlet Agent is similar to OpenClaw but takes a different approach:

1. Can we provide "code-like" automation in a cleaner interface that organizes sessions like traditional directories?

2. Can we offer advanced views, cron-like tasks, and dynamic tool usage already built into coding CLIs?

3. Can we achieve a potentially zero CVE deployment in a single container?

Directory View

CLI View

Task Sytem

Styled HTML Views

Shareable Links

Most importantly, an autonomous agent must be secure. Strong authentication measures are essential, and we use Auth0 for this purpose. Additionally, can we achieve the ideal scenario of having zero CVEs? Fortunately, we managed to shift many components from the standard node:20-slim, use a multi-stage build, and heavily prune packages, which significantly reduced CVEs (see the Trivy scan below)!

It starts with knowledge gathering

The first thing a user typically does is learn about the platform's dynamics, data structures, and goals by looking for relevant documentation. This includes query languages, UI mechanics, APIs, and more. For our expert, we had our agent thoroughly explore platform documentation and indexed knowledge. Our agent then organizes this information into clear markdown and creates visual diagrams for future reference (as shown below).

With a grounded knowledge base, our agent can effectively explore and play around with platform capabilities.

It progresses to deterministic tools

Once our agent understands "how to access" a platform, using credentials or OAuth just like a user would, it can move on to creating precise tools. While MCP is a good general platform for building and distributing tools, we can also rely on OS first principles and let the agent execute code directly.

In the example above, we demonstrate one tool. Notice the importance of precision to help the agent recover from errors and its own mistakes. We can also create highly specific tools that follow a certain user structure (such as querying only this namespace or using these filters). We find particular value in these structured implementations, as they provide a strong framework around dynamic query languages.

It culminates in sophisticated processes

Finally, what if we link all these precise tools together while allowing for dynamic exploration and model analysis? This is where we can develop systematic processes to achieve a goal within the platform. This could involve a detailed investigation or a comprehensive overview of trends.

Our main goal, using targeted tools and advanced workflows, is to let observability data move beyond the limits of generic presentation and use. That's why, within our Agent OS platform, these components can be tailored to fit internal domain knowledge while also working with other Agents. Imagine one agent gathering information and skillfully querying, while another performs analysis and assesses multiple hypotheses.

Data and Systems Discovery

The most important step is creating a map of what exists and how everything interacts. For us, this begins with discovering tables, namespaces, components, and how services call each other. We use specialized "template queries" to guide the model accurately. Sampling can significantly limit the ability to effectively perform data discovery.

“Path-finding” Guidance

Deep dive investigations should be conducted one at a time, in isolation. A common mistake many AI SRE solutions make is getting stuck in the wrong context and overlooking important components and interactions. We enhance this process by offering path-finding expertise to the agent.

Incident Documentation Format

Finally, summaries, findings, and trends must be documented in a proper format for future agent runs.

An open, robust, and well-connected AI SRE

Our core belief, supported by our design partners, is that an AI SRE should first be liberated from rigid tools. This begins by using standard data formats like OTel and creating strong tool layers with MCP.

Exploring Maximas in the AI Space

Sometimes, it's beneficial to wait and see how different parts of the ecosystem, especially in AI, develop. The image shows a pattern in various stages of the current AI phase and partly in past AI phases. At different times, the industry has claimed that true automation is finally here and that AI will solve all the problems that trouble enterprises. However, this has often fallen short due to:

1. Integration challenges

2. Inconsistent usage

3. Inaccuracy and scalability issues from a hardware perspective

I believe we are seeing some signs of a "global" peak in the usefulness of AI for data analysis.

Less is more for MCP tools

We quickly realized that humans are significantly affected by context switching and having "too many tools." From our past experience with LLMs and observability data, we also knew that using different payloads could confuse the LLM. With this in mind, we focused on the essentials for our MCP server:

1. Read data from OTel formatted tables

2. Read data and cluster

This "confinement" allows a reasoning model with tool-use abilities to focus on the basics without getting distracted.

What’s next?

As we mentioned, things are moving quickly, but we can use this growing knowledge to adapt fast. The guiding principle should always be: Are we helping engineering, architecture, and ops teams solve issues quickly and consistently? Are we assisting teams in resolving issues when they're unavailable or need breaks? Are we doing this throughout the organization?

Object Storage

Object storage represents a "bottomless" and horizontally concurrent way to store batches of data, such as otlp_json. The awss3exporter within the OTel Collector Contrib provides a great way to batch write to S3.

An example config:

receivers:
  otlp:
    protocols:
      http:
        endpoint: '0.0.0.0:4318'

processors:
  batch:

exporters:
  awss3:
    s3uploader:
      region: "us-west-2"
      s3_bucket: "omlet-logs"
      # root “directory” inside the bucket; you can change "logs" to whatever prefix you like
      s3_prefix: "logs"
      # partition by year/month/day/hour/minute
      s3_partition_format: "year=%Y/month=%m/day=%d/hour=%H/minute=%M"
       # gzip compression is optional
      compression: gzip
    sending_queue:
      enabled: true
      num_consumers: 5
      queue_size: 50
    timeout: 10s
  debug:

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, awss3]

The above configuration takes OTLP data (in this example, only logs) and batches it to an S3 bucket. While straightforward, this setup unlocks major capabilities.

Lakehouse DB

OLAP databases like DuckDB and Clickhouse (among others) can easily connect to this storage system. We use CHDB (embedded Clickhouse) to access this object storage.

import os
import json
import gzip
import logging
import boto3
from datetime import datetime, timedelta
import chdb
from itertools import islice
from concurrent.futures import ThreadPoolExecutor, as_completed

# ───────────────────────────────
# Configure logging
# ───────────────────────────────
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)
log = logging.getLogger("otel_ingest")

# ───────────────────────────────
# AWS creds
# ───────────────────────────────
assert os.environ.get("AWS_ACCESS_KEY_ID"), "Missing AWS_ACCESS_KEY_ID"
assert os.environ.get("AWS_SECRET_ACCESS_KEY"), "Missing AWS_SECRET_ACCESS_KEY"
log.info("AWS credentials loaded")

# ───────────────────────────────
# List S3 files for last day
# ───────────────────────────────
s3 = boto3.client("s3", region_name="us-west-2")
now = datetime.utcnow()
start = now - timedelta(days=1)
paginator = s3.get_paginator("list_objects_v2")
pages = paginator.paginate(Bucket="omlet-logs", Prefix="logs/")

all_keys = [
    obj["Key"]
    for page in pages
    for obj in page.get("Contents", [])
    if start <= obj["LastModified"].replace(tzinfo=None) <= now and obj["Key"].endswith(".json.gz")
]
log.info(f"Total files found: {len(all_keys)}")
if not all_keys:
    exit(0)

# ───────────────────────────────
# Create target table
# ───────────────────────────────
conn = chdb.connect()
cur = conn.cursor()
cur.execute("""
CREATE TABLE IF NOT EXISTS otel_logs (
    Timestamp           DateTime64(9),
    TraceId             String,
    SpanId              String,
    TraceFlags          UInt8,
    SeverityText        String,
    SeverityNumber      UInt8,
    ServiceName         String,
    Body                String,
    ResourceAttributes  Map(String, String),
    ScopeAttributes     Map(String, String),
    LogAttributes       Map(String, String)
) ENGINE = Memory
""")
log.info("Table otel_logs ready")

# ───────────────────────────────
# Helper to process a single file → list of dicts
# ───────────────────────────────
def parse_file(key):
    log.info(f"Starting download of {key}")
    blob = s3.get_object(Bucket="omlet-logs", Key=key)["Body"].read()
    log.info(f"Downloaded {key} ({len(blob)} bytes)")
    payload = json.loads(gzip.decompress(blob))
    entries = payload.get("resourceLogs", [])
    log.info(f"Parsed JSON for {key}, found {len(entries)} resourceLogs entries")

    out = []
    for r in entries:
        res_attrs = {a["key"]: a["value"]["stringValue"] for a in r["resource"]["attributes"]}
        for sl in r["scopeLogs"]:
            scope_attrs = {a["key"]: a["value"]["stringValue"] for a in sl["scope"].get("attributes", [])}
            service = sl["scope"].get("name", "")
            for logrec in sl["logRecords"]:
                ts = datetime.utcfromtimestamp(int(logrec["timeUnixNano"]) / 1e9).isoformat(sep=' ')
                rec = {
                    "Timestamp": ts,
                    "TraceId": logrec.get("traceId", ""),
                    "SpanId": logrec.get("spanId", ""),
                    "TraceFlags": logrec.get("traceFlags", 0),
                    "SeverityText": logrec.get("severityText", ""),
                    "SeverityNumber": logrec.get("severityNumber", 0),
                    "ServiceName": service,
                    "Body": logrec["body"]["stringValue"],
                    "ResourceAttributes": res_attrs,
                    "ScopeAttributes": scope_attrs,
                    "LogAttributes": {a["key"]: a["value"]["stringValue"] for a in logrec.get("attributes", [])}
                }
                out.append(rec)
    log.info(f"Processed {key}, constructed {len(out)} records")
    return out

# ───────────────────────────────
# Chunking utilities
# ───────────────────────────────
def chunked(iterable, size):
    it = iter(iterable)
    while True:
        batch = list(islice(it, size))
        if not batch:
            break
        yield batch

batch_size = 100
row_batch_size = 100000  # JSONEachRow batch size
total = 0

# ───────────────────────────────
# Main loop: batches of files
# ───────────────────────────────
for batch_keys in chunked(all_keys, batch_size):
    log.info(f"Processing batch of {len(batch_keys)} files")
    # parallel parse → list of dicts
    rows_accum = []
    with ThreadPoolExecutor(max_workers=8) as ex:
        futures = {ex.submit(parse_file, key): key for key in batch_keys}
        for fut in as_completed(futures):
            rows_accum.extend(fut.result())

    # bulk insert in JSONEachRow chunks
    for chunk in chunked(rows_accum, row_batch_size):
        payload = "\n".join(json.dumps(r, default=str) for r in chunk)
        sql = "INSERT INTO otel_logs FORMAT JSONEachRow\n" + payload
        cur.execute(sql)
        log.info(f"Inserted JSONEachRow batch of {len(chunk)} rows")
        total += len(chunk)

log.info(f"Total rows inserted: {total}")

# Example select query
import pandas as pd
df = pd.read_sql("SELECT SeverityText, count() AS cnt FROM otel_logs GROUP BY SeverityText", conn)
log.info("Counts by severity:\n" + df.to_string(index=False))

conn.close()
log.info("Done.")

In the code above, we load a series of compressed (gzip) otlp_json files from a specific time window (the past day), create an in-memory table, insert rows in batches, and query for counts by severity level.

Going “Wide”

The code above shows a simple way to query using one worker. However, the advantage of object storage is that it allows multiple workers to access partitions in the bucket. For example:

1. Access log/trace data by minute per worker for the past hour for large-scale data and aggregate it.

2. Partition by "Service" and "Time" to query high-resolution data at scale during live incidents.

Our vision at Omlet is to use OTel as a tool to create the best Observability Lakehouse experience.

Configuration and Administration

Config and Admin are usually the first pain points engineers confront with OSS Observability solutions. Think endless YAML, user management (auth), service integration, boilerplate queries and dashboards, cluster configuration, permissions, etc.

Networking

Beyond initial configuration, networking components need to be addressed. How will users access the UIs? What DNS configurations do you need to think about? What about certificate management? Although these seem like simple, common-sense things, they really add up.

Scale

Different services and teams, at different times, may send unpredictable amounts of volume. How do you sustain peak load and not under-provision? How do you enforce resource quotas and minimize noisy neighbors? Can resources spin up (and down) quickly?

Multi-Tenancy and Orchestration

In the enterprise, various teams/groups will want their own tenants. How do you properly isolate for ownership and compliance? Can a central team manage and govern sub-units effectively?

Omlet House

The above factors, especially regarding enterprise multi-tenancy and orchestration, really pushed us to consider the ideal stack of open source observability components. There are various combinations one could think of, but the above challenges still remained. With our Omlet House service that enterprises can deploy in their own Kubernetes cluster, we wanted the “stack” spin-up to be fluid and ready to go.

If you are on AWS, GCP, or Azure, we’d love to talk to you about trying it out! Send us a note at info@omlet.co!

The “Bitter Lesson” as it concerns Observability

The “Bitter Lesson” is a significant observation on machine learning improvement trends:

In the long run, general methods that leverage computation (like deep learning) outperform human-designed, specialized algorithms—even in domains where we think we have deep expertise.

Of course, the “bitterness” arises in individuals who spend considerable time examining carefully crafted processes due to the sheer attachment they feel to them over time. This happens quite often in the sciences (physicists becoming emotionally attached to theories). Observability practitioners suffer from the same fate, in my opinion. Not because of the state of the environment, however, but because the mantra from a commercial sense has largely propagated a varied, uniform, “perfect” set of telemetry. Maybe it's self-created, forcing us to endlessly worry about what is or isn’t ingested or structured. From a field perspective, one painstakingly tries to promote an ideal collection of telemetry, only to find that people are fine with just logs.

The “Fog of War” in Observability

When I first deployed tracing, I felt the power of what I thought of as “automatic and structured logging”. I saw spans as these efficient structures where I could stash metadata for whatever analytical purpose. What I should have realized at this point: this is what happens when you stick to a consistent pattern in your observability strategy. I was fortunate to have done this early. The temptation to collect extraneous datapoints via other ingestion frameworks wasn’t there. I was fine “extending my events”.

From a game-theory perspective, this is analogous to illuminating the “fog of war” in RTS (real-time strategy) games

Common in these games, players have to regularly send out “scouts” or unit groups to explore and path find. I was doing this in my own way for my services. Consequently, we must accept that individuals/groups at enterprises (small and large) do this in their own way. We must ultimately be considerate of this.

Regardless of how this fog is being removed, we are at an interesting moment. In our RTS scenario, a constant need to explore can be mentally taxing and cost us units during the game. In observability, there is a compounding cognitive toll on querying and reading. Would there be a way to eliminate the “fog” from reappearing all together? Explore everything in parallel?

*I want to highlight that a diverse group of units is a good strategy to combat opposing players in RTS, but that’s more akin to looking at this from a security perspective..

Get in touch at info@omlet.co, our website contact form, or leave a comment!

Approach

One of the amazing benefits of the OpenTelemetry semantic conventions is the solid data platform it unlocks. With this great data foundation, we can creatively approach integrating AI and LLMs. As discussed in one of the previous posts, however, it is extremely tempting to look at Observability AI Agents from a “greedy” perspective. In that post, we mentioned that reliance on only “top-down” AI systems can miss crucial long-tail problems and disregard important information. Very quickly, this aproach devolves into a rules-based approach. At Omlet, we wanted to take a different, more straight-forward approach to incorporating a powerful, lightweight AI agent into the observability landscape.

Observability is an “Always-on” endeavor

Observability practices, and the automation aligned to it, do not have the comfort of being active only sometimes. In order to be maximize effectiveness, observability is required to be always active and all encompassing. Practitioners are often forced into taking a “collect everything” mentality, which can result in massive increases in cost and noise. Fortunately, there’s a better way.

Bottom-up: AI in the pipeline, at the “Edge”

A core principle of Observability is the concept of “entities”. This could be reflected, oftentimes, as a “Service”, “Host”, “Pod”, etc. These cohorts unpack grouping methodologies that can be further supplemented by “Version”, “Release”, “Experiment”, etc. We can use these groupings as opportunities to examine issues in isolated groups. This covers a few giant pain points:

1. Making sense of large volumes of data in a methodical way

2. Allowing “top-down” methodologies (AI or Human) to focus on larger impact of interacting parts, not why or how the parts are breaking. Humans spend too much time focusing on individual entities currently, resulting in high cognitive fatigue.

Efficient “Always-on” AI

You’re probably wondering: wouldn’t event data structures (logs / spans), and their associated volume of tokens, lead to a cost/noise explosion within LLMs? We attacked this problem in three parts, specifically for LLM architectures:

1. Better Entity Based Grouping

2. Clustering and Masking

3. Selective extraction on Span Data Structures

What allowed these techniques in the first place? A plug-and-play data platform built on OTel. OTel semantic conventions are very well structured, which makes the data engineering and formatting task a breeze. We had to also consider carefully: Should we build yet another pane of glass for this type of view, or, could we benefit from the interoperability of OTel and bring the intelligence to the existing platform views? We chose the latter.

Dynamic Incident Analysis and Log Quality

Our early experiments showed exciting results across specific OpenAI (gpt4o-mini) calls. As we expanded the capabilities, we proceeded to add other models, like Claude, Llama, Mistral, and now 100s of others via “bring-you-own-LLM”. Some of the examples:

What’s Next?

Unsurprisingly, observability use-cases aren’t limited to log or span incident analysis. Over the past months, we have been delighted to explore a diverse range of powerful use-cases presented by design partners working with us to develop Skillet:

1. Log Line Reduction Recommendation

2. Log Quality Analysis and Scoring

3. Complex Sensitive Data Analysis

4. Performance Optimization Opportunities

5. SQL Query Optimization

6. Key Points in Metrics Data

As an always-on AI agent, Skillet represents a massive leap forward in Omlet’s capacity to simplify and amplify your observability workflows. This is a feature still in active development, and we’re eager to continue exploring customer and partner-led ideas on how we can hone on in other use cases with this data (hint: security). Contact us at info@omlet.co and let’s get cooking!

The power of Open Source

We're experiencing a kind of software "enlightenment" era with various Open Source movements like OTel, Apache Iceberg, Deepseek, and more. For us at Omlet, we believe that the “telemetry highway” will be OTel. The “on” and “off” ramps could be diverse, but the “highway”: OTel

We want the Datadog Agent to transmit data smoothly, and we are excited to introduce the "Datadog-to-OTel" Service as a container that you can host yourself.

• Container: psharma1989/datadog-to-otel

• Associated Docs: https://omlet-1.gitbook.io/omlet-docs/datadog-agent-s-to-otel#omlet-datadog-to-otel-service

  • We use the Omlet Gateway as an example within the docs

This opens up various use cases that benefit from dual-shipping or proxying alongside your main Datadog platform usage.

Use Cases

High-compliance scenarios

Sensitive observability data requires teams to buffer or store data on-premises or in-VPC. Redaction or filtration rules can be applied after the fact.

“Edge” Alerting and AIOPs

OTel is quickly becoming the universal standard for Observability understanding. The Datadog Agent outputs a wealth of structured telemetry that can be augmented with AI Agents at the “Edge”. These agents could output back to the Datadog platform for alerting / orchestration

Observability Pipelining and Observability Data Lake

You might want full data resolution with a specific retention period. You may also want to control the amount of noisy data that impacts MTTR and significantly increases costs. Long-term data analysis over weeks, months, or quarters can be done on complete observability data stored in object storage. Key observability data signals can be enhanced to reduce noise in the Datadog platform.

What next?

We’d love to hear from the community about what else you'd like to see. Visit us at https://www.omlet.co/ or email us at info@omlet.co to share your thoughts!

The questions above really help illustrate the gulf that is developing between the various users of LLM tools. For some, its sparsely used and a toy, for programmers, it’s like discovering the mental combustion engine.

Knowledge and Verification Bubbles

It certainly helps to examine different knowledge spheres in terms of scope and subjectivity. This is where “programming” as a sphere has various advantages:

1. Brevity: Programming Languages have the distinct advantage of having very few terms (keywords) and those keywords fixed in their function.

2. Reasonably Verifiable: Execution of the program results in a healthy population of “correct” solutions. Compare this to more subjective (and equally, or more important to humans) knowledge, like literature.

3. Modularity in problem and solution: The above has indirectly resulted in QA destinations (like Stack Overflow) that provide excellent training signals.

4. Logical Structure: Program structure is fairly logical and dependencies (even in complex situations) can be reasonably discerned

Compressing the feedback loop

Here we arrive to, what I think due to personal experience…the true philosophical moment. Recall the programming experience, lets say in the 2010s, for small tasks within a large task (“build an app, dashboard, components, API, etc”)

1. Decide on paradigm: Programming Language opinions, frameworks, documentation maturity, etc

2. Execute feedback loop:

   1. Write code

   2. Run: Verify correctness

   3. Debug: Read stack-traces, logs, etc

   4. Search: Documentation, Google, etc

   5. Repeat

3. Refactor: clean-up with architectural best practices

This, of course, is a slim example of the general feedback loop. The steps can vary, but the general loop is fairly consistent. Here is where the boundlessness, fuzziness, and variability contribute so heavily. At each of these steps, the cost of finding information, integrating, and verifying can be high. Furthermore, the cost of the wrong answer (or “path”), can be extremely high as this cascades into prior and subsequent efforts. To help visualize:

You can imagine the vertical axis as being possible start paths, and the horizontal axis as distance in time and effort to the next task. The issue mainly lies in choosing and evaluating various paths, one that could be ultimately more time consuming or erroneous. What’s changed:

Suddenly, because of LLMs in programming, there is more of a “straight-line” effect between choices and tasks. It cannot be overstated how impactful this is. It compresses the exhausting feedback loop greatly. Although this may seem fully apparent and taken for granted, we must appreciate the significance of this for software engineering.

Pathfinding: From Learning to Search

We can think of this common feedback loop as a pathfinding problem. What’s interesting is that these “ideal” paths for all sorts of practical programming functions, due to human and synthetic data, are being quickly discovered. Higher level tasks such as “building an app, but with these variations” is starting to transition from a time-consuming learning problem, to a cached search problem. The implications of this are enormous, but why did this happen?

“Good-enough-ness”

People unfamiliar or uninterested in software engineering tasks sometimes consider this type of work to be esoteric and/or arduous. Software engineers can also have a habit of touting the willpower and intellect needed to do what they do. But in reality, the work isn’t that complicated. Furthermore, things are getting quite “good enough”.

Consider examples like SaaS applications with user management, dashboards, notifications, etc. At a lower-level, lets also consider their interface components and CRUD APIs. Might seem simplistic, but this represents the vast majority of generic SaaS applications that have verticalized in different industries. Even the smallest models can attack this problem set in components, with new architectures attacking this problem from high-level specification to components. Sure there are variations and opinions that humans need to inject, but the integration time has become fairly constant. The delta between a large team building a sophisticated app, vs one with new tools, is shrinking significantly. SaaS, unfortunately, also relies on the belief that customers are too dumb and lazy to do it themselves. This is changing fast.

This is a massive industry, and if you add lower-level cloud architecture tools, its even more massive. At the end, only the physical substrate access (compute, storage) really matters. This presents a difficulty for incumbents, where now, customers look for extreme differentiation, or just proclaim: “good-enough”

• Obsessed, almost reliant, on network and entity complexity

• Addicted to promoting big-data digestion

• Suffers from, or completely disregards, context gaps. Or worse, assume you have the all right data.

Engineers, and observability practitioners, ideally like to manage and forecast for complexity. It’s for this reason we are also seeing services and tools that are combating code complexity:

• Optimization for runtime execution via profiling

• Code cleanup for brevity/developer experience

• Dependency and configuration cleanup

I always like to take a gentle, systems theory approach to these things. We can see a “shift-left” moment happening in programming to make the system and its functioning in the environment more predictable and less “reactive” to changes and drift. Let’s consider a variable C that represents complexity and N that can be considered independently but influences C. The purpose of this exercise is to examine how these variables can increase in magnitude, greatly influence compute time factors, and subsequently increase the probability of false negatives and positives.

Factors influencing C:

1. Quantity of entities (micro-services, hosts, etc). Increases the “cohorts” potentially needed to be analyzed

2. Variety of request types. Could be simple HTTP requests, or long-running jobs.

3. Variety of request attributes. Metadata decoration.

4. The interaction between entities.

Factors influencing N:

1. Volume of requests

2. Repetition rate of requests

What one can imagine is a large network visualization with interacting nodes modeled by weighted edges. This has become a hallmark of Observability marketing propaganda…

I believe that it's necessary to accurately model the system by projecting it as this data structure. However, much of the Observability industry has clung to it as the "finished" product, even when it becomes unintelligible ("network hairball"). Others have tried to add some layer of meaning but feel: "if you don't show it in all its glory, you are missing out." I believe the days of using UI/UX to parse this are over, mostly due to the phase shift we are seeing because of LLMs.

Here lies another problem, however. As C and N increase, so will the resources required to distill. The approach I’m seeing is very top-down, similar to the human SRE approach, which suffers from the same fixation on the holy network visualization. What these systems are doing on this graph structure is running a selective/greedy or full depth-first search for the sake of it. Furthermore, there lies the critical instantiation problem. When should this search trigger? A typical metrics alert? An event?

The above image helps illustrate how boundless this agentic search can become. Worse yet, there is a high chance of missing context (unknown unknowns). The problem with observability isn’t only disjointed data; it's not having the data, as illustrated below.

In this case, a top-down agentic system might get “greedy” and miss edge cases or catastrophically disregard them. It’s worth noting, also, these types of agentic systems are highly incentivized to crave this complexity (“we correlate millions of signals, and charge per run”). This will quickly run opposite to what observability practitioners will want to budget for (efficiency).

This post highlights why we must critically examine the new breed of top-down agents, even if they spur intellectual curiosity. In subsequent posts, we will examine alternative approaches.

Various reasons:

1. No apparent use case for broad spectrum data, long-term storage, or both

2. Inability or hesitation to take different data structures and normalize (via ETL, etc)

3. Standardization

There, finally, may be a way to build a so called Observability Data Lake, with OpenTelemetry schemas being the avenue. The data models specified by OTeL are extremely conducive to sustainable broad and long-term storage and unlock exciting capabilities:

1. Organization wide visibility into incidents, incident impacts, resource usage, and cost forecasts

2. Using observability data for user analytics and making it “joinable” with other analytics (advertising performance, partner data, etc)

3. Large datasets for machine learning

We seek to enable this through OTeL standard pipelining into object storage as a primary destination. This strikes a balance between the real-time and nuanced views that current observability vendors critically provide, and the full-resolution observability datasets that can be used for strategic purposes.

Luckily, OTeL has a robust C++ SDK. An opinionated way to instrument is to be minimalistic on library components and rely on a host collector to access the wealth of processors and exporters. With that in mind, one can simply use the OTLP GRPC exporter to send to a local collector, creating a decoupled architecture with minimal potential impact.

A particularly valuable use case for the C++ OTeL SDK is tracing, judiciously, intensive computing jobs. We can treat spans as “structured logs” in this case and methodically add them in sections. Here is an example “epidemic simulation” :

#include <iostream>
#include <vector>
#include <random>
#include <memory>
#include <thread>
#include <chrono>
#include <algorithm>  // for std::count

// OpenTelemetry headers
#include "opentelemetry/exporters/otlp/otlp_grpc_exporter_factory.h"
#include "opentelemetry/sdk/trace/simple_processor_factory.h"
#include "opentelemetry/sdk/trace/tracer_provider_factory.h"
#include "opentelemetry/trace/provider.h"

namespace trace     = opentelemetry::trace;
namespace trace_sdk = opentelemetry::sdk::trace;
namespace otlp      = opentelemetry::exporter::otlp;

namespace
{
opentelemetry::exporter::otlp::OtlpGrpcExporterOptions opts;
std::shared_ptr<opentelemetry::sdk::trace::TracerProvider> provider;

void InitTracer()
{
    // Create OTLP exporter instance
    auto exporter  = otlp::OtlpGrpcExporterFactory::Create(opts);
    auto processor = trace_sdk::SimpleSpanProcessorFactory::Create(std::move(exporter));
    provider       = trace_sdk::TracerProviderFactory::Create(std::move(processor));

    // Set the global trace provider
    std::shared_ptr<opentelemetry::trace::TracerProvider> api_provider = provider;
    trace::Provider::SetTracerProvider(api_provider);
}

void CleanupTracer()
{
    // We call ForceFlush to prevent canceling running exports, but it's optional.
    if (provider)
    {
        provider->ForceFlush();
    }

    provider.reset();
    std::shared_ptr<opentelemetry::trace::TracerProvider> none;
    trace::Provider::SetTracerProvider(none);
}
}  // namespace

int main()
{
    opts.endpoint = "localhost:4317";  // Default OTLP gRPC endpoint
    opts.use_ssl_credentials = false;  // Disable SSL/TLS for local communication

    // Initialize the tracer
    InitTracer();

    auto tracer = trace::Provider::GetTracerProvider()->GetTracer("epidemic_simulation");

    // Start the main simulation span
    auto simulation_span = tracer->StartSpan("Simulation");
    auto scoped_simulation = trace::Scope(simulation_span);

    // Simulation parameters
    const int population_size = 1000;
    const int initial_infected = 10;
    const double infection_rate = 0.05;
    const double recovery_rate = 0.01;
    const double mortality_rate = 0.005;

    // States: false = susceptible, true = infected, 2 = recovered, 3 = dead
    std::vector<int> population(population_size, 0);
    std::fill(population.begin(), population.begin() + initial_infected, 1);

    std::default_random_engine generator;
    std::uniform_real_distribution<double> infection_dist(0.0, 1.0);
    std::uniform_real_distribution<double> recovery_dist(0.0, 1.0);
    std::uniform_real_distribution<double> mortality_dist(0.0, 1.0);

    int day = 1;
    while (true)  // Infinite loop, change condition to stop based on your needs
    {
        auto day_span = tracer->StartSpan("Day " + std::to_string(day));
        auto scoped_day = trace::Scope(day_span);

        int new_infections = 0;
        int recoveries = 0;
        int deaths = 0;

        // Copy of the population to avoid modifying while iterating
        std::vector<int> new_population = population;

        for (int i = 0; i < population_size; ++i)
        {
            if (population[i] == 1)  // Infected
            {
                // Chance to recover or die
                if (recovery_dist(generator) < recovery_rate)
                {
                    new_population[i] = 2;  // Recovered
                    recoveries++;
                }
                else if (mortality_dist(generator) < mortality_rate)
                {
                    new_population[i] = 3;  // Dead
                    deaths++;
                }
                else
                {
                    // Try to infect others
                    for (int j = 0; j < population_size; ++j)
                    {
                        if (population[j] == 0 && infection_dist(generator) < infection_rate)
                        {
                            new_population[j] = 1;  // Newly infected
                            new_infections++;
                        }
                    }
                }
            }
        }

        population = new_population;

        int total_infected = std::count(population.begin(), population.end(), 1);
        int total_recovered = std::count(population.begin(), population.end(), 2);
        int total_dead = std::count(population.begin(), population.end(), 3);

        // Log the day's results
        std::cout << "Day " << day << ": " << new_infections << " new infections, " 
                  << recoveries << " recoveries, " << deaths << " deaths.\n";

        // Add events and attributes to the span
        day_span->AddEvent("Day summary", {
            {"new_infections", new_infections},
            {"recoveries", recoveries},
            {"deaths", deaths},
        });
        day_span->SetAttribute("day_number", day);
        day_span->SetAttribute("total_infected", total_infected);
        day_span->SetAttribute("total_recovered", total_recovered);
        day_span->SetAttribute("total_dead", total_dead);

        day_span->End();
        day++;

        // Sleep to slow down the simulation
        std::this_thread::sleep_for(std::chrono::seconds(1));  // Sleep for 1 second between days
    }

    simulation_span->End();

    // Clean up and flush tracer
    CleanupTracer();

    return 0;
}

Key points to note:

1. InitTracer : where we create our tracer instance and pass options (such as OTLP endpoint)

2. tracer->StartSpan declarations to create spans

3. AddEvent and SetAttribute to add crucial metadata to the spans. This can be valuable to store counts, facets for later processing in storage.

4. Ending and Cleaning up to flush

Leveraging spans as a rich data structure provides a versatile way to instrument C++ applications and get a detailed inspection of behavior and performance.

Time to Data/Repeated Time to Data

In frantic situations, the last thing observability users need is slow queries and overall “slow time to data”. Here we define “Data” as crucial information that assists in investigation. This can take various forms:

1. Slow Queries from Observability Backend: Metrics/logs/trace/etc queries are resulting in timeouts, errors, overload from concurrency, etc. This can cause a repeating pattern of further querying, frustration, loss of context

2. Obstacles to “Data”: Sometimes, due to product opinions, “Data” can be occluded intentionally to steer the user and force engagement. This is also immensely frustrating.

3. Malformed “Data”: Often, the expectation of how the data should be displayed deviates greatly, also leading to enormous context loss.

Investigations usually show compounding patterns, and all of these can add up incrementally.

Data Structure Disjointedness

Inconsistency in data structures is another avenue of enormous frustration. Consider log formatting:

1. Deeply Nested “JSONic” Logs

2. System Components Logs (Apache, NGINX, etc)

3. Free-form Application Logs

4. Stack Traces

Furthermore, if observability initiatives embrace tracing, detachment of spans due to missing trace ids (missing headers) can lead to gaps in observability for micro-service environments.

All of this results in “Data” that is less “joinable” and not pointing to causative factors for an incident.

Cognitive Load of Noise and Context Shifting

As humans, cognitive and emotional strain starts to pile up during active incidents and system failure. Little does tooling consider this during design. Noise, in the form of extraneous “Data”, too much “Data” doesn’t translate well to the unfortunately narrowing perception budget of the observability practitioner. This is further aggravated by UIs that are of an entirely different design philosophy and must be paired together during incident analysis.

Minimal Engagement/Communication Loss

Finally, gaps in communication or simply, observability not being utilized, is a significant problem. Very often, visualization is configured without considering that human operators won’t constantly be observing. Unfortunately, alerting configuration is non-intuitively designed vs its graphing counterpart. Furthermore, unreliable or ignored communication channels culminate into an ultimate panic event.

SRE-experience metric

When examining these frequent obstacles to incident resolution, tooling must start to consider metrics for emotional fatigue and general organizational chaos. Only then can we accurately have tooling comprehend what observability users have to endure during these types of events.

Trace Data Model

Just like Logs and Metrics, the OTeL Spec defines a Data Model (for Spans). Let's look at a few key fields:

• Time

  • start_time and end_time: together these can calculate duration

• Context

  • trace_id and span_id: unique identifiers that help link spans together as part of a trace

• Kind

  • Can be one either Client, Server, Internal, Producer, or Consumer: provides clues to how spans should be linked. Represents outgoing calls, incoming calls, or calls that are nested internally.

• Status

  • status_code and status_message: provides information on wether the span represents an error or without issue ("ok").

• Attribution

  • resource and attributes: just like other telemetry, resource attribution can represent metadata about the runtime environment (host, pod, etc). Attributes can help add metadata context to spans, like User ID.

• Links

  • "Links" can help bridge different traces. Especially useful for asynchronous and distributed operations.

• Events

  • Additional data that can represent a singular event of significance.

Instrumentation

Zero-Code Instrumentation

Zero-Code instrumentation libraries provide an effective way for practitioners to gather trace telemetry without code changes. Here is an example with Node.Js:

env OTEL_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=your-endpoint \
node --require @opentelemetry/auto-instrumentations-node/register app.js

Note how OTEL_EXPORTER_OTLP_TRACES_ENDPOINT points to your collector. There are various environment variables one can set, examples as outlined here.

Additionally, one can verify supported libraries (Node.Js example).

Output

Once this is configured, you can start receiving traces, as seen below:

Logs Data Model

There are various fields within the data model specification. Let's cover them categorically:

• Time

  • Timestamp: The timestamp from the source (for example, when a log was generated by an SDK. Preferred)

  • ObservedTimestamp: The timestamp from the collection system (for example, when the collector receives the log message)

• Trace

  • TraceId, SpanId, TraceFlags: For correlation to traces and spans. TraceFlags provides recommendations such as 'sampling' level, etc

• Severity

  • SeverityText, SeverityNumber: "log level". Optional fields but OTeL provides numerical representations for granular log levels (TRACE, DEBUG, etc)

• Body

  • Body: the log message body.

• Attribution

  • Resource: Key / Value pairs that define and scope where messages originate from (for example: pods, instances, services, etc)

  • Attributes: Key / Value pairs that give additional information about the message (for example: http method, status, etc)

  • InstrumentationScope: Provides information on emitting libraries (for example: OTeL SDK library and version)

File Log Receiver

This receiver has a multitude of configuration options. Again, let's explore categorically and highlight important ones:

• File Match Patterns

  • include, exclude: glob patterns that define files to be read, and specific ones to be excluded (for example: /var/log/folder/*.json )

• Processing

  • multiline: Specifies a match pattern that can be used to specify new records. Incredibly useful for Error / Stack Trace type logs.

  • storage: Critical to offset tracking (file storage). Points to an "extension" in the OTeL spec.

  • operators: There are various "operators" that can take one of the fields (like the message body) and parse it (for example, parse json, time, uri, etc). These can be chained together via the output option. You can use if conditionals and also embed processors

• Performance

  • poll_interval: Duration between File System polls

  • max_log_size: Max size of log entry to read to protect from large memory usage

• Metadata Decoration

  • include_file_*: Various config parameters (boolean) like include_file_name, include_file_path, etc that can be added as useful attributes

  • attributes / resource: Directly add key / value pairs for all log messages from a file referenced in a receiver.

Offset Tracking for Reliability

Offset tracking is crucial to ensure log collection is reliable and accurate if the collector(s) restart. Using the file_storage extension, one can specify a directory for storage.

receivers:
  filelog:
    include: [/var/log/service/sample.log]
    storage: file_storage/filelogreceiver
extensions:
  file_storage/filelogreceiver:
    directory: /var/lib/otelcol/mydir

Configuration

receivers:
  hostmetrics:
    collection_interval: 10s
    scrapers:
      process:
          metrics:
            process.cpu.utilization:
              enabled: true
            process.disk.operations:
              enabled: true
            process.memory.utilization:
              enabled: true
            process.threads:
              enabled: true

As part of the hostmetrics receiver, the process scraper can easily be added. This example configuration provides a sufficient instantiation, with gauge like metrics such as cpu and memory utilization being enabled.

Resource Attributes

Valuable resource attributes are populated alongside the metrics. Of these, process.pid, process.owner, process.command and process.command_line are particularly interesting. Process identifiers and owners can help highlight resource contention within a host. Changes in process command line arguments can highlight anomalous changes in metrics due to different command line strings.

Sensitive Data Scrubbing

One thing to note is the potential to leak sensitive command line arguments as part of data collection. It is imperative to add the transform processor

transform:
  metric_statements:
    - context: metric
      statements:
        - replace_pattern(resource.attributes["process.command_line"], "password\\=[^\\s]*(\\s?)", "password=***")

An example where the command_line resource attribute with a password argument is redacted via OTTL (OpenTelemetry Transformation Language).

Conclusion

Process monitoring is an excellent way to isolate long-running processes that are exhausting resources or ephemeral processes that cause sudden anomalies. OTeL provides a clean and concise way to do just this.

opentelemetry-collector:
  presets:
    logsCollection:
      enabled: true
    kubernetesAttributes:
      enabled: true
    kubeletMetrics:
      enabled: true
    hostMetrics:
      enabled: true
    clusterMetrics:
      enabled: true
    kubernetesEvents:
      enabled: true

The above concisely covers most of what observability practitioners would want to cover. We can dive into each individually.

Logs Collection

The logsCollection preset is essentially the filelogreceiver. By default it will collect /var/log/pods/*/*/*.log

Kubernetes Attributes

The Kubernetes Attributes Processor is probably considered the most important component. It specifically adds Kubernetes context to telemetry for effective correlation and filtration. Additional options should also be examined for further metadata extraction:

opentelemetry-collector:
  presets:
    kubernetesAttributes:
      enabled: true
      # When enabled the processor will extra all labels for an associated pod and add them as resource attributes.
      # The label's exact name will be the key.
      extractAllPodLabels: false
      # When enabled the processor will extra all annotations for an associated pod and add them as resource attributes.
      # The annotation's exact name will be the key.
      extractAllPodAnnotations: false

This is recommended for most deployments to maximize metadata decoration. In our minimal example we can expect these attributes by default:

• k8s.namespace.name

• k8s.pod.name

• k8s.pod.uid

• k8s.pod.start_time

• k8s.deployment.name

• k8s.node.name

Kublet Metrics

This preset adds the kubeletstatsreceiver to the configuration and collects node, pod and container metrics. The metrics definitions, such as container.cpu.time, k8s.pod.cpu.utilization can be found here.

Host Metrics

Similar to Part 1, we can get details about the host (or node) by adding a hostmetricsreceiver

Cluster Metrics

Cluster level metrics can be collected by enabling this (adding a k8sclusterreceiver). Similarly to Kublet Metrics, we have definitions of metrics here

Kubernetes Events

This adds the k8sobjectsreceiver and pulls/watches (via the K8s API) for K8s events. This is outputted as individual logs in the collector pipeline.

Config Section

Beyond the presets, within the values.yaml, one can provide specific configurations in the config section:

opentelemetry-collector:
  presets:
    kubernetesAttributes:
      enabled: true
    kubeletMetrics:
      enabled: true
    hostMetrics:
      enabled: true
    clusterMetrics:
      enabled: true
    kubernetesEvents:
      enabled: true
  config:
    receivers:
      kubeletstats:
        insecure_skip_verify: true
    extensions:
      bearertokenauth:
        token: $API_KEY
    exporters:
      otlphttp/example:
        endpoint: "https://intake.test_domain.com"
        auth:
          authenticator: bearertokenauth

    service:
      pipelines:
        traces:
          exporters: [spanmetrics, otlphttp/example]
        metrics:
          exporters: [otlphttp/example]
        logs:
          exporters: [otlphttp/example]
      extensions: [health_check, bearertokenauth]
      telemetry:
        metrics:
          level: "none"

In this configuration example, we are defining exporter configurations with authentication. Additionally, we are modifying the kubeletstatsreceiver from the preset, and adding a config option (insecure_skip_verify: true).

Conclusion

In Part 1 and 2, we introduce ideal configuration options that add health metrics and attributes that can benefit observability practitioners as soon as they deploy an OTeL collector. In future blog posts, we will continue to explore configuration profiles.

Host Metrics and Attributes

The hostmetricsreceiver is a mainstay of the OTeL collector repository and provides system health metrics. There are various components within the configuration patterns that can be enabled selectively.

Collection Interval

Receivers can scrape at various intervals:

hostmetrics:
        collection_interval: 10s

Depending on data retention windows and granularity needed, one should expect to set this from 5s-30s (in increments of 5s), although any frequency can be used.

CPU and Memory

CPU and Memory metrics can be enabled, however, it is important to know which ones are on by "default". Let's walkthrough a config:

hostmetrics:
        collection_interval: 10s
        scrapers:
          cpu:
            metrics:
              system.cpu.utilization:
                enabled: true
          memory:
            metrics:
              system.memory.utilization:
                enabled: true

In the above code, just enabling cpu populates system.cpu.time time metrics as a sum. system.cpu.utilization is recommended as it provides percentage based metrics as a gauge. Operators may be more familiar with this.

It's important to consider that states such as idle, wait, user are in the form of attributes, not unique metric names. So instead of system.cpu.utilization.user, depending on the backend, one must filter by an attribute / tag

Similarly, system.memory.utilization offers gauge like metrics that may be more appetizing. Attributes such as used and free are also available rather than unique metric names.

Disk and Filesystem

Disk performance and overall filesystem usage can be added as seen in the below config:

hostmetrics:
        collection_interval: 10s
        scrapers:
          disk:
          filesystem:
            metrics:
              system.filesystem.utilization:
                enabled: true

Important metrics, especially system.disk.io, are populated via the disk scraper. Attributes provide read and write info by device. Filesystem usage (space used vs free) is also added, with system.filesystem.utilization providing gauge like metrics. Again, attributes exist for free and used.

Network

To get network I/O, connections, and errors/dropped packets:

hostmetrics:
        collection_interval: 10s
        scrapers:
          network:

These metrics can be quite valuable as they can pinpoint network issues by device, protocol and receive / transmit

Resource Attributes

Resource attribution decoration is crucial to query metrics by information such as "host", "region", "provider", etc:

processors:
  resourcedetection:
    detectors: [gcp, ecs, ec2, azure, system]
    override: true

Keep in mind: order matters and the first detector to insert wins.

Part 2

In Part 2, we will explore relevant metrics and metadata decoration within a Kubernetes environment.

If I were to start a high school course around the topic, what would it look like..?

To me, it fundamentally speaks to the importance of understanding proper "feedback from a system" to accurately answer questions like:

• Why is it deviating from a baseline?

• If I were to make X or Y change, what data points would be forecasted to change alongside that?

• What is the system's "homeostasis"?

• How can I easily gather system output data that is best reflective of internal states?

If you strip away all the vernacular that we use in the industry, it becomes simply about being able to "ask" something about the system. A stable and open standard is crucial to that succeeding. Furthermore, fostering an environment that facilitates an enticing entry for future generations is even more significant.

OpenTelemetry (OTeL)

I think everyone in the Observability space has explored OTeL from the perspective of a practitioner, vendor, architect, engineer, IT/OPs, analyst, the list goes on. Opinions have varied from it being a liberator, misguided initiative, "not ready yet", indifference, or even adversarial.

With all that said, if we consider a student's (even beginner) approach to comfortably approaching this subject, we can all agree that humans hate context shifting. So whatever the state of the standard or what it looks like, the fact that the space is relentlessly heading towards it is good.

Every individual within the space can claim the cognitive burden induced by obscure data structures, unlinked data, and arbitrary config patterns. In fact, vendors have the most to benefit. Why is this? Sales cycles, onboarding and continuous engagement.

Ask every account person and customer in Observability, the main toil is re-education and re-absorption of concepts that should be quite fluid. Very often, the engineering and product teams focus on patterns that allow them to iterate quickly and unfortunately emerge as hyper-opinionated. This comes at a direct cost to the mental bandwidth of the account teams responsible for promoting to and maintaining the users. Not only are they consistently playing catch-up to the core product/eng teams, they are hearing the brunt of the complaints from customers.

OMLET

OMLET was started because we wanted to be part of the OTeL journey, but especially because promotion and evangelism of an open standard in this space is such a need. We plan to focus on the pipeline problem that restricts observability users from kickstarting their OTeL journey, or keeping it alive. Beyond the tech, standardization is very much an organizational and cultural need.

"You can't make an omelette without breaking eggs"