Ideal OTeL Configurations for Bare-Metal and Kubernetes (Part 1)

Search for a command to run...

An SRE agent starts every investigation missing the same thing: a model of how the system is wired. Which services call which, what each one reads from, where workloads are co-resident, how much redun

OpenClaw's role as a constantly running agent that performs various encoded tasks has been exciting. Its focus on prioritizing communication channels and its strong emphasis on sessions are particularly noteworthy. However, security concerns have inc...

As AI agents flood the enterprise software landscape, companies must integrate their existing data hubs. In the data analytics world, this can include platforms like Snowflake. In CRM / front-office land, platforms like Salesforce. In Observability, ...

There have been several posts about AI SRE solutions being deployed in a "fire and forget" manner, even for complex setups. One post that tested this idea is from the Clickhouse team. They concluded that "Models are not ready" and needed guidance thr...

A while ago, we explored different LLM use cases through “Skillet”. During our experiments, we quickly realized the advantages of using tools, precise integration with observability data sources, and sparse tokenization. The ecosystem has been evolvi...

The intention of this post is to be an opinionated suggestion on collector configurations to gather comprehensive metrics for two fundamental environment types. Fortunately, the OTeL community has made it fairly straight-forward. In "Part 1", we will explore simple host monitoring.
The hostmetricsreceiver is a mainstay of the OTeL collector repository and provides system health metrics. There are various components within the configuration patterns that can be enabled selectively.
Receivers can scrape at various intervals:
hostmetrics:
collection_interval: 10s
Depending on data retention windows and granularity needed, one should expect to set this from 5s-30s (in increments of 5s), although any frequency can be used.
CPU and Memory metrics can be enabled, however, it is important to know which ones are on by "default". Let's walkthrough a config:
hostmetrics:
collection_interval: 10s
scrapers:
cpu:
metrics:
system.cpu.utilization:
enabled: true
memory:
metrics:
system.memory.utilization:
enabled: true
In the above code, just enabling cpu populates system.cpu.time time metrics as a sum. system.cpu.utilization is recommended as it provides percentage based metrics as a gauge. Operators may be more familiar with this.
It's important to consider that states such as idle, wait, user are in the form of attributes, not unique metric names. So instead of system.cpu.utilization.user, depending on the backend, one must filter by an attribute / tag
Similarly, system.memory.utilization offers gauge like metrics that may be more appetizing. Attributes such as used and free are also available rather than unique metric names.
Disk performance and overall filesystem usage can be added as seen in the below config:
hostmetrics:
collection_interval: 10s
scrapers:
disk:
filesystem:
metrics:
system.filesystem.utilization:
enabled: true
Important metrics, especially system.disk.io, are populated via the disk scraper. Attributes provide read and write info by device. Filesystem usage (space used vs free) is also added, with system.filesystem.utilization providing gauge like metrics. Again, attributes exist for free and used.
To get network I/O, connections, and errors/dropped packets:
hostmetrics:
collection_interval: 10s
scrapers:
network:
These metrics can be quite valuable as they can pinpoint network issues by device, protocol and receive / transmit
Resource attribution decoration is crucial to query metrics by information such as "host", "region", "provider", etc:
processors:
resourcedetection:
detectors: [gcp, ecs, ec2, azure, system]
override: true
Keep in mind: order matters and the first detector to insert wins.
In Part 2, we will explore relevant metrics and metadata decoration within a Kubernetes environment.