OTeL Setup for Bare-Metal and Kubernetes (Pt 2)

In Part 1, we explored the hostmetricsreceiver as an excellent way to get system metrics. Moving onto a K8s environment, we must consider the importance of additional metadata and the importance of metrics at a component level. Before diving into the conventional ways of configuring (verbose configurations), one can utilize ready-made presets via Helm. Here is an example values.yaml:

opentelemetry-collector:
  presets:
    logsCollection:
      enabled: true
    kubernetesAttributes:
      enabled: true
    kubeletMetrics:
      enabled: true
    hostMetrics:
      enabled: true
    clusterMetrics:
      enabled: true
    kubernetesEvents:
      enabled: true

The above concisely covers most of what observability practitioners would want to cover. We can dive into each individually.

Logs Collection

The logsCollection preset is essentially the filelogreceiver. By default it will collect /var/log/pods/*/*/*.log

Kubernetes Attributes

The Kubernetes Attributes Processor is probably considered the most important component. It specifically adds Kubernetes context to telemetry for effective correlation and filtration. Additional options should also be examined for further metadata extraction:

opentelemetry-collector:
  presets:
    kubernetesAttributes:
      enabled: true
      # When enabled the processor will extra all labels for an associated pod and add them as resource attributes.
      # The label's exact name will be the key.
      extractAllPodLabels: false
      # When enabled the processor will extra all annotations for an associated pod and add them as resource attributes.
      # The annotation's exact name will be the key.
      extractAllPodAnnotations: false

This is recommended for most deployments to maximize metadata decoration. In our minimal example we can expect these attributes by default:

Kublet Metrics

This preset adds the kubeletstatsreceiver to the configuration and collects node, pod and container metrics. The metrics definitions, such as container.cpu.time, k8s.pod.cpu.utilization can be found here.

Host Metrics

Similar to Part 1, we can get details about the host (or node) by adding a hostmetricsreceiver

Cluster Metrics

Cluster level metrics can be collected by enabling this (adding a k8sclusterreceiver). Similarly to Kublet Metrics, we have definitions of metrics here

Kubernetes Events

This adds the k8sobjectsreceiver and pulls/watches (via the K8s API) for K8s events. This is outputted as individual logs in the collector pipeline.

Config Section

Beyond the presets, within the values.yaml, one can provide specific configurations in the config section:

opentelemetry-collector:
  presets:
    kubernetesAttributes:
      enabled: true
    kubeletMetrics:
      enabled: true
    hostMetrics:
      enabled: true
    clusterMetrics:
      enabled: true
    kubernetesEvents:
      enabled: true
  config:
    receivers:
      kubeletstats:
        insecure_skip_verify: true
    extensions:
      bearertokenauth:
        token: $API_KEY
    exporters:
      otlphttp/example:
        endpoint: "https://intake.test_domain.com"
        auth:
          authenticator: bearertokenauth

    service:
      pipelines:
        traces:
          exporters: [spanmetrics, otlphttp/example]
        metrics:
          exporters: [otlphttp/example]
        logs:
          exporters: [otlphttp/example]
      extensions: [health_check, bearertokenauth]
      telemetry:
        metrics:
          level: "none"

In this configuration example, we are defining exporter configurations with authentication. Additionally, we are modifying the kubeletstatsreceiver from the preset, and adding a config option (insecure_skip_verify: true).

Conclusion

In Part 1 and 2, we introduce ideal configuration options that add health metrics and attributes that can benefit observability practitioners as soon as they deploy an OTeL collector. In future blog posts, we will continue to explore configuration profiles.

Ideal OTeL Configurations for Bare-Metal and Kubernetes (Part 2)

Table of contents