How open-telemetry/opentelemetry-collector works
1. This repository is the OpenTelemetry Collector core codebase: a Go multi-module workspace that provides the runtime, shared APIs, signal data model, component implementations, and code generators for collector distributions.
2. The main execution path is:
config loading → service assembly → pipeline graph wiring → signal ingestion/processing/export.
The otelcol package is the generic runtime/CLI, while cmd/otelcorecol is a concrete distribution binary built from a fixed component set.
3. service/ is the orchestration layer. It turns a parsed configuration into running graphs of receivers, processors, connectors, exporters, and extensions. It also manages lifecycle, capability negotiation, component wiring, internal telemetry, and zPages/service diagnostics.
4. pipeline/, receiver/, processor/, exporter/, connector/, scraper/, and extension/ define the component abstractions and helper utilities for each role in the data path. These packages split the collector into clear boundaries: input adapters, transformation stages, routing/fanout, output adapters, periodic scrapers, and out-of-band service extensions.
5. The data model layer lives in pdata/. It contains the in-memory representations and codecs for logs, metrics, traces, and profiles, plus OTLP/gRPC request/response types and JSON/protobuf support. Most runtime components depend on pdata rather than on raw protobuf messages.
6. confmap/ is the configuration ingestion subsystem. It resolves config from multiple providers such as files, env vars, HTTP/HTTPS, and YAML, then expands, merges, validates, and unmarshals it into typed component configs. This is the boundary between external config sources and the service graph.
7. component/, consumer/, producer-like helpers, and host/telemetry packages define the shared runtime contracts: component identity, status, host capabilities, error semantics, and observability hooks. internal/sharedcomponent, internal/statusutil, and similar packages hold cross-cutting implementation details that should not leak into public APIs.
8. Many subdirectories are concrete component implementations:
- receiver/otlpreceiver for OTLP ingest over gRPC/HTTP
- exporter/otlpexporter and exporter/otlphttpexporter for OTLP emission
- processor/batchprocessor and processor/memorylimiterprocessor
- extension/memorylimiterextension, extension/zpagesextension, extension/extensionauth
- scraper/* for metric/profile scraping
These are assembled by the service graph according to configured pipelines.
9. Connectors are a distinct boundary in the graph. The connector/ packages bridge pipelines by acting as both consumer and producer, enabling fanout, routing, and cross-signal joins without collapsing the pipeline abstraction.
10. The repository heavily uses generators and metadata files. cmd/mdatagen, cmd/schemagen, internal/cmd/pdatagen, and many metadata.yaml files generate configs, schemas, docs, status artifacts, and test fixtures. This means much of the repo’s “shape” is defined declaratively and materialized into code during build time.
11. cmd/builder is a distribution factory: it takes a component manifest and emits a tailor-made collector binary and supporting source. The reports/ and distributions.yaml files capture curated component sets used to define official distributions like core, contrib, k8s, and otlp.
12. internal/e2e, *_test helper modules, and the examples/ and docs/ trees provide the integration boundary. They validate that configuration, service wiring, data-path behavior, and component contracts work together across signals and platforms.
13. In architecture-diagram terms, the repo is best drawn as layered:
external config sources and telemetry inputs at the top/left; config resolution and service graph in the middle; signal model/runtime helpers beneath; concrete receivers/processors/connectors/exporters/extensions/scrapers around the graph; and pdata plus code generators as the foundational substrate underneath everything else.
Main entry points: pdata pdata · graph graph · service service · consumer consumer
Components 34
Runtime
Collector execution, service assembly, and distribution binaries.
- otelcol · Generic CLI/runtime that loads config and runs collector services.
otelcol - otelcorecol · Concrete collector distribution built from a fixed component set.
cmd/otelcorecol - service · Turns configuration into running pipelines, extensions, and lifecycle-managed graphs.
service - graph · Builds component graphs, capabilities, fanout, host plumbing, and zPages wiring.
service/internal/graph - pipelines · Assembles signal pipelines from configured receivers, processors, connectors, and exporters.
service/pipelines
Contracts
Shared component interfaces and helper boundaries for the data path.
- component · Core component identity, host, telemetry, and status contracts shared across the repo.
component - consumer · Signal consumer interfaces and error semantics for downstream delivery.
consumer - pipeline · Shared pipeline abstractions and signal boundary definitions.
pipeline - receiver · Receiver interfaces and reusable receiver helper contracts.
receiver - processor · Processor interfaces and helper patterns for in-flight signal transformation.
processor - exporter · Exporter interfaces plus helper patterns for delivering signals outward.
exporter - connector · Connectors bridge pipelines by acting as both consumers and producers.
connector - scraper · Scraper interfaces for periodically collecting metrics and profiles.
scraper - extension · Extensions that add service-side capabilities without participating in pipelines.
extension
Data Model
In-memory signal representations and OTLP codecs.
- pdata · In-memory logs, metrics, traces, and profiles plus OTLP request/response codecs.
pdata - pcommon · Shared primitives, resources, IDs, timestamps, and value types.
pdata/pcommon - plog · Logs data structures, JSON/proto support, and OTLP request wrappers.
pdata/plog - pmetric · Metrics data structures, aggregation types, and OTLP request wrappers.
pdata/pmetric - ptrace · Trace/span data structures, status types, and OTLP request wrappers.
pdata/ptrace - pprofile · Profile data structures, dictionaries, and OTLP request wrappers.
pdata/pprofile
Config
Configuration resolution, validation, and typed unmarshaling.
Generators
Build-time tools and metadata-driven code generation.
- pdatagen · Generates pdata wrappers, request types, and proto/json helpers from metadata and templates.
internal/cmd/pdatagen - mdatagen · Generates configs, docs, status artifacts, and fixtures from component metadata.
cmd/mdatagen - schemagen · Builds configuration schemas from Go structs and overlays.
cmd/schemagen - builder · Emits tailored collector binaries from a manifest of selected components.
cmd/builder - distributions · Curated distribution definitions for core, contrib, k8s, and otlp builds.
reports/distributions
Components
Concrete receivers, processors, exporters, connectors, extensions, and scrapers.
- OTLP recv · OTLP ingest over gRPC/HTTP for logs, metrics, traces, and profiles.
receiver/otlpreceiver - batch proc · Batches and splits signals before forwarding downstream.
processor/batchprocessor - mem limiter · Enforces memory pressure limits and sheds load when needed.
processor/memorylimiterprocessor - OTLP exp · OTLP exporter implementation, primarily gRPC-based.
exporter/otlpexporter - OTLP HTTP · OTLP exporter over HTTP for signal emission.
exporter/otlphttpexporter - forward · Forwarding connector for pipeline bridging and fanout flows.
connector/forwardconnector - mem ext · Memory limiter extension for service-side protection and coordination.
extension/memorylimiterextension
Connections 33
- confmap → otelcol: loads config · Config resolution feeds the runtime CLI.
- otelcol → service: assembles · The runtime creates services from resolved configuration.
- otelcorecol → service: ships · Concrete distribution binary is built on the same service layer.
- service → graph: wires · Service turns configs into component graphs.
- service → pipelines: builds · Pipeline definitions are instantiated into running data paths.
- graph → receiver: uses · Graph wiring depends on receiver contracts.
- graph → processor: uses · Graph wiring depends on processor contracts.
- graph → exporter: uses · Graph wiring depends on exporter contracts.
- graph → connector: uses · Graph wiring depends on connector contracts.
- graph → extension: uses · Graph wiring includes extensions and host capabilities.
- pipeline → consumer: defines · Pipeline boundaries are built around consumers.
- receiver → consumer: pushes to · Receivers deliver data into consumers.
- processor → consumer: accepts · Processors consume and forward signals.
- exporter → consumer: implements · Exporters present consumer-facing send paths.
- connector → consumer: bridges · Connectors act as consumers on one side of a pipeline.
- connector → pipeline: bridges · Connectors bridge pipelines without flattening the graph.
- scraper → pdata: emits · Scrapers populate pdata-native signal structures.
- pdata → pcommon: shares · Signal types reuse common resource and value primitives.
- pdata → plog: contains · Logs model is a first-class pdata subsystem.
- pdata → pmetric: contains · Metrics model is a first-class pdata subsystem.
- pdata → ptrace: contains · Traces model is a first-class pdata subsystem.
- pdata → pprofile: contains · Profiles model is a first-class pdata subsystem.
- OTLP recv → pdata: decodes to · OTLP receiver converts wire payloads into pdata.
- OTLP exp → pdata: encodes from · OTLP exporter serializes pdata to wire payloads.
- OTLP HTTP → pdata: encodes from · HTTP exporter serializes pdata to wire payloads.
- batch proc → pdata: rewrites · Batch processor groups pdata before downstream delivery.
- mem limiter → graph: protects · Memory limiter participates in service-level protection.
- mem ext → service: extends · Extension augments service behavior outside pipelines.
- pdatagen → pdata: generates · Generator materializes pdata types and codecs.
- mdatagen → component: generates metadata · Metadata generator produces component docs, configs, and status artifacts.
- schemagen → confmap: generates schemas · Schema generator supports typed config ingestion.
- builder → distributions: builds from · Distribution builder consumes curated component manifests.
- builder → otelcorecol: emits · Builder produces tailored binaries such as the core collector.