How bitfused/opentelemetry-collector works
Project Type & Purpose
• This is a full-stack, extensible telemetry service (“OpenTelemetry Collector”) written in Go.
• It receives telemetry (traces, metrics, logs) over multiple protocols, processes it through configurable pipelines, and exports it to various back-ends.
• Key subsystems: CLI/build tools, code generators (mdatagen, pdatagen), core service (“service” & “otelcol”), component frameworks (receiver, processor, exporter, connector, extension), internal helpers, and integration tests.
File-Structure Analysis
• Top-level binaries in cmd/ (e.g., otelcol, builder, mdatagen, githubgen).
• Core libraries under service/ (service builder, pipeline builder, component registry), otelcol/ (collector runner), and pipeline/.
• Component categories each live in its own directory: receiver/, processor/, exporter/, connector/, extension/, scraper/, etc., each with subpackages for helper libraries and metadata.
• Configuration lives in config/ and in YAML metadata files per component (e.g., metadata.yaml).
• Code-generation tools under cmd/mdatagen and pdata/ for protocol data types.
• Tests and examples scattered under each module and in internal/e2e.
Architecture Patterns & Principles
• Plugin + Factory Pattern: each component (receiver, exporter, etc.) registers a Factory that the service builder loads via reflection on metadata.
• Builder Pattern: service/internal/builders constructs the service graph from a YAML config.
• Separation of concerns: CLI/build, config loading, service core, component frameworks, component implementations, and code generation are clearly decoupled.
• Configuration-as-Metadata: YAML schemas drive code generation for strong-typed config.
Main Components
a. CLI & Build Tools (cmd/otelcol, cmd/builder, cmd/mdatagen)
b. Configuration Loader (confmap, config modules)
c. Service Core (service package):
– Component Registrar (loads all Factories)
– Pipeline Builder (constructs pipeline graph)
– Lifecycle Manager (start/stop)
d. Telemetry Pipelines:
– Receivers (e.g., OTLP gRPC/HTTP, Prometheus, Jaeger)
– Processors (e.g., batchprocessor, memorylimiterprocessor)
– Exporters (e.g., OTLP exporter, logging exporter, debugexporter)
– Connectors (chain multiple pipelines)
– Scrapers
e. Extensions (zPages, memorylimiterextension, auth extensions)
f. Component Helpers & Shared Libraries (collectorhelper, emitter, confmap)
g. Back-ends / External Systems (third-party telemetry back-ends, observability dashboards)
Relationships & Data Flows
• CLI → reads command-line flags & config file → invokes Configuration Loader.
• Configuration Loader → outputs typed config objects → passed to Service Core.
• Service Core → registers Factories from all component modules → pipelines are instantiated per config.
External Telemetry Sources → Receiver components → Processor chain → Exporter components → External Back-ends
• Extensions hook into the service lifecycle (before/after start) and can expose HTTP endpoints (e.g., zPages).
• Internal telemetry (service metrics/logs) is emitted via exporterhelper back into the pipeline or a dedicated telemetry pipeline.
1. Sketch top-level layers: Build Tools, Config, Core, Components, External.
3. Under Components, create five grouped columns: Receivers, Processors, Exporters, Connectors, Extensions.
7. Internal Telemetry path: Service Core metrics → exporterhelper → pipeline.
8. Code-generation tools feeding into component metadata (cmd/mdatagen → metadata files → component helpers).