# bitfused/opentelemetry-collector architecture

> Architecture diagram of the GitHub repository bitfused/opentelemetry-collector, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bitfused/opentelemetry-collector
- Repository: https://github.com/bitfused/opentelemetry-collector

## Overview

##### Project Type & Purpose
- This is a full-stack, extensible telemetry service (“OpenTelemetry Collector”) written in Go.
- It receives telemetry (traces, metrics, logs) over multiple protocols, processes it through configurable pipelines, and exports it to various back-ends.
- Key subsystems: CLI/build tools, code generators (mdatagen, pdatagen), core service (“service” & “otelcol”), component frameworks (receiver, processor, exporter, connector, extension), internal helpers, and integration tests.

##### File-Structure Analysis
- Top-level binaries in `cmd/` (e.g., `otelcol`, `builder`, `mdatagen`, `githubgen`).
- Core libraries under `service/` (service builder, pipeline builder, component registry), `otelcol/` (collector runner), and `pipeline/`.
- Component categories each live in its own directory: `receiver/`, `processor/`, `exporter/`, `connector/`, `extension/`, `scraper/`, etc., each with subpackages for helper libraries and metadata.
- Configuration lives in `config/` and in YAML metadata files per component (e.g., `metadata.yaml`).
- Code-generation tools under `cmd/mdatagen` and `pdata/` for protocol data types.
- Tests and examples scattered under each module and in `internal/e2e`.

##### Architecture Patterns & Principles
- Plugin + Factory Pattern: each component (receiver, exporter, etc.) registers a Factory that the service builder loads via reflection on metadata.
- Builder Pattern: `service/internal/builders` constructs the service graph from a YAML config.
- Separation of concerns: CLI/build, config loading, service core, component frameworks, component implementations, and code generation are clearly decoupled.
- Configuration-as-Metadata: YAML schemas drive code generation for strong-typed config.

##### Main Components
a. CLI & Build Tools (cmd/otelcol, cmd/builder, cmd/mdatagen)
b. Configuration Loader (confmap, config modules)
c. Service Core (service package):
– Component Registrar (loads all Factories)
– Pipeline Builder (constructs pipeline graph)
– Lifecycle Manager (start/stop)
d. Telemetry Pipelines:
– Receivers (e.g., OTLP gRPC/HTTP, Prometheus, Jaeger)
– Processors (e.g., batchprocessor, memorylimiterprocessor)
– Exporters (e.g., OTLP exporter, logging exporter, debugexporter)
– Connectors (chain multiple pipelines)
– Scrapers
e. Extensions (zPages, memorylimiterextension, auth extensions)
f. Component Helpers & Shared Libraries (collectorhelper, emitter, confmap)
g. Back-ends / External Systems (third-party telemetry back-ends, observability dashboards)

##### Relationships & Data Flows
- CLI → reads command-line flags & config file → invokes Configuration Loader.
- Configuration Loader → outputs typed config objects → passed to Service Core.
- Service Core → registers Factories from all component modules → pipelines are instantiated per config.
External Telemetry Sources → Receiver components → Processor chain → Exporter components → External Back-ends
- Extensions hook into the service lifecycle (before/after start) and can expose HTTP endpoints (e.g., zPages).
- Internal telemetry (service metrics/logs) is emitted via exporterhelper back into the pipeline or a dedicated telemetry pipeline.

1. Sketch top-level layers: Build Tools, Config, Core, Components, External.
3. Under Components, create five grouped columns: Receivers, Processors, Exporters, Connectors, Extensions.
7. Internal Telemetry path: Service Core metrics → exporterhelper → pipeline.
8. Code-generation tools feeding into component metadata (cmd/mdatagen → metadata files → component helpers).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Build & Codegen Layer
    subgraph "Build & Code Generation"
        direction TB
        CLI_Otelcol["otelcol CLI"]:::build
        CLI_Builder["Builder CLI"]:::build
        CLI_Mdatagen["Metadata Generator (mdatagen)"]:::build
        Pdatagen["Protocol Data Generator (pdatagen)"]:::build
    end

    %% Configuration Layer
    subgraph "Configuration"
        direction TB
        Confmap["Configuration Loader (confmap)"]:::core
        Config_Modules["Config Modules"]:::metadata
    end

    %% Service Core Layer
    subgraph "Service Core"
        direction TB
        Core_Service["Core Service"]:::core
        Factory_Registry["Factory Registry"]:::core
        Pipeline_Builder["Pipeline Builder"]:::core
        Lifecycle_Manager["Lifecycle Manager"]:::core
    end

    %% Component Frameworks Layer
    subgraph "Component Frameworks"
        direction TB
        Receiver_Framework["Receiver Framework"]:::framework
        Processor_Framework["Processor Framework"]:::framework
        Exporter_Framework["Exporter Framework"]:::framework
        Connector_Framework["Connector Framework"]:::framework
        Scraper_Framework["Scraper Framework"]:::framework
        Extension_Framework["Extension Framework"]:::framework
    end

    %% Component Implementations Layer
    subgraph "Component Implementations"
        direction TB
        subgraph Receivers
            direction TB
            OTLP_Receiver["OTLP Receiver"]:::framework
            Nop_Receiver["No-op Receiver"]:::framework
            Receiver_Helper["Receiver Helper"]:::framework
        end
        subgraph Processors
            direction TB
            Batch_Processor["Batch Processor"]:::framework
            MemoryLimiter_Processor["Memory Limiter Processor"]:::framework
            Processor_Helper["Processor Helper"]:::framework
        end
        subgraph Exporters
            direction TB
            OTLP_Exporter["OTLP Exporter"]:::framework
            OTLPHTTP_Exporter["HTTP OTLP Exporter"]:::framework
            Debug_Exporter["Debug Exporter"]:::framework
            Nop_Exporter["No-op Exporter"]:::framework
            Exporter_Helper["Exporter Helper"]:::framework
        end
        subgraph Connectors
            direction TB
            Forward_Connector["Forward Connector"]:::framework
            X_Connector["Example Connector"]:::framework
        end
        subgraph Scrapers
            direction TB
            Scraper_Impl["Sample Scraper"]:::framework
            Scraper_Helper["Scraper Helper"]:::framework
        end
        subgraph Extensions
            direction TB
            MemoryLimiter_Ext["Memory Limiter Extension"]:::framework
            ZPages_Ext["zPages Extension"]:::framework
            Auth_Ext["Auth Extension"]:::framework
        end
    end

    %% Helpers & Shared Libraries
    subgraph "Helpers & Shared Libraries"
        direction TB
        Confmap_Dir["Confmap Library"]:::framework
        FeatureGate["Feature Gate"]:::framework
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        Telemetry_Sources["External Telemetry Sources"]:::external
        External_Backends["External Back-ends"]:::external
        ZPages_UI["zPages UI"]:::external
    end

    %% Control Flow
    CLI_Otelcol -->|"invokes"| Confmap
    CLI_Builder --> Confmap
    CLI_Mdatagen --> Config_Modules
    Pdatagen --> Config_Modules

    Confmap -->|"provides config"| Core_Service
    Config_Modules --> Core_Service

    Core_Service --> Factory_Registry
    Core_Service --> Pipeline_Builder
    Core_Service --> Lifecycle_Manager

    Factory_Registry -.-> Receiver_Framework
    Factory_Registry -.-> Processor_Framework
    Factory_Registry -.-> Exporter_Framework
    Factory_Registry -.-> Connector_Framework
    Factory_Registry -.-> Scraper_Framework
    Factory_Registry -.-> Extension_Framework

    Pipeline_Builder -->|"builds pipelines"| OTLP_Receiver
    Pipeline_Builder --> Nop_Receiver
    OTLP_Receiver --> Processor_Framework
    Nop_Receiver --> Processor_Framework

    Processor_Framework --> Batch_Processor
    Processor_Framework --> MemoryLimiter_Processor

    Batch_Processor --> Exporter_Framework
    MemoryLimiter_Processor --> Exporter_Framework

    Exporter_Framework --> OTLP_Exporter
    Exporter_Framework --> OTLPHTTP_Exporter
    Exporter_Framework --> Debug_Exporter
    Exporter_Framework --> Nop_Exporter

    OTLP_Exporter -->|"exports"| External_Backends
    OTLPHTTP_Exporter --> External_Backends
    Debug_Exporter --> External_Backends
    Nop_Exporter --> External_Backends

    Lifecycle_Manager -.-> MemoryLimiter_Ext
    Lifecycle_Manager -.-> ZPages_Ext
    Lifecycle_Manager -.-> Auth_Ext

    Telemetry_Sources --> OTLP_Receiver
    Telemetry_Sources --> Nop_Receiver

    %% Internal Telemetry Path
    Core_Service --> Exporter_Helper
    Exporter_Helper --> Batch_Processor

    %% Codegen to Metadata
    CLI_Mdatagen -->|"generates metadata"| Config_Modules
    Pdatagen -->|"generates pdata"| FeatureGate

    %% Click Events
    click CLI_Otelcol "https://github.com/bitfused/opentelemetry-collector/blob/main/cmd/otelcorecol/main.go"
    click CLI_Builder "https://github.com/bitfused/opentelemetry-collector/blob/main/cmd/builder/main.go"
    click CLI_Mdatagen "https://github.com/bitfused/opentelemetry-collector/blob/main/cmd/mdatagen/main.go"
    click Pdatagen "https://github.com/bitfused/opentelemetry-collector/blob/main/pdata/internal/cmd/pdatagen/main.go"
    click Confmap "https://github.com/bitfused/opentelemetry-collector/blob/main/confmap/confmap.go"
    click Config_Modules "https://github.com/bitfused/opentelemetry-collector/tree/main/config/"
    click Core_Service "https://github.com/bitfused/opentelemetry-collector/blob/main/service/service.go"
    click Factory_Registry "https://github.com/bitfused/opentelemetry-collector/blob/main/service/attributes.go"
    click Pipeline_Builder "https://github.com/bitfused/opentelemetry-collector/blob/main/service/internal/builders/builders.go"
    click Receiver_Framework "https://github.com/bitfused/opentelemetry-collector/blob/main/receiver/receiver.go"
    click Receiver_Helper "https://github.com/bitfused/opentelemetry-collector/blob/main/receiver/receiverhelper/receiverhelper.go"
    click OTLP_Receiver "https://github.com/bitfused/opentelemetry-collector/blob/main/receiver/otlpreceiver/factory.go"
    click Nop_Receiver "https://github.com/bitfused/opentelemetry-collector/blob/main/receiver/nopreceiver/nop_receiver.go"
    click Processor_Framework "https://github.com/bitfused/opentelemetry-collector/blob/main/processor/processor.go"
    click Processor_Helper "https://github.com/bitfused/opentelemetry-collector/blob/main/processor/processorhelper/processorhelper.go"
    click Batch_Processor "https://github.com/bitfused/opentelemetry-collector/blob/main/processor/batchprocessor/factory.go"
    click MemoryLimiter_Processor "https://github.com/bitfused/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/factory.go"
    click Exporter_Framework "https://github.com/bitfused/opentelemetry-collector/blob/main/exporter/exporter.go"
    click Exporter_Helper "https://github.com/bitfused/opentelemetry-collector/blob/main/exporter/exporterhelper/common.go"
    click OTLP_Exporter "https://github.com/bitfused/opentelemetry-collector/blob/main/exporter/otlpexporter/factory.go"
    click OTLPHTTP_Exporter "https://github.com/bitfused/opentelemetry-collector/blob/main/exporter/otlphttpexporter/factory.go"
    click Debug_Exporter "https://github.com/bitfused/opentelemetry-collector/blob/main/exporter/debugexporter/factory.go"
    click Nop_Exporter "https://github.com/bitfused/opentelemetry-collector/blob/main/exporter/nopexporter/nop_exporter.go"
    click Connector_Framework "https://github.com/bitfused/opentelemetry-collector/blob/main/connector/connector.go"
    click Forward_Connector "https://github.com/bitfused/opentelemetry-collector/blob/main/connector/forwardconnector/forward.go"
    click X_Connector "https://github.com/bitfused/opentelemetry-collector/blob/main/connector/xconnector/connector.go"
    click Scraper_Impl "https://github.com/bitfused/opentelemetry-collector/blob/main/scraper/scraper.go"
    click Scraper_Helper "https://github.com/bitfused/opentelemetry-collector/blob/main/scraper/scraperhelper/controller.go"
    click Extension_Framework "https://github.com/bitfused/opentelemetry-collector/blob/main/extension/extension.go"
    click MemoryLimiter_Ext "https://github.com/bitfused/opentelemetry-collector/blob/main/extension/memorylimiterextension/factory.go"
    click ZPages_Ext "https://github.com/bitfused/opentelemetry-collector/blob/main/extension/zpagesextension/factory.go"
    click Auth_Ext "https://github.com/bitfused/opentelemetry-collector/blob/main/extension/extensionauth/server.go"
    click Confmap_Dir "https://github.com/bitfused/opentelemetry-collector/tree/main/confmap/"
    click FeatureGate "https://github.com/bitfused/opentelemetry-collector/blob/main/featuregate/flag.go"

    %% Styles
    classDef build fill:#FFA500,stroke:#333,stroke-width:1px
    classDef core fill:#8FBC8F,stroke:#333,stroke-width:1px
    classDef framework fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef metadata fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef external fill:#DDA0DD,stroke:#333,stroke-width:1px
```
