# signalfx/splunk-otel-collector architecture

> Architecture diagram of the GitHub repository signalfx/splunk-otel-collector, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/signalfx/splunk-otel-collector
- Repository: https://github.com/signalfx/splunk-otel-collector

## Overview

##### Project type and purpose
- This is an open-source, opinionated distribution of the OpenTelemetry Collector.
- Purpose: receive, process, and export metrics, traces, and logs to Splunk Observability Cloud (and other backends).

##### File structure
- cmd/otelcol: the main Collector binary and default configs (agent, full, gateway).
- internal/: core libraries—configuration sources, config‐map providers, config conversion, discovery, receivers, settings, versioning.
- pkg/extension & pkg/receiver: SignalFx SmartAgent extension & receiver plugins.
- deployments/: Ansible, Chef, Puppet, Salt, Kubernetes, Nomad, CloudFoundry, etc.
- packaging/: scripts and manifests to produce OS-specific packages (deb, rpm, MSI, Docker images).
- examples/: sample Docker Compose & Helm deployments.
- docs/: architecture, component reference, getting-started guides.

##### README
- Describes architecture (see docs/architecture.md), deployment modes (agent vs. gateway), and supported configuration sources & exporters.
- Highlights core pipelines: receivers → processors → exporters, plus extensions.

##### Main components & relationships
a. Core Collector Service (cmd/otelcol binary)
– Configuration Engine
- Config Sources (envvar, file include, etcd2, Vault, Zookeeper)
- Config‐map provider (Kubernetes style)
- Config converter (backward compatibility & normalizations)
– Extensions (e.g., memory_ballast, debug config server)
– Receivers (OTLP/gRPC/HTTP, SmartAgent, Prometheus remote_write gateway, discovery receiver, scripted inputs, etc.)
– Processors (batching, memory limiter, resourcedetection, filters)
– Exporters (signalfx, splunk_hec, otlphttp, logging, etc.)
– Service pipeline definitions (metrics, traces, logs)
b. Deployment & Packaging Layer
– Packaging scripts and platform-specific installers (packaging/fpm, msi, choco, bundle scripts)
– Configuration-management integrations (deployments/ansible, chef, puppet, salt)
– Container images (docker/, k8s/) and orchestration artifacts (Helm, Nomad, CloudFoundry, ECS).
c. Extension & Receiver Plugins
– pkg/extension/smartagentextension: bridges legacy SmartAgent Extensions (Collectd, Python)
– pkg/receiver/smartagentreceiver: loads legacy SmartAgent monitors as OTLP data

##### Important architectural patterns & principles
- Modular “factory” pattern for adding new receivers, processors, exporters, extensions.
- Pluggable config sources & discovery.
- Single-binary distribution with embedded default configs and platform-specific overrides.
- Backward-compatibility layer (config converter).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Telemetry Sources
    subgraph "External Sources"
        direction TB
        TS1["OTLP/gRPC"]:::external
        TS2["Prometheus"]:::external
        TS3["Logs & Metrics"]:::external
    end

    %% Core Collector Service
    subgraph "Core Collector Service" 
        direction TB
        CCS["Single-Binary Runtime"]:::runtime
        click CCS "https://github.com/signalfx/splunk-otel-collector/tree/main/cmd/otelcol/"

        subgraph "Configuration Layer"
            direction TB
            CSources["Config Sources"]:::plugin
            click CSources "https://github.com/signalfx/splunk-otel-collector/tree/main/internal/configsource/"
            CMapProvider["Config-Map Provider"]:::plugin
            click CMapProvider "https://github.com/signalfx/splunk-otel-collector/blob/main/internal/confmapprovider/configsource/provider.go"
            CConverter["Config Converter"]:::plugin
            click CConverter "https://github.com/signalfx/splunk-otel-collector/tree/main/internal/configconverter/"
        end

        subgraph "Extensions"
            direction TB
            ExtSA["SmartAgent Extension"]:::pluginOpt
            click ExtSA "https://github.com/signalfx/splunk-otel-collector/tree/main/pkg/extension/smartagentextension/"
        end

        subgraph "Data Pipelines"
            direction TB
            subgraph "Receivers"
                direction TB
                RCore["Core Receivers"]:::runtime
                click RCore "https://github.com/signalfx/splunk-otel-collector/tree/main/internal/receiver/"
                RDisc["Discovery Receiver"]:::plugin
                click RDisc "https://github.com/signalfx/splunk-otel-collector/tree/main/internal/receiver/discoveryreceiver/"
                RProm["Prometheus Receiver"]:::plugin
                click RProm "https://github.com/signalfx/splunk-otel-collector/tree/main/internal/receiver/lightprometheusreceiver/"
                RScript["Scripted Inputs"]:::plugin
                click RScript "https://github.com/signalfx/splunk-otel-collector/tree/main/internal/receiver/scriptedinputsreceiver/"
                RRemote["Remote Write Gateway"]:::plugin
                click RRemote "https://github.com/signalfx/splunk-otel-collector/tree/main/internal/receiver/signalfxgatewayprometheusremotewritereceiver/"
                SAR["SmartAgent Receiver"]:::pluginOpt
                click SAR "https://github.com/signalfx/splunk-otel-collector/tree/main/pkg/receiver/smartagentreceiver/"
            end

            subgraph "Processors"
                direction TB
                ProcTS["Timestamp Processor"]:::plugin
                click ProcTS "https://github.com/signalfx/splunk-otel-collector/tree/main/pkg/processor/timestampprocessor/"
            end

            subgraph "Exporters"
                direction TB
                ExpSFx["Signalfx Exporter"]:::plugin
                ExpHEC["Splunk HEC Exporter"]:::plugin
                ExpOTLP["OTLP HTTP Exporter"]:::plugin
                ExpLog["Logging Exporter"]:::plugin
                click ExpSFx "https://github.com/signalfx/splunk-otel-collector/tree/main/tests/exporters/signalfx/"
            end
        end

        DebugServer["Debug/Config HTTP Server"]:::runtime
        Settings["Service Settings & Flags"]:::plugin
        click Settings "https://github.com/signalfx/splunk-otel-collector/tree/main/internal/settings/"
    end

    %% Backends
    subgraph "Backends"
        direction TB
        SplunkCloud["Splunk Observability Cloud"]:::external
        UpstreamOTel["Upstream OTel Collector"]:::external
    end

    %% Deployment & Packaging
    subgraph "Deployment & Packaging" 
        direction TB
        subgraph "Config Mgmt"
            direction TB
            Ansible["Ansible Roles"]:::deploy
            click Ansible "https://github.com/signalfx/splunk-otel-collector/tree/main/deployments/ansible/roles/collector/"
            Chef["Chef Cookbooks"]:::deploy
            click Chef "https://github.com/signalfx/splunk-otel-collector/tree/main/deployments/chef/recipes/"
            Puppet["Puppet Modules"]:::deploy
            click Puppet "https://github.com/signalfx/splunk-otel-collector/tree/main/deployments/puppet/manifests/"
            Salt["Salt States"]:::deploy
            click Salt "https://github.com/signalfx/splunk-otel-collector/tree/main/deployments/salt/splunk-otel-collector/"
        end

        subgraph "OS Installers"
            direction TB
            FPM["FPM Packaging"]:::package
            click FPM "https://github.com/signalfx/splunk-otel-collector/tree/main/packaging/fpm/"
            MSI["MSI Packaging"]:::package
            click MSI "https://github.com/signalfx/splunk-otel-collector/tree/main/packaging/msi/"
            Choco["Chocolatey"]:::package
            click Choco "https://github.com/signalfx/splunk-otel-collector/tree/main/packaging/choco/"
        end

        subgraph "Containers & Orchestration"
            direction TB
            Docker{"Docker Compose Samples"}:::deploy
            click Docker "https://github.com/signalfx/splunk-otel-collector/tree/main/examples/"
            K8s["Kubernetes Manifests"]:::deploy
            click K8s "https://github.com/signalfx/splunk-otel-collector/tree/main/k8s/"
            Nomad["Nomad Jobs"]:::deploy
            click Nomad "https://github.com/signalfx/splunk-otel-collector/tree/main/deployments/nomad/"
            CF["CloudFoundry BOSH"]:::deploy
            click CF "https://github.com/signalfx/splunk-otel-collector/tree/main/deployments/cloudfoundry/bosh/"
        end
    end

    %% Connections
    TS1 --> Receivers
    TS2 --> Receivers
    TS3 --> Receivers

    CSources --> CCS
    CMapProvider --> CCS
    CConverter --> CCS

    CCS --> Extensions
    CCS --> Receivers
    Receivers --> Processors
    Processors --> Exporters
    Exporters --> SplunkCloud
    Exporters --> UpstreamOTel

    CCS --> DebugServer

    %% Styles
    classDef runtime fill:#c8e6c9,stroke:#2e7d32,color:#2e7d32
    classDef plugin fill:#bbdefb,stroke:#1e88e5,color:#1e88e5
    classDef pluginOpt fill:#e3f2fd,stroke:#64b5f6,stroke-dasharray: 5 5,color:#1e88e5
    classDef external fill:#fff9c4,stroke:#f9a825,color:#f57f17
    classDef deploy fill:#ffe0b2,stroke:#fb8c00,color:#e65100
    classDef package fill:#ffccbc,stroke:#d84315,color:#bf360c
```
