How liatrio/liatrio-otel-collector works
Main Components
• OTEL Collector Core: The primary executable is built from the source in the “cmd/otel-compgen” directory. This is the central component that initializes and runs the telemetry collection pipelines.
• Receivers: Multiple custom receivers are provided under the “receiver” directory (e.g., githubactionsreceiver, githubreceiver, gitlabreceiver). These receivers are responsible for collecting data from Git provider events, triggering scrapers and collectors.
• Extensions: Under the “extension” directory (such as “githubappauthextension”), you have authentication and perhaps other middleware extensions that support secure integrations with external services.
• Configuration & Manifests: The “config” directory (with files like config.yaml and manifest.yaml) defines the behavior and pipeline composition—declaring which receivers, processors, exporters, and extensions are used.
• Exporters & Pipeline: The README details configuration for exporting telemetry (e.g., using OTLP exporter with authentication for Grafana Cloud). Although not explicitly in a subdirectory, this element is part of the overall OTEL pipeline.
• Internal Tools: The “internal” directories (like “internal/tools”) house supporting utilities used during build and runtime.
Relationships and Data Flow
• The main application (from the “cmd/otel-compgen” binary) spins up the OTEL Collector service. This service instantiates various components as defined in the configuration.
• Receivers are the first stage—they listen to specific endpoints (e.g., GitHub, GitLab endpoints) and ingest telemetry data (debug logs, trace data, or metric events).
• The ingested data then flows through pipelines (defined in the config file under “service/pipelines”), where it may be processed/transformed.
• Extensions (such as authentication modules) interact with both receivers and exporters to handle secure communication (e.g., for scrapers or when exporting data).
• Finally, the processed telemetry data is sent to external endpoints—for example, to Grafana Cloud via an OTLP exporter as shown in the README.
Architectural Patterns and Design Principles
• Modularization: The structure clearly separates the main collector logic, receivers, and extensions. This modular approach improves extensibility and maintainability.
• Pipeline Architecture: The design aligns with the typical OTEL model, where data flows from receivers to processors and finally to exporters.
• Factory Pattern: Many areas (notably in receivers and extensions) implement the factory design pattern to create and manage instances of different components.
• Emphasis on Configuration-Driven Behavior: The use of extensive configuration (in config/config.yaml) means that behavior is driven by external files and environment variables.
Relevant Technologies, Frameworks, and Libraries
• Go: The entire project is written in Go (with modules like go.mod found in multiple directories).
• OpenTelemetry Collector: The core framework upon which this custom collector is built.
• GitHub/GitLab APIs: The custom receivers/scrapers interface with external Git provider APIs.
• CI/CD and Pre-commit Hooks: Notice the configuration for GitHub workflows (in “.github/workflows”), pre-commit hooks, and tools like Go Report Card, Codecov, and renovate.
a. OTEL Collector Core as the primary central block.
d. In the lower section, represent the OTEL pipeline that handles processing and exporting of data. Link the core to an “OTLP Exporter” block which then connects to an external service (e.g., Grafana Cloud).
e. A configuration block (representing config.yaml and manifest.yaml) that feeds into the core, showing that it configures the seekers, receivers, and extensions.
f. Optionally, incorporate development and tooling components (such as internal tools, CI pipelines, etc.) as side modules that support the development lifecycle.