How telefonica/prometheus-kafka-adapter works
Main Components
• HTTP Server / API Layer: The core service starts in main.go. It includes the HTTP endpoint ("/receive") that listens for incoming Prometheus metrics. This is implemented using the Gin framework.
• Metric Handlers: The logic that handles incoming metrics (handlers.go) and possibly validation or transformation logic.
• Serialization Layer: Components (serializers.go and schemas/metric.avsc) responsible for converting Prometheus metrics into JSON or Avro-JSON depending on the configuration.
• Kafka Integration: The module or function that connects to Kafka and publishes messages. This component needs to factor in configuration including SSL, SASL settings, topic configurations, and batching (as mentioned in environment variables).
• Configuration Management: The config.go file along with environment variable settings defines how the application behaves and connects to external services.
• Metrics Tracking & Logging: Files like metrics.go and the usage of logrus (configured via LOG_LEVEL) highlight that the application is instrumented for logging and possibly for internal metrics.
• Deployment Concerns: The Helm charts (located under helm/prometheus-kafka-adapter) indicate that the service is designed to be deployed in a Kubernetes cluster, and configuration like hostAliases for external Kafka resolution is supported.
• CI/CD and Build/Dev Tools: The Makefile and scripts in tools/ show that the project uses standardized processes for testing, formatting, and building.
Relationships and Interactions
• HTTP Server → Handler/Processing: Represent the flow from the API endpoint to the business logic/handlers that process the incoming metric data.
• Handler → Serialization Module: Indicate that the data is passed to the serialization layer to be marshaled into either JSON or Avro-JSON.
• Serialization Module → Kafka Publisher: Show the output from the serializer being sent to the Kafka publishing component, including batching if applicable.
• Configuration & Environment Variables: Add a side component or overlay that points out that many aspects (e.g., topic naming with Go templating) are configurable via environment variables.
• Deployment Context (Kubernetes): Illustrate that the service is often deployed via Helm in Kubernetes, altering how endpoints (like NodePort exposure) might be set up.
Architectural Patterns and Key Technologies
• Service-Oriented Architecture: The project decouples data ingestion (HTTP endpoint) from data serialization and message publishing.
• Configuration-Driven Design: The service behavior is extensively controlled by environment variables.
• Use of Microservice Tools: Gin for HTTP routing, logrus for logging, and the use of Helm charts for deployment illustrate standard cloud-native patterns.
• Build and CI/CD Practices: Usage of Makefile and GitHub Actions (as visible in the workflow configuration) emphasizes standard development practices.
• Optionally, add small descriptions next to each component that mention key libraries or technologies (e.g., "Gin-HTTP" or "logrus logging").