How redborder/rb-druid-indexer works
Project Type & Purpose
• This is a distributed task manager for Kafka ingestion into Druid. Its main goal is to simplify and manage the indexing process for streaming Kafka data into Druid clusters.
• It features multi-druid router compatibility, auto-discovery of routers, cluster-compatibility with failover support through ZooKeeper, and centralized task management.
Main Components
a. Frontend/External Interfaces:
• Although not a web “frontend,” the service’s external inputs include Kafka brokers (specified in each task’s configuration) and connections to ZooKeeper for leadership and coordination.
• Also, the Druid Router is an externally managed component that the indexer will communicate with.
b. Core Service:
• The entry point (main.go) encapsulates the core service logic.
• The main business logic is split into several modules:
• Configuration Module: Files under /config and /druid/config manage reading and validation of configuration (from YAML like example_config.yml) used for setting up tasks.
• Task Management & Datasources: The /druid/datasources directory holds code defining each task’s metrics and dimensions. These components prepare the configuration for each data source (such as “rb_flow”, “rb_monitor”, etc.) that will be sent to the Druid Supervisor.
• Real-time Processing: Modules like druid/realtime.go and druid/router.go handle the submission and routing of tasks to the Druid system.
c. Cluster Coordination & Failover:
• The zkclient module (in /zkclient) handles ZooKeeper-based tasks – managing leadership election, task announcements, and ensuring high availability in a clustered environment.
d. Logging & Observability:
• The logger component (in /logger) is responsible for logging events and errors, which is key to tracking the workflow and diagnosing issues.
e. Testing and Integration:
• There are integration test scripts (e.g., in the /integration directory) and GitHub workflows that indicate both unit and integration test automation.
• Packaging (in /packaging/rpm) is used for deployment and can be noted as a deployment artifact rather than a runtime component.
Relationships and Data Flow
• ZooKeeper: Represent ZooKeeper servers (configured in the YAML) interacting with the “Cluster Coordination” (zkclient) component.
• Druid Router: Represent the external Druid Router which receives tasks.
• Inside the core service:
• Config Reader component ingesting YAML configuration.
• That the Task Manager handles submission of tasks to the Druid Router via the realtime processing components.
• Data flow from Kafka into the processing pipeline.
• Communication from the Task Manager to the Druid Router.
• Coordination between the Task Manager and the ZooKeeper client for failover/leadership tasks.
• Logging outputs flowing to the Logger module.
• A container or boundary that represents the entire rb-druid-indexer service.
Important Architectural Patterns & Technologies
• Micro-modular design: Different responsibilities are placed in separate packages (configuration, datasources, logging, zkclient).
• Distributed systems and dynamic discovery: Use of ZooKeeper for leader election, and auto-discovery of Druid Routers.
• Separation of concerns: Clear distinction between configuration, domain logic (task/datasource management), external communication, and deployment/integration.
Additional Notes
• While the integration tests and packaging are essential for deployment and CI/CD, you may note them separately as “Deployment & Testing Artifacts” if needed.