# redborder/rb-druid-indexer architecture

> Architecture diagram of the GitHub repository redborder/rb-druid-indexer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/redborder/rb-druid-indexer
- Repository: https://github.com/redborder/rb-druid-indexer

## Overview

##### Project Type & Purpose
- This is a distributed task manager for Kafka ingestion into Druid. Its main goal is to simplify and manage the indexing process for streaming Kafka data into Druid clusters.
- It features multi-druid router compatibility, auto-discovery of routers, cluster-compatibility with failover support through ZooKeeper, and centralized task management.

##### Main Components
a. Frontend/External Interfaces:
- Although not a web “frontend,” the service’s external inputs include Kafka brokers (specified in each task’s configuration) and connections to ZooKeeper for leadership and coordination.
- Also, the Druid Router is an externally managed component that the indexer will communicate with.

b. Core Service:
- The entry point (main.go) encapsulates the core service logic.
- The main business logic is split into several modules:
- Configuration Module: Files under /config and /druid/config manage reading and validation of configuration (from YAML like example_config.yml) used for setting up tasks.
- Task Management & Datasources: The /druid/datasources directory holds code defining each task’s metrics and dimensions. These components prepare the configuration for each data source (such as “rb_flow”, “rb_monitor”, etc.) that will be sent to the Druid Supervisor.
- Real-time Processing: Modules like druid/realtime.go and druid/router.go handle the submission and routing of tasks to the Druid system.

c. Cluster Coordination & Failover:
- The zkclient module (in /zkclient) handles ZooKeeper-based tasks – managing leadership election, task announcements, and ensuring high availability in a clustered environment.

d. Logging & Observability:
- The logger component (in /logger) is responsible for logging events and errors, which is key to tracking the workflow and diagnosing issues.

e. Testing and Integration:
- There are integration test scripts (e.g., in the /integration directory) and GitHub workflows that indicate both unit and integration test automation.
- Packaging (in /packaging/rpm) is used for deployment and can be noted as a deployment artifact rather than a runtime component.

##### Relationships and Data Flow
- ZooKeeper: Represent ZooKeeper servers (configured in the YAML) interacting with the “Cluster Coordination” (zkclient) component.
- Druid Router: Represent the external Druid Router which receives tasks.
- Inside the core service:
- Config Reader component ingesting YAML configuration.
- That the Task Manager handles submission of tasks to the Druid Router via the realtime processing components.
- Data flow from Kafka into the processing pipeline.
- Communication from the Task Manager to the Druid Router.
- Coordination between the Task Manager and the ZooKeeper client for failover/leadership tasks.
- Logging outputs flowing to the Logger module.
- A container or boundary that represents the entire rb-druid-indexer service.

##### Important Architectural Patterns & Technologies
- Micro-modular design: Different responsibilities are placed in separate packages (configuration, datasources, logging, zkclient).
- Distributed systems and dynamic discovery: Use of ZooKeeper for leader election, and auto-discovery of Druid Routers.
- Separation of concerns: Clear distinction between configuration, domain logic (task/datasource management), external communication, and deployment/integration.

##### Additional Notes
- While the integration tests and packaging are essential for deployment and CI/CD, you may note them separately as “Deployment & Testing Artifacts” if needed.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Components
    kafka["Kafka Brokers"]:::external
    zkExt["ZooKeeper Cluster"]:::external
    druidRouter["Druid Router / Druid Supervisor"]:::external

    %% Internal rb-druid-indexer Service
    subgraph "rb-druid-indexer Service"
        mainService["Main Service Entry Point"]:::core
        configManager["Configuration Manager"]:::core
        taskManager["Task Manager & Datasources"]:::core
        realTime["Real-time Processing Engine"]:::process
        zkClient["ZooKeeper Client Module"]:::coord
        logger["Logger"]:::util
    end

    %% Additional Artifacts
    subgraph "Deployment & Testing"
        integration["Integration Testing"]:::test
        packaging["Deployment & Packaging Artifacts"]:::test
        workflows["CI/CD Workflows"]:::test
    end

    %% Relationships and Data Flow
    kafka -->|"ingest_data"| taskManager
    mainService -->|"init_config"| configManager
    configManager -->|"load_config"| taskManager
    taskManager -->|"submit_tasks"| realTime
    realTime -->|"send_tasks"| druidRouter
    taskManager -->|"coordinate"| zkClient
    zkClient -->|"leader_election"| zkExt
    taskManager -->|"log_events"| logger

    %% Additional relationships for deployment/testing artifacts
    mainService -->|"trigger_tests"| integration
    integration -->|"build_artifact"| packaging
    packaging -->|"deploy_via"| workflows

    %% Click Events
    click mainService "https://github.com/redborder/rb-druid-indexer/blob/master/main.go"
    click configManager "https://github.com/redborder/rb-druid-indexer/blob/master/config/config.go"
    click configManager "https://github.com/redborder/rb-druid-indexer/blob/master/druid/config/config.go"
    click taskManager "https://github.com/redborder/rb-druid-indexer/tree/master/druid/datasources/"
    click realTime "https://github.com/redborder/rb-druid-indexer/blob/master/druid/realtime.go"
    click realTime "https://github.com/redborder/rb-druid-indexer/blob/master/druid/router.go"
    click zkClient "https://github.com/redborder/rb-druid-indexer/blob/master/zkclient/client.go"
    click zkClient "https://github.com/redborder/rb-druid-indexer/blob/master/zkclient/election.go"
    click zkClient "https://github.com/redborder/rb-druid-indexer/blob/master/zkclient/task_announcer.go"
    click logger "https://github.com/redborder/rb-druid-indexer/blob/master/logger/logger.go"
    click integration "https://github.com/redborder/rb-druid-indexer/blob/master/integration/config.yml"
    click integration "https://github.com/redborder/rb-druid-indexer/blob/master/integration/docker-compose.yml"
    click packaging "https://github.com/redborder/rb-druid-indexer/tree/master/packaging/rpm/Makefile"
    click packaging "https://github.com/redborder/rb-druid-indexer/blob/master/packaging/rpm/rb-druid-indexer.service"
    click packaging "https://github.com/redborder/rb-druid-indexer/blob/master/packaging/rpm/rb-druid-indexer.spec"
    click workflows "https://github.com/redborder/rb-druid-indexer/blob/master/.github/workflows/go.yml"
    click workflows "https://github.com/redborder/rb-druid-indexer/blob/master/.github/workflows/integration.yml"

    %% Styles
    classDef external fill:#FFE6E6,stroke:#CC3333,stroke-width:2px,color:#7A0000;
    classDef core fill:#E0FFE0,stroke:#008000,stroke-width:2px,color:#004D00;
    classDef process fill:#E6F7FF,stroke:#0066CC,stroke-width:2px,color:#003366;
    classDef coord fill:#FFF2CC,stroke:#CC9900,stroke-width:2px,color:#664400;
    classDef util fill:#DDEEFF,stroke:#336699,stroke-width:2px,color:#003366;
    classDef test fill:#FFF5CC,stroke:#CC9900,stroke-width:2px,color:#664400;
```
