# datainfrahq/druid-operator architecture

> Architecture diagram of the GitHub repository datainfrahq/druid-operator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datainfrahq/druid-operator
- Repository: https://github.com/datainfrahq/druid-operator

## Overview

##### Project type and purpose
- This is a Kubernetes Operator, built with Go/kubebuilder, whose job is to provision and manage distributed Apache Druid clusters on Kubernetes.
- It defines two CRDs: Druid and DruidIngestion (group “druid.apache.org”, version v1alpha1).
- Reads/writes CustomResources, and creates/manages StatefulSets, Deployments, Services, ConfigMaps, RBAC, etc., to run a Druid cluster.

##### File structure
Top-level components:
- apis/ – Go types and generated deepcopy code for the two CRDs
- controllers/ – Reconciliation logic for Druid and DruidIngestion resources
- config/ – Kustomize manifests for CRDs, RBAC, manager deployment, webhooks, Prometheus
- chart/ – Helm chart for packaging and deploying the operator & CRDs
- pkg/ – Auxiliary Go packages: HTTP server, API client helpers, utilities
- e2e/ and examples/ – end-to-end tests and sample CRs for various Druid deployment scenarios
- main.go & Dockerfile & Makefile – operator entrypoint, containerization, build/release scripts

##### Key insights from README
- Confirms operator pattern, kubebuilder v3, target Kubernetes versions, supported CRs.

##### Main components
A. Control Plane (Operator)
1. Kubernetes API Server (CRD registration & CustomResource store)
2. Operator Manager Pod
- apis module (CRD types)
- controller-runtime machinery
- controllers/druid (watches Druid CRs)
- controllers/ingestion (watches DruidIngestion CRs)
- pkg/http (metrics & health endpoints)
3. RBAC & Leader Election (config/rbac & rbac_*.yaml)
4. Webhooks (if used; config/crd/patches)
5. Helm Chart (chart/Chart.yaml, templates) for operator deployment

B. Data Plane (Druid Cluster)
1. Druid CustomResources (instances of Druid, DruidIngestion)
2. Generated Kubernetes objects:
- StatefulSets (coordinator, overlord, broker, historical, middlemanager)
- ConfigMaps & Secrets (configuration, dynamic configuration patches)
- Services & Headless Services (for inter-pod communication)
- PodDisruptionBudgets, AutoScaling (HPA)
3. Dependencies/Extensions:
- Zookeeper (ext/zookeeper_default_ext.go or external ZK)
- Metadata store (e.g. MySQL/PostgreSQL)
- Deep Storage (e.g. S3, HDFS)
4. Metrics Endpoint (prometheus config in config/prometheus/)

C. External Services
- AI/O storage (deep storage backend)
- Metadata DB
- Zookeeper cluster
- (Optional) Kafka for ingestion

- Operator Manager watches API Server for CR events → Reconcile loop
- Reconcile → creates/updates Druid cluster objects via API Server
- Druid Pods talk to ZK, metadata store, deep storage
- Operator exposes metrics & health to Prometheus & K8s readiness/liveness

##### Patterns & technologies
- Kubernetes Operator Pattern (Controller-Manager, Reconciliation)
- Kubebuilder / controller-runtime
- Kustomize for CRDs and RBAC
- Helm for packaging/deployment
- Go modules (go.mod, go.sum)
- E2E tests using kind / Docker

## Diagram (Mermaid)

```mermaid
flowchart TB
  classDef control fill:#D0E8FF,stroke:#0366d6,color:#0366d6
  classDef data fill:#E6F4EA,stroke:#22863A,color:#22863A
  classDef external fill:#FFEFE6,stroke:#D73A49,color:#D73A49
  classDef config fill:#F1F8FF,stroke:#6F42C1,color:#6F42C1
  classDef helm fill:#FFF5B1,stroke:#DBAB09,color:#735C0F

  subgraph "Control Plane"
    apis["CRD Types\napis/druid/v1alpha1"]:::control
    controllers_druid["DruidReconciler\ncontrollers/druid"]:::control
    controllers_ing["IngestionReconciler\ncontrollers/ingestion"]:::control
    main["Operator Manager\nmain.go"]:::control
    http["Metrics & Health Server\npkg/http/http.go"]:::control
    api_helpers["API Helpers\npkg/druidapi/druidapi.go"]:::control
    util["Utilities\npkg/util/util.go"]:::control

    subgraph RBAC
      rbac_kust["Kustomize RBAC\nconfig/rbac/kustomization.yaml"]:::config
      rbac_yamls["RBAC YAMLs\nconfig/rbac/*.yaml"]:::config
      helm_rbac["Helm RBAC Templates\nchart/templates/rbac_*.yaml"]:::helm
    end

    subgraph Webhooks
      webhook_ing["Webhook Patches Ingestion\nconfig/crd/patches/webhook_in_druid_druidingestions.yaml"]:::config
      webhook_druid["Webhook Patches Druid\nconfig/crd/patches/webhook_in_druids.yaml"]:::config
    end

    subgraph CRD Registration
      crd_bases["CRD Manifests\nconfig/crd/bases/*.yaml"]:::config
      crd_kust["Kustomize CRDs\nconfig/crd/kustomization.yaml"]:::config
    end

    subgraph Helm Chart
      chart_def["Chart Definition\nchart/Chart.yaml, values.yaml"]:::helm
      chart_tpl["Operator Deployment Templates\nchart/templates/deployment.yaml,\nservice_account.yaml,\nservice.yaml,\n_helpers.tpl,\nNOTES.txt"]:::helm
    end
  end

  subgraph "Data Plane"
    druid_cr["Druid CustomResources"]:::data
    statefulsets["StatefulSets\n(coordinator, overlord,\nbroker, historical, mm)"]:::data
    configmaps["ConfigMaps & Secrets"]:::data
    services["Services & Headless Services"]:::data
    pdb_hpa["PodDisruptionBudgets & HPA"]:::data

    subgraph Extensions
      zk_ext["Zookeeper Extension\ncontrollers/druid/ext/zookeeper_default_ext.go"]:::data
      meta_ext["MetadataStore Extension\ncontrollers/druid/ext/metadata_store_default_ext.go"]:::data
      ds_ext["DeepStorage Extension\ncontrollers/druid/ext/deep_storage_default_ext.go"]:::data
    end
  end

  subgraph "External Services"
    zookeeper[(Zookeeper)]:::external
    metadata[(Metadata DB)]:::external
    deep_storage[(Deep Storage)]:::external
    kafka[(Kafka)]:::external
  end

  prometheus[(Prometheus)]:::external

  main -->|uses| apis
  main --> controllers_druid
  main --> controllers_ing
  main --> http
  controllers_druid --> zk_ext
  controllers_druid --> meta_ext
  controllers_druid --> ds_ext
  controllers_druid --> statefulsets
  controllers_ing --> statefulsets
  main --> api_helpers
  main --> util
  main -->|leaderElect| rbac_yamls
  main -->|leaderElect| helm_rbac
  main -->|register CRDs| crd_bases
  main -->|kustomize| crd_kust
  main --> chart_def
  main --> chart_tpl
  druid_cr --> statefulsets
  druid_cr --> configmaps
  druid_cr --> services
  druid_cr --> pdb_hpa
  statefulsets --> zookeeper
  statefulsets --> metadata
  statefulsets --> deep_storage
  statefulsets --> kafka
  http --> prometheus

  click apis "https://github.com/datainfrahq/druid-operator/tree/master/apis/druid/v1alpha1"
  click controllers_druid "https://github.com/datainfrahq/druid-operator/tree/master/controllers/druid"
  click controllers_ing "https://github.com/datainfrahq/druid-operator/tree/master/controllers/ingestion"
  click main "https://github.com/datainfrahq/druid-operator/blob/master/main.go"
  click http "https://github.com/datainfrahq/druid-operator/blob/master/pkg/http/http.go"
  click api_helpers "https://github.com/datainfrahq/druid-operator/blob/master/pkg/druidapi/druidapi.go"
  click util "https://github.com/datainfrahq/druid-operator/blob/master/pkg/util/util.go"
  click rbac_kust "https://github.com/datainfrahq/druid-operator/blob/master/config/rbac/kustomization.yaml"
  click rbac_yamls "https://github.com/datainfrahq/druid-operator/tree/master/config/rbac/"
  click helm_rbac "https://github.com/datainfrahq/druid-operator/blob/master/chart/templates/rbac_leader_election.yaml"
  click helm_rbac "https://github.com/datainfrahq/druid-operator/blob/master/chart/templates/rbac_manager.yaml"
  click helm_rbac "https://github.com/datainfrahq/druid-operator/blob/master/chart/templates/rbac_metrics.yaml"
  click helm_rbac "https://github.com/datainfrahq/druid-operator/blob/master/chart/templates/rbac_proxy.yaml"
  click webhook_ing "https://github.com/datainfrahq/druid-operator/blob/master/config/crd/patches/webhook_in_druid_druidingestions.yaml"
  click webhook_druid "https://github.com/datainfrahq/druid-operator/blob/master/config/crd/patches/webhook_in_druids.yaml"
  click crd_bases "https://github.com/datainfrahq/druid-operator/blob/master/config/crd/bases/druid.apache.org_druids.yaml"
  click crd_bases "https://github.com/datainfrahq/druid-operator/blob/master/config/crd/bases/druid.apache.org_druidingestions.yaml"
  click crd_kust "https://github.com/datainfrahq/druid-operator/blob/master/config/crd/kustomization.yaml"
  click chart_def "https://github.com/datainfrahq/druid-operator/blob/master/chart/Chart.yaml"
  click chart_def "https://github.com/datainfrahq/druid-operator/blob/master/chart/values.yaml"
  click chart_tpl "https://github.com/datainfrahq/druid-operator/blob/master/chart/templates/deployment.yaml"
  click chart_tpl "https://github.com/datainfrahq/druid-operator/blob/master/chart/templates/service_account.yaml"
  click chart_tpl "https://github.com/datainfrahq/druid-operator/blob/master/chart/templates/service.yaml"
  click chart_tpl "https://github.com/datainfrahq/druid-operator/blob/master/chart/templates/_helpers.tpl"
  click chart_tpl "https://github.com/datainfrahq/druid-operator/blob/master/chart/templates/NOTES.txt"
  click zk_ext "https://github.com/datainfrahq/druid-operator/blob/master/controllers/druid/ext/zookeeper_default_ext.go"
  click meta_ext "https://github.com/datainfrahq/druid-operator/blob/master/controllers/druid/ext/metadata_store_default_ext.go"
  click ds_ext "https://github.com/datainfrahq/druid-operator/blob/master/controllers/druid/ext/deep_storage_default_ext.go"
```
