How datainfrahq/druid-operator works
Project type and purpose
• This is a Kubernetes Operator, built with Go/kubebuilder, whose job is to provision and manage distributed Apache Druid clusters on Kubernetes.
• It defines two CRDs: Druid and DruidIngestion (group “druid.apache.org”, version v1alpha1).
• Reads/writes CustomResources, and creates/manages StatefulSets, Deployments, Services, ConfigMaps, RBAC, etc., to run a Druid cluster.
File structure
Top-level components:
• apis/ – Go types and generated deepcopy code for the two CRDs
• controllers/ – Reconciliation logic for Druid and DruidIngestion resources
• config/ – Kustomize manifests for CRDs, RBAC, manager deployment, webhooks, Prometheus
• chart/ – Helm chart for packaging and deploying the operator & CRDs
• pkg/ – Auxiliary Go packages: HTTP server, API client helpers, utilities
• e2e/ and examples/ – end-to-end tests and sample CRs for various Druid deployment scenarios
• main.go & Dockerfile & Makefile – operator entrypoint, containerization, build/release scripts
Key insights from README
• Confirms operator pattern, kubebuilder v3, target Kubernetes versions, supported CRs.
Main components
A. Control Plane (Operator)
1. Kubernetes API Server (CRD registration & CustomResource store)
2. Operator Manager Pod
• apis module (CRD types)
• controller-runtime machinery
• controllers/druid (watches Druid CRs)
• controllers/ingestion (watches DruidIngestion CRs)
• pkg/http (metrics & health endpoints)
3. RBAC & Leader Election (config/rbac & rbac_*.yaml)
4. Webhooks (if used; config/crd/patches)
5. Helm Chart (chart/Chart.yaml, templates) for operator deployment
B. Data Plane (Druid Cluster)
1. Druid CustomResources (instances of Druid, DruidIngestion)
2. Generated Kubernetes objects:
• StatefulSets (coordinator, overlord, broker, historical, middlemanager)
• ConfigMaps & Secrets (configuration, dynamic configuration patches)
• Services & Headless Services (for inter-pod communication)
• PodDisruptionBudgets, AutoScaling (HPA)
3. Dependencies/Extensions:
• Zookeeper (ext/zookeeper_default_ext.go or external ZK)
• Metadata store (e.g. MySQL/PostgreSQL)
• Deep Storage (e.g. S3, HDFS)
4. Metrics Endpoint (prometheus config in config/prometheus/)
C. External Services
• AI/O storage (deep storage backend)
• Metadata DB
• Zookeeper cluster
• (Optional) Kafka for ingestion
• Operator Manager watches API Server for CR events → Reconcile loop
• Reconcile → creates/updates Druid cluster objects via API Server
• Druid Pods talk to ZK, metadata store, deep storage
• Operator exposes metrics & health to Prometheus & K8s readiness/liveness
Patterns & technologies
• Kubernetes Operator Pattern (Controller-Manager, Reconciliation)
• Kubebuilder / controller-runtime
• Kustomize for CRDs and RBAC
• Helm for packaging/deployment
• Go modules (go.mod, go.sum)
• E2E tests using kind / Docker