How aniruddh-a/flink-kubernetes-operator works
Project type & purpose
• This is a multi‐module Java project implementing a Kubernetes Operator for Apache Flink.
• It provides:
• Custom Resource Definitions (CRDs) for FlinkDeployment, FlinkSessionJob, FlinkStateSnapshot
• A controller/reconciler loop to manage those resources
• An admission webhook for validation/mutation
• A standalone autoscaler (with JDBC and in-memory state store plugins)
• Helm charts and docs
High-level architecture components
a. Kubernetes Control Plane
• API Server: stores CRDs and custom resources
• etcd: backing store
b. Operator Components
1. Custom Resource Definitions (flink-kubernetes-operator-api)
• Defines CRDs and Java API types
2. Controller (flink-kubernetes-operator)
• Informers / SharedIndexInformers → watch CRs
• Reconciler factories (ReconcilerFactory → DeploymentReconciler, SessionJobReconciler, SnapshotReconciler)
• Controllers invoke FlinkService to create/patch Kubernetes resources (Deployments, Services, ConfigMaps, Jobs)
• Kubernetes client decorators (mounts, sidecars, config overrides)
3. Admission Webhook (flink-kubernetes-webhook)
• Validates/mutates CRs on creation/update
• Uses informer caches to enrich/mutate requests
4. Autoscaler
• Modules:
• flink-autoscaler (core logic)
• flink-autoscaler-plugin-jdbc (JDBC state store)
• flink-autoscaler-standalone (standalone entrypoint using Hikari + Flink REST API)
• Fetches job metrics from Flink REST endpoint
• Persists scaling state (ConfigMap or JDBC)
• Computes scaling actions via ScalingRealizer → patches CRs or directly scales pods
5. Standalone Cluster Descriptor (flink-kubernetes-standalone)
• Implements Fabric8KubeClient to launch a standalone Flink cluster
c. Supporting Modules
• helm/flink-kubernetes-operator: Helm chart for operator deployment with CRDs, RBAC, Deployment, Webhook
• docs, examples, e2e-tests, tools
Relationships & interactions
• User “kubectl apply”
→ Admission Webhook (mutate/validate)
→ API Server (store CR)
→ Controller Informer receives event
→ Reconciler computes desired state → calls FlinkService & KubeClient → creates/patches Kubernetes resources
→ Optional: Autoscaler poller fetches metrics from Flink JobManager REST → writes state to ConfigMap/JDBC → triggers scale via Kubernetes API
→ Status updates on CR → API Server
Architectural patterns & principles
• Kubernetes Operator pattern: Informer + Reconciler loops, eventual consistency
• Modular multi-module Maven structure: separation of API, controller, webhook, autoscaler, standalone
• Plugin architecture for state store (JDBC)
• Dependency Injection: factories for event handlers, reconciler, state store
• Use of Fabric8 or native Kubernetes Java client
Essential elements
• Clear component labels: “FlinkDeploymentController”, “SharedIndexInformer”, “FlinkService”, “ArtifactFetcher”, “ConfigMapStore”
Putting it all together
• Admission Webhook (handles AdmissionReview requests)
• Controller + Reconcilers (three controllers)
• Autoscaler (core + plugin modules)
• Standalone cluster launcher
• On the right, show Flink Cluster REST endpoint, and on bottom right, optional external JDBC DB.
• Helm chart as deployment descriptor installing the Operator Pod, CRDs, RBAC.