# aniruddh-a/flink-kubernetes-operator architecture

> Architecture diagram of the GitHub repository aniruddh-a/flink-kubernetes-operator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aniruddh-a/flink-kubernetes-operator
- Repository: https://github.com/aniruddh-a/flink-kubernetes-operator

## Overview

##### Project type & purpose
- This is a multi‐module Java project implementing a Kubernetes Operator for Apache Flink.
- It provides:
- Custom Resource Definitions (CRDs) for FlinkDeployment, FlinkSessionJob, FlinkStateSnapshot
- A controller/reconciler loop to manage those resources
- An admission webhook for validation/mutation
- A standalone autoscaler (with JDBC and in-memory state store plugins)
- Helm charts and docs

##### High-level architecture components
a. Kubernetes Control Plane
- API Server: stores CRDs and custom resources
- etcd: backing store
b. Operator Components
1. Custom Resource Definitions (flink-kubernetes-operator-api)
- Defines CRDs and Java API types
2. Controller (flink-kubernetes-operator)
- Informers / SharedIndexInformers → watch CRs
- Reconciler factories (ReconcilerFactory → DeploymentReconciler, SessionJobReconciler, SnapshotReconciler)
- Controllers invoke FlinkService to create/patch Kubernetes resources (Deployments, Services, ConfigMaps, Jobs)
- Kubernetes client decorators (mounts, sidecars, config overrides)
3. Admission Webhook (flink-kubernetes-webhook)
- Validates/mutates CRs on creation/update
- Uses informer caches to enrich/mutate requests
4. Autoscaler
- Modules:
- flink-autoscaler (core logic)
- flink-autoscaler-plugin-jdbc (JDBC state store)
- flink-autoscaler-standalone (standalone entrypoint using Hikari + Flink REST API)
- Fetches job metrics from Flink REST endpoint
- Persists scaling state (ConfigMap or JDBC)
- Computes scaling actions via ScalingRealizer → patches CRs or directly scales pods
5. Standalone Cluster Descriptor (flink-kubernetes-standalone)
- Implements Fabric8KubeClient to launch a standalone Flink cluster

c. Supporting Modules
- helm/flink-kubernetes-operator: Helm chart for operator deployment with CRDs, RBAC, Deployment, Webhook
- docs, examples, e2e-tests, tools

##### Relationships & interactions
- User “kubectl apply”
→ Admission Webhook (mutate/validate)
→ API Server (store CR)
→ Controller Informer receives event
→ Reconciler computes desired state → calls FlinkService & KubeClient → creates/patches Kubernetes resources
→ Optional: Autoscaler poller fetches metrics from Flink JobManager REST → writes state to ConfigMap/JDBC → triggers scale via Kubernetes API
→ Status updates on CR → API Server

##### Architectural patterns & principles
- Kubernetes Operator pattern: Informer + Reconciler loops, eventual consistency
- Modular multi-module Maven structure: separation of API, controller, webhook, autoscaler, standalone
- Plugin architecture for state store (JDBC)
- Dependency Injection: factories for event handlers, reconciler, state store
- Use of Fabric8 or native Kubernetes Java client

##### Essential elements
- Clear component labels: “FlinkDeploymentController”, “SharedIndexInformer”, “FlinkService”, “ArtifactFetcher”, “ConfigMapStore”

##### Putting it all together
- Admission Webhook (handles AdmissionReview requests)
- Controller + Reconcilers (three controllers)
- Autoscaler (core + plugin modules)
- Standalone cluster launcher
- On the right, show Flink Cluster REST endpoint, and on bottom right, optional external JDBC DB.
- Helm chart as deployment descriptor installing the Operator Pod, CRDs, RBAC.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Kubernetes Control Plane
    subgraph "Kubernetes Control Plane"
        API["API Server"]:::controlPlane
        ETCD["etcd"]:::external
    end
    API -->|"stores CRs/CRDs"| ETCD

    %% Operator Pod
    subgraph "Operator Pod"
        subgraph "API & CRDs"
            CRDModule["Custom Resource Definitions"]:::operator
        end

        subgraph "Admission Webhook"
            Webhook["FlinkOperatorWebhook"]:::operator
            AdmissionHandler["AdmissionHandler"]:::operator
            FlinkValidator["FlinkValidator"]:::operator
            InformerManager["InformerManager"]:::operator
        end

        subgraph "Controller & Reconcilers"
            SharedInformer["SharedIndexInformer"]:::operator
            FDepCtrl["FlinkDeploymentController"]:::operator
            FSessionCtrl["FlinkSessionJobController"]:::operator
            FStateSnapCtrl["FlinkStateSnapshotController"]:::operator
            ReconcilerFactory["ReconcilerFactory"]:::operator
            DeploymentRecon["ApplicationReconciler"]:::operator
            SessionRecon["SessionReconciler"]:::operator
            SnapshotRecon["StateSnapshotReconciler"]:::operator
            FlinkService["FlinkService"]:::operator
            Decorator["FlinkConfMountDecorator"]:::operator
        end

        subgraph "Autoscaler"
            Core["JobAutoScaler"]:::autoscaler
            MetricsCollector["RestApiMetricsCollector"]:::autoscaler
            ScalingRealizer["ScalingRealizer"]:::autoscaler
            InMemoryStore["InMemoryAutoScalerStateStore"]:::autoscaler
            JDBCStore["JdbcAutoScalerStateStore"]:::autoscaler
        end

        subgraph "Standalone Cluster Descriptor"
            ClusterDesc["KubernetesStandaloneClusterDescriptor"]:::operator
            Fabric8Client["Fabric8FlinkStandaloneKubeClient"]:::operator
        end
    end

    %% Flink Cluster and External DB
    subgraph "Flink Cluster"
        FlinkREST["Flink JobManager REST API"]:::external
    end
    subgraph "External State Store"
        JDBCDB["External JDBC DB"]:::external
    end

    %% Helm deployment
    Helm["Helm Chart"]:::helm

    %% Interactions
    kubectl["\"kubectl apply\""]:::external --> Webhook
    Webhook -->|"AdmissionReview"| API
    API -->|"List/Watch"| SharedInformer
    SharedInformer --> FDepCtrl
    SharedInformer --> FSessionCtrl
    SharedInformer --> FStateSnapCtrl
    FDepCtrl --> ReconcilerFactory
    FSessionCtrl --> ReconcilerFactory
    FStateSnapCtrl --> ReconcilerFactory
    ReconcilerFactory --> DeploymentRecon
    ReconcilerFactory --> SessionRecon
    ReconcilerFactory --> SnapshotRecon
    DeploymentRecon -->|"Reconcile→Create/Update"| FlinkService
    SessionRecon -->|"Reconcile→Create/Update"| FlinkService
    SnapshotRecon -->|"Reconcile→Create/Update"| FlinkService
    FlinkService -->|"Kubernetes API Calls"| API
    Decorator --> FlinkService

    Core --> MetricsCollector
    MetricsCollector -->|"GET /job/metrics"| FlinkREST
    Core -->|"persist state"| InMemoryStore
    Core -->|"persist state"| JDBCStore
    Core --> ScalingRealizer
    ScalingRealizer -->|"scale action"| API
    InMemoryStore --> API
    JDBCStore -->|JDBC write| JDBCDB

    ClusterDesc --> Fabric8Client
    Fabric8Client -->|"launch pods"| API

    Helm -->|"deploy operator+CRDs"| API

    %% Click events
    click CRDModule "https://github.com/aniruddh-a/flink-kubernetes-operator/tree/main/flink-kubernetes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api"
    click FDepCtrl "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkDeploymentController.java"
    click FSessionCtrl "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkSessionJobController.java"
    click FStateSnapCtrl "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkStateSnapshotController.java"
    click ReconcilerFactory "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/ReconcilerFactory.java"
    click SessionRecon "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/SessionReconciler.java"
    click DeploymentRecon "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/ApplicationReconciler.java"
    click SnapshotRecon "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/snapshot/StateSnapshotReconciler.java"
    click FlinkService "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/FlinkService.java"
    click Decorator "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/FlinkConfMountDecorator.java"
    click AdmissionHandler "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-kubernetes-webhook/src/main/java/org/apache/flink/kubernetes/operator/admission/AdmissionHandler.java"
    click Webhook "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-kubernetes-webhook/src/main/java/org/apache/flink/kubernetes/operator/admission/FlinkOperatorWebhook.java"
    click FlinkValidator "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-kubernetes-webhook/src/main/java/org/apache/flink/kubernetes/operator/admission/FlinkValidator.java"
    click InformerManager "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-kubernetes-webhook/src/main/java/org/apache/flink/kubernetes/operator/admission/informer/InformerManager.java"
    click Core "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobAutoScaler.java"
    click MetricsCollector "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/RestApiMetricsCollector.java"
    click InMemoryStore "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/InMemoryAutoScalerStateStore.java"
    click JDBCStore "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-autoscaler-plugin-jdbc/src/main/java/org/apache/flink/autoscaler/jdbc/JdbcAutoScalerStateStore.java"
    click ClusterDesc "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-kubernetes-standalone/src/main/java/org/apache/flink/kubernetes/operator/standalone/KubernetesStandaloneClusterDescriptor.java"
    click Fabric8Client "https://github.com/aniruddh-a/flink-kubernetes-operator/blob/main/flink-kubernetes-standalone/src/main/java/org/apache/flink/kubernetes/operator/standalone/Fabric8FlinkStandaloneKubeClient.java"

    %% Styles
    classDef controlPlane fill:#cce5ff,stroke:#005cbf;
    classDef operator fill:#d4edda,stroke:#155724;
    classDef autoscaler fill:#fff3cd,stroke:#856404;
    classDef external fill:#e2e3e5,stroke:#6c757d;
    classDef helm fill:#f8f9fa,stroke:#343a40;
    class kubectl external;
    class Helm helm;
```
