# kubeflow/spark-operator architecture

> Architecture diagram of the GitHub repository kubeflow/spark-operator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kubeflow/spark-operator
- Repository: https://github.com/kubeflow/spark-operator

## Overview

##### Main Components

a. Core Components:
- Spark Operator Controller
- Webhook Server
- Custom Resource Definitions (CRDs)
- Metrics Collection System
- Scheduler Integration Components

b. External Components:
- Kubernetes API Server
- Apache Spark Applications
- Prometheus (for metrics)
- Storage Systems (GCS, S3)

##### Control Plane Layer
- Spark Operator Controller as the central component
- It to the Webhook Server (both Mutating and Validating)
- CRD definitions (SparkApplication and ScheduledSparkApplication)

##### Resource Management Layer
- Scheduler integrations (Kubernetes, Volcano, YuniKorn)
- Resource quota management
- Pod template management

##### Execution Layer
- Spark Driver pods
- Spark Executor pods
- Service accounts and RBAC

##### Monitoring Layer
- Prometheus integration
- Metrics collection
- Application monitoring

##### Specific Implementation Details

a. Controller Flow:
**CustomResourceDefinition → Controller → Spark-Submit → Driver Pod → Executor Pods**

b. Webhook Flow:
**Pod Creation Request → Mutating Webhook → Validating Webhook → Modified Pod Spec**

c. Monitoring Flow:
**Spark Metrics → Operator Metrics → Prometheus → Monitoring System**

##### Key Interactions

- Bidirectional communication between the operator and Kubernetes API
- Webhook intercept path for pod modifications
- Metrics collection flow
- Scheduler integration points

##### Security and Access Control Flow

Remember to include annotations explaining key components and their roles, and consider creating separate detailed views for complex subsystems like the webhook configuration or scheduler integration.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Control_Plane["Control Plane Layer"]
        SparkController["Spark Operator Controller\n(Manages Spark Applications)"]:::controller
        WebhookServer["Webhook Server\n(Pod Mutation/Validation)"]:::webhook
        CRDs["Custom Resource Definitions\n(SparkApplication & ScheduledSparkApplication)"]:::crd
        ConfigMaps["ConfigMaps\n(Operator Configuration)"]:::controller
    end

    subgraph Resource_Management["Resource Management Layer"]
        Scheduler["Scheduler Integration\n(K8s/Volcano/YuniKorn)"]:::scheduler
        ResourceQuota["Resource Quota Management\n(CPU/Memory Limits)"]:::resource
        PodTemplate["Pod Template Management\n(Container Specs)"]:::resource
        AutoScaler["Auto Scaling\n(Dynamic Resource Adjustment)"]:::resource
    end

    subgraph Execution_Layer["Execution Layer"]
        Driver["Spark Driver Pods\n(Application Control)"]:::spark
        Executor["Spark Executor Pods\n(Task Processing)"]:::spark
        RBAC["RBAC & Service Accounts\n(Security Context)"]:::security
        Secrets["Secrets Management\n(Credentials/Tokens)"]:::security
    end

    subgraph Monitoring["Monitoring Layer"]
        Prometheus["Prometheus\n(Metrics Storage)"]:::monitoring
        MetricsCollector["Metrics Collection\n(Performance Data)"]:::monitoring
        AppMonitor["Application Monitoring\n(Status/Health)"]:::monitoring
        Alerts["Alert Management\n(Notifications)"]:::monitoring
    end

    subgraph External["External Systems"]
        K8sAPI["Kubernetes API Server\n(Cluster Management)"]:::external
        Storage["Storage Systems\n(GCS/S3/HDFS)"]:::external
        LogSystem["Logging System\n(ELK/Stackdriver)"]:::external
    end

    %% Core Relationships
    K8sAPI <--> SparkController
    SparkController --> WebhookServer
    CRDs --> SparkController
    ConfigMaps --> SparkController
    
    %% Resource Management Flow
    SparkController --> Scheduler
    Scheduler --> Driver
    Driver --> Executor
    ResourceQuota --> |Enforces Limits| Driver
    ResourceQuota --> |Enforces Limits| Executor
    AutoScaler --> Driver
    AutoScaler --> Executor
    
    %% Security Flow
    RBAC --> Driver
    RBAC --> Executor
    Secrets --> Driver
    Secrets --> Executor
    
    %% Storage and Logging
    Driver --> Storage
    Executor --> Storage
    Driver --> LogSystem
    Executor --> LogSystem
    
    %% Monitoring Flow
    Driver --> MetricsCollector
    Executor --> MetricsCollector
    MetricsCollector --> Prometheus
    Prometheus --> Alerts
    AppMonitor --> MetricsCollector
    
    %% Template Management
    WebhookServer --> PodTemplate
    PodTemplate --> Driver
    PodTemplate --> Executor

    %% Styles
    classDef controller fill:#3498db,color:white
    classDef webhook fill:#2ecc71,color:white
    classDef crd fill:#95a5a6,color:white
    classDef scheduler fill:#e74c3c,color:white
    classDef resource fill:#f1c40f
    classDef spark fill:#e67e22,color:white
    classDef security fill:#8e44ad,color:white
    classDef monitoring fill:#9b59b6,color:white
    classDef external fill:#7f8c8d,color:white

    %% Click Events
    click SparkController "https://github.com/kubeflow/spark-operator/blob/master/internal/controller/sparkapplication/controller.go"
    click WebhookServer "https://github.com/kubeflow/spark-operator/blob/master/internal/webhook/webhook.go"
    click CRDs "https://github.com/kubeflow/spark-operator/blob/master/config/crd/bases/sparkoperator.k8s.io_sparkapplications.yaml"
    click Scheduler "https://github.com/kubeflow/spark-operator/blob/master/internal/scheduler/kubescheduler/scheduler.go"
    click MetricsCollector "https://github.com/kubeflow/spark-operator/blob/master/internal/metrics/metrcis.go"
    click RBAC "https://github.com/kubeflow/spark-operator/blob/master/config/rbac/role.yaml"
    click WebhookServer "https://github.com/kubeflow/spark-operator/blob/master/config/webhook/manifests.yaml"

    %% Legend
    subgraph Legend
        L1["Controller Components"]:::controller
        L2["Webhook Components"]:::webhook
        L3["CRD Components"]:::crd
        L4["Scheduler Components"]:::scheduler
        L5["Resource Management"]:::resource
        L6["Spark Components"]:::spark
        L7["Security Components"]:::security
        L8["Monitoring Components"]:::monitoring
        L9["External Systems"]:::external
    end
```
