# aixhunter/spark-k8s-pod-template architecture

> Architecture diagram of the GitHub repository aixhunter/spark-k8s-pod-template, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aixhunter/spark-k8s-pod-template
- Repository: https://github.com/aixhunter/spark-k8s-pod-template

## Overview

##### Main Components

a) Infrastructure Layer:
- Kubernetes Cluster
- Master Node (Control Plane)
- Worker Nodes
- Docker Registry (for storing the custom Spark image)

b) Spark Components:
- Spark Driver Pod
- Spark Executor Pods
- Service Account (sa-spark)

c) Application Components:
- Custom Spark Application (app.jar)
- Docker Container (docker-spark:0.0.0.1)

##### Specific Elements

- Rectangles: For Kubernetes components
- Hexagons: For Spark components
- Rounded rectangles: For application components

- Blue: Kubernetes components
- Orange: Spark components
- Green: Application components
- Gray: Infrastructure components

c) Connections:

##### Key Relationships to Highlight

- How the Docker image is built and pushed to the registry
- How Kubernetes pulls the image and creates pods
- Communication between Spark Driver and Executors
- Role of service account in pod creation and management

##### Important Details

- Namespace (default)
- Cluster role binding between sa-spark and edit role
- Container image path
- Spark.executor.instances configuration (2 instances)

##### Notes Section

A small notes section mentioning:
- Spark version (3.3.2)
- Important configuration parameters
- Volume mounts for file upload path (/tmp/)

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles and Classes
    classDef kubernetes fill:#326CE5,color:white
    classDef spark fill:#E25A1C,color:white
    classDef app fill:#4CAF50,color:white
    classDef infra fill:#808080,color:white
    classDef storage fill:#FFA500,color:white

    %% Application Layer
    subgraph Application
        App["Custom Spark Application (app.jar)"]:::app
        Docker["Docker Container<br/>docker-spark:0.0.0.1"]:::app
    end

    %% Spark Layer
    subgraph Spark
        Driver["Spark Driver Pod"]:::spark
        Exec1["Spark Executor Pod 1"]:::spark
        Exec2["Spark Executor Pod 2"]:::spark
        SA["Service Account<br/>(sa-spark)"]:::spark
    end

    %% Kubernetes Layer
    subgraph Kubernetes
        Control["Kubernetes Control Plane<br/>k8s://[cluster-ip]:8443"]:::kubernetes
        Worker1["Worker Node 1"]:::kubernetes
        Worker2["Worker Node 2"]:::kubernetes
    end

    %% Infrastructure
    Registry[(Docker Registry)]:::storage

    %% Relationships
    App --> Docker
    Docker --> Registry
    Registry --> Control
    Control --> SA
    SA --> Driver
    Driver --> Exec1
    Driver --> Exec2
    Control --> Worker1
    Control --> Worker2
    Worker1 --> Exec1
    Worker2 --> Exec2

    %% Click Events
    click App "https://github.com/AIxHunter/Spark-k8s-pod-template/blob/main/app.jar"
    click Docker "https://github.com/AIxHunter/Spark-k8s-pod-template/tree/main/Dockerfile"
    click Driver "https://github.com/AIxHunter/Spark-k8s-pod-template/blob/main/pod.yaml"
    click SA "https://github.com/AIxHunter/Spark-k8s-pod-template/blob/main/entrypoint.sh"

    %% Legend
    subgraph Legend
        L1["Kubernetes Component"]:::kubernetes
        L2["Spark Component"]:::spark
        L3["Application Component"]:::app
        L4["Storage Component"]:::storage
    end

    %% Notes
    subgraph Notes
        N1["Spark Version: 3.3.2"]
        N2["File Upload Path: /tmp/"]
        N3["Executor Instances: 2"]
    end
```
