# hussein-awala/spark-on-k8s architecture

> Architecture diagram of the GitHub repository hussein-awala/spark-on-k8s, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hussein-awala/spark-on-k8s
- Repository: https://github.com/hussein-awala/spark-on-k8s

## Overview

Spark On Kubernetes submits and manages Spark workloads through a Python client or CLI, creating driver pods and observing their status in Kubernetes. Optional layers expose those operations through a REST API and web console, Airflow operators and a task tab, plus dbt/Connect integrations. The graph emphasizes the submission-to-Kubernetes path and the documented management surfaces. Edges are limited to relationships visible in excerpts, the source index, or README; unsampled subsystem wiring is not inferred.

Main entry points: [`spark_on_k8s/client.py`](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/client.py) (Python client), [`spark_on_k8s/utils/app_manager.py`](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/utils/app_manager.py) (Application manager), [`spark_on_k8s/cli/app.py`](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/cli/app.py) (CLI commands), [`spark_on_k8s/api/main.py`](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/api/main.py) (REST API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_submission["Submission interfaces"]
  node_client["Python client<br/>[client.py]"]
  node_cli["CLI commands<br/>[app.py]"]
  node_cliwork["CLI automation<br/>[automation.py]"]
end

subgraph group_runtime["Workload runtime"]
  node_config["Configuration<br/>[configuration.py]"]
  node_planning["Resource planning<br/>[planning.py]"]
  node_podtemplate["Driver pod template"]
  node_manager["Application manager<br/>[app_manager.py]"]
  node_k8sclient["Kubernetes client<br/>[sync_client.py]"]
end

subgraph group_apiui["API and console"]
  node_api["REST API<br/>[main.py]"]
  node_apiapps["App endpoints<br/>[apps.py]"]
  node_apiapp["App actions<br/>[app.py]"]
  node_webconsole["Web console<br/>[__init__.py]"]
  node_reactconsole["Applications console<br/>[Applications.tsx]"]
  node_frontendclient["Console API client<br/>[client.ts]"]
end

subgraph group_integrations["Integrations"]
  node_airflow["Airflow operators<br/>[operators.py]"]
  node_airflowtab["Airflow task tab<br/>[backend.py]"]
  node_dbtlifecycle["dbt lifecycle<br/>[lifecycle.py]"]
  node_connect["Spark Connect<br/>[connect.py]"]
  node_connectserver["Connect server<br/>[connect_server.py]"]
end

subgraph group_state["Run state"]
  node_status["Status and identity"]
  node_run["Run handles and waits<br/>[run.py]"]
  node_history[("Run history store<br/>[store.py]")]
  node_collector["History collector<br/>[collector.py]"]
end

node_user(("Engineer or scheduler"))
node_k8s[("Kubernetes cluster")]

node_user -->|"submits through"| node_client
node_user -->|"runs commands"| node_cli
node_client -->|"plans resources"| node_planning
node_client -->|"builds pod spec"| node_podtemplate
node_client -->|"manages app"| node_manager
node_client -->|"uses client"| node_k8sclient
node_client -->|"creates run handle"| node_run
node_k8sclient -->|"loads settings"| node_config
node_manager -->|"uses client"| node_k8sclient
node_manager -->|"reads status"| node_status
node_manager -->|"uses wait deadline"| node_run
node_manager -->|"creates and manages pods"| node_k8s
node_cli -->|"runs automation"| node_cliwork
node_cli -->|"submits app"| node_client
node_cli -->|"manages app"| node_manager
node_collector -->|"writes history"| node_history
node_api -->|"registers routes"| node_apiapps
node_api -->|"registers routes"| node_apiapp
node_api -->|"serves console"| node_webconsole
node_apiapps -->|"lists driver pods"| node_k8s
node_apiapp -->|"kills or deletes"| node_manager
node_webconsole -.->|"serves UI"| node_reactconsole
node_reactconsole -->|"requests data"| node_frontendclient
node_user -->|"uses optional API"| node_api
node_airflow -->|"submits workloads"| node_client
node_airflow -->|"manages workloads"| node_manager
node_airflowtab -->|"reuses console"| node_reactconsole
node_dbtlifecycle -->|"submits workloads"| node_client
node_dbtlifecycle -->|"uses Connect"| node_connect
node_dbtlifecycle -->|"uses server"| node_connectserver

click node_client "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/client.py"
click node_cli "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/cli/app.py"
click node_cliwork "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/cli/automation.py"
click node_config "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/utils/configuration.py"
click node_planning "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/planning.py"
click node_podtemplate "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/utils/driver_pod_template.py"
click node_manager "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/utils/app_manager.py"
click node_k8sclient "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/k8s/sync_client.py"
click node_status "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/utils/spark_app_status.py"
click node_run "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/utils/run.py"
click node_history "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/history/store.py"
click node_collector "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/history/collector.py"
click node_api "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/api/main.py"
click node_apiapps "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/api/apps.py"
click node_apiapp "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/api/app.py"
click node_webconsole "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/api/webserver/__init__.py"
click node_reactconsole "https://github.com/hussein-awala/spark-on-k8s/blob/main/frontend/src/Applications.tsx"
click node_frontendclient "https://github.com/hussein-awala/spark-on-k8s/blob/main/frontend/src/client.ts"
click node_airflow "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/airflow/operators.py"
click node_airflowtab "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/airflow/ui/backend.py"
click node_dbtlifecycle "https://github.com/hussein-awala/spark-on-k8s/blob/main/integrations/dbt/src/spark_on_k8s_dbt/lifecycle.py"
click node_connect "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/connect.py"
click node_connectserver "https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/connect_server.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_client,node_cli,node_cliwork toneBlue
class node_config,node_planning,node_podtemplate,node_manager,node_k8sclient,node_k8s toneAmber
class node_api,node_apiapps,node_apiapp,node_webconsole,node_reactconsole,node_frontendclient toneMint
class node_airflow,node_airflowtab,node_dbtlifecycle,node_connect,node_connectserver,node_user toneRose
class node_status,node_run,node_history,node_collector toneIndigo
```

## Components

### Submission interfaces

- [Python client](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/client.py): `spark_on_k8s/client.py`
- [CLI commands](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/cli/app.py): `spark_on_k8s/cli/app.py`
- [CLI automation](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/cli/automation.py): `spark_on_k8s/cli/automation.py`

### Workload runtime

- [Configuration](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/utils/configuration.py): `spark_on_k8s/utils/configuration.py`
- [Resource planning](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/planning.py): `spark_on_k8s/planning.py`
- [Driver pod template](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/utils/driver_pod_template.py): `spark_on_k8s/utils/driver_pod_template.py`
- [Application manager](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/utils/app_manager.py): `spark_on_k8s/utils/app_manager.py`
- [Kubernetes client](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/k8s/sync_client.py): `spark_on_k8s/k8s/sync_client.py`

### API and console

- [REST API](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/api/main.py): `spark_on_k8s/api/main.py`
- [App endpoints](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/api/apps.py): `spark_on_k8s/api/apps.py`
- [App actions](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/api/app.py): `spark_on_k8s/api/app.py`
- [Web console](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/api/webserver/__init__.py): `spark_on_k8s/api/webserver/__init__.py`
- [Applications console](https://github.com/hussein-awala/spark-on-k8s/blob/main/frontend/src/Applications.tsx): `frontend/src/Applications.tsx`
- [Console API client](https://github.com/hussein-awala/spark-on-k8s/blob/main/frontend/src/client.ts): `frontend/src/client.ts`

### Integrations

- [Airflow operators](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/airflow/operators.py): `spark_on_k8s/airflow/operators.py`
- [Airflow task tab](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/airflow/ui/backend.py): `spark_on_k8s/airflow/ui/backend.py`
- [dbt lifecycle](https://github.com/hussein-awala/spark-on-k8s/blob/main/integrations/dbt/src/spark_on_k8s_dbt/lifecycle.py): `integrations/dbt/src/spark_on_k8s_dbt/lifecycle.py`
- [Spark Connect](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/connect.py): `spark_on_k8s/connect.py`
- [Connect server](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/connect_server.py): `spark_on_k8s/connect_server.py`

### Run state

- [Status and identity](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/utils/spark_app_status.py): `spark_on_k8s/utils/spark_app_status.py`
- [Run handles and waits](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/utils/run.py): `spark_on_k8s/utils/run.py`
- [Run history store](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/history/store.py): `spark_on_k8s/history/store.py`
- [History collector](https://github.com/hussein-awala/spark-on-k8s/blob/main/spark_on_k8s/history/collector.py): `spark_on_k8s/history/collector.py`

### Other components

- **Engineer or scheduler**
- **Kubernetes cluster**

## Connections

- Engineer or scheduler → Python client: submits through
- Engineer or scheduler → CLI commands: runs commands
- Python client → Resource planning: plans resources (evidence: `spark_on_k8s/client.py`)
- Python client → Driver pod template: builds pod spec (evidence: `spark_on_k8s/client.py`)
- Python client → Application manager: manages app (evidence: `spark_on_k8s/client.py`)
- Python client → Kubernetes client: uses client (evidence: `spark_on_k8s/client.py`)
- Python client → Run handles and waits: creates run handle (evidence: `spark_on_k8s/client.py`)
- Kubernetes client → Configuration: loads settings (evidence: `spark_on_k8s/k8s/sync_client.py`)
- Application manager → Kubernetes client: uses client (evidence: `spark_on_k8s/utils/app_manager.py`)
- Application manager → Status and identity: reads status (evidence: `spark_on_k8s/utils/app_manager.py`)
- Application manager → Run handles and waits: uses wait deadline (evidence: `spark_on_k8s/utils/app_manager.py`)
- Application manager → Kubernetes cluster: creates and manages pods
- CLI commands → CLI automation: runs automation (evidence: `spark_on_k8s/cli/app.py`)
- CLI commands → Python client: submits app (evidence: `spark_on_k8s/cli/app.py`)
- CLI commands → Application manager: manages app (evidence: `spark_on_k8s/cli/app.py`)
- History collector → Run history store: writes history (evidence: `spark_on_k8s/history/collector.py`)
- REST API → App endpoints: registers routes (evidence: `spark_on_k8s/api/main.py`)
- REST API → App actions: registers routes (evidence: `spark_on_k8s/api/main.py`)
- REST API → Web console: serves console (evidence: `spark_on_k8s/api/main.py`)
- App endpoints → Kubernetes cluster: lists driver pods (evidence: `spark_on_k8s/api/apps.py`)
- App actions → Application manager: kills or deletes (evidence: `spark_on_k8s/api/app.py`)
- Web console → Applications console: serves UI
- Applications console → Console API client: requests data (evidence: `frontend/src/Applications.tsx`)
- Engineer or scheduler → REST API: uses optional API
- Airflow operators → Python client: submits workloads (evidence: `spark_on_k8s/airflow/operators.py`)
- Airflow operators → Application manager: manages workloads (evidence: `spark_on_k8s/airflow/operators.py`)
- Airflow task tab → Applications console: reuses console
- dbt lifecycle → Python client: submits workloads (evidence: `integrations/dbt/src/spark_on_k8s_dbt/lifecycle.py`)
- dbt lifecycle → Spark Connect: uses Connect (evidence: `integrations/dbt/src/spark_on_k8s_dbt/lifecycle.py`)
- dbt lifecycle → Connect server: uses server (evidence: `integrations/dbt/src/spark_on_k8s_dbt/lifecycle.py`)