How hussein-awala/spark-on-k8s works
Spark On Kubernetes submits and manages Spark workloads through a Python client or CLI, creating driver pods and observing their status in Kubernetes. Optional layers expose those operations through a REST API and web console, Airflow operators and a task tab, plus dbt/Connect integrations. The graph emphasizes the submission-to-Kubernetes path and the documented management surfaces. Edges are limited to relationships visible in excerpts, the source index, or README; unsampled subsystem wiring is not inferred.
Main entry points: client.py Python client · app_manager.py Application manager · app.py CLI commands · main.py REST API
Components 25
Submission interfaces
- Python client
spark_on_k8s/client.py - CLI commands
spark_on_k8s/cli/app.py - CLI automation
spark_on_k8s/cli/automation.py
Workload runtime
- Configuration
spark_on_k8s/utils/configuration.py - Resource planning
spark_on_k8s/planning.py - Driver pod template
spark_on_k8s/utils/driver_pod_template.py - Application manager
spark_on_k8s/utils/app_manager.py - Kubernetes client
spark_on_k8s/k8s/sync_client.py
API and console
- REST API
spark_on_k8s/api/main.py - App endpoints
spark_on_k8s/api/apps.py - App actions
spark_on_k8s/api/app.py - Web console
spark_on_k8s/api/webserver/__init__.py - Applications console
frontend/src/Applications.tsx - Console API client
frontend/src/client.ts
Integrations
- Airflow operators
spark_on_k8s/airflow/operators.py - Airflow task tab
spark_on_k8s/airflow/ui/backend.py - dbt lifecycle
integrations/dbt/src/spark_on_k8s_dbt/lifecycle.py - Spark Connect
spark_on_k8s/connect.py - Connect server
spark_on_k8s/connect_server.py
Run state
- Status and identity
spark_on_k8s/utils/spark_app_status.py - Run handles and waits
spark_on_k8s/utils/run.py - Run history store
spark_on_k8s/history/store.py - History collector
spark_on_k8s/history/collector.py
Other components
- Engineer or scheduler
- Kubernetes cluster
Connections 30
- Engineer or scheduler → Python client: submits through
- Engineer or scheduler → CLI commands: runs commands
- Python client → Resource planning: plans resources
spark_on_k8s/client.py - Python client → Driver pod template: builds pod spec
spark_on_k8s/client.py - Python client → Application manager: manages app
spark_on_k8s/client.py - Python client → Kubernetes client: uses client
spark_on_k8s/client.py - Python client → Run handles and waits: creates run handle
spark_on_k8s/client.py - Kubernetes client → Configuration: loads settings
spark_on_k8s/k8s/sync_client.py - Application manager → Kubernetes client: uses client
spark_on_k8s/utils/app_manager.py - Application manager → Status and identity: reads status
spark_on_k8s/utils/app_manager.py - Application manager → Run handles and waits: uses wait deadline
spark_on_k8s/utils/app_manager.py - Application manager → Kubernetes cluster: creates and manages pods
- CLI commands → CLI automation: runs automation
spark_on_k8s/cli/app.py - CLI commands → Python client: submits app
spark_on_k8s/cli/app.py - CLI commands → Application manager: manages app
spark_on_k8s/cli/app.py - History collector → Run history store: writes history
spark_on_k8s/history/collector.py - REST API → App endpoints: registers routes
spark_on_k8s/api/main.py - REST API → App actions: registers routes
spark_on_k8s/api/main.py - REST API → Web console: serves console
spark_on_k8s/api/main.py - App endpoints → Kubernetes cluster: lists driver pods
spark_on_k8s/api/apps.py - App actions → Application manager: kills or deletes
spark_on_k8s/api/app.py - Web console → Applications console: serves UI
- Applications console → Console API client: requests data
frontend/src/Applications.tsx - Engineer or scheduler → REST API: uses optional API
- Airflow operators → Python client: submits workloads
spark_on_k8s/airflow/operators.py - Airflow operators → Application manager: manages workloads
spark_on_k8s/airflow/operators.py - Airflow task tab → Applications console: reuses console
- dbt lifecycle → Python client: submits workloads
integrations/dbt/src/spark_on_k8s_dbt/lifecycle.py - dbt lifecycle → Spark Connect: uses Connect
integrations/dbt/src/spark_on_k8s_dbt/lifecycle.py - dbt lifecycle → Connect server: uses server
integrations/dbt/src/spark_on_k8s_dbt/lifecycle.py