How aixhunter/spark-k8s-pod-template works
Main Components
a) Infrastructure Layer:
• Kubernetes Cluster
• Master Node (Control Plane)
• Worker Nodes
• Docker Registry (for storing the custom Spark image)
b) Spark Components:
• Spark Driver Pod
• Spark Executor Pods
• Service Account (sa-spark)
c) Application Components:
• Custom Spark Application (app.jar)
• Docker Container (docker-spark:0.0.0.1)
Specific Elements
• Rectangles: For Kubernetes components
• Hexagons: For Spark components
• Rounded rectangles: For application components
• Blue: Kubernetes components
• Orange: Spark components
• Green: Application components
• Gray: Infrastructure components
c) Connections:
Key Relationships to Highlight
• How the Docker image is built and pushed to the registry
• How Kubernetes pulls the image and creates pods
• Communication between Spark Driver and Executors
• Role of service account in pod creation and management
Important Details
• Namespace (default)
• Cluster role binding between sa-spark and edit role
• Container image path
• Spark.executor.instances configuration (2 instances)
Notes Section
A small notes section mentioning:
• Spark version (3.3.2)
• Important configuration parameters
• Volume mounts for file upload path (/tmp/)