How apple/batch-processing-gateway works
Project type and purpose
• BPG is a backend microservice (a RESTful API) that sits in front of one or more Spark-on-Kubernetes clusters.
• It exposes REST endpoints for users to upload artifacts to S3, submit Spark jobs, query status, fetch logs, and delete jobs.
• It persists metadata in a relational App Submission DB and routes CRDs to Spark K8s clusters via the fabric8 Kubernetes client.
File structure and key technologies
• src/main/java/com/apple/spark/rest → REST controllers (Dropwizard)
• src/main/java/com/apple/spark/core → core logic (cluster selection, monitoring, queueing)
• src/main/java/com/apple/spark/api → request/response DTOs (OpenAPI spec)
• src/main/java/com/apple/spark/clients → external client wrappers (Spark History server)
• src/main/java/com/apple/spark/security → pluggable auth filters
• src/main/bash & entrypoint.sh → container bootstrap scripts
• pom.xml → Maven build, dependencies include Dropwizard, fabric8 K8s client, AWS SDK, Micrometer, Swagger
• helm/ & dev-setup/ → Helm charts and scripts for Kubernetes deployment of BPG, DB, S3 emulator, Spark Operator, etc.
• openapi/ → OpenAPI spec used to generate client libraries
Extract architecture insights from README
• High-level view: clients → BPG → Spark K8s Operator → Spark pods
• External systems: S3 artifacts bucket, App Submission relational DB, (optional) S3 logs bucket & log mover/indexer
• Key flow: POST /spark → validate & store request in DB → upload artifacts to S3 → choose cluster/namespace via queue & weight config → generate SparkApplication CRD → submit CRD to Kubernetes API → operator invokes spark-submit → monitor status, update DB
• Log retrieval: query live pod logs or fallback to S3 via pre-configured prefix & optional indexer
Main components
• Client (CLI or SDK), Swagger UI or REST client
• BPG Service (Docker container)
• REST API layer (Dropwizard endpoints)
• Request validation & authentication filter
• OpenAPI-generated request/response models
• Core modules
– Submission Helper (uploads to S3, writes DB)
– Cluster Router (queue lookup, weight algorithm)
– Kubernetes Helper (fabric8 client → API Server)
– Application Monitor (polls SparkApplication CRDs & Events)
– Log Retriever (pod logs or S3 fallback)
• Infrastructure
– AWS S3 Client (artifacts & logs)
– JDBC Client (App Submission DB)
– Metrics (Micrometer / StatsD)
• App Submission DB (e.g., PostgreSQL)
• S3 Artifacts Bucket
• S3 Logs Bucket (optional)
• Kubernetes API Server(s) / Spark on K8s Operator
• Spark Driver & Executor Pods
• Optional: external log-mover & indexer services
1. Client → BPG REST API (POST /spark, GET status, GET logs, DELETE)
2. BPG → S3 (upload artifacts)
3. BPG → App Submission DB (INSERT submission metadata)
4. BPG → Cluster Router → selected K8s API → create SparkApplication CRD
5. Spark Operator → Spark pods (driver, executors) → run job
6. Application Monitor → fetch CRD status & pod events → update DB
7. Client → BPG → fetch logs from live pods, else → S3 logs bucket
Architectural patterns and design principles
• Layered separation: API layer, service(core) layer, persistence layer, external connector layer
• Configuration-driven routing: queue and weight based cluster selection
• CRD-driven orchestration: modeling Spark jobs as Kubernetes custom resources
• Asynchronous status monitoring via Kubernetes watch / polling
• Fail-over for logs via dual source (pods + S3)