# apple/batch-processing-gateway architecture

> Architecture diagram of the GitHub repository apple/batch-processing-gateway, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apple/batch-processing-gateway
- Repository: https://github.com/apple/batch-processing-gateway

## Overview

##### Project type and purpose
- BPG is a backend microservice (a RESTful API) that sits in front of one or more Spark-on-Kubernetes clusters.
- It exposes REST endpoints for users to upload artifacts to S3, submit Spark jobs, query status, fetch logs, and delete jobs.
- It persists metadata in a relational App Submission DB and routes CRDs to Spark K8s clusters via the fabric8 Kubernetes client.

##### File structure and key technologies
- src/main/java/com/apple/spark/rest → REST controllers (Dropwizard)
- src/main/java/com/apple/spark/core → core logic (cluster selection, monitoring, queueing)
- src/main/java/com/apple/spark/api → request/response DTOs (OpenAPI spec)
- src/main/java/com/apple/spark/clients → external client wrappers (Spark History server)
- src/main/java/com/apple/spark/security → pluggable auth filters
- src/main/bash & entrypoint.sh → container bootstrap scripts
- pom.xml → Maven build, dependencies include Dropwizard, fabric8 K8s client, AWS SDK, Micrometer, Swagger
- helm/ & dev-setup/ → Helm charts and scripts for Kubernetes deployment of BPG, DB, S3 emulator, Spark Operator, etc.
- openapi/ → OpenAPI spec used to generate client libraries

##### Extract architecture insights from README
- High-level view: clients → BPG → Spark K8s Operator → Spark pods
- External systems: S3 artifacts bucket, App Submission relational DB, (optional) S3 logs bucket & log mover/indexer
- Key flow: POST /spark → validate & store request in DB → upload artifacts to S3 → choose cluster/namespace via queue & weight config → generate SparkApplication CRD → submit CRD to Kubernetes API → operator invokes spark-submit → monitor status, update DB
- Log retrieval: query live pod logs or fallback to S3 via pre-configured prefix & optional indexer

##### Main components
- Client (CLI or SDK), Swagger UI or REST client
- BPG Service (Docker container)
- REST API layer (Dropwizard endpoints)
- Request validation & authentication filter
- OpenAPI-generated request/response models
- Core modules
– Submission Helper (uploads to S3, writes DB)
– Cluster Router (queue lookup, weight algorithm)
– Kubernetes Helper (fabric8 client → API Server)
– Application Monitor (polls SparkApplication CRDs & Events)
– Log Retriever (pod logs or S3 fallback)
- Infrastructure
– AWS S3 Client (artifacts & logs)
– JDBC Client (App Submission DB)
– Metrics (Micrometer / StatsD)
- App Submission DB (e.g., PostgreSQL)
- S3 Artifacts Bucket
- S3 Logs Bucket (optional)
- Kubernetes API Server(s) / Spark on K8s Operator
- Spark Driver & Executor Pods
- Optional: external log-mover & indexer services

1. Client → BPG REST API (POST /spark, GET status, GET logs, DELETE)
2. BPG → S3 (upload artifacts)
3. BPG → App Submission DB (INSERT submission metadata)
4. BPG → Cluster Router → selected K8s API → create SparkApplication CRD
5. Spark Operator → Spark pods (driver, executors) → run job
6. Application Monitor → fetch CRD status & pod events → update DB
7. Client → BPG → fetch logs from live pods, else → S3 logs bucket

##### Architectural patterns and design principles
- Layered separation: API layer, service(core) layer, persistence layer, external connector layer
- Configuration-driven routing: queue and weight based cluster selection
- CRD-driven orchestration: modeling Spark jobs as Kubernetes custom resources
- Asynchronous status monitoring via Kubernetes watch / polling
- Fail-over for logs via dual source (pods + S3)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% BPG Service Internal Modules
    subgraph "BPG Service" 
        direction TB
        REST_API["REST API Layer (Dropwizard)"]:::internal
        AuthFilter1["UserNameAuthFilter"]:::internal
        AuthFilter2["UserNameBasicAuthenticator"]:::internal
        AuthFilter3["UserUnauthorizedHandler"]:::internal
        DTOs["OpenAPI DTOs"]:::internal
        Helper["Submission Helper"]:::internal
        Router1["ZonePicker"]:::internal
        Router2["RoundRobinZonePicker"]:::internal
        Router3["ZoneManager"]:::internal
        K8s_Helper["Kubernetes Helper"]:::internal
        Monitor1["ApplicationMonitor"]:::internal
        Monitor2["RunningApplicationMonitor"]:::internal
        LogDao["LogDao"]:::internal
        LogRest["ApplicationGetLogRest"]:::internal
        AWS_Client["CloudStorageRest"]:::internal
        JDBC["DBConnection"]:::internal
        Metrics["BPGStatsdConfig"]:::internal
        BootstrapApp["BPGApplication"]:::internal
        Entrypoint["entrypoint.sh"]:::internal
    end

    %% External Managed Services
    subgraph "Persistence & Storage"
        direction TB
        DB["App Submission DB"]:::external
        S3_Artifacts["S3 Artifacts Bucket"]:::external
        S3_Logs["S3 Logs Bucket"]:::external
    end

    %% Kubernetes Cluster
    subgraph "Kubernetes / Spark On K8s"
        direction TB
        API_Server["Kubernetes API Server"]:::k8s
        SparkOperator["Spark Operator"]:::k8s
        DriverPod["Spark Driver & Executor Pods"]:::k8s
    end

    %% Client
    Client["Client (CLI/SDK/Swagger UI)"]:::external

    %% Flows
    Client -->|1 POST/GET/DELETE| REST_API
    REST_API -->|2 Validate| AuthFilter1
    AuthFilter1 -->|2 Validate| AuthFilter2
    AuthFilter2 -->|2 Validate| AuthFilter3
    AuthFilter3 -->|3 Map to DTOs| DTOs
    DTOs -->|4 Upload artifacts| AWS_Client
    AWS_Client -->|4 Store in S3| S3_Artifacts
    DTOs -->|5 Write metadata| JDBC
    JDBC -->|5 Insert| DB
    DTOs -->|6 Pick cluster| Router1
    Router1 -->|6 Algorithm| Router2
    Router2 -->|6 Zone list| Router3
    Router3 -->|6 Selected zone| K8s_Helper
    K8s_Helper -->|7 Create CRD| API_Server
    API_Server -->|7 Hand off| SparkOperator
    SparkOperator -->|8 Launch| DriverPod
    DriverPod -->|9 Monitor status| Monitor1
    Monitor1 -->|9 Update status| Monitor2
    Monitor2 -->|9 Write back| JDBC
    Client -->|10 Fetch logs| LogRest
    LogRest -->|10 Live pod logs| DriverPod
    LogRest -->|10 or fallback| S3_Logs

    %% Click Events
    click REST_API "https://github.com/apple/batch-processing-gateway/tree/main/src/main/java/com/apple/spark/rest"
    click AuthFilter1 "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/security/UserNameAuthFilter.java"
    click AuthFilter2 "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/security/UserNameBasicAuthenticator.java"
    click AuthFilter3 "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/security/UserUnauthorizedHandler.java"
    click DTOs "https://github.com/apple/batch-processing-gateway/tree/main/src/main/java/com/apple/spark/api"
    click Helper "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/core/ApplicationSubmissionHelper.java"
    click Router1 "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/core/ZonePicker.java"
    click Router2 "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/core/RoundRobinZonePicker.java"
    click Router3 "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/core/ZoneManager.java"
    click K8s_Helper "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/core/KubernetesHelper.java"
    click Monitor1 "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/core/ApplicationMonitor.java"
    click Monitor2 "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/core/RunningApplicationMonitor.java"
    click LogDao "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/core/LogDao.java"
    click LogRest "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/rest/ApplicationGetLogRest.java"
    click AWS_Client "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/rest/CloudStorageRest.java"
    click JDBC "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/core/DBConnection.java"
    click Metrics "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/core/BPGStatsdConfig.java"
    click BootstrapApp "https://github.com/apple/batch-processing-gateway/blob/main/src/main/java/com/apple/spark/BPGApplication.java"
    click Entrypoint "https://github.com/apple/batch-processing-gateway/blob/main/entrypoint.sh"
    click S3_Artifacts "https://github.com/apple/batch-processing-gateway/tree/main/helm/batch-processing-gateway"
    click DB "https://github.com/apple/batch-processing-gateway/tree/main/helm/batch-processing-gateway"
    click S3_Logs "https://github.com/apple/batch-processing-gateway/tree/main/helm/batch-processing-gateway"

    %% Styles
    classDef internal fill:#cce5ff,stroke:#004085;
    classDef external fill:#d4edda,stroke:#155724;
    classDef k8s fill:#ffe5b4,stroke:#cc8400;
```
