# apache/spark-docker architecture

> Architecture diagram of the GitHub repository apache/spark-docker, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/spark-docker
- Repository: https://github.com/apache/spark-docker

## Overview

##### Main Components
- Dockerfile Templates & Tools: Files such as Dockerfile.template, add-dockerfiles.sh, tools/* (e.g., manifest.py, template.py), and versions.json represent configuration inputs and automation utilities that generate new Dockerfiles and manage version/tag information.
- CI/CD Workflows: The .github/workflows directory contains numerous YAML files for build, test, and publish jobs. These workflows are responsible for verifying every change, building docker images, and pushing them to the repositories (including publishing to Docker Hub).
- GitHub as the source code and automation orchestrator.
- Docker Hub for publishing Apache Spark images (both the apache/spark and the official spark image).
- Possibly the Docker Official Images repository, as mentioned in the README, which involves a PR for inclusion.

##### Relationships and Interactions
- Code Changes/PRs → GitHub Workflows: Illustrate how a commit or PR triggers multiple workflows (for building various versions, running tests, publishing images, and even manifest generation).
- Version Directory Generation: Show how running add-dockerfiles.sh and updating versions.json and workflow files enables the generation of new version directories that subsequently feed into the CI/CD pipelines.
- Publishing: After successful testing, the images are pushed to Docker Hub. For the official images, an additional PR submission step to the Docker Official Images repository is required.

##### Architectural Patterns
- Modularization & Separation of Concerns: Separate directory structure by version and environment clearly indicates a modular design. Similarly, distinct GitHub Actions for build, test, and publish responsibilities illustrate the separation of concerns.
- Template-based Generation: Incorporating Dockerfile templates and shell scripts (e.g., add-dockerfiles.sh) suggests an architecture that leverages templating to standardize Docker configurations across multiple runtime environments.
- Automation via CI/CD: The use of GitHub workflows (with distinct YAML configuration files per version and job type) exemplifies a continuous integration/continuous deployment (CI/CD) pattern for ensuring image consistency and quality.

##### Relevant Technologies, Frameworks, or Libraries
- Docker & Dockerfiles (for container images)
- GitHub Actions (for building, testing, and publishing)
- Shell scripting (for automation in tools like add-dockerfiles.sh, free_disk_space.sh, etc)
- Python scripts (tools/manifest.py and merge_spark_docker_pr.py) are used for manifest and PR automation tasks.

##### Final Advice
- Be very detailed in capturing how each element interacts and contributes to the overall goal of building and publishing Docker images for Apache Spark.
- Avoid overcomplicating the design – break down the project into its major components and show their interactions clearly.

Happy diagramming!

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Services
    GH["GitHub"]:::external

    %% Tools & Automation Scripts Subgraph
    subgraph "Tools & Automation Scripts"
        T1["Dockerfile.template"]:::automation
        T2["add-dockerfiles.sh"]:::automation
        T3["versions.json"]:::automation
        T4["tools/manifest.py"]:::automation
        T5["tools/template.py"]:::automation
        T6["tools/requirements.txt"]:::automation
        T7["entrypoint.sh.template"]:::automation
        T8["free_disk_space.sh"]:::automation
        T9["free_disk_space_container.sh"]:::automation
    end

    %% Version Directories & Dockerfiles Subgraph
    subgraph "Version Directories & Dockerfiles"
        V330["3.3.0"]:::source
        V331["3.3.1"]:::source
        V332["3.3.2"]:::source
        V333["3.3.3"]:::source
        V340["3.4.0"]:::source
        V341["3.4.1"]:::source
        V342["3.4.2"]:::source
        V343["3.4.3"]:::source
        V344["3.4.4"]:::source
        V350["3.5.0"]:::source
        V351["3.5.1"]:::source
        V352["3.5.2"]:::source
        V353["3.5.3"]:::source
        V354["3.5.4"]:::source
        V400p1["4.0.0-preview1"]:::source
        V400p2["4.0.0-preview2"]:::source
    end

    %% CI/CD Pipeline Subgraph
    subgraph "CI/CD Pipeline (GitHub Actions)"
        WF[".github/workflows"]:::cicd
        B["Build Workflows"]:::cicd
        Tst["Test Workflows"]:::cicd
        P["Publish Workflows"]:::cicd
    end

    %% External Registries Subgraph
    subgraph "External Services"
        DH["Docker Hub"]:::external
        DOI["Docker Official Images Repo"]:::external
    end

    %% Connections
    GH -->|"PR/Commit triggers"| WF
    GH -->|"Hosts repository"| T1
    T2 -->|"Generates Dockerfiles"| V330
    T3 -->|"Updates version info"| V330
    T4 -->|"Updates manifest"| B
    T5 -->|"Processes templates"| B
    V330 ---|"Source artifact"| B
    B -->|"Build complete"| Tst
    Tst -->|"Tests passed"| P
    P -->|"Push images"| DH
    P -->|"Submit PR"| DOI

    %% Styles
    classDef source fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef automation fill:#A9DFBF,stroke:#145A32,stroke-width:2px;
    classDef cicd fill:#F9E79F,stroke:#7D6608,stroke-width:2px;
    classDef external fill:#F5B7B1,stroke:#922B21,stroke-width:2px;

    %% Click Events for Version Directories
    click V330 "https://github.com/apache/spark-docker/blob/master/3.3.0"
    click V331 "https://github.com/apache/spark-docker/blob/master/3.3.1"
    click V332 "https://github.com/apache/spark-docker/blob/master/3.3.2"
    click V333 "https://github.com/apache/spark-docker/blob/master/3.3.3"
    click V340 "https://github.com/apache/spark-docker/blob/master/3.4.0"
    click V341 "https://github.com/apache/spark-docker/blob/master/3.4.1"
    click V342 "https://github.com/apache/spark-docker/blob/master/3.4.2"
    click V343 "https://github.com/apache/spark-docker/blob/master/3.4.3"
    click V344 "https://github.com/apache/spark-docker/blob/master/3.4.4"
    click V350 "https://github.com/apache/spark-docker/blob/master/3.5.0"
    click V351 "https://github.com/apache/spark-docker/blob/master/3.5.1"
    click V352 "https://github.com/apache/spark-docker/blob/master/3.5.2"
    click V353 "https://github.com/apache/spark-docker/blob/master/3.5.3"
    click V354 "https://github.com/apache/spark-docker/blob/master/3.5.4"
    click V400p1 "https://github.com/apache/spark-docker/blob/master/4.0.0-preview1"
    click V400p2 "https://github.com/apache/spark-docker/blob/master/4.0.0-preview2"

    %% Click Events for Tools & Automation Scripts
    click T1 "https://github.com/apache/spark-docker/blob/master/Dockerfile.template"
    click T2 "https://github.com/apache/spark-docker/blob/master/add-dockerfiles.sh"
    click T3 "https://github.com/apache/spark-docker/blob/master/versions.json"
    click T4 "https://github.com/apache/spark-docker/blob/master/tools/manifest.py"
    click T5 "https://github.com/apache/spark-docker/blob/master/tools/template.py"
    click T6 "https://github.com/apache/spark-docker/blob/master/tools/requirements.txt"
    click T7 "https://github.com/apache/spark-docker/blob/master/entrypoint.sh.template"
    click T8 "https://github.com/apache/spark-docker/blob/master/tools/ci_runner_cleaner/free_disk_space.sh"
    click T9 "https://github.com/apache/spark-docker/blob/master/tools/ci_runner_cleaner/free_disk_space_container.sh"

    %% Click Event for CI/CD Workflows directory
    click WF "https://github.com/apache/spark-docker/tree/master/.github/workflows"
```
