# rockthejvm/spark-essentials architecture

> Architecture diagram of the GitHub repository rockthejvm/spark-essentials, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rockthejvm/spark-essentials
- Repository: https://github.com/rockthejvm/spark-essentials

## Overview

##### Project Type & Purpose
– This is a “big data” demo/training application built in Scala using Apache Spark.
– It includes:
- A local SBT/Scala codebase with multiple Spark exercises (part1→part7, playground).
- A Dockerized PostgreSQL instance for Spark to read/write.
- A Dockerized Spark cluster (master + multiple workers) for distributed job execution.
- Sample data under src/main/resources/data (JSON, CSV, Parquet).

##### Main Components
A. Developer Environment
– IntelliJ / sbt build (build.sbt, project/*).
– Local code in src/main/scala and src/main/resources.
B. Data Sources
– Local resource files (JSON, CSV, Parquet) mounted into the cluster.
C. Docker Compose Network
1. PostgreSQL container (docker-compose.yml)
2. Spark Cluster (spark-cluster/docker-compose.yml)
- Spark Master
- Spark Submit image (entry point for jobs)
- Spark Worker nodes (scale=3)
D. Execution Flow
– spark-submit (from developer machine or container) to Spark Master
– Master distributes tasks to Workers
– Workers read input data (from mounted volume)
– Processes DataFrames/RDDs
– Optionally writes results to PostgreSQL over JDBC

##### Relationships & Interactions
– Developer → runs sbt/spark-submit → Spark Master
– Spark Master ↔ Spark Workers (job scheduling & task execution)
– Workers → Data (mounted HDFS‐style directory)
– Spark jobs → PostgreSQL (JDBC network call)

##### Architectural Patterns & Design Principles
– Distributed computation (Master/Workers)
– Containerized services (Docker Compose, isolation, easy reproduction)
– Modular code organization (by course modules: recap, dataframes, datasets, SQL, low‐level RDDs, practical, big data)
– External service integration (Postgres via JDBC)

##### Final Touches
– Ensure every component has a clear, unique label.
Layer 1: Developer & sbt project
Layer 2: Docker orchestration
Layer 3: Spark cluster & Postgres
Layer 4: Data volume

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Layer 1: Developer Environment
    subgraph "Developer Environment"
        sbtConfig["build.sbt"]:::code
        sbtProps["project/build.properties"]:::code
        bsp[".bsp/sbt.json"]:::code
        scalaCode["src/main/scala"]:::code
    end

    %% Layer 2: Orchestration
    subgraph "Docker Orchestration"
        composeTopo["docker-compose.yml"]:::orchestration
        cleanScripts["docker-clean.sh/.bat"]:::orchestration
        sparkCompose["spark-cluster/docker-compose.yml"]:::orchestration
    end

    %% Layer 3: Spark Cluster & Postgres
    subgraph "Runtime Services"
        subgraph "PostgreSQL Service"
            postgres["PostgreSQL Container"]:::datastore
            sqlInit["sql/db.sql"]:::datastore
        end
        subgraph "Spark Cluster"
            master["Spark Master"]:::compute
            submit["Spark Submit"]:::compute
            workers["3× Spark Worker"]:::compute
        end
    end

    %% Layer 4: Data Volume
    vol["src/main/resources/data"]:::data

    %% Connections
    sbtConfig --> scalaCode
    sbtProps --> scalaCode
    bsp --> scalaCode
    scalaCode -->|runs spark-submit| composeTopo
    cleanScripts --> composeTopo
    composeTopo --> sparkCompose
    sparkCompose --> master
    master -->|schedules tasks| workers
    workers -->|reads data| vol
    workers -->|JDBC read/write| postgres
    postgres -->|initialized by| sqlInit

    %% Click Events
    click sbtConfig "https://github.com/rockthejvm/spark-essentials/blob/master/build.sbt"
    click sbtProps "https://github.com/rockthejvm/spark-essentials/blob/master/project/build.properties"
    click bsp "https://github.com/rockthejvm/spark-essentials/blob/master/.bsp/sbt.json"
    click scalaCode "https://github.com/rockthejvm/spark-essentials/tree/master/src/main/scala"
    click vol "https://github.com/rockthejvm/spark-essentials/tree/master/src/main/resources/data"
    click composeTopo "https://github.com/rockthejvm/spark-essentials/blob/master/docker-compose.yml"
    click sqlInit "https://github.com/rockthejvm/spark-essentials/blob/master/sql/db.sql"
    click cleanScripts "https://github.com/rockthejvm/spark-essentials/blob/master/docker-clean.sh"
    click cleanScripts "https://github.com/rockthejvm/spark-essentials/blob/master/docker-clean.bat"
    click sparkCompose "https://github.com/rockthejvm/spark-essentials/blob/master/spark-cluster/docker-compose.yml"
    click master "https://github.com/rockthejvm/spark-essentials/tree/master/spark-cluster/docker/spark-master/Dockerfile"
    click master "https://github.com/rockthejvm/spark-essentials/blob/master/spark-cluster/docker/spark-master/start-master.sh"
    click workers "https://github.com/rockthejvm/spark-essentials/tree/master/spark-cluster/docker/spark-worker/Dockerfile"
    click workers "https://github.com/rockthejvm/spark-essentials/blob/master/spark-cluster/docker/spark-worker/start-worker.sh"
    click submit "https://github.com/rockthejvm/spark-essentials/tree/master/spark-cluster/docker/spark-submit/Dockerfile"
    click submit "https://github.com/rockthejvm/spark-essentials/blob/master/spark-cluster/docker/spark-submit/spark-submit.sh"
    click base "https://github.com/rockthejvm/spark-essentials/tree/master/spark-cluster/docker/base/Dockerfile"
    click env "https://github.com/rockthejvm/spark-essentials/blob/master/spark-cluster/env/spark-worker.sh"
    click buildScripts "https://github.com/rockthejvm/spark-essentials/blob/master/spark-cluster/build-images.sh"
    click buildScripts "https://github.com/rockthejvm/spark-essentials/blob/master/spark-cluster/build-images.bat"

    %% Styles
    classDef code fill:#e0bbff,stroke:#a060ff,color:#000
    classDef orchestration fill:#ffeb99,stroke:#ffaa00,color:#000
    classDef compute fill:#cce5ff,stroke:#3399ff,color:#000
    classDef datastore fill:#d5fdd5,stroke:#33cc33,color:#000
    classDef data fill:#f0f0f0,stroke:#888,color:#000
```
