# giuliodamico/spark_local architecture

> Architecture diagram of the GitHub repository giuliodamico/spark_local, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/giuliodamico/spark_local
- Repository: https://github.com/giuliodamico/spark_local

## Overview

This repository provides a local classroom Spark environment: students use JupyterLab to work with notebooks, whose jobs run on a standalone Spark cluster. The documented stack also exposes the Spark master, History Server, and the current notebook’s Spark UI. The graph preserves the sample notebooks and datasets, but their contents were not inspected, so dataset-to-notebook usage is not asserted. Service wiring is documented in the README; source excerpts do not verify the deployment configuration.

Main entry points: [`docker-compose.yml`](https://github.com/giuliodamico/spark_local/blob/main/docker-compose.yml) (Spark master), [`notebooks/RDD_pyspark_lesson.ipynb`](https://github.com/giuliodamico/spark_local/blob/main/notebooks/RDD_pyspark_lesson.ipynb) (RDD lesson), [`notebooks/spark_analytics_deliverable.ipynb`](https://github.com/giuliodamico/spark_local/blob/main/notebooks/spark_analytics_deliverable.ipynb) (Analytics notebook)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_classroom["Classroom interface"]
  node_jupyter["JupyterLab<br/>[docker-compose.yml]"]
  node_spark_ui["Notebook Spark UI<br/>[docker-compose.yml]"]
end

subgraph group_compute["Spark compute"]
  node_master["Spark master<br/>[docker-compose.yml]"]
  node_workers["Spark workers<br/>[docker-compose.yml]"]
  node_history["History Server<br/>[docker-compose.yml]"]
end

subgraph group_learning["Learning materials"]
  node_notebook_lesson["RDD lesson"]
  node_notebook_deliverable["Analytics notebook"]
  node_bike_data["Bike datasets"]
  node_cdc_data["CDC sample"]
end

subgraph group_deployment["Stack deployment"]
  node_deployment["Docker stack<br/>[docker-compose.yml]"]
end

node_student(("Student"))

node_student -->|"uses browser"| node_jupyter
node_jupyter -->|"works with"| node_notebook_lesson
node_jupyter -->|"works with"| node_notebook_deliverable
node_notebook_lesson -->|"submits jobs"| node_master
node_notebook_deliverable -->|"submits jobs"| node_master
node_master -->|"coordinates cluster"| node_workers
node_student -->|"views UI"| node_master
node_student -->|"views history"| node_history
node_student -->|"views UI"| node_spark_ui
node_deployment -.->|"starts service"| node_jupyter
node_deployment -.->|"starts service"| node_master
node_deployment -.->|"starts services"| node_workers
node_deployment -.->|"starts service"| node_history

click node_jupyter "https://github.com/giuliodamico/spark_local/blob/main/docker-compose.yml"
click node_notebook_lesson "https://github.com/giuliodamico/spark_local/blob/main/notebooks/RDD_pyspark_lesson.ipynb"
click node_notebook_deliverable "https://github.com/giuliodamico/spark_local/blob/main/notebooks/spark_analytics_deliverable.ipynb"
click node_master "https://github.com/giuliodamico/spark_local/blob/main/docker-compose.yml"
click node_workers "https://github.com/giuliodamico/spark_local/blob/main/docker-compose.yml"
click node_history "https://github.com/giuliodamico/spark_local/blob/main/docker-compose.yml"
click node_spark_ui "https://github.com/giuliodamico/spark_local/blob/main/docker-compose.yml"
click node_bike_data "https://github.com/giuliodamico/spark_local/tree/main/book_data/bike"
click node_cdc_data "https://github.com/giuliodamico/spark_local/blob/main/book_data/cdc/mort2021us_sample.txt"
click node_deployment "https://github.com/giuliodamico/spark_local/blob/main/docker-compose.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_jupyter,node_spark_ui toneBlue
class node_master,node_workers,node_history toneAmber
class node_notebook_lesson,node_notebook_deliverable,node_bike_data,node_cdc_data toneMint
class node_deployment toneRose
class node_student toneIndigo
```

## Components

### Classroom interface

- [JupyterLab](https://github.com/giuliodamico/spark_local/blob/main/docker-compose.yml): `docker-compose.yml`
- [Notebook Spark UI](https://github.com/giuliodamico/spark_local/blob/main/docker-compose.yml): `docker-compose.yml`

### Spark compute

- [Spark master](https://github.com/giuliodamico/spark_local/blob/main/docker-compose.yml): `docker-compose.yml`
- [Spark workers](https://github.com/giuliodamico/spark_local/blob/main/docker-compose.yml): `docker-compose.yml`
- [History Server](https://github.com/giuliodamico/spark_local/blob/main/docker-compose.yml): `docker-compose.yml`

### Learning materials

- [RDD lesson](https://github.com/giuliodamico/spark_local/blob/main/notebooks/RDD_pyspark_lesson.ipynb): `notebooks/RDD_pyspark_lesson.ipynb`
- [Analytics notebook](https://github.com/giuliodamico/spark_local/blob/main/notebooks/spark_analytics_deliverable.ipynb): `notebooks/spark_analytics_deliverable.ipynb`
- [Bike datasets](https://github.com/giuliodamico/spark_local/tree/main/book_data/bike): `book_data/bike`
- [CDC sample](https://github.com/giuliodamico/spark_local/blob/main/book_data/cdc/mort2021us_sample.txt): `book_data/cdc/mort2021us_sample.txt`

### Stack deployment

- [Docker stack](https://github.com/giuliodamico/spark_local/blob/main/docker-compose.yml): `docker-compose.yml`

### Other components

- **Student**

## Connections

- Student → JupyterLab: uses browser (evidence: `README.md`)
- JupyterLab → RDD lesson: works with (evidence: `README.md`)
- JupyterLab → Analytics notebook: works with (evidence: `README.md`)
- RDD lesson → Spark master: submits jobs (evidence: `README.md`)
- Analytics notebook → Spark master: submits jobs (evidence: `README.md`)
- Spark master → Spark workers: coordinates cluster (evidence: `README.md`)
- Student → Spark master: views UI (evidence: `README.md`)
- Student → History Server: views history (evidence: `README.md`)
- Student → Notebook Spark UI: views UI (evidence: `README.md`)
- Docker stack → JupyterLab: starts service
- Docker stack → Spark master: starts service
- Docker stack → Spark workers: starts services
- Docker stack → History Server: starts service
