How giuliodamico/spark_local works
This repository provides a local classroom Spark environment: students use JupyterLab to work with notebooks, whose jobs run on a standalone Spark cluster. The documented stack also exposes the Spark master, History Server, and the current notebook’s Spark UI. The graph preserves the sample notebooks and datasets, but their contents were not inspected, so dataset-to-notebook usage is not asserted. Service wiring is documented in the README; source excerpts do not verify the deployment configuration.
Main entry points: docker-compose.yml Spark master · RDD_pyspark_lesson.ipynb RDD lesson · spark_analytics_deliverable.ipynb Analytics notebook
Components 11
Classroom interface
- JupyterLab
docker-compose.yml - Notebook Spark UI
docker-compose.yml
Spark compute
- Spark master
docker-compose.yml - Spark workers
docker-compose.yml - History Server
docker-compose.yml
Learning materials
- RDD lesson
notebooks/RDD_pyspark_lesson.ipynb - Analytics notebook
notebooks/spark_analytics_deliverable.ipynb - Bike datasets
book_data/bike - CDC sample
book_data/cdc/mort2021us_sample.txt
Stack deployment
- Docker stack
docker-compose.yml
Other components
- Student
Connections 13
- Student → JupyterLab: uses browser
README.md - JupyterLab → RDD lesson: works with
README.md - JupyterLab → Analytics notebook: works with
README.md - RDD lesson → Spark master: submits jobs
README.md - Analytics notebook → Spark master: submits jobs
README.md - Spark master → Spark workers: coordinates cluster
README.md - Student → Spark master: views UI
README.md - Student → History Server: views history
README.md - Student → Notebook Spark UI: views UI
README.md - Docker stack → JupyterLab: starts service
- Docker stack → Spark master: starts service
- Docker stack → Spark workers: starts services
- Docker stack → History Server: starts service