How rockthejvm/spark-essentials works
Project Type & Purpose
– This is a “big data” demo/training application built in Scala using Apache Spark.
– It includes:
• A local SBT/Scala codebase with multiple Spark exercises (part1→part7, playground).
• A Dockerized PostgreSQL instance for Spark to read/write.
• A Dockerized Spark cluster (master + multiple workers) for distributed job execution.
• Sample data under src/main/resources/data (JSON, CSV, Parquet).
Main Components
A. Developer Environment
– IntelliJ / sbt build (build.sbt, project/*).
– Local code in src/main/scala and src/main/resources.
B. Data Sources
– Local resource files (JSON, CSV, Parquet) mounted into the cluster.
C. Docker Compose Network
1. PostgreSQL container (docker-compose.yml)
2. Spark Cluster (spark-cluster/docker-compose.yml)
• Spark Master
• Spark Submit image (entry point for jobs)
• Spark Worker nodes (scale=3)
D. Execution Flow
– spark-submit (from developer machine or container) to Spark Master
– Master distributes tasks to Workers
– Workers read input data (from mounted volume)
– Processes DataFrames/RDDs
– Optionally writes results to PostgreSQL over JDBC
Relationships & Interactions
– Developer → runs sbt/spark-submit → Spark Master
– Spark Master ↔ Spark Workers (job scheduling & task execution)
– Workers → Data (mounted HDFS‐style directory)
– Spark jobs → PostgreSQL (JDBC network call)
Architectural Patterns & Design Principles
– Distributed computation (Master/Workers)
– Containerized services (Docker Compose, isolation, easy reproduction)
– Modular code organization (by course modules: recap, dataframes, datasets, SQL, low‐level RDDs, practical, big data)
– External service integration (Postgres via JDBC)
Final Touches
– Ensure every component has a clear, unique label.
Layer 1: Developer & sbt project
Layer 2: Docker orchestration
Layer 3: Spark cluster & Postgres
Layer 4: Data volume