How jlsilva01/spark-iceberg works
Project type and purpose
• Type: A local data-processing demonstration/lab project rather than a traditional full-stack application. It leverages Apache Spark (PySpark) to read/write data, and Apache Iceberg as the storage format.
• Purpose: Show how to configure and use Spark locally (via WSL/Linux), run a Jupyter notebook with PySpark code to create and manipulate Iceberg tables, and observe the output on the local filesystem.
File structure
• Top-level files:
• README.md – project description and setup instructions
• spark-iceberg.ipynb – the demonstration notebook containing all Spark/Iceberg code
• pyproject.toml, uv.lock – Python project config and lockfile created by UV (dependency/version manager)
• .python-version – Python version for tools like pyenv
• .gitignore – excludes virtual environment, caches, etc.
• assets/ – screenshots and documentation images
• No “frontend” or “backend” folders; this is a single-process, notebook-driven data pipeline demo.
• No separate deployment scripts; you run everything locally under WSL/Linux.
Insights from README
• Environment setup uses UV commands: uv init, uv venv, uv add pyspark, jupyterlab, ipykernel.
• Targets Linux (Ubuntu 24.04) and shows how to select the .venv kernel in Jupyter/VS Code.
• Core dependencies: pyspark 3.5.3, Apache Iceberg Spark runtime.
Main components and interactions (
a. Developer workstation (WSL/Linux)
– Python runtime & UV toolchain (.venv)
– JupyterLab/VS Code
b. Jupyter Notebook (spark-iceberg.ipynb)
– Serves as the interactive “front end” for executing PySpark code
c. PySpark application
– Spark Driver process (runs notebook cells)
– Spark local Master & Executor(s) (in “local” mode)
d. Apache Iceberg
– Catalog & metadata layer (managed locally, files in a warehouse directory)
– Data files (Parquet/Avro) written to the local filesystem
e. Local filesystem
– Acts as both storage for Iceberg tables and for the virtual environment, notebook, and assets
Data flow:
Jupyter Notebook → PySpark Driver → Spark local cluster → Iceberg catalog APIs → Metadata & data files on disk
Architectural patterns and design principles
• Single-process “local” mode Spark cluster (no networked cluster)
• Separation of concerns:
• Notebook/UI layer
• Spark compute layer
• Iceberg storage layer
• Dependency management & reproducibility using UV (pyproject.toml + uv.lock)
Elements
– Developer Machine
– Python/.venv (UV)
– JupyterLab
– Spark Driver & Executors
– Iceberg Catalog & Data Files
– Local File System
– “Execute code” from Notebook → Driver
– “Dispatch tasks” from Driver → Executors
– “Read/write metadata & data” between Spark and Iceberg on disk
– e.g., blue for compute components (Spark), green for storage (Iceberg/files), gray for tooling (UV, Jupyter)