# jlsilva01/spark-iceberg architecture

> Architecture diagram of the GitHub repository jlsilva01/spark-iceberg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jlsilva01/spark-iceberg
- Repository: https://github.com/jlsilva01/spark-iceberg

## Overview

##### Project type and purpose
- Type: A local data-processing demonstration/lab project rather than a traditional full-stack application. It leverages Apache Spark (PySpark) to read/write data, and Apache Iceberg as the storage format.
- Purpose: Show how to configure and use Spark locally (via WSL/Linux), run a Jupyter notebook with PySpark code to create and manipulate Iceberg tables, and observe the output on the local filesystem.

##### File structure
- Top-level files:
- README.md            – project description and setup instructions
- spark-iceberg.ipynb  – the demonstration notebook containing all Spark/Iceberg code
- pyproject.toml, uv.lock – Python project config and lockfile created by UV (dependency/version manager)
- .python-version      – Python version for tools like pyenv
- .gitignore           – excludes virtual environment, caches, etc.
- assets/              – screenshots and documentation images
- No “frontend” or “backend” folders; this is a single-process, notebook-driven data pipeline demo.
- No separate deployment scripts; you run everything locally under WSL/Linux.

##### Insights from README
- Environment setup uses UV commands: uv init, uv venv, uv add pyspark, jupyterlab, ipykernel.
- Targets Linux (Ubuntu 24.04) and shows how to select the .venv kernel in Jupyter/VS Code.
- Core dependencies: pyspark 3.5.3, Apache Iceberg Spark runtime.

##### Main components and interactions (
a. Developer workstation (WSL/Linux)
– Python runtime & UV toolchain (.venv)
– JupyterLab/VS Code
b. Jupyter Notebook (spark-iceberg.ipynb)
– Serves as the interactive “front end” for executing PySpark code
c. PySpark application
– Spark Driver process (runs notebook cells)
– Spark local Master & Executor(s) (in “local” mode)
d. Apache Iceberg
– Catalog & metadata layer (managed locally, files in a warehouse directory)
– Data files (Parquet/Avro) written to the local filesystem
e. Local filesystem
– Acts as both storage for Iceberg tables and for the virtual environment, notebook, and assets

Data flow:
Jupyter Notebook → PySpark Driver → Spark local cluster → Iceberg catalog APIs → Metadata & data files on disk

##### Architectural patterns and design principles
- Single-process “local” mode Spark cluster (no networked cluster)
- Separation of concerns:
- Notebook/UI layer
- Spark compute layer
- Iceberg storage layer
- Dependency management & reproducibility using UV (pyproject.toml + uv.lock)

##### Elements
– Developer Machine
– Python/.venv (UV)
– JupyterLab
– Spark Driver & Executors
– Iceberg Catalog & Data Files
– Local File System
– “Execute code” from Notebook → Driver
– “Dispatch tasks” from Driver → Executors
– “Read/write metadata & data” between Spark and Iceberg on disk
– e.g., blue for compute components (Spark), green for storage (Iceberg/files), gray for tooling (UV, Jupyter)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Developer Machine"
        subgraph "Environment"
            PYaml["pyproject.toml"]:::tool
            ULock["uv.lock"]:::tool
            PVer[".python-version"]:::tool
        end
        README["README.md"]:::doc
        Assets["assets/"]:::doc
        Jupyter["spark-iceberg.ipynb"]:::frontend
        subgraph "Spark Runtime"
            Driver["Spark Driver"]:::compute
            Executors["Spark Executors (local)"]:::compute
        end
        subgraph "Iceberg Storage"
            Catalog["Iceberg Catalog"]:::storage
            DataFiles["Data Files (Parquet/Avro)"]:::storage
        end
        FS["Local File System"]:::storage
    end

    PYaml -->|"defines dependencies"| Jupyter
    ULock -->|"locks versions"| Jupyter
    PVer -->|"specifies Python version"| Jupyter
    Jupyter -->|"execute code"| Driver
    Driver -->|"dispatch tasks"| Executors
    Driver -->|"read/write metadata"| Catalog
    Executors -->|"read/write data files"| DataFiles
    Catalog -->|"persists metadata"| FS
    DataFiles -->|"persists data"| FS

    click Jupyter "https://github.com/jlsilva01/spark-iceberg/blob/main/spark-iceberg.ipynb"
    click PYaml "https://github.com/jlsilva01/spark-iceberg/blob/main/pyproject.toml"
    click ULock "https://github.com/jlsilva01/spark-iceberg/blob/main/uv.lock"
    click PVer "https://github.com/jlsilva01/spark-iceberg/blob/main/.python-version"
    click README "https://github.com/jlsilva01/spark-iceberg/blob/main/README.md"
    click Assets "https://github.com/jlsilva01/spark-iceberg/tree/main/assets/"

    classDef tool fill:#D3D3D3,stroke:#333;
    classDef doc fill:#F5DEB3,stroke:#333;
    classDef frontend fill:#ADD8E6,stroke:#333;
    classDef compute fill:#87CEFA,stroke:#333;
    classDef storage fill:#90EE90,stroke:#333;
```
