# hnobakht/iceberg-practice-lab architecture

> Architecture diagram of the GitHub repository hnobakht/iceberg-practice-lab, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hnobakht/iceberg-practice-lab
- Repository: https://github.com/hnobakht/iceberg-practice-lab

## Overview

Iceberg Practice Lab is a local data-engineering environment for orchestrating Spark ETL and querying Iceberg tables across HDFS, Hive Metastore, and Trino, with both Spark-on-Kubernetes and standalone Spark examples. The principal workflow is an engineer-triggered Airflow DAG that checks and prepares a table, inserts generated records, then queries and summarizes them. The README documents additional stack services; their wiring is represented only where the README or sampled DAG configuration supports it, since their implementation files were not inspected.

Main entry points: [`airflow/dags/iceberg_random_records_dag.py`](https://github.com/hnobakht/iceberg-practice-lab/blob/main/airflow/dags/iceberg_random_records_dag.py) (Spark on K8s), [`spark/jobs/iceberg_prepare_table.py`](https://github.com/hnobakht/iceberg-practice-lab/blob/main/spark/jobs/iceberg_prepare_table.py) (Iceberg Tables), [`docker-compose.yaml`](https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml) (Airflow), [`configs`](https://github.com/hnobakht/iceberg-practice-lab/tree/main/configs) (Stack Configuration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_random_dag["Iceberg Pipeline DAG"]
  node_example_dag["K8s Example DAG"]
  node_airflow["Airflow"]
end

subgraph group_processing["Spark Processing"]
  node_k3s["K3s Cluster"]
  node_spark_k8s["Spark on K8s"]
  node_spark_standalone["Spark Standalone"]
  node_check["Check Table"]
  node_prepare["Prepare Table"]
  node_insert["Generate and Insert"]
  node_select["Query and Summarize"]
  node_hello["Hello World Job<br/>[hello_world.py]"]
end

subgraph group_lakehouse["Lakehouse Storage"]
  node_hdfs[("HDFS Warehouse")]
  node_metastore[("Hive Metastore")]
  node_iceberg[("Iceberg Tables")]
end

subgraph group_access["Query and Services"]
  node_trino["Trino"]
  node_postgres[("PostgreSQL")]
  node_redis["Redis Broker"]
  node_stack_config["Stack Configuration"]
end

node_engineer(("Engineer"))

node_engineer -->|"uses UI"| node_airflow
node_airflow -->|"runs DAG"| node_random_dag
node_airflow -->|"runs DAG"| node_example_dag
node_random_dag -->|"submits jobs"| node_spark_k8s
node_example_dag -->|"submits job"| node_spark_k8s
node_spark_k8s -->|"submits to"| node_k3s
node_spark_standalone -.->|"runs job"| node_hello
node_spark_k8s -->|"executes"| node_check
node_spark_k8s -->|"executes"| node_prepare
node_spark_k8s -->|"executes"| node_insert
node_spark_k8s -->|"executes"| node_select
node_check -->|"checks existence"| node_iceberg
node_prepare -->|"creates or clears"| node_iceberg
node_insert -->|"appends and reads"| node_iceberg
node_select -->|"queries"| node_iceberg
node_spark_k8s -->|"uses warehouse"| node_hdfs
node_spark_k8s -->|"uses catalog"| node_metastore
node_spark_standalone -.->|"uses warehouse"| node_hdfs
node_spark_standalone -.->|"uses catalog"| node_metastore
node_trino -->|"queries tables"| node_iceberg
node_metastore -->|"catalogs metadata"| node_iceberg
node_airflow -->|"uses database"| node_postgres
node_airflow -->|"uses broker"| node_redis
node_engineer -->|"runs SQL"| node_trino
node_stack_config -->|"configures"| node_spark_k8s
node_stack_config -->|"configures"| node_spark_standalone
node_stack_config -->|"configures"| node_trino
node_stack_config -->|"configures"| node_metastore

click node_random_dag "https://github.com/hnobakht/iceberg-practice-lab/blob/main/airflow/dags/iceberg_random_records_dag.py"
click node_example_dag "https://github.com/hnobakht/iceberg-practice-lab/blob/main/airflow/dags/spark_k8s_example.py"
click node_airflow "https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml"
click node_k3s "https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml"
click node_spark_k8s "https://github.com/hnobakht/iceberg-practice-lab/blob/main/airflow/dags/iceberg_random_records_dag.py"
click node_spark_standalone "https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml"
click node_check "https://github.com/hnobakht/iceberg-practice-lab/blob/main/spark/jobs/iceberg_check_table.py"
click node_prepare "https://github.com/hnobakht/iceberg-practice-lab/blob/main/spark/jobs/iceberg_prepare_table.py"
click node_insert "https://github.com/hnobakht/iceberg-practice-lab/blob/main/spark/jobs/iceberg_insert_random_records.py"
click node_select "https://github.com/hnobakht/iceberg-practice-lab/blob/main/spark/jobs/iceberg_select_records.py"
click node_hello "https://github.com/hnobakht/iceberg-practice-lab/blob/main/spark/jobs/hello_world.py"
click node_hdfs "https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml"
click node_metastore "https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml"
click node_iceberg "https://github.com/hnobakht/iceberg-practice-lab/blob/main/spark/jobs/iceberg_prepare_table.py"
click node_trino "https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml"
click node_postgres "https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml"
click node_redis "https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml"
click node_stack_config "https://github.com/hnobakht/iceberg-practice-lab/tree/main/configs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_random_dag,node_example_dag,node_airflow toneBlue
class node_k3s,node_spark_k8s,node_spark_standalone,node_check,node_prepare,node_insert,node_select,node_hello toneAmber
class node_hdfs,node_metastore,node_iceberg toneMint
class node_trino,node_postgres,node_redis,node_stack_config toneRose
class node_engineer toneIndigo
```

## Components

### Orchestration

- [Iceberg Pipeline DAG](https://github.com/hnobakht/iceberg-practice-lab/blob/main/airflow/dags/iceberg_random_records_dag.py): `airflow/dags/iceberg_random_records_dag.py`
- [K8s Example DAG](https://github.com/hnobakht/iceberg-practice-lab/blob/main/airflow/dags/spark_k8s_example.py): `airflow/dags/spark_k8s_example.py`
- [Airflow](https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml): `docker-compose.yaml`

### Spark Processing

- [K3s Cluster](https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml): `docker-compose.yaml`
- [Spark on K8s](https://github.com/hnobakht/iceberg-practice-lab/blob/main/airflow/dags/iceberg_random_records_dag.py): `airflow/dags/iceberg_random_records_dag.py`
- [Spark Standalone](https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml): `docker-compose.yaml`
- [Check Table](https://github.com/hnobakht/iceberg-practice-lab/blob/main/spark/jobs/iceberg_check_table.py): `spark/jobs/iceberg_check_table.py`
- [Prepare Table](https://github.com/hnobakht/iceberg-practice-lab/blob/main/spark/jobs/iceberg_prepare_table.py): `spark/jobs/iceberg_prepare_table.py`
- [Generate and Insert](https://github.com/hnobakht/iceberg-practice-lab/blob/main/spark/jobs/iceberg_insert_random_records.py): `spark/jobs/iceberg_insert_random_records.py`
- [Query and Summarize](https://github.com/hnobakht/iceberg-practice-lab/blob/main/spark/jobs/iceberg_select_records.py): `spark/jobs/iceberg_select_records.py`
- [Hello World Job](https://github.com/hnobakht/iceberg-practice-lab/blob/main/spark/jobs/hello_world.py): `spark/jobs/hello_world.py`

### Lakehouse Storage

- [HDFS Warehouse](https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml): `docker-compose.yaml`
- [Hive Metastore](https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml): `docker-compose.yaml`
- [Iceberg Tables](https://github.com/hnobakht/iceberg-practice-lab/blob/main/spark/jobs/iceberg_prepare_table.py): `spark/jobs/iceberg_prepare_table.py`

### Query and Services

- [Trino](https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml): `docker-compose.yaml`
- [PostgreSQL](https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml): `docker-compose.yaml`
- [Redis Broker](https://github.com/hnobakht/iceberg-practice-lab/blob/main/docker-compose.yaml): `docker-compose.yaml`
- [Stack Configuration](https://github.com/hnobakht/iceberg-practice-lab/tree/main/configs): `configs`

### Other components

- **Engineer**

## Connections

- Engineer → Airflow: uses UI
- Airflow → Iceberg Pipeline DAG: runs DAG
- Airflow → K8s Example DAG: runs DAG
- Iceberg Pipeline DAG → Spark on K8s: submits jobs
- K8s Example DAG → Spark on K8s: submits job
- Spark on K8s → K3s Cluster: submits to
- Spark Standalone → Hello World Job: runs job
- Spark on K8s → Check Table: executes
- Spark on K8s → Prepare Table: executes
- Spark on K8s → Generate and Insert: executes
- Spark on K8s → Query and Summarize: executes
- Check Table → Iceberg Tables: checks existence
- Prepare Table → Iceberg Tables: creates or clears
- Generate and Insert → Iceberg Tables: appends and reads
- Query and Summarize → Iceberg Tables: queries
- Spark on K8s → HDFS Warehouse: uses warehouse
- Spark on K8s → Hive Metastore: uses catalog
- Spark Standalone → HDFS Warehouse: uses warehouse
- Spark Standalone → Hive Metastore: uses catalog
- Trino → Iceberg Tables: queries tables
- Hive Metastore → Iceberg Tables: catalogs metadata
- Airflow → PostgreSQL: uses database
- Airflow → Redis Broker: uses broker
- Engineer → Trino: runs SQL
- Stack Configuration → Spark on K8s: configures
- Stack Configuration → Spark Standalone: configures
- Stack Configuration → Trino: configures
- Stack Configuration → Hive Metastore: configures