How hnobakht/iceberg-practice-lab works
Iceberg Practice Lab is a local data-engineering environment for orchestrating Spark ETL and querying Iceberg tables across HDFS, Hive Metastore, and Trino, with both Spark-on-Kubernetes and standalone Spark examples. The principal workflow is an engineer-triggered Airflow DAG that checks and prepares a table, inserts generated records, then queries and summarizes them. The README documents additional stack services; their wiring is represented only where the README or sampled DAG configuration supports it, since their implementation files were not inspected.
Main entry points: iceberg_random_records_dag.py Spark on K8s · iceberg_prepare_table.py Iceberg Tables · docker-compose.yaml Airflow · configs Stack Configuration
Components 19
Orchestration
- Iceberg Pipeline DAG
airflow/dags/iceberg_random_records_dag.py - K8s Example DAG
airflow/dags/spark_k8s_example.py - Airflow
docker-compose.yaml
Spark Processing
- K3s Cluster
docker-compose.yaml - Spark on K8s
airflow/dags/iceberg_random_records_dag.py - Spark Standalone
docker-compose.yaml - Check Table
spark/jobs/iceberg_check_table.py - Prepare Table
spark/jobs/iceberg_prepare_table.py - Generate and Insert
spark/jobs/iceberg_insert_random_records.py - Query and Summarize
spark/jobs/iceberg_select_records.py - Hello World Job
spark/jobs/hello_world.py
Lakehouse Storage
- HDFS Warehouse
docker-compose.yaml - Hive Metastore
docker-compose.yaml - Iceberg Tables
spark/jobs/iceberg_prepare_table.py
Query and Services
- Trino
docker-compose.yaml - PostgreSQL
docker-compose.yaml - Redis Broker
docker-compose.yaml - Stack Configuration
configs
Other components
- Engineer
Connections 28
- Engineer → Airflow: uses UI
- Airflow → Iceberg Pipeline DAG: runs DAG
- Airflow → K8s Example DAG: runs DAG
- Iceberg Pipeline DAG → Spark on K8s: submits jobs
- K8s Example DAG → Spark on K8s: submits job
- Spark on K8s → K3s Cluster: submits to
- Spark Standalone → Hello World Job: runs job
- Spark on K8s → Check Table: executes
- Spark on K8s → Prepare Table: executes
- Spark on K8s → Generate and Insert: executes
- Spark on K8s → Query and Summarize: executes
- Check Table → Iceberg Tables: checks existence
- Prepare Table → Iceberg Tables: creates or clears
- Generate and Insert → Iceberg Tables: appends and reads
- Query and Summarize → Iceberg Tables: queries
- Spark on K8s → HDFS Warehouse: uses warehouse
- Spark on K8s → Hive Metastore: uses catalog
- Spark Standalone → HDFS Warehouse: uses warehouse
- Spark Standalone → Hive Metastore: uses catalog
- Trino → Iceberg Tables: queries tables
- Hive Metastore → Iceberg Tables: catalogs metadata
- Airflow → PostgreSQL: uses database
- Airflow → Redis Broker: uses broker
- Engineer → Trino: runs SQL
- Stack Configuration → Spark on K8s: configures
- Stack Configuration → Spark Standalone: configures
- Stack Configuration → Trino: configures
- Stack Configuration → Hive Metastore: configures