How stalpha2708/datalake works
Project type and purpose
• This is a data-engineering pipeline project combining two main subsystems:
a. A datalake powered by Trino (with an Iceberg connector), MinIO (object store), and Postgres (Iceberg metastore).
b. An Airflow orchestration layer that runs dbt transformations on that same datalake.
• The root README describes how to spin up both subsystems via separate Docker‐Compose stacks.
File structure
• Top-level:
• ReadMe.md – project overview
• airflow-db/ – custom Airflow image, DAGs, dbt project, Docker compose for orchestration
• trino-dlake/ – Trino cluster configuration, MinIO, Postgres metastore, sample data, Docker compose for the datalake
• airflow-db/ contains:
• Dockerfile + docker-compose.yml for Airflow components (webserver, scheduler, workers, metadata DB)
• config/airflow.cfg and plugins/ for custom operator or email hooks
• dags/dl_etl_dag.py – orchestrates dbt runs
• datalake/ – dbt project structure (models/staging, models/marts, macros, profiles)
• trino-dlake/ contains:
• etc/catalog/iceberg.properties – configures Iceberg connector pointing to Postgres metastore and MinIO
• data/ – sample Parquet files
• docker-compose.yml – services: Trino controller & workers, MinIO, Postgres
README for additional insights
• It shows a high-level image (datalake.jpg) depicting Trino, Iceberg, Postgres, MinIO.
• Commands to launch both datalake stack and Airflow stack are given.