# stalpha2708/datalake architecture

> Architecture diagram of the GitHub repository stalpha2708/datalake, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stalpha2708/datalake
- Repository: https://github.com/stalpha2708/datalake

## Overview

##### Project type and purpose
- This is a data-engineering pipeline project combining two main subsystems:
a. A datalake powered by Trino (with an Iceberg connector), MinIO (object store), and Postgres (Iceberg metastore).
b. An Airflow orchestration layer that runs dbt transformations on that same datalake.
- The root README describes how to spin up both subsystems via separate Docker‐Compose stacks.

##### File structure
- Top-level:
- ReadMe.md – project overview
- airflow-db/ – custom Airflow image, DAGs, dbt project, Docker compose for orchestration
- trino-dlake/ – Trino cluster configuration, MinIO, Postgres metastore, sample data, Docker compose for the datalake
- airflow-db/ contains:
- Dockerfile + docker-compose.yml for Airflow components (webserver, scheduler, workers, metadata DB)
- config/airflow.cfg and plugins/ for custom operator or email hooks
- dags/dl_etl_dag.py – orchestrates dbt runs
- datalake/ – dbt project structure (models/staging, models/marts, macros, profiles)
- trino-dlake/ contains:
- etc/catalog/iceberg.properties – configures Iceberg connector pointing to Postgres metastore and MinIO
- data/ – sample Parquet files
- docker-compose.yml – services: Trino controller & workers, MinIO, Postgres

##### README for additional insights
- It shows a high-level image (datalake.jpg) depicting Trino, Iceberg, Postgres, MinIO.
- Commands to launch both datalake stack and Airflow stack are given.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External User
    User["User / Data Engineer"]:::external

    %% Airflow Orchestration Layer
    subgraph "Airflow Orchestration"
        AFCompose["Docker Compose"]:::orchestration
        AFDockerfile["Airflow Dockerfile"]:::orchestration
        AirflowCfg["airflow.cfg"]:::orchestration
        Plugins["Custom Plugins"]:::orchestration
        subgraph "Airflow Services"
            Webserver["Webserver"]:::orchestration
            Scheduler["Scheduler"]:::orchestration
            Workers["Workers"]:::orchestration
            MetadataDB["Metadata DB (Postgres)"]:::metadata
        end
        DAG["dl_etl_dag.py"]:::orchestration
    end

    %% dbt Transformation Layer
    subgraph "dbt Transformations"
        dbtProject["dbt_project.yml"]:::transformation
        Profiles["profiles.yml"]:::transformation
        subgraph "Models"
            Staging["staging"]:::transformation
            Marts["marts"]:::transformation
        end
        Macros["generate_schema_name.sql"]:::transformation
        dbtCLI["dbt CLI"]:::transformation
    end

    %% Storage Layer
    subgraph "Datalake Storage"
        MinIO["MinIO (Object Store)"]:::storage
        PostgresIce["Postgres (Iceberg Metastore)"]:::metadata
        SampleData["Sample Parquet Data"]:::storage
    end

    %% Query Engine Layer
    subgraph "Query Engine (Trino)"
        IcebergConf["iceberg.properties"]:::query
        TrinoCoordinator["Coordinator"]:::query
        TrinoWorkers["Workers"]:::query
    end

    %% User interactions
    User -->|"access UI/CLI"| Webserver
    Webserver -->|"schedule jobs"| Scheduler

    %% Airflow orchestration flow
    Scheduler -->|"trigger DAG"| DAG
    DAG -->|"invoke"| dbtCLI

    %% dbt transformation flow
    dbtCLI -->|"use project"| dbtProject
    dbtCLI -->|"load models"| Staging
    dbtCLI -->|"load models"| Marts
    dbtCLI -->|"use macros"| Macros
    dbtCLI -->|"use profiles"| Profiles
    dbtCLI -->|"write Parquet"| MinIO
    dbtCLI -->|"register metadata"| PostgresIce

    %% Trino query flow
    User -->|"submit SQL query"| TrinoCoordinator
    TrinoCoordinator -->|"distribute"| TrinoWorkers
    TrinoWorkers -->|"read data"| MinIO
    TrinoWorkers -->|"read metadata"| PostgresIce
    TrinoCoordinator -->|"use catalog config"| IcebergConf

    %% Click events for component mapping
    click AFCompose "https://github.com/stalpha2708/datalake/blob/main/airflow-db/docker-compose.yml"
    click AFDockerfile "https://github.com/stalpha2708/datalake/tree/main/airflow-db/Dockerfile"
    click AirflowCfg "https://github.com/stalpha2708/datalake/blob/main/airflow-db/config/airflow.cfg"
    click Plugins "https://github.com/stalpha2708/datalake/blob/main/airflow-db/plugins/email_config.py"
    click DAG "https://github.com/stalpha2708/datalake/blob/main/airflow-db/dags/dl_etl_dag.py"
    click dbtProject "https://github.com/stalpha2708/datalake/blob/main/airflow-db/datalake/dbt_project.yml"
    click Staging "https://github.com/stalpha2708/datalake/tree/main/airflow-db/datalake/models/staging"
    click Marts "https://github.com/stalpha2708/datalake/tree/main/airflow-db/datalake/models/marts"
    click Macros "https://github.com/stalpha2708/datalake/blob/main/airflow-db/datalake/macros/generate_schema_name.sql"
    click Profiles "https://github.com/stalpha2708/datalake/blob/main/airflow-db/profile/profiles.yml"
    click MinIO "https://github.com/stalpha2708/datalake/blob/main/trino-dlake/docker-compose.yml"
    click PostgresIce "https://github.com/stalpha2708/datalake/blob/main/trino-dlake/docker-compose.yml"
    click TrinoCoordinator "https://github.com/stalpha2708/datalake/blob/main/trino-dlake/docker-compose.yml"
    click TrinoWorkers "https://github.com/stalpha2708/datalake/blob/main/trino-dlake/docker-compose.yml"
    click IcebergConf "https://github.com/stalpha2708/datalake/blob/main/trino-dlake/etc/catalog/iceberg.properties"
    click SampleData "https://github.com/stalpha2708/datalake/blob/main/trino-dlake/data/sales_summary_updated.parquet"

    %% Styles
    classDef orchestration fill:#D0E8FF,stroke:#0275d8,stroke-width:1px;
    classDef transformation fill:#DFF2BF,stroke:#4F8A10,stroke-width:1px;
    classDef storage fill:#FFE8BA,stroke:#E87E04,stroke-width:1px;
    classDef metadata fill:#E2E3E5,stroke:#6C757D,stroke-width:1px;
    classDef query fill:#F0D9FF,stroke:#6F42C1,stroke-width:1px;
    classDef external fill:#FFD1DC,stroke:#C12F3D,stroke-width:1px;
```
