# dain55788/elt-data-pipeline architecture

> Architecture diagram of the GitHub repository dain55788/elt-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dain55788/elt-data-pipeline
- Repository: https://github.com/dain55788/elt-data-pipeline

## Overview

##### Project type and purpose
- This is an ELT (Extract-Load-Transform) data-pipeline project for a data-warehousing environment.
- It orchestrates ingestion from the OpenAQ API, staging in an on-premise data lake (MinIO + Hive Metastore), runs data‐quality checks (Great Expectations), transforms data (DBT) into a star schema in PostgreSQL, and supports analytics via Trino and PowerBI.

##### File structure
- Top-level directories:
- airflow/ – Apache Airflow configuration and DAG definitions for orchestration
- data_validation/ – Great Expectations configurations and validations
- dbt_openaq/ – DBT project for modeling and transformations
- trino-minio/ + storage-docker-compose.yaml – On-premise data lake setup (MinIO + Hive Metastore + Trino)
- scripts/ – Utility scripts to create schemas/tables and manage PostgreSQL
- docker-compose.yaml, airflow-docker-compose.yaml – Container orchestration for the entire stack
- Configuration files reveal use of Docker, Python dependencies (requirements.txt), and DBT profiles.

##### README for additional insights
- The “System Architecture” section sketches an ELT flow:
OpenAQ API → Airflow DAGs → MinIO/Hive → Spark processing → Trino queries → PostgreSQL warehouse → DBT modelling → PowerBI
- Technologies explicitly referenced: Apache Spark (for heavy processing), DBeaver (querying), PowerBI (visualization).

##### Detail
- Sub-component: Airflow scheduler vs workers, Spark executor, GE checkpoints, DBT models vs seeds
- Don’t hesitate to split out storage (raw vs curated), schema registry (Hive Metastore), and catalog (Trino)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Source
    subgraph "External Data Source"
        OpenAQAPI["OpenAQ API"]:::external
    end

    %% Orchestration Layer
    subgraph "Orchestration (Airflow)"
        IngestDAG["airquality_pipeline_dag.py"]:::orchestrator
        MigrationDAG["data_migration_dag.py"]:::orchestrator
        Collector["airquality_collector.py"]:::orchestrator
        MigrateUtil["migrate_data.py"]:::orchestrator
        AirflowPlugins["plugins/"]:::orchestrator
    end

    %% Data Lake
    subgraph "Data Lake (MinIO + Hive)"
        MinIO["MinIO Object Store"]:::storage
        HiveMetastore["Hive Metastore"]:::storage
    end

    %% Processing Engine
    subgraph "Processing (Spark)"
        Spark["Apache Spark"]:::processing
    end

    %% Data Quality
    subgraph "Data Quality (Great Expectations)"
        GE["Great Expectations"]:::quality
    end

    %% Query Engine
    subgraph "Query Engine (Trino)"
        Trino["Trino"]:::query
    end

    %% Transformation & Modeling
    subgraph "Transformation (DBT)"
        DBTProject["dbt_project.yml"]:::modeling
        DBTProfiles["profiles.yml"]:::modeling
    end

    %% Data Warehouse
    subgraph "Data Warehouse (PostgreSQL)"
        CreateSchema["create_schema.py"]:::warehouse
        CreateTable["create_table.py"]:::warehouse
        PGClient["postgresql_client.py"]:::warehouse
        PostgresDB["PostgreSQL"]:::warehouse
    end

    %% Analytics & BI
    subgraph "Analytics & BI"
        PowerBI["PowerBI"]:::analytics
        DBeaver["DBeaver"]:::analytics
    end

    %% Container Orchestration & Environment
    subgraph "Infrastructure & Deployment"
        DockerComposeAll["docker-compose.yaml"]:::infra
        AirflowCompose["airflow-docker-compose.yaml"]:::infra
        StorageCompose["storage-docker-compose.yaml"]:::infra
        DockerfileNode["Dockerfile"]:::infra
        Requirements["requirements.txt"]:::infra
    end

    %% Connections
    OpenAQAPI -->|"REST API"| IngestDAG
    IngestDAG -->|"S3 API"| MinIO
    MigrationDAG -->|"S3 API"| MinIO
    MigrationDAG -->|"Metastore"| HiveMetastore
    IngestDAG -->|"trigger Spark"| Spark
    Spark -->|"write cleansed"| MinIO
    MinIO -->|"staged data"| GE
    GE -->|"validation"| HiveMetastore
    GE -->|"on success"| DBTProject
    DBTProject -->|"read via JDBC"| PostgresDB
    DBTProfiles -->|"config"| DBTProject
    DBTProject -->|"models star schema"| PostgresDB
    MinIO -->|"catalog"| Trino
    Trino -->|"ad-hoc SQL"| PostgresDB
    PostgresDB -->|"reports"| PowerBI
    PostgresDB -->|"queries"| DBeaver
    Trino -->|"queries"| DBeaver

    %% Click Events
    click IngestDAG "https://github.com/dain55788/elt-data-pipeline/blob/main/airflow/dags/airquality_pipeline_dag.py"
    click MigrationDAG "https://github.com/dain55788/elt-data-pipeline/blob/main/airflow/dags/data_migration_dag.py"
    click Collector "https://github.com/dain55788/elt-data-pipeline/blob/main/airflow/dags/utils/airquality_collector.py"
    click MigrateUtil "https://github.com/dain55788/elt-data-pipeline/blob/main/airflow/dags/utils/migrate_data.py"
    click AirflowPlugins "https://github.com/dain55788/elt-data-pipeline/tree/main/airflow/plugins/"
    click MinIO "https://github.com/dain55788/elt-data-pipeline/blob/main/trino-minio/etc/catalog/minio.properties"
    click HiveMetastore "https://github.com/dain55788/elt-data-pipeline/blob/main/trino-minio/conf/metastore-site.xml"
    click GE "https://github.com/dain55788/elt-data-pipeline/blob/main/data_validation/gx/great_expectations.yml"
    click Trino "https://github.com/dain55788/elt-data-pipeline/blob/main/trino-minio/etc/config.properties"
    click DBTProject "https://github.com/dain55788/elt-data-pipeline/blob/main/dbt_openaq/dbt_project.yml"
    click DBTProfiles "https://github.com/dain55788/elt-data-pipeline/blob/main/dbt_openaq/profiles.yml"
    click CreateSchema "https://github.com/dain55788/elt-data-pipeline/blob/main/scripts/create_schema.py"
    click CreateTable "https://github.com/dain55788/elt-data-pipeline/blob/main/scripts/create_table.py"
    click PGClient "https://github.com/dain55788/elt-data-pipeline/blob/main/scripts/postgresql_client.py"
    click DockerComposeAll "https://github.com/dain55788/elt-data-pipeline/blob/main/docker-compose.yaml"
    click AirflowCompose "https://github.com/dain55788/elt-data-pipeline/blob/main/airflow-docker-compose.yaml"
    click StorageCompose "https://github.com/dain55788/elt-data-pipeline/blob/main/storage-docker-compose.yaml"
    click DockerfileNode "https://github.com/dain55788/elt-data-pipeline/tree/main/Dockerfile"
    click Requirements "https://github.com/dain55788/elt-data-pipeline/blob/main/requirements.txt"

    %% Styles
    classDef external fill:#f9f,stroke:#333,stroke-width:1px
    classDef orchestrator fill:#bbf,stroke:#333,stroke-width:1px
    classDef storage fill:#d0f0c0,stroke:#333,stroke-width:1px
    classDef processing fill:#ffeb99,stroke:#333,stroke-width:1px
    classDef quality fill:#fdd0a2,stroke:#333,stroke-width:1px
    classDef query fill:#c0e0ff,stroke:#333,stroke-width:1px
    classDef modeling fill:#b2f2bb,stroke:#333,stroke-width:1px
    classDef warehouse fill:#f2b0c0,stroke:#333,stroke-width:1px
    classDef analytics fill:#e0e0e0,stroke:#333,stroke-width:1px
    classDef infra fill:#d3d3d3,stroke:#333,stroke-width:1px
```
