How dain55788/elt-data-pipeline works
Project type and purpose
• This is an ELT (Extract-Load-Transform) data-pipeline project for a data-warehousing environment.
• It orchestrates ingestion from the OpenAQ API, staging in an on-premise data lake (MinIO + Hive Metastore), runs data‐quality checks (Great Expectations), transforms data (DBT) into a star schema in PostgreSQL, and supports analytics via Trino and PowerBI.
File structure
• Top-level directories:
• airflow/ – Apache Airflow configuration and DAG definitions for orchestration
• data_validation/ – Great Expectations configurations and validations
• dbt_openaq/ – DBT project for modeling and transformations
• trino-minio/ + storage-docker-compose.yaml – On-premise data lake setup (MinIO + Hive Metastore + Trino)
• scripts/ – Utility scripts to create schemas/tables and manage PostgreSQL
• docker-compose.yaml, airflow-docker-compose.yaml – Container orchestration for the entire stack
• Configuration files reveal use of Docker, Python dependencies (requirements.txt), and DBT profiles.
README for additional insights
• The “System Architecture” section sketches an ELT flow:
OpenAQ API → Airflow DAGs → MinIO/Hive → Spark processing → Trino queries → PostgreSQL warehouse → DBT modelling → PowerBI
• Technologies explicitly referenced: Apache Spark (for heavy processing), DBeaver (querying), PowerBI (visualization).
Detail
• Sub-component: Airflow scheduler vs workers, Spark executor, GE checkpoints, DBT models vs seeds
• Don’t hesitate to split out storage (raw vs curated), schema registry (Hive Metastore), and catalog (Trino)