How hamzaoujja/datamart works
Project Type & Purpose
• This is a data-engineering pipeline / datamart project built on the Cookiecutter Data Science template.
• It orchestrates ETL jobs (via Apache Airflow) that extract data from external sources and MinIO, transform it through raw→interim→processed layers, load it into a warehouse and datamart, then exposes data for visualization or reporting.
File Structure
• Top-level dirs:
• airflow/ — airflow.cfg, dags, logs, plugins
• data/ — storage layers (external, interim, processed, datamart)
• sql/ — raw SQL schemas & inserts (creation.sql, insertion.sql, etc.)
• src/data/ — Python ETL modules (grab, write, transform, create_Marts)
• src/visualization/ — Python scripts + CSS for charting
• docs/, notebooks/, reports/ — documentation, exploratory analysis, final reports
• docker-compose.yml & Makefile — bring up containers, run pipeline commands
• Config files: configuration.yml, check.yaml, requirements.txt, setup.py, tox.ini.
Pull Insights from README
• Confirms use of Airflow DAGs for orchestration.
• Describes data directory layering (external→interim→processed→raw [implicit]).
• References make targets (make data, make train) that invoke scripts in src/.
a. Main Components
1. Orchestration: Airflow scheduler + webserver + workers
2. Object Store: MinIO (for raw and intermediate data dumps)
3. Data Warehouse: Postgres (or other RDBMS defined in docker-compose)
4. Datamart: Organizational subset built by create_Marts.py
5. ETL Code: Python modules in src/data
6. Visualization Layer: Python scripts in src/visualization (and optional notebooks/reports)
7. External Sources: Third-party APIs or CSVs (e.g. taxi_zone_lookup.csv)
8. Developer Interface: Makefile, CLI, docker-compose
b. Relationships & Data Flow
• External source → Airflow grabs data (grab_Data_From_Source) → writes to MinIO raw
• Airflow tasks → transform raw→interim→processed → load into Postgres warehouse
• Airflow task → create_Marts → materialize datamart tables in warehouse
• Visualization scripts fetch from warehouse/datamart → generate reports & figures → served via notebooks or static HTML/PDF
• Airflow logs all task runs in airflow/logs
• Docker-Compose wires up Airflow services, MinIO, Postgres, and any other containerized dependencies
c. Architectural Patterns & Principles
• Data-lake / data-warehouse layering (raw/interim/processed/datamart)
• Orchestration-as-code via Airflow DAGs (modular, event-driven jobs)
• Micro-services style: each container (Airflow, MinIO, Postgres) encapsulates a well-defined role
• Python package structure (src/ namespace, setup.py for installability)
d. Key Technologies
• Apache Airflow (scheduler, DAGs, plugins)
• MinIO (S3-compatible object store)
• PostgreSQL (or other SQL DB)
• Python (ETL & visualization modules)
• Docker Compose (container orchestration)
• Sphinx (docs), Jupyter (notebooks), Makefile (CLI)