# hamzaoujja/datamart architecture

> Architecture diagram of the GitHub repository hamzaoujja/datamart, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hamzaoujja/datamart
- Repository: https://github.com/hamzaoujja/datamart

## Overview

##### Project Type & Purpose
- This is a data-engineering pipeline / datamart project built on the Cookiecutter Data Science template.
- It orchestrates ETL jobs (via Apache Airflow) that extract data from external sources and MinIO, transform it through raw→interim→processed layers, load it into a warehouse and datamart, then exposes data for visualization or reporting.

##### File Structure
- Top-level dirs:
- airflow/ — airflow.cfg, dags, logs, plugins
- data/ — storage layers (external, interim, processed, datamart)
- sql/ — raw SQL schemas & inserts (creation.sql, insertion.sql, etc.)
- src/data/ — Python ETL modules (grab, write, transform, create_Marts)
- src/visualization/ — Python scripts + CSS for charting
- docs/, notebooks/, reports/ — documentation, exploratory analysis, final reports
- docker-compose.yml & Makefile — bring up containers, run pipeline commands
- Config files: configuration.yml, check.yaml, requirements.txt, setup.py, tox.ini.

##### Pull Insights from README
- Confirms use of Airflow DAGs for orchestration.
- Describes data directory layering (external→interim→processed→raw [implicit]).
- References make targets (make data, make train) that invoke scripts in src/.

a. Main Components
1. Orchestration: Airflow scheduler + webserver + workers
2. Object Store: MinIO (for raw and intermediate data dumps)
3. Data Warehouse: Postgres (or other RDBMS defined in docker-compose)
4. Datamart: Organizational subset built by `create_Marts.py`
5. ETL Code: Python modules in src/data
6. Visualization Layer: Python scripts in src/visualization (and optional notebooks/reports)
7. External Sources: Third-party APIs or CSVs (e.g. taxi_zone_lookup.csv)
8. Developer Interface: Makefile, CLI, docker-compose
b. Relationships & Data Flow
- External source → Airflow grabs data (grab_Data_From_Source) → writes to MinIO raw
- Airflow tasks → transform raw→interim→processed → load into Postgres warehouse
- Airflow task → create_Marts → materialize datamart tables in warehouse
- Visualization scripts fetch from warehouse/datamart → generate reports & figures → served via notebooks or static HTML/PDF
- Airflow logs all task runs in airflow/logs
- Docker-Compose wires up Airflow services, MinIO, Postgres, and any other containerized dependencies
c. Architectural Patterns & Principles
- Data-lake / data-warehouse layering (raw/interim/processed/datamart)
- Orchestration-as-code via Airflow DAGs (modular, event-driven jobs)
- Micro-services style: each container (Airflow, MinIO, Postgres) encapsulates a well-defined role
- Python package structure (src/ namespace, setup.py for installability)
d. Key Technologies
- Apache Airflow (scheduler, DAGs, plugins)
- MinIO (S3-compatible object store)
- PostgreSQL (or other SQL DB)
- Python (ETL & visualization modules)
- Docker Compose (container orchestration)
- Sphinx (docs), Jupyter (notebooks), Makefile (CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    subgraph "External Systems" 
        direction TB
        API["Third-Party APIs"]:::external
        CSV["taxi_zone_lookup.csv"]:::external
        EXDIR["data/external/"]:::datastore
    end

    %% Developer Interface
    subgraph "Developer Interface" 
        direction TB
        MAKE["Makefile"]:::config
        DC["docker-compose.yml"]:::config
        CFG["configuration.yml"]:::config
        CHECK["check.yaml"]:::config
        SETUP["setup.py"]:::config
        REQ["requirements.txt"]:::config
    end

    %% Docker Network Boundary
    subgraph "Docker Network" 
        direction TB
        %% Orchestration: Airflow
        subgraph "Airflow Service" 
            direction TB
            SCHED["Airflow Scheduler"]:::service
            WEB["Airflow Webserver"]:::service
            WORK["Airflow Worker"]:::service
            DAG["minio_workflow.py DAG"]:::service
        end

        %% Object Store
        MINIO["MinIO (Object Store)"]:::service

        %% Data Warehouse
        PG["Postgres (Warehouse DB)"]:::service
    end

    %% Data Lake / Storage Layers
    subgraph "Data Lake & Processed Layers" 
        direction TB
        RAW["data/external/raw"]:::datastore
        INT["data/interim/"]:::datastore
        PROC["data/processed/"]:::datastore
        DMFS["data/datamart/"]:::datastore
    end

    %% SQL Schemas
    subgraph "SQL Definitions" 
        direction TB
        SCHEMA["creation.sql"]:::config
        INSERT1["insertion.sql"]:::config
        INSERT2["insertion2.sql"]:::config
    end

    %% ETL Code
    subgraph "ETL Modules (src/data)" 
        direction TB
        G1["grab_Data_From_Source.py"]:::code
        G2["grab_Data_From_MinIO.py"]:::code
        W1["write_Data_To_MinIO.py"]:::code
        W2["write_Data_To_Warehouse.py"]:::code
        WD2M["warehouse_to_datamart.py"]:::code
        CM["create_Marts.py"]:::code
    end

    %% Visualization Layer
    subgraph "Visualization (src/visualization)" 
        direction TB
        FETCH["fetch_Data.py"]:::code
        VIS["visualize.py"]:::code
        STYLE["style.css"]:::code
        NB["notebooks/"]:::code
        RPT["reports/"]:::code
    end

    %% Data & Control Flows
    API -->|"pull data"| G1
    CSV --> G1
    EXDIR --> G1
    G1 -->|"raw data"| RAW
    SCHED -->|"trigger DAG"| DAG
    DAG -->|"ingest task"| G1
    DAG -->|"ingest task"| G2
    G2 -->|"reads raw"| RAW
    G2 -->|"writes interim"| INT
    DAG -->|"transform task"| W2
    W2 -->|"load warehouse"| PG
    DAG -->|"transform task"| W1
    W1 -->|"writes processed"| PROC
    DAG -->|"datamart build"| CM
    CM -->|"materialize tables"| PG
    CM -->|"write datamart FS"| DMFS
    PG --> FETCH
    PG --> VIS
    FETCH --> VIS
    VIS --> NB
    VIS --> RPT
    VIS --> STYLE
    MAKE --> DC
    DC --> SCHED
    DC --> WEB
    DC --> WORK
    DC --> MINIO
    DC --> PG

    %% Click Events
    click API "https://github.com/hamzaoujja/datamart/tree/main/data/external/"
    click CSV "https://github.com/hamzaoujja/datamart/blob/main/data/taxi_zone_lookup.csv"
    click EXDIR "https://github.com/hamzaoujja/datamart/tree/main/data/external/"
    click MAKE "https://github.com/hamzaoujja/datamart/tree/main/Makefile"
    click DC "https://github.com/hamzaoujja/datamart/blob/main/docker-compose.yml"
    click CFG "https://github.com/hamzaoujja/datamart/blob/main/configuration.yml"
    click CHECK "https://github.com/hamzaoujja/datamart/blob/main/check.yaml"
    click SETUP "https://github.com/hamzaoujja/datamart/blob/main/setup.py"
    click REQ "https://github.com/hamzaoujja/datamart/blob/main/requirements.txt"
    click SCHED "https://github.com/hamzaoujja/datamart/tree/main/airflow/"
    click WEB "https://github.com/hamzaoujja/datamart/tree/main/airflow/"
    click WORK "https://github.com/hamzaoujja/datamart/tree/main/airflow/"
    click DAG "https://github.com/hamzaoujja/datamart/blob/main/airflow/dags/minio_workflow.py"
    click MINIO "https://github.com/hamzaoujja/datamart/blob/main/docker-compose.yml"
    click PG "https://github.com/hamzaoujja/datamart/blob/main/docker-compose.yml"
    click RAW "https://github.com/hamzaoujja/datamart/tree/main/data/external/"
    click INT "https://github.com/hamzaoujja/datamart/tree/main/data/interim/"
    click PROC "https://github.com/hamzaoujja/datamart/tree/main/data/processed/"
    click DMFS "https://github.com/hamzaoujja/datamart/tree/main/data/datamart/"
    click G1 "https://github.com/hamzaoujja/datamart/blob/main/src/data/grab_Data_From_Source.py"
    click G2 "https://github.com/hamzaoujja/datamart/blob/main/src/data/grab_Data_From_MinIO.py"
    click W1 "https://github.com/hamzaoujja/datamart/blob/main/src/data/write_Data_To_MinIO.py"
    click W2 "https://github.com/hamzaoujja/datamart/blob/main/src/data/write_Data_To_Warehouse.py"
    click WD2M "https://github.com/hamzaoujja/datamart/blob/main/src/data/warehouse_to_datamart.py"
    click CM "https://github.com/hamzaoujja/datamart/blob/main/src/data/create_Marts.py"
    click FETCH "https://github.com/hamzaoujja/datamart/blob/main/src/visualization/fetch_Data.py"
    click VIS "https://github.com/hamzaoujja/datamart/blob/main/src/visualization/visualize.py"
    click STYLE "https://github.com/hamzaoujja/datamart/blob/main/src/visualization/style.css"
    click NB "https://github.com/hamzaoujja/datamart/tree/main/notebooks/"
    click RPT "https://github.com/hamzaoujja/datamart/tree/main/reports/"
    click SCHEMA "https://github.com/hamzaoujja/datamart/blob/main/sql/creation.sql"
    click INSERT1 "https://github.com/hamzaoujja/datamart/blob/main/sql/insertion.sql"
    click INSERT2 "https://github.com/hamzaoujja/datamart/blob/main/sql/insertion2.sql"

    %% Styles
    classDef service fill:#AED6F1,stroke:#2980B9,color:#1B2631;
    classDef datastore fill:#ABEBC6,stroke:#239B56,color:#145A32,stroke-dasharray: 5 5;
    classDef code fill:#F9E79F,stroke:#B7950B,color:#7D6608;
    classDef external fill:#D5D8DC,stroke:#7F8C8D,color:#2C3E50,stroke-dasharray: 2 2;
    classDef config fill:#E5E7E9,stroke:#A6ACAF,color:#212F3D,stroke-dasharray: 2 2;
```
