# bambanza/chai_assessment architecture

> Architecture diagram of the GitHub repository bambanza/chai_assessment, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bambanza/chai_assessment
- Repository: https://github.com/bambanza/chai_assessment

## Overview

##### Project Type and Purpose
- Type: A containerized, end-to-end ETL pipeline with orchestration (Airflow), a relational database (PostgreSQL), and a BI/visualization tool (Metabase).

##### Main Architectural Components
a. External Data Source
– Open-Meteo Weather API (REST endpoint)
b. Container Orchestration
– Docker Compose as the top-level orchestrator
c. ETL Pipelines
– One-time runner at startup (pipeline/run_pipeline.py)
– Airflow DAG “weather_etl_pipeline” (airflow/dags/weather_etl_dag.py)
d. Data Storage
– Host-mounted Volumes:
- data/raw (raw CSV)
- data/processed (clean CSV)
– PostgreSQL Container (db/init.sql to create tables)
- staging_weather table
- mart_daily_city_weather table
e. Orchestration & Scheduling
– Airflow Scheduler & Webserver Container (airflow.cfg, Dockerfile.airflow)
f. Visualization
– Metabase Container (metabase-init for init script, metabase-data volume)

##### Relationships & Data Flow
- At startup: Docker Compose spins up
1. PostgreSQL → “pipeline” container runs ingest → transform → model (writes to volumes + DB)
2. Airflow Scheduler + Webserver container
3. Metabase container
- External call: Pipeline.ingest hits Open-Meteo API → writes raw CSV
- Pipeline.transform reads raw CSV → writes clean CSV
- Pipeline.model loads CSVs + cities.csv into Postgres (staging → mart)
- Airflow DAG tasks (ingest_weather, transform_weather, model_weather) invoke the same pipeline functions on schedule
- Metabase polls Postgres for mart_daily_city_weather to display dashboards

##### Architectural Patterns & Principles
- ETL pattern (Extract → Transform → Load)
- DAG orchestration (Airflow directed acyclic graph)
- Infrastructure as Code / Containerization (Dockerfile, Docker Compose)
- Volume-based data persistence for CSV artifacts
- Clear separation of concerns: ingestion, transformation, modeling, orchestration, storage, visualization

## Diagram (Mermaid)

```mermaid
flowchart TD
    classDef etlCompute fill:#cfe2f3,stroke:#0000ff,color:#0000ff
    classDef storage fill:#d9ead3,stroke:#00a000,color:#00a000
    classDef orchestration fill:#f9cb9c,stroke:#ff6600,color:#ff6600
    classDef external fill:#d9d2e9,stroke:#666666,color:#666666

    compose["Docker Compose"]:::orchestration
    click compose "https://github.com/bambanza/chai_assessment/blob/main/docker-compose.yml"

    subgraph "Docker Compose"
        PSQL[(PostgreSQL<br/>localhost:5433)]:::storage
        initScript["db/init.sql"]:::storage
        click initScript "https://github.com/bambanza/chai_assessment/blob/main/db/init.sql"

        subgraph "Pipeline Container"
            runner["One-time Runner<br/>(run_pipeline.py)"]:::etlCompute
            click runner "https://github.com/bambanza/chai_assessment/blob/main/pipeline/run_pipeline.py"
            ingest["Ingest Logic"]:::etlCompute
            click ingest "https://github.com/bambanza/chai_assessment/blob/main/pipeline/ingest.py"
            transform["Transform Logic"]:::etlCompute
            click transform "https://github.com/bambanza/chai_assessment/blob/main/pipeline/transform.py"
            model["Load/Model Logic"]:::etlCompute
            click model "https://github.com/bambanza/chai_assessment/blob/main/pipeline/model.py"
            config["Pipeline Config"]:::etlCompute
            click config "https://github.com/bambanza/chai_assessment/blob/main/pipeline/config.py"
            pipelineDf["Dockerfile"]:::etlCompute
            click pipelineDf "https://github.com/bambanza/chai_assessment/tree/main/Dockerfile"
            req["requirements.txt"]:::etlCompute
            click req "https://github.com/bambanza/chai_assessment/blob/main/requirements.txt"
        end

        subgraph "Airflow Container"
            airflow["Airflow<br/>Scheduler & Webserver"]:::orchestration
            dag["weather_etl_dag.py"]:::orchestration
            click dag "https://github.com/bambanza/chai_assessment/blob/main/airflow/dags/weather_etl_dag.py"
            airflowCfg["airflow.cfg"]:::orchestration
            click airflowCfg "https://github.com/bambanza/chai_assessment/blob/main/airflow.cfg"
            dockerfileAirflow["Dockerfile.airflow"]:::orchestration
            click dockerfileAirflow "https://github.com/bambanza/chai_assessment/blob/main/Dockerfile.airflow"
        end

        subgraph "Metabase Container"
            metabase["Metabase"]:::orchestration
            metabaseInitScript["init-metabase.sh"]:::orchestration
            click metabaseInitScript "https://github.com/bambanza/chai_assessment/blob/main/metabase-init/init-metabase.sh"
            metabaseDockerfile["Dockerfile"]:::orchestration
            click metabaseDockerfile "https://github.com/bambanza/chai_assessment/tree/main/metabase-init/Dockerfile"
            metabaseDataVol["metabase-data/"]:::storage
            click metabaseDataVol "https://github.com/bambanza/chai_assessment/tree/main/metabase-data/"
        end

        RAW[(weather_raw.csv)]:::storage
        click RAW "https://github.com/bambanza/chai_assessment/blob/main/data/raw/weather_raw.csv"
        CLEAN[(weather_clean.csv)]:::storage
        click CLEAN "https://github.com/bambanza/chai_assessment/blob/main/data/processed/weather_clean.csv"
    end

    api["Open-Meteo API"]:::external
    click api "https://github.com/bambanza/chai_assessment/blob/main/pipeline/ingest.py"
    cities["cities.csv"]:::storage
    click cities "https://github.com/bambanza/chai_assessment/blob/main/data/cities.csv"

    api -->|HTTP GET| ingest
    ingest -->|CSV write| RAW
    RAW -->|read| transform
    transform -->|CSV write| CLEAN
    cities -->|read| model
    model -->|INSERT staging_weather| PSQL
    model -->|INSERT mart_daily_city_weather| PSQL
    initScript -->|init tables| PSQL

    airflow -->|ingest_weather| ingest
    airflow -->|transform_weather| transform
    airflow -->|model_weather| model

    metabase -->|reads mart_daily_city_weather| PSQL

    PSQL -->|serves DB| pipelineDf
    PSQL -->|serves DB| metabase

    compose --> PSQL
    compose --> runner
    compose --> airflow
    compose --> metabase
```
