How bambanza/chai_assessment works
Project Type and Purpose
• Type: A containerized, end-to-end ETL pipeline with orchestration (Airflow), a relational database (PostgreSQL), and a BI/visualization tool (Metabase).
Main Architectural Components
a. External Data Source
– Open-Meteo Weather API (REST endpoint)
b. Container Orchestration
– Docker Compose as the top-level orchestrator
c. ETL Pipelines
– One-time runner at startup (pipeline/run_pipeline.py)
– Airflow DAG “weather_etl_pipeline” (airflow/dags/weather_etl_dag.py)
d. Data Storage
– Host-mounted Volumes:
• data/raw (raw CSV)
• data/processed (clean CSV)
– PostgreSQL Container (db/init.sql to create tables)
• staging_weather table
• mart_daily_city_weather table
e. Orchestration & Scheduling
– Airflow Scheduler & Webserver Container (airflow.cfg, Dockerfile.airflow)
f. Visualization
– Metabase Container (metabase-init for init script, metabase-data volume)
Relationships & Data Flow
• At startup: Docker Compose spins up
1. PostgreSQL → “pipeline” container runs ingest → transform → model (writes to volumes + DB)
2. Airflow Scheduler + Webserver container
3. Metabase container
• External call: Pipeline.ingest hits Open-Meteo API → writes raw CSV
• Pipeline.transform reads raw CSV → writes clean CSV
• Pipeline.model loads CSVs + cities.csv into Postgres (staging → mart)
• Airflow DAG tasks (ingest_weather, transform_weather, model_weather) invoke the same pipeline functions on schedule
• Metabase polls Postgres for mart_daily_city_weather to display dashboards
Architectural Patterns & Principles
• ETL pattern (Extract → Transform → Load)
• DAG orchestration (Airflow directed acyclic graph)
• Infrastructure as Code / Containerization (Dockerfile, Docker Compose)
• Volume-based data persistence for CSV artifacts
• Clear separation of concerns: ingestion, transformation, modeling, orchestration, storage, visualization