How sraaaas/orchestration_airflow works
This repository is an hourly weather-data ELT pipeline. Airflow schedules extraction from Open-Meteo, loading into PostgreSQL, dbt transformations, dbt tests, and a final data check. The README also documents Metabase visualization. The main scripts and DAG are excerpted; the Metabase connection and some deployment wiring are not shown, so that integration is represented as unverified.
Main entry points: etl_dags.py Weather DAG · extract.py Weather extraction · run_dbt.py dbt runner · load.py Weather loader
Components 13
Orchestration
- Weather DAG
dags/etl_dags.py
Ingestion
- Weather extraction
scripts/extract.py - Weather CSV
data/raw_weather_data.csv - Weather loader
scripts/load.py
Analytics
- PostgreSQL weather DB
- dbt runner
scripts/run_dbt.py - Weather staging
dbt_weather/models/staging/stg_weather.sql - Daily temperature mart
dbt_weather/models/marts/daily_temperature.sql - dbt tests
scripts/test_dbt.py - Data verification
scripts/check_data.py
Other components
- Airflow scheduler
- Open-Meteo API
- Metabase
Connections 20
- Airflow scheduler → Weather DAG: schedules
dags/etl_dags.py - Weather DAG → Weather extraction: runs first
dags/etl_dags.py - Weather DAG → Weather loader: runs next
dags/etl_dags.py - Weather DAG → dbt runner: runs next
dags/etl_dags.py - Weather DAG → dbt tests: runs next
dags/etl_dags.py - Weather DAG → Data verification: runs last
dags/etl_dags.py - Weather extraction → PostgreSQL weather DB: reads latest date
scripts/extract.py - Weather extraction → Open-Meteo API: requests observations
scripts/extract.py - Weather extraction → Weather CSV: writes CSV
scripts/extract.py - Weather CSV → Weather loader: supplies rows
README.md - Weather loader → PostgreSQL weather DB: upserts rows
scripts/load.py - dbt runner → Weather staging: runs models
README.md - dbt runner → Daily temperature mart: runs models
README.md - Weather staging → Daily temperature mart: prepares data
README.md - PostgreSQL weather DB → Weather staging: stores model data
README.md - PostgreSQL weather DB → Daily temperature mart: stores model data
README.md - Weather DAG → dbt tests: tests after run
dags/etl_dags.py - dbt tests → PostgreSQL weather DB: tests models
- Data verification → PostgreSQL weather DB: queries tables
scripts/check_data.py - PostgreSQL weather DB → Metabase: serves analytics