How fmiles-2006/ingdatos-clima-sidney-grupo06 works
This is a daily Sydney-weather data pipeline: Open-Meteo data is incrementally ingested into Delta bronze, transformed and quality-checked by dbt, published to PostgreSQL, and served to Metabase. Airflow orchestrates the runtime workflow. The graph also retains the documented Grafana observability capability. dbt model-level connections follow the README’s documented lineage; unsampled model internals are not inferred beyond that.
Main entry points: pipeline_grupo06.py Pipeline DAG · stg_clima_diario.sql Daily climate staging · publicar_gold.py Gold publisher · ingesta.py Weather ingestion
Components 16
Ingestion
- Weather ingestion
scripts/ingesta.py - Delta bronze
Transformation
- Daily climate staging
dbt/models/staging/stg_clima_diario.sql - Weather code staging
dbt/models/staging/stg_codigos_clima.sql - ENSO staging
dbt/models/staging/stg_fases_enso.sql - Monthly climate facts
dbt/models/marts/fct_clima_mensual.sql - Daily climate features
dbt/models/marts/feat_clima_diario.sql - dbt quality checks
dbt/tests - dbt project
dbt
Serving
- Gold publisher
scripts/publicar_gold.py - DuckDB warehouse
- PostgreSQL gold
- Metabase dashboard
- Grafana monitoring
docs/grafana/observabilidad_grupo06.json
Orchestration
- Pipeline DAG
dags/pipeline_grupo06.py
Other components
- Open-Meteo
Connections 14
- Pipeline DAG → Weather ingestion: runs
dags/pipeline_grupo06.py - Open-Meteo → Weather ingestion: provides weather
README.md - Weather ingestion → Delta bronze: writes Delta
README.md - Pipeline DAG → dbt project: runs dbt
dags/pipeline_grupo06.py - Delta bronze → Daily climate staging: feeds staging
README.md - Daily climate staging → Monthly climate facts: builds marts
README.md - Daily climate staging → Daily climate features: builds marts
README.md - Pipeline DAG → dbt quality checks: runs tests
dags/pipeline_grupo06.py - dbt project → DuckDB warehouse: builds models
README.md - Pipeline DAG → Gold publisher: runs publication
dags/pipeline_grupo06.py - Gold publisher → DuckDB warehouse: reads gold
scripts/publicar_gold.py - Gold publisher → PostgreSQL gold: publishes tables
scripts/publicar_gold.py - PostgreSQL gold → Metabase dashboard: serves dashboard
README.md - Grafana monitoring → Pipeline DAG: monitors runs