How lanafren/weather-data-platform works
This repository implements a scheduled weather-data pipeline: Airflow fetches current conditions and forecasts, lands raw records in GCS and BigQuery, and dbt builds normalized, unified, and analytics-ready tables. A separate Airflow DAG checks table counts and freshness and records monitoring metadata. The README also describes schema and custom data-quality tests. dbt model SQL was not sampled, so model-level dependencies are shown only at the documented layer boundaries; Looker remains planned.
Main entry points: int_weather_unified.sql Unified weather · owm_batch_bq.py Batch ingestion DAG · stg_current.sql Current staging · dbt_monitor.py Quality monitor DAG
Components 14
Weather ingestion
- Batch ingestion DAG
airflow/dags/owm_batch_bq.py
Raw warehouse
- Raw GCS
- Raw BigQuery
dbt transformations
- Current staging
dbt/models/staging/stg_current.sql - Forecast staging
dbt/models/staging/stg_forecast.sql - Unified weather
dbt/models/intermediate/int_weather_unified.sql - Weather fact
dbt/models/marts/fact_weather.sql - Datetime dimension
dbt/models/marts/dim_datetime.sql - Condition dimension
dbt/models/marts/dim_weather_condition.sql - Schema tests
dbt/models/marts/schema.yml - Custom validations
dbt/tests/check_duplicate_records.sql
Quality monitoring
- Quality monitor DAG
airflow/dags/dbt_monitor.py - Quality metadata
Other components
- OpenWeatherMap
Connections 14
- Batch ingestion DAG → OpenWeatherMap: fetches weather
airflow/dags/owm_batch_bq.py - Batch ingestion DAG → Raw GCS: writes raw files
README.md - Batch ingestion DAG → Raw BigQuery: loads raw records
airflow/dags/owm_batch_bq.py - Raw BigQuery → Current staging: stages current
README.md - Raw BigQuery → Forecast staging: stages forecast
README.md - Current staging → Unified weather: unifies data
README.md - Forecast staging → Unified weather: unifies data
README.md - Unified weather → Weather fact: models facts
README.md - Unified weather → Datetime dimension: models dimension
README.md - Unified weather → Condition dimension: models dimension
README.md - Quality monitor DAG → Raw BigQuery: checks tables
airflow/dags/dbt_monitor.py - Quality monitor DAG → Quality metadata: records metrics
airflow/dags/dbt_monitor.py - Schema tests → Current staging: tests models
README.md - Custom validations → Weather fact: validates data
README.md