# lanafren/weather-data-platform architecture

> Architecture diagram of the GitHub repository lanafren/weather-data-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lanafren/weather-data-platform
- Repository: https://github.com/lanafren/weather-data-platform

## Overview

This repository implements a scheduled weather-data pipeline: Airflow fetches current conditions and forecasts, lands raw records in GCS and BigQuery, and dbt builds normalized, unified, and analytics-ready tables. A separate Airflow DAG checks table counts and freshness and records monitoring metadata. The README also describes schema and custom data-quality tests. dbt model SQL was not sampled, so model-level dependencies are shown only at the documented layer boundaries; Looker remains planned.

Main entry points: [`dbt/models/intermediate/int_weather_unified.sql`](https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/intermediate/int_weather_unified.sql) (Unified weather), [`airflow/dags/owm_batch_bq.py`](https://github.com/lanafren/weather-data-platform/blob/main/airflow/dags/owm_batch_bq.py) (Batch ingestion DAG), [`dbt/models/staging/stg_current.sql`](https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/staging/stg_current.sql) (Current staging), [`airflow/dags/dbt_monitor.py`](https://github.com/lanafren/weather-data-platform/blob/main/airflow/dags/dbt_monitor.py) (Quality monitor DAG)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Weather ingestion"]
  node_ingestion["Batch ingestion DAG<br/>[owm_batch_bq.py]"]
end

subgraph group_warehouse["Raw warehouse"]
  node_gcs[("Raw GCS")]
  node_raw_bq[("Raw BigQuery")]
end

subgraph group_transform["dbt transformations"]
  node_stg_current["Current staging<br/>[stg_current.sql]"]
  node_stg_forecast["Forecast staging<br/>[stg_forecast.sql]"]
  node_int_weather["Unified weather"]
  node_fact_weather["Weather fact<br/>[fact_weather.sql]"]
  node_dim_datetime["Datetime dimension<br/>[dim_datetime.sql]"]
  node_dim_condition["Condition dimension"]
  node_schema_tests["Schema tests<br/>[schema.yml]"]
  node_custom_tests["Custom validations"]
end

subgraph group_monitor["Quality monitoring"]
  node_monitor_dag["Quality monitor DAG<br/>[dbt_monitor.py]"]
  node_metadata[("Quality metadata")]
end

node_owm(("OpenWeatherMap"))

node_ingestion -->|"fetches weather"| node_owm
node_ingestion -->|"writes raw files"| node_gcs
node_ingestion -->|"loads raw records"| node_raw_bq
node_raw_bq -->|"stages current"| node_stg_current
node_raw_bq -->|"stages forecast"| node_stg_forecast
node_stg_current -->|"unifies data"| node_int_weather
node_stg_forecast -->|"unifies data"| node_int_weather
node_int_weather -->|"models facts"| node_fact_weather
node_int_weather -->|"models dimension"| node_dim_datetime
node_int_weather -->|"models dimension"| node_dim_condition
node_monitor_dag -->|"checks tables"| node_raw_bq
node_monitor_dag -->|"records metrics"| node_metadata
node_schema_tests -->|"tests models"| node_stg_current
node_custom_tests -->|"validates data"| node_fact_weather

click node_ingestion "https://github.com/lanafren/weather-data-platform/blob/main/airflow/dags/owm_batch_bq.py"
click node_stg_current "https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/staging/stg_current.sql"
click node_stg_forecast "https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/staging/stg_forecast.sql"
click node_int_weather "https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/intermediate/int_weather_unified.sql"
click node_fact_weather "https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/marts/fact_weather.sql"
click node_dim_datetime "https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/marts/dim_datetime.sql"
click node_dim_condition "https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/marts/dim_weather_condition.sql"
click node_schema_tests "https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/marts/schema.yml"
click node_custom_tests "https://github.com/lanafren/weather-data-platform/blob/main/dbt/tests/check_duplicate_records.sql"
click node_monitor_dag "https://github.com/lanafren/weather-data-platform/blob/main/airflow/dags/dbt_monitor.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingestion toneBlue
class node_gcs,node_raw_bq toneAmber
class node_stg_current,node_stg_forecast,node_int_weather,node_fact_weather,node_dim_datetime,node_dim_condition,node_schema_tests,node_custom_tests toneMint
class node_monitor_dag,node_metadata toneRose
class node_owm toneIndigo
```

## Components

### Weather ingestion

- [Batch ingestion DAG](https://github.com/lanafren/weather-data-platform/blob/main/airflow/dags/owm_batch_bq.py): `airflow/dags/owm_batch_bq.py`

### Raw warehouse

- **Raw GCS**
- **Raw BigQuery**

### dbt transformations

- [Current staging](https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/staging/stg_current.sql): `dbt/models/staging/stg_current.sql`
- [Forecast staging](https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/staging/stg_forecast.sql): `dbt/models/staging/stg_forecast.sql`
- [Unified weather](https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/intermediate/int_weather_unified.sql): `dbt/models/intermediate/int_weather_unified.sql`
- [Weather fact](https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/marts/fact_weather.sql): `dbt/models/marts/fact_weather.sql`
- [Datetime dimension](https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/marts/dim_datetime.sql): `dbt/models/marts/dim_datetime.sql`
- [Condition dimension](https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/marts/dim_weather_condition.sql): `dbt/models/marts/dim_weather_condition.sql`
- [Schema tests](https://github.com/lanafren/weather-data-platform/blob/main/dbt/models/marts/schema.yml): `dbt/models/marts/schema.yml`
- [Custom validations](https://github.com/lanafren/weather-data-platform/blob/main/dbt/tests/check_duplicate_records.sql): `dbt/tests/check_duplicate_records.sql`

### Quality monitoring

- [Quality monitor DAG](https://github.com/lanafren/weather-data-platform/blob/main/airflow/dags/dbt_monitor.py): `airflow/dags/dbt_monitor.py`
- **Quality metadata**

### Other components

- **OpenWeatherMap**

## Connections

- Batch ingestion DAG → OpenWeatherMap: fetches weather (evidence: `airflow/dags/owm_batch_bq.py`)
- Batch ingestion DAG → Raw GCS: writes raw files (evidence: `README.md`)
- Batch ingestion DAG → Raw BigQuery: loads raw records (evidence: `airflow/dags/owm_batch_bq.py`)
- Raw BigQuery → Current staging: stages current (evidence: `README.md`)
- Raw BigQuery → Forecast staging: stages forecast (evidence: `README.md`)
- Current staging → Unified weather: unifies data (evidence: `README.md`)
- Forecast staging → Unified weather: unifies data (evidence: `README.md`)
- Unified weather → Weather fact: models facts (evidence: `README.md`)
- Unified weather → Datetime dimension: models dimension (evidence: `README.md`)
- Unified weather → Condition dimension: models dimension (evidence: `README.md`)
- Quality monitor DAG → Raw BigQuery: checks tables (evidence: `airflow/dags/dbt_monitor.py`)
- Quality monitor DAG → Quality metadata: records metrics (evidence: `airflow/dags/dbt_monitor.py`)
- Schema tests → Current staging: tests models (evidence: `README.md`)
- Custom validations → Weather fact: validates data (evidence: `README.md`)
