# chen729689ougk/weather-air-quality-etl architecture

> Architecture diagram of the GitHub repository chen729689ougk/weather-air-quality-etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chen729689ougk/weather-air-quality-etl
- Repository: https://github.com/chen729689ougk/weather-air-quality-etl

## Overview

This is a scheduled weather and air-quality forecast ETL for Fes: the Python entry point runs extraction, transformation, loading, and validation, with PostgreSQL as the durable store and SQL analytics as a downstream capability. The sampled code confirms stage orchestration and file/database interactions; analytics and database deployment details are represented from README documentation, since their implementation files were not excerpted. The README mentions duplicate checks, but the sampled validator does not show one.

Main entry points: [`src/main.py`](https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/main.py) (Pipeline runner), [`src/extract.py`](https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/extract.py) (Forecast extraction), [`src/load.py`](https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/load.py) (Database loader), [`src/transform.py`](https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/transform.py) (Transform forecasts)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_runner["Pipeline runner<br/>[main.py]"]
end

subgraph group_ingestion["Forecast ingestion"]
  node_weather_api{{"Open-Meteo weather"}}
  node_air_api{{"Open-Meteo air quality"}}
  node_extract["Forecast extraction<br/>[extract.py]"]
  node_raw_json["Raw JSON files<br/>[extract.py]"]
end

subgraph group_preparation["Data preparation"]
  node_transform["Transform forecasts<br/>[transform.py]"]
  node_processed_csv["Processed CSV files<br/>[transform.py]"]
end

subgraph group_storage["Storage"]
  node_loader["Database loader<br/>[load.py]"]
  node_schema["Database schema<br/>[schema.sql]"]
  node_postgres[("PostgreSQL<br/>[docker-compose.yml]")]
end

subgraph group_quality_analytics["Quality and analytics"]
  node_validator["Data validation<br/>[validate.py]"]
  node_analytics["SQL analytics<br/>[analytics.sql]"]
end

node_operator(("Operator / cron"))

node_operator -->|"starts"| node_runner
node_runner -->|"runs"| node_extract
node_runner -->|"runs"| node_transform
node_runner -->|"runs"| node_loader
node_runner -->|"runs"| node_validator
node_extract -->|"fetches forecasts"| node_weather_api
node_extract -->|"fetches forecasts"| node_air_api
node_extract -->|"writes"| node_raw_json
node_transform -->|"reads latest"| node_raw_json
node_transform -->|"writes"| node_processed_csv
node_loader -->|"reads"| node_processed_csv
node_loader -->|"executes"| node_schema
node_loader -->|"upserts observations"| node_postgres
node_validator -->|"checks tables"| node_postgres
node_analytics -.->|"queries summaries"| node_postgres

click node_runner "https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/main.py"
click node_extract "https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/extract.py"
click node_raw_json "https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/extract.py"
click node_transform "https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/transform.py"
click node_processed_csv "https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/transform.py"
click node_loader "https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/load.py"
click node_schema "https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/sql/schema.sql"
click node_postgres "https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/docker-compose.yml"
click node_validator "https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/validate.py"
click node_analytics "https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/sql/analytics.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner toneBlue
class node_weather_api,node_air_api,node_extract,node_raw_json toneAmber
class node_transform,node_processed_csv toneMint
class node_loader,node_schema,node_postgres toneRose
class node_validator,node_analytics,node_operator toneIndigo
```

## Components

### Orchestration

- [Pipeline runner](https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/main.py): `src/main.py`

### Forecast ingestion

- **Open-Meteo weather**
- **Open-Meteo air quality**
- [Forecast extraction](https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/extract.py): `src/extract.py`
- [Raw JSON files](https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/extract.py): `src/extract.py`

### Data preparation

- [Transform forecasts](https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/transform.py): `src/transform.py`
- [Processed CSV files](https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/transform.py): `src/transform.py`

### Storage

- [Database loader](https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/load.py): `src/load.py`
- [Database schema](https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/sql/schema.sql): `sql/schema.sql`
- [PostgreSQL](https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/docker-compose.yml): `docker-compose.yml`

### Quality and analytics

- [Data validation](https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/src/validate.py): `src/validate.py`
- [SQL analytics](https://github.com/chen729689ougk/weather-air-quality-etl/blob/main/sql/analytics.sql): `sql/analytics.sql`

### Other components

- **Operator / cron**

## Connections

- Operator / cron → Pipeline runner: starts (evidence: `README.md`)
- Pipeline runner → Forecast extraction: runs (evidence: `src/main.py`)
- Pipeline runner → Transform forecasts: runs (evidence: `src/main.py`)
- Pipeline runner → Database loader: runs (evidence: `src/main.py`)
- Pipeline runner → Data validation: runs (evidence: `src/main.py`)
- Forecast extraction → Open-Meteo weather: fetches forecasts (evidence: `src/extract.py`)
- Forecast extraction → Open-Meteo air quality: fetches forecasts (evidence: `src/extract.py`)
- Forecast extraction → Raw JSON files: writes (evidence: `src/extract.py`)
- Transform forecasts → Raw JSON files: reads latest (evidence: `src/transform.py`)
- Transform forecasts → Processed CSV files: writes (evidence: `src/transform.py`)
- Database loader → Processed CSV files: reads (evidence: `src/load.py`)
- Database loader → Database schema: executes (evidence: `src/load.py`)
- Database loader → PostgreSQL: upserts observations (evidence: `src/load.py`)
- Data validation → PostgreSQL: checks tables (evidence: `src/validate.py`)
- SQL analytics → PostgreSQL: queries summaries (evidence: `README.md`)
