How chen729689ougk/weather-air-quality-etl works
This is a scheduled weather and air-quality forecast ETL for Fes: the Python entry point runs extraction, transformation, loading, and validation, with PostgreSQL as the durable store and SQL analytics as a downstream capability. The sampled code confirms stage orchestration and file/database interactions; analytics and database deployment details are represented from README documentation, since their implementation files were not excerpted. The README mentions duplicate checks, but the sampled validator does not show one.
Main entry points: main.py Pipeline runner · extract.py Forecast extraction · load.py Database loader · transform.py Transform forecasts
Components 13
Orchestration
- Pipeline runner
src/main.py
Forecast ingestion
- Open-Meteo weather
- Open-Meteo air quality
- Forecast extraction
src/extract.py - Raw JSON files
src/extract.py
Data preparation
- Transform forecasts
src/transform.py - Processed CSV files
src/transform.py
Storage
- Database loader
src/load.py - Database schema
sql/schema.sql - PostgreSQL
docker-compose.yml
Quality and analytics
- Data validation
src/validate.py - SQL analytics
sql/analytics.sql
Other components
- Operator / cron
Connections 15
- Operator / cron → Pipeline runner: starts
README.md - Pipeline runner → Forecast extraction: runs
src/main.py - Pipeline runner → Transform forecasts: runs
src/main.py - Pipeline runner → Database loader: runs
src/main.py - Pipeline runner → Data validation: runs
src/main.py - Forecast extraction → Open-Meteo weather: fetches forecasts
src/extract.py - Forecast extraction → Open-Meteo air quality: fetches forecasts
src/extract.py - Forecast extraction → Raw JSON files: writes
src/extract.py - Transform forecasts → Raw JSON files: reads latest
src/transform.py - Transform forecasts → Processed CSV files: writes
src/transform.py - Database loader → Processed CSV files: reads
src/load.py - Database loader → Database schema: executes
src/load.py - Database loader → PostgreSQL: upserts observations
src/load.py - Data validation → PostgreSQL: checks tables
src/validate.py - SQL analytics → PostgreSQL: queries summaries
README.md