How akashdeep-bit-gif/30-day-data-engineering-marathon works
This repository implements a weather-data ETL pipeline. An operator starts the orchestrator, which runs five stages in order: fetch current weather into raw JSON, load it into staging, transform records into analytics data, aggregate daily summaries, and audit data quality. PostgreSQL is the shared store. The source excerpts show the stage sequence and each stage’s storage operations; there is no README describing additional capabilities.
Main entry points: run_pipeline.py Pipeline runner · extract.py Weather extractor · load_staging.py Staging loader · transform_data.py Weather transformer
Components 10
Pipeline orchestration
- Pipeline runner
code/day-06-orchestrator/run_pipeline.py
Weather ingestion
- Raw JSON files
- Staging loader
code/day-02-db-stage/load_staging.py - Weather extractor
code/day-01-api-ingest/extract.py
Analytics processing
- Weather transformer
code/day-03-transform/transform_data.py - Daily aggregator
code/day-04-aggregate/aggregate_daily.py
Data quality
- Quality validator
code/day-05-quality/validate_data.py
Other components
- Operator
- Open-Meteo API
- Weather database
Connections 13
- Operator → Pipeline runner: starts
code/day-06-orchestrator/run_pipeline.py - Pipeline runner → Weather extractor: runs first
code/day-06-orchestrator/run_pipeline.py - Pipeline runner → Staging loader: runs second
code/day-06-orchestrator/run_pipeline.py - Pipeline runner → Weather transformer: runs third
code/day-06-orchestrator/run_pipeline.py - Pipeline runner → Daily aggregator: runs fourth
code/day-06-orchestrator/run_pipeline.py - Pipeline runner → Quality validator: runs fifth
code/day-06-orchestrator/run_pipeline.py - Weather extractor → Open-Meteo API: fetches weather
code/day-01-api-ingest/extract.py - Weather extractor → Raw JSON files: writes JSON
code/day-01-api-ingest/extract.py - Raw JSON files → Staging loader: reads JSON
code/day-02-db-stage/load_staging.py - Staging loader → Weather database: writes staging
code/day-02-db-stage/load_staging.py - Weather transformer → Weather database: reads staging; writes analytics
code/day-03-transform/transform_data.py - Daily aggregator → Weather database: reads analytics; writes summaries
code/day-04-aggregate/aggregate_daily.py - Quality validator → Weather database: checks staging and analytics
code/day-05-quality/validate_data.py