How george-souza/carbon-intensity-pipeline works
This is an early-stage UK carbon-intensity ETL pipeline. Its principal workflow is a CLI-selected backfill or incremental run: the runner creates an ingestion record, requests data from the public API, stores raw responses, and finalizes run status. A separate migration command prepares the database schema. The README describes validation, transformation/loading into the half-hourly table, and a future REST API, but those stages are not implemented in the shown runtime. Edges reflect calls shown in excerpts or the README; unsampled schema behavior is not inferred.
Main entry points: __main__.py Extraction CLI · db.py Database operations · migrate.py Migration runner · client.py API client
Components 15
Extraction
- Extraction CLI
src/extract/__main__.py - Run orchestration
src/extract/runner.py - API client
src/extract/client.py - Validation
- Transform and load
Persistence
- Database operations
src/db.py - PostgreSQL
- SQL migrations
sql - Half-hourly data
Operations
- Configuration
src/config.py - Logging setup
src/logging_setup.py - Migration runner
src/migrate.py - Future REST API
Other components
- Operator
- Carbon Intensity API
Connections 20
- Operator → Extraction CLI: runs
README.md - Extraction CLI → Configuration: loads settings
src/extract/__main__.py - Extraction CLI → Logging setup: configures
src/extract/__main__.py - Extraction CLI → Run orchestration: dispatches run
src/extract/__main__.py - Extraction CLI → API client: creates client
src/extract/__main__.py - Extraction CLI → Database operations: creates engine
src/extract/__main__.py - API client → Configuration: reads settings
src/extract/client.py - API client → Carbon Intensity API: requests data
src/extract/client.py - Run orchestration → API client: fetches data
src/extract/runner.py - Run orchestration → Database operations: records run and responses
src/extract/runner.py - Database operations → Configuration: gets database URL
src/db.py - Database operations → PostgreSQL: reads and writes
src/db.py - Operator → Migration runner: runs
README.md - Migration runner → Database operations: gets engine
src/migrate.py - Migration runner → SQL migrations: applies
src/migrate.py - Migration runner → PostgreSQL: tracks schema
src/migrate.py - Carbon Intensity API → Validation: planned input
README.md - Validation → Transform and load: planned stage
README.md - Transform and load → Half-hourly data: planned load
README.md - Half-hourly data → Future REST API: future serving
README.md