How kharchuai/bigdata_electrivcity_forecast works
This repository is an early-stage electricity-forecasting pipeline: the implemented runtime covers configuration, data contracts, PostgreSQL/Iceberg initialization and infrastructure checks, and run/job tracking. The README describes ingestion, processing, feature engineering, ML, streaming and demo/benchmark work as future scaffolding, so those are not represented as working pipeline stages. Edges are limited to calls shown in excerpts or initialization relationships documented in the README; unsampled analytics wiring is omitted.
Main entry points: infrastructure.py Infrastructure checks · cli.py CLI commands · metrics.py Run/job tracking · config.py Configuration
Components 13
Runtime foundation
- CLI commands
src/electricity_forecast/cli.py - Configuration
src/electricity_forecast/config.py - Reading contracts
src/electricity_forecast/contracts.py - Infrastructure checks
src/electricity_forecast/infrastructure.py - Storage primitives
src/electricity_forecast/storage.py
Run operations
- Run/job tracking
src/electricity_forecast/metrics.py - Application schema
sql/001_initial.sql
Analytics models
- Forecast error model
dbt/models/forecast_errors.sql - Daily metrics model
dbt/models/forecast_metrics_daily.sql
Other components
- Engineer
- PostgreSQL
- Iceberg on HDFS
- Kafka
Connections 15
- Engineer → CLI commands: invokes
src/electricity_forecast/cli.py - CLI commands → Configuration: loads settings
src/electricity_forecast/cli.py - CLI commands → Infrastructure checks: runs commands
src/electricity_forecast/cli.py - CLI commands → Run/job tracking: sanitizes errors
src/electricity_forecast/cli.py - Infrastructure checks → Configuration: validates settings
src/electricity_forecast/infrastructure.py - Infrastructure checks → Storage primitives: initializes tables
src/electricity_forecast/infrastructure.py - Storage primitives → Configuration: reads settings
src/electricity_forecast/storage.py - Storage primitives → Reading contracts: uses keys
src/electricity_forecast/storage.py - Infrastructure checks → PostgreSQL: initializes schema
README.md - Infrastructure checks → Iceberg on HDFS: initializes tables
README.md - Infrastructure checks → Kafka: waits for bootstrap
README.md - Run/job tracking → Configuration: reads settings
src/electricity_forecast/metrics.py - Run/job tracking → Reading contracts: validates timestamps
src/electricity_forecast/metrics.py - Run/job tracking → PostgreSQL: records run state
src/electricity_forecast/metrics.py - Infrastructure checks → Application schema: executes SQL