How samithcsachi/grid-forecast works
1. GridForecast is a dual-path forecasting system: an offline ML lifecycle for training/retraining electricity-demand models, and an online serving layer that exposes predictions and health/metrics to consumers. The domain data source is ENTSO-E operational electricity data, with the end goal of 24-hour demand forecasts for grid operations and trading.
2. The public runtime surfaces are a FastAPI service and a Streamlit dashboard. The FastAPI app under api/ provides health, metrics, and predict routes, plus middleware for logging and rate limiting. The Streamlit app under app/ is the human-facing visualization layer for forecasts, confidence intervals, and model performance.
3. Online inference flows through services/prediction_service.py and pipelines/inference_pipeline.py. Requests are converted into features, scored by the active model, and returned via the API. A Redis cache exists in cache/redis_client.py, likely used to reduce repeated lookups or speed up serving-path operations.
4. The training path is organized as a staged pipeline in components/ and pipelines/training_pipeline.py. The stages are explicit: data ingestion, validation, transformation, model training, evaluation, drift monitoring, and model registry. This makes the offline lifecycle easy to diagram as a left-to-right sequence from raw data to promoted model artifacts.
5. Data access is separated from modeling. data_access/entsoe_client.py pulls external electricity data, while database_client.py and storage_client.py abstract persistence for processed datasets, artifacts, or metrics. This boundary keeps external I/O out of the training and serving logic.
6. Feature preparation is handled by feature_store/feature_engineering.py and feature_store/feature_store.py, with registry.json acting as a local feature definition registry. This layer sits between raw time-series data and model inputs, and is a key boundary for reproducibility and consistency between training and inference.
7. Model governance is handled by registry/model_registry.py and registry/promotion.py, with components/stage_07_model_registry.py tying registry updates into the pipeline. The repository is designed around explicit model versioning and promotion so that retraining can produce a candidate model before it replaces the currently served one.
8. Monitoring is a first-class subsystem. monitoring/ covers drift detection, performance monitoring, and alerting, while services/monitoring_service.py and pipelines/monitoring_pipeline.py wire these checks into the operational flow. The API also exposes a metrics endpoint, indicating the monitoring stack is meant to be consumed by external observability tools.
9. Observability is backed by separate logging, metrics, and tracing modules under observability/, plus Prometheus and Grafana configs in observability/prometheus/ and observability/grafana/. At the HTTP layer there is also request logging middleware, so the serving path produces operational telemetry in addition to predictions.
10. Retraining is treated as an operational workflow, not a manual notebook task. pipelines/retraining_pipeline.py, services/retraining_service.py, and workers/scheduler.py support periodic or triggered retraining, likely coordinated with the GitHub Actions retrain.yaml workflow. This closes the loop from monitoring signals back to model updates.
11. Deployment and runtime infrastructure are present in infra/, docker-compose.yaml, and .github/workflows/. There is a Dockerfile, nginx config, and AWS deployment documentation, which suggests containerized deployment behind a reverse proxy with CI/CD for build, deploy, and retrain automation. load_testing/locustfile.py indicates the serving API is also performance-tested.
12. Configuration and validation are centralized. configs/*.yaml and the config/ package define system parameters and schemas, while validation/ contains data-quality and schema-validation logic. This creates a clear boundary between environment/config inputs and the core ML pipeline.
13. tests/ covers API, components, and pipelines, so the repository is structured to validate each architectural layer independently. The research notebooks under research/ are exploratory artifacts and are separate from the production execution path.
14. In diagram form, the main boundaries are: external data ingestion, feature engineering, model lifecycle/promotion, online serving, monitoring/observability, and deployment automation. The main control points are FastAPI for inference, Streamlit for visualization, the staged pipelines for offline work, and the registry/monitoring loop that keeps the system continuously updated.
Main entry points: stage_01_data_ingestion.py Ingest · model_registry.py Registry · prediction_service.py Prediction svc · inference_pipeline.py Infer pipeline
Components 26
Runtime
Online prediction and human-facing interfaces
- Prediction svc · Converts requests to features and model outputs
SRC/GridForecast_Electricity_Demand_Predictor/services/prediction_service.py - Infer pipeline · Orchestrates online feature prep and scoring
SRC/GridForecast_Electricity_Demand_Predictor/pipelines/inference_pipeline.py - Redis cache · Speeds repeated serving-path lookups
SRC/GridForecast_Electricity_Demand_Predictor/cache/redis_client.py - FastAPI · Serving API with health, metrics, and predict routes
SRC/GridForecast_Electricity_Demand_Predictor/api/fastapi_app.py - Streamlit · Human-facing visualization and analysis app
SRC/GridForecast_Electricity_Demand_Predictor/app/streamlit_app.py
Offline ML
Training, retraining, and batch lifecycle
- Train pipeline · Coordinates staged model training lifecycle
SRC/GridForecast_Electricity_Demand_Predictor/pipelines/training_pipeline.py - Ingest · Pulls raw electricity data into the system
SRC/GridForecast_Electricity_Demand_Predictor/components/stage_01_data_ingestion.py - Transform · Cleans and prepares training features
SRC/GridForecast_Electricity_Demand_Predictor/components/stage_03_data_transformation.py - Train · Fits the forecasting model
SRC/GridForecast_Electricity_Demand_Predictor/components/stage_04_model_training.py - Evaluate · Scores candidate models and metrics
SRC/GridForecast_Electricity_Demand_Predictor/components/stage_05_model_evaluation.py - Retrain pipe · Operational retraining workflow
SRC/GridForecast_Electricity_Demand_Predictor/pipelines/retraining_pipeline.py - Retraining svc · Triggers and manages retraining runs
SRC/GridForecast_Electricity_Demand_Predictor/services/retraining_service.py
Data & Features
External ingestion, persistence, and feature preparation
- ENTSO-E · Domain electricity data source for demand forecasting
SRC/GridForecast_Electricity_Demand_Predictor/data_access/entsoe_client.py - DB client · Abstraction for database-backed storage
SRC/GridForecast_Electricity_Demand_Predictor/data_access/database_client.py - Storage client · Abstraction for datasets, models, and metrics storage
SRC/GridForecast_Electricity_Demand_Predictor/data_access/storage_client.py - Feature eng · Transforms raw time-series data into model inputs
SRC/GridForecast_Electricity_Demand_Predictor/feature_store/feature_engineering.py - Feature store · Local feature definitions and reproducible feature access
SRC/GridForecast_Electricity_Demand_Predictor/feature_store/feature_store.py - Feature schema · Static feature definition registry
SRC/GridForecast_Electricity_Demand_Predictor/feature_store/registry.json
Governance
Registry, validation, drift, and performance controls
- Data validation · Schema and data-quality validation layer
SRC/GridForecast_Electricity_Demand_Predictor/validation/data_quality.py - Schema checks · Validates expected structure and fields
SRC/GridForecast_Electricity_Demand_Predictor/validation/schema_validation.py - Registry · Stores model versions and active artifacts
SRC/GridForecast_Electricity_Demand_Predictor/registry/model_registry.py - Promotion · Promotes evaluated candidates to serving
SRC/GridForecast_Electricity_Demand_Predictor/registry/promotion.py - Monitor pipe · Runs drift and performance checks
SRC/GridForecast_Electricity_Demand_Predictor/pipelines/monitoring_pipeline.py - Monitoring svc · Coordinates drift, performance, and alerting
SRC/GridForecast_Electricity_Demand_Predictor/services/monitoring_service.py
Connections 27
- ENTSO-E → Ingest: source data · Raw electricity data enters the ingestion stage
- Ingest → Data validation: checks · Ingested data is validated before downstream use
- Data validation → Schema checks: schema · Validation enforces expected structure
- Ingest → Feature eng: prep input · Raw data is prepared for feature construction
- Feature eng → Feature store: publish · Engineered features are stored for reuse
- Feature store → Feature schema: defs · Feature definitions are tracked in the local registry
- DB client → Storage client: persistence · Persistence layer separates database and artifact storage
- Train pipeline → Ingest: orchestrates · Training pipeline starts with ingestion
- Train pipeline → Transform: orchestrates · Training pipeline prepares features
- Transform → Train: feeds · Transformed features feed model training
- Train → Evaluate: scores · Trained candidate is evaluated
- Evaluate → Registry: registers · Accepted artifacts are versioned in the registry
- Promotion → Registry: promotes · Promotion updates the active served model
- Monitor pipe → Monitoring svc: runs · Monitoring pipeline delegates operational checks
- Monitoring svc → Telemetry: emits · Monitoring outputs telemetry and alerts
- FastAPI → Prediction svc: serves · API routes request prediction logic
- Prediction svc → Infer pipeline: executes · Prediction service uses the inference pipeline
- Infer pipeline → Feature store: reads features · Online inference reuses the feature layer
- Infer pipeline → Registry: loads model · Scoring uses the active registered model
- Prediction svc → Redis cache: cache · Serving path can use Redis for repeated lookups
- FastAPI → Telemetry: telemetry · HTTP layer contributes logs, metrics, and traces
- Streamlit → Registry: visualizes · Dashboard shows forecast and performance views
- Retraining svc → Retrain pipe: coordinates · Retraining service orchestrates workflow execution
- Retrain pipe → Monitoring svc: triggered by · Monitoring signals can initiate retraining
- Scheduler → Retraining svc: triggers · Scheduled workers initiate retraining runs
- Scheduler → Monitor pipe: runs · Workers can execute monitoring jobs
- Telemetry → FastAPI: observes · Telemetry stack covers API runtime behavior