How zniii/bess-arbitrage-pipeline works
This is a GB battery-arbitrage forecasting and dispatch product, with both an incremental data pipeline and a walk-forward backtest feeding a precomputed-results Streamlit app. The principal user workflow is to inspect a day’s forecast and battery schedule, then review expected earnings and historical performance. The graph preserves the documented ingestion, UTC/Silver, forecasting, MILP, settlement/KPI, and app stages. Some subsystem connections are documented in the README rather than shown in sampled call sites; those edges cite the README.
Main entry points: pipeline_runner.py Batch pipeline · daily_update.py Daily updater · dispatch_optimizer.py MILP dispatcher · elexon_client.py Elexon client
Components 22
Market ingestion
- Elexon client
src/bess_pipeline/ingestion/elexon_client.py - NESO client
src/bess_pipeline/ingestion/neso_client.py
Data lake
- UTC normalization
src/bess_pipeline/transformations/temporal.py - Silver Parquet
src/bess_pipeline/transformations/storage.py - S3 lake manager
src/bess_pipeline/common/database.py
Forecast and dispatch
- Exogenous features
src/bess_pipeline/forecasting/exogenous.py - Forecast features
src/bess_pipeline/forecasting/features.py - Price forecasters
src/bess_pipeline/forecasting/model.py - MILP dispatcher
src/bess_pipeline/optimization/dispatch_optimizer.py - Rolling backtest
src/bess_pipeline/forecasting/rolling.py - Dispatch strategies
src/bess_pipeline/forecasting/strategies.py
Settlement and orchestration
- Walk-forward backtest
src/bess_pipeline/forecasting/backtest.py - Financial KPIs
src/bess_pipeline/analytics/financial_kpis.py - Batch pipeline
src/bess_pipeline/orchestration/pipeline_runner.py - Daily updater
src/bess_pipeline/orchestration/daily_update.py - Backtest report
scripts/run_backtest_report.py
Results app
- Precomputed results
app/data - Max Revenue
app/Max_Revenue.py - Findings
app/pages/Findings.py
Other components
- Engineer / analyst
- Elexon Insights
- NESO data
Connections 27
- Elexon client → Elexon Insights: fetches market data
README.md - NESO client → NESO data: fetches auction prices
README.md - Batch pipeline → Elexon client: requests prices
src/bess_pipeline/orchestration/pipeline_runner.py - Batch pipeline → UTC normalization: normalizes intervals
src/bess_pipeline/orchestration/pipeline_runner.py - Batch pipeline → Silver Parquet: writes Silver data
src/bess_pipeline/orchestration/pipeline_runner.py - Batch pipeline → S3 lake manager: syncs partitions
src/bess_pipeline/orchestration/pipeline_runner.py - Daily updater → Elexon client: fetches inputs
src/bess_pipeline/orchestration/daily_update.py - Daily updater → NESO client: fetches auction data
src/bess_pipeline/orchestration/daily_update.py - Daily updater → S3 lake manager: checks lake state
src/bess_pipeline/orchestration/daily_update.py - Daily updater → Price forecasters: fits and predicts
src/bess_pipeline/orchestration/daily_update.py - Daily updater → Exogenous features: loads forecasts
src/bess_pipeline/orchestration/daily_update.py - Daily updater → MILP dispatcher: optimizes schedule
src/bess_pipeline/orchestration/daily_update.py - Price forecasters → Forecast features: builds features
src/bess_pipeline/forecasting/model.py - Batch pipeline → MILP dispatcher: optimizes schedule
src/bess_pipeline/orchestration/pipeline_runner.py - Batch pipeline → Financial KPIs: calculates KPIs
src/bess_pipeline/orchestration/pipeline_runner.py - Walk-forward backtest → Price forecasters: fits and predicts
src/bess_pipeline/forecasting/backtest.py - Walk-forward backtest → MILP dispatcher: settles dispatch
src/bess_pipeline/forecasting/backtest.py - Walk-forward backtest → Financial KPIs: measures earnings
src/bess_pipeline/forecasting/backtest.py - Rolling backtest → MILP dispatcher: optimizes schedule
src/bess_pipeline/forecasting/rolling.py - Dispatch strategies → MILP dispatcher: dispatches strategy
src/bess_pipeline/forecasting/strategies.py - Backtest report → NESO client: loads auction prices
scripts/run_backtest_report.py - Backtest report → Elexon client: loads wind and demand
scripts/run_backtest_report.py - Backtest report → Precomputed results: writes app results
README.md - Engineer / analyst → Max Revenue: views revenue plan
README.md - Engineer / analyst → Findings: views backtest results
README.md - Max Revenue → Precomputed results: reads results
README.md - Findings → Precomputed results: reads results
README.md