# zniii/bess-arbitrage-pipeline architecture

> Architecture diagram of the GitHub repository zniii/bess-arbitrage-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zniii/bess-arbitrage-pipeline
- Repository: https://github.com/zniii/bess-arbitrage-pipeline

## Overview

This is a GB battery-arbitrage forecasting and dispatch product, with both an incremental data pipeline and a walk-forward backtest feeding a precomputed-results Streamlit app. The principal user workflow is to inspect a day’s forecast and battery schedule, then review expected earnings and historical performance. The graph preserves the documented ingestion, UTC/Silver, forecasting, MILP, settlement/KPI, and app stages. Some subsystem connections are documented in the README rather than shown in sampled call sites; those edges cite the README.

Main entry points: [`src/bess_pipeline/orchestration/pipeline_runner.py`](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/orchestration/pipeline_runner.py) (Batch pipeline), [`src/bess_pipeline/orchestration/daily_update.py`](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/orchestration/daily_update.py) (Daily updater), [`src/bess_pipeline/optimization/dispatch_optimizer.py`](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/optimization/dispatch_optimizer.py) (MILP dispatcher), [`src/bess_pipeline/ingestion/elexon_client.py`](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/ingestion/elexon_client.py) (Elexon client)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Market ingestion"]
  node_elexon_client["Elexon client<br/>[elexon_client.py]"]
  node_neso_client["NESO client<br/>[neso_client.py]"]
end

subgraph group_lake["Data lake"]
  node_temporal["UTC normalization<br/>[temporal.py]"]
  node_silver[("Silver Parquet<br/>[storage.py]")]
  node_lake_manager[("S3 lake manager<br/>[database.py]")]
end

subgraph group_forecast["Forecast and dispatch"]
  node_exogenous["Exogenous features<br/>[exogenous.py]"]
  node_forecast_features["Forecast features<br/>[features.py]"]
  node_forecaster["Price forecasters<br/>[model.py]"]
  node_optimizer["MILP dispatcher"]
  node_rolling["Rolling backtest<br/>[rolling.py]"]
  node_strategies["Dispatch strategies<br/>[strategies.py]"]
end

subgraph group_results["Settlement and orchestration"]
  node_backtest["Walk-forward backtest<br/>[backtest.py]"]
  node_kpis["Financial KPIs<br/>[financial_kpis.py]"]
  node_pipeline["Batch pipeline<br/>[pipeline_runner.py]"]
  node_daily["Daily updater<br/>[daily_update.py]"]
  node_report["Backtest report"]
end

subgraph group_serving["Results app"]
  node_appdata[("Precomputed results")]
  node_revenue_app["Max Revenue<br/>[Max_Revenue.py]"]
  node_findings["Findings<br/>[Findings.py]"]
end

node_operator(("Engineer / analyst"))
node_elexon{{"Elexon Insights"}}
node_neso{{"NESO data"}}

node_elexon_client -->|"fetches market data"| node_elexon
node_neso_client -->|"fetches auction prices"| node_neso
node_pipeline -->|"requests prices"| node_elexon_client
node_pipeline -->|"normalizes intervals"| node_temporal
node_pipeline -->|"writes Silver data"| node_silver
node_pipeline -->|"syncs partitions"| node_lake_manager
node_daily -->|"fetches inputs"| node_elexon_client
node_daily -->|"fetches auction data"| node_neso_client
node_daily -->|"checks lake state"| node_lake_manager
node_daily -->|"fits and predicts"| node_forecaster
node_daily -->|"loads forecasts"| node_exogenous
node_daily -->|"optimizes schedule"| node_optimizer
node_forecaster -->|"builds features"| node_forecast_features
node_pipeline -->|"optimizes schedule"| node_optimizer
node_pipeline -->|"calculates KPIs"| node_kpis
node_backtest -->|"fits and predicts"| node_forecaster
node_backtest -->|"settles dispatch"| node_optimizer
node_backtest -->|"measures earnings"| node_kpis
node_rolling -->|"optimizes schedule"| node_optimizer
node_strategies -->|"dispatches strategy"| node_optimizer
node_report -->|"loads auction prices"| node_neso_client
node_report -->|"loads wind and demand"| node_elexon_client
node_report -->|"writes app results"| node_appdata
node_operator -->|"views revenue plan"| node_revenue_app
node_operator -->|"views backtest results"| node_findings
node_revenue_app -->|"reads results"| node_appdata
node_findings -->|"reads results"| node_appdata

click node_elexon_client "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/ingestion/elexon_client.py"
click node_neso_client "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/ingestion/neso_client.py"
click node_temporal "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/transformations/temporal.py"
click node_silver "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/transformations/storage.py"
click node_lake_manager "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/common/database.py"
click node_exogenous "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/forecasting/exogenous.py"
click node_forecast_features "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/forecasting/features.py"
click node_forecaster "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/forecasting/model.py"
click node_optimizer "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/optimization/dispatch_optimizer.py"
click node_rolling "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/forecasting/rolling.py"
click node_strategies "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/forecasting/strategies.py"
click node_backtest "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/forecasting/backtest.py"
click node_kpis "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/analytics/financial_kpis.py"
click node_pipeline "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/orchestration/pipeline_runner.py"
click node_daily "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/orchestration/daily_update.py"
click node_report "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/scripts/run_backtest_report.py"
click node_appdata "https://github.com/zniii/bess-arbitrage-pipeline/tree/main/app/data"
click node_revenue_app "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/app/Max_Revenue.py"
click node_findings "https://github.com/zniii/bess-arbitrage-pipeline/blob/main/app/pages/Findings.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_elexon_client,node_neso_client toneBlue
class node_temporal,node_silver,node_lake_manager toneAmber
class node_exogenous,node_forecast_features,node_forecaster,node_optimizer,node_rolling,node_strategies toneMint
class node_backtest,node_kpis,node_pipeline,node_daily,node_report toneRose
class node_appdata,node_revenue_app,node_findings,node_operator,node_elexon,node_neso toneIndigo
```

## Components

### Market ingestion

- [Elexon client](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/ingestion/elexon_client.py): `src/bess_pipeline/ingestion/elexon_client.py`
- [NESO client](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/ingestion/neso_client.py): `src/bess_pipeline/ingestion/neso_client.py`

### Data lake

- [UTC normalization](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/transformations/temporal.py): `src/bess_pipeline/transformations/temporal.py`
- [Silver Parquet](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/transformations/storage.py): `src/bess_pipeline/transformations/storage.py`
- [S3 lake manager](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/common/database.py): `src/bess_pipeline/common/database.py`

### Forecast and dispatch

- [Exogenous features](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/forecasting/exogenous.py): `src/bess_pipeline/forecasting/exogenous.py`
- [Forecast features](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/forecasting/features.py): `src/bess_pipeline/forecasting/features.py`
- [Price forecasters](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/forecasting/model.py): `src/bess_pipeline/forecasting/model.py`
- [MILP dispatcher](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/optimization/dispatch_optimizer.py): `src/bess_pipeline/optimization/dispatch_optimizer.py`
- [Rolling backtest](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/forecasting/rolling.py): `src/bess_pipeline/forecasting/rolling.py`
- [Dispatch strategies](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/forecasting/strategies.py): `src/bess_pipeline/forecasting/strategies.py`

### Settlement and orchestration

- [Walk-forward backtest](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/forecasting/backtest.py): `src/bess_pipeline/forecasting/backtest.py`
- [Financial KPIs](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/analytics/financial_kpis.py): `src/bess_pipeline/analytics/financial_kpis.py`
- [Batch pipeline](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/orchestration/pipeline_runner.py): `src/bess_pipeline/orchestration/pipeline_runner.py`
- [Daily updater](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/src/bess_pipeline/orchestration/daily_update.py): `src/bess_pipeline/orchestration/daily_update.py`
- [Backtest report](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/scripts/run_backtest_report.py): `scripts/run_backtest_report.py`

### Results app

- [Precomputed results](https://github.com/zniii/bess-arbitrage-pipeline/tree/main/app/data): `app/data`
- [Max Revenue](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/app/Max_Revenue.py): `app/Max_Revenue.py`
- [Findings](https://github.com/zniii/bess-arbitrage-pipeline/blob/main/app/pages/Findings.py): `app/pages/Findings.py`

### Other components

- **Engineer / analyst**
- **Elexon Insights**
- **NESO data**

## Connections

- Elexon client → Elexon Insights: fetches market data (evidence: `README.md`)
- NESO client → NESO data: fetches auction prices (evidence: `README.md`)
- Batch pipeline → Elexon client: requests prices (evidence: `src/bess_pipeline/orchestration/pipeline_runner.py`)
- Batch pipeline → UTC normalization: normalizes intervals (evidence: `src/bess_pipeline/orchestration/pipeline_runner.py`)
- Batch pipeline → Silver Parquet: writes Silver data (evidence: `src/bess_pipeline/orchestration/pipeline_runner.py`)
- Batch pipeline → S3 lake manager: syncs partitions (evidence: `src/bess_pipeline/orchestration/pipeline_runner.py`)
- Daily updater → Elexon client: fetches inputs (evidence: `src/bess_pipeline/orchestration/daily_update.py`)
- Daily updater → NESO client: fetches auction data (evidence: `src/bess_pipeline/orchestration/daily_update.py`)
- Daily updater → S3 lake manager: checks lake state (evidence: `src/bess_pipeline/orchestration/daily_update.py`)
- Daily updater → Price forecasters: fits and predicts (evidence: `src/bess_pipeline/orchestration/daily_update.py`)
- Daily updater → Exogenous features: loads forecasts (evidence: `src/bess_pipeline/orchestration/daily_update.py`)
- Daily updater → MILP dispatcher: optimizes schedule (evidence: `src/bess_pipeline/orchestration/daily_update.py`)
- Price forecasters → Forecast features: builds features (evidence: `src/bess_pipeline/forecasting/model.py`)
- Batch pipeline → MILP dispatcher: optimizes schedule (evidence: `src/bess_pipeline/orchestration/pipeline_runner.py`)
- Batch pipeline → Financial KPIs: calculates KPIs (evidence: `src/bess_pipeline/orchestration/pipeline_runner.py`)
- Walk-forward backtest → Price forecasters: fits and predicts (evidence: `src/bess_pipeline/forecasting/backtest.py`)
- Walk-forward backtest → MILP dispatcher: settles dispatch (evidence: `src/bess_pipeline/forecasting/backtest.py`)
- Walk-forward backtest → Financial KPIs: measures earnings (evidence: `src/bess_pipeline/forecasting/backtest.py`)
- Rolling backtest → MILP dispatcher: optimizes schedule (evidence: `src/bess_pipeline/forecasting/rolling.py`)
- Dispatch strategies → MILP dispatcher: dispatches strategy (evidence: `src/bess_pipeline/forecasting/strategies.py`)
- Backtest report → NESO client: loads auction prices (evidence: `scripts/run_backtest_report.py`)
- Backtest report → Elexon client: loads wind and demand (evidence: `scripts/run_backtest_report.py`)
- Backtest report → Precomputed results: writes app results (evidence: `README.md`)
- Engineer / analyst → Max Revenue: views revenue plan (evidence: `README.md`)
- Engineer / analyst → Findings: views backtest results (evidence: `README.md`)
- Max Revenue → Precomputed results: reads results (evidence: `README.md`)
- Findings → Precomputed results: reads results (evidence: `README.md`)