# erikstream1/reproducible-ml-trading-pipeline architecture

> Architecture diagram of the GitHub repository erikstream1/reproducible-ml-trading-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/erikstream1/reproducible-ml-trading-pipeline
- Repository: https://github.com/erikstream1/reproducible-ml-trading-pipeline

## Overview

This repository is a reproducible ML trading research and execution system. Its principal runtime flow collects market quotes and historical data, checks quote quality, builds features, loads a trained model, and turns predictions into a strategy decision. That decision can feed real-time simulation, shadow or paper execution, or a live Bitso order path; separate workflows support training, walk-forward validation, backtesting, and composite-signal research. The README documents several capabilities whose implementations were not sampled; those are represented at their known directory paths, with uncertain connections omitted.

Main entry points: [`src/pipelines/realtime_simulation_pipeline.py`](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/realtime_simulation_pipeline.py) (Real-time decision step), [`src/pipelines/live_broker_pipeline.py`](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/live_broker_pipeline.py) (Live broker pipeline), [`src/pipelines/paper_trading_pipeline.py`](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/paper_trading_pipeline.py) (Paper trading), [`src/pipelines/collect_quotes_pipeline.py`](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/collect_quotes_pipeline.py) (Quote collection)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_research["Research workflows"]
  node_validation["Walk-forward validation"]
  node_scoring["Validation scoring<br/>[scoring.py]"]
  node_signals["Composite signals"]
  node_signal_research["Signal research"]
  node_backtest["Backtest ledger<br/>[ledger.py]"]
end

subgraph group_market["Market data"]
  node_data_pipe["Data pipeline<br/>[data_pipeline.py]"]
  node_quote_collect["Quote collection"]
  node_bitso_data["Bitso market client<br/>[bitso_client.py]"]
  node_quote_store[("Quote history")]
  node_quote_quality["Quote quality gates<br/>[quality_gates.py]"]
end

subgraph group_ml["Modeling"]
  node_features["Feature engineering<br/>[builder.py]"]
  node_training["Model training"]
  node_models["Model implementations"]
  node_inference["Model inference"]
end

subgraph group_trading["Strategy and execution"]
  node_realtime["Real-time decision step"]
  node_strategy["Trading strategy"]
  node_shadow["Shadow execution"]
  node_paper["Paper trading"]
  node_simulator["Fill simulator<br/>[simulator.py]"]
  node_risk["Pre-trade risk<br/>[risk_limits.py]"]
  node_live["Live broker pipeline"]
  node_broker["Bitso broker<br/>[bitso_brokers.py]"]
end

subgraph group_operations["Monitoring and artifacts"]
  node_circuit["Circuit breakers"]
  node_incidents["Incident bundles"]
  node_divergence["Execution divergence"]
  node_experiments["Experiment artifacts"]
end

node_user(("Pipeline operator"))
node_external_bitso{{"Bitso API"}}

node_user -.->|"runs"| node_quote_collect
node_user -.->|"runs"| node_training
node_user -.->|"runs"| node_validation
node_user -.->|"runs"| node_realtime
node_quote_collect -.->|"fetches quotes"| node_bitso_data
node_quote_collect -.->|"stores snapshots"| node_quote_store
node_realtime -->|"loads history"| node_quote_store
node_realtime -->|"checks quality"| node_quote_quality
node_realtime -->|"loads market data"| node_data_pipe
node_realtime -->|"builds features"| node_features
node_realtime -->|"loads model"| node_models
node_realtime -->|"generates position"| node_strategy
node_realtime -->|"persists step"| node_backtest
node_training -->|"builds model"| node_models
node_training -.->|"saves runs"| node_experiments
node_validation -->|"validates models"| node_models
node_validation -->|"scores folds"| node_scoring
node_signal_research -->|"loads data"| node_data_pipe
node_signal_research -->|"builds features"| node_features
node_signal_research -->|"builds signals"| node_signals
node_signal_research -->|"evaluates results"| node_scoring
node_shadow -->|"optionally collects"| node_quote_collect
node_shadow -->|"runs decision step"| node_realtime
node_shadow -->|"simulates fills"| node_simulator
node_paper -->|"optionally collects"| node_quote_collect
node_paper -->|"runs decision step"| node_realtime
node_paper -->|"checks trade risk"| node_risk
node_paper -->|"simulates fills"| node_simulator
node_live -->|"optionally collects"| node_quote_collect
node_live -->|"runs decision step"| node_realtime
node_live -->|"checks trade risk"| node_risk
node_live -->|"submits order"| node_broker
node_broker -.->|"places order"| node_external_bitso
node_live -->|"checks failures"| node_circuit
node_live -->|"records incident"| node_incidents
node_paper -->|"checks failures"| node_circuit
node_paper -->|"records incident"| node_incidents
node_divergence -.->|"compares fills"| node_shadow
node_divergence -.->|"compares fills"| node_live
node_backtest -.->|"uses signals"| node_strategy

click node_data_pipe "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/data_pipeline.py"
click node_quote_collect "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/collect_quotes_pipeline.py"
click node_bitso_data "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/data/providers/bitso_client.py"
click node_quote_store "https://github.com/erikstream1/reproducible-ml-trading-pipeline/tree/main/src/data/quotes"
click node_quote_quality "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/data/quotes/quality_gates.py"
click node_features "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/features/builder.py"
click node_training "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/training_pipeline.py"
click node_models "https://github.com/erikstream1/reproducible-ml-trading-pipeline/tree/main/src/models"
click node_validation "https://github.com/erikstream1/reproducible-ml-trading-pipeline/tree/main/src/validation"
click node_scoring "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/validation/scoring.py"
click node_inference "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/inference_pipeline.py"
click node_realtime "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/realtime_simulation_pipeline.py"
click node_signals "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/strategy/composite_signals.py"
click node_signal_research "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/signal_research_pipeline.py"
click node_strategy "https://github.com/erikstream1/reproducible-ml-trading-pipeline/tree/main/src/strategy"
click node_backtest "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/backtest/ledger.py"
click node_shadow "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/end_to_end_execution_pipeline.py"
click node_paper "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/paper_trading_pipeline.py"
click node_simulator "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/execution/simulator.py"
click node_risk "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/execution/risk_limits.py"
click node_live "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/live_broker_pipeline.py"
click node_broker "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/execution/brokers/bitso_brokers.py"
click node_circuit "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/execution/circuit_breakers.py"
click node_incidents "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/execution/incident_artifacts.py"
click node_divergence "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/execution/divergence_monitor.py"
click node_experiments "https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/utils/experiments_artifacts.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_validation,node_scoring,node_signals,node_signal_research,node_backtest toneBlue
class node_data_pipe,node_quote_collect,node_bitso_data,node_quote_store,node_quote_quality toneAmber
class node_features,node_training,node_models,node_inference,node_external_bitso toneMint
class node_realtime,node_strategy,node_shadow,node_paper,node_simulator,node_risk,node_live,node_broker toneRose
class node_circuit,node_incidents,node_divergence,node_experiments,node_user toneIndigo
```

## Components

### Research workflows

- [Walk-forward validation](https://github.com/erikstream1/reproducible-ml-trading-pipeline/tree/main/src/validation): `src/validation`
- [Validation scoring](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/validation/scoring.py): `src/validation/scoring.py`
- [Composite signals](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/strategy/composite_signals.py): `src/strategy/composite_signals.py`
- [Signal research](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/signal_research_pipeline.py): `src/pipelines/signal_research_pipeline.py`
- [Backtest ledger](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/backtest/ledger.py): `src/backtest/ledger.py`

### Market data

- [Data pipeline](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/data_pipeline.py): `src/pipelines/data_pipeline.py`
- [Quote collection](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/collect_quotes_pipeline.py): `src/pipelines/collect_quotes_pipeline.py`
- [Bitso market client](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/data/providers/bitso_client.py): `src/data/providers/bitso_client.py`
- [Quote history](https://github.com/erikstream1/reproducible-ml-trading-pipeline/tree/main/src/data/quotes): `src/data/quotes`
- [Quote quality gates](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/data/quotes/quality_gates.py): `src/data/quotes/quality_gates.py`

### Modeling

- [Feature engineering](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/features/builder.py): `src/features/builder.py`
- [Model training](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/training_pipeline.py): `src/pipelines/training_pipeline.py`
- [Model implementations](https://github.com/erikstream1/reproducible-ml-trading-pipeline/tree/main/src/models): `src/models`
- [Model inference](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/inference_pipeline.py): `src/pipelines/inference_pipeline.py`

### Strategy and execution

- [Real-time decision step](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/realtime_simulation_pipeline.py): `src/pipelines/realtime_simulation_pipeline.py`
- [Trading strategy](https://github.com/erikstream1/reproducible-ml-trading-pipeline/tree/main/src/strategy): `src/strategy`
- [Shadow execution](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/end_to_end_execution_pipeline.py): `src/pipelines/end_to_end_execution_pipeline.py`
- [Paper trading](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/paper_trading_pipeline.py): `src/pipelines/paper_trading_pipeline.py`
- [Fill simulator](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/execution/simulator.py): `src/execution/simulator.py`
- [Pre-trade risk](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/execution/risk_limits.py): `src/execution/risk_limits.py`
- [Live broker pipeline](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/pipelines/live_broker_pipeline.py): `src/pipelines/live_broker_pipeline.py`
- [Bitso broker](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/execution/brokers/bitso_brokers.py): `src/execution/brokers/bitso_brokers.py`

### Monitoring and artifacts

- [Circuit breakers](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/execution/circuit_breakers.py): `src/execution/circuit_breakers.py`
- [Incident bundles](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/execution/incident_artifacts.py): `src/execution/incident_artifacts.py`
- [Execution divergence](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/execution/divergence_monitor.py): `src/execution/divergence_monitor.py`
- [Experiment artifacts](https://github.com/erikstream1/reproducible-ml-trading-pipeline/blob/main/src/utils/experiments_artifacts.py): `src/utils/experiments_artifacts.py`

### Other components

- **Pipeline operator**
- **Bitso API**

## Connections

- Pipeline operator → Quote collection: runs
- Pipeline operator → Model training: runs
- Pipeline operator → Walk-forward validation: runs
- Pipeline operator → Real-time decision step: runs
- Quote collection → Bitso market client: fetches quotes
- Quote collection → Quote history: stores snapshots
- Real-time decision step → Quote history: loads history (evidence: `src/pipelines/realtime_simulation_pipeline.py`)
- Real-time decision step → Quote quality gates: checks quality (evidence: `src/pipelines/realtime_simulation_pipeline.py`)
- Real-time decision step → Data pipeline: loads market data (evidence: `src/pipelines/realtime_simulation_pipeline.py`)
- Real-time decision step → Feature engineering: builds features (evidence: `src/pipelines/realtime_simulation_pipeline.py`)
- Real-time decision step → Model implementations: loads model (evidence: `src/pipelines/realtime_simulation_pipeline.py`)
- Real-time decision step → Trading strategy: generates position (evidence: `src/pipelines/realtime_simulation_pipeline.py`)
- Real-time decision step → Backtest ledger: persists step (evidence: `src/pipelines/realtime_simulation_pipeline.py`)
- Model training → Model implementations: builds model (evidence: `src/pipelines/training_pipeline.py`)
- Model training → Experiment artifacts: saves runs
- Walk-forward validation → Model implementations: validates models (evidence: `src/validation/cv.py`)
- Walk-forward validation → Validation scoring: scores folds (evidence: `src/validation/cv.py`)
- Signal research → Data pipeline: loads data (evidence: `src/pipelines/signal_research_pipeline.py`)
- Signal research → Feature engineering: builds features (evidence: `src/pipelines/signal_research_pipeline.py`)
- Signal research → Composite signals: builds signals (evidence: `src/pipelines/signal_research_pipeline.py`)
- Signal research → Validation scoring: evaluates results (evidence: `src/pipelines/signal_research_pipeline.py`)
- Shadow execution → Quote collection: optionally collects (evidence: `src/pipelines/end_to_end_execution_pipeline.py`)
- Shadow execution → Real-time decision step: runs decision step (evidence: `src/pipelines/end_to_end_execution_pipeline.py`)
- Shadow execution → Fill simulator: simulates fills (evidence: `src/pipelines/end_to_end_execution_pipeline.py`)
- Paper trading → Quote collection: optionally collects (evidence: `src/pipelines/paper_trading_pipeline.py`)
- Paper trading → Real-time decision step: runs decision step (evidence: `src/pipelines/paper_trading_pipeline.py`)
- Paper trading → Pre-trade risk: checks trade risk (evidence: `src/pipelines/paper_trading_pipeline.py`)
- Paper trading → Fill simulator: simulates fills (evidence: `src/pipelines/paper_trading_pipeline.py`)
- Live broker pipeline → Quote collection: optionally collects (evidence: `src/pipelines/live_broker_pipeline.py`)
- Live broker pipeline → Real-time decision step: runs decision step (evidence: `src/pipelines/live_broker_pipeline.py`)
- Live broker pipeline → Pre-trade risk: checks trade risk (evidence: `src/pipelines/live_broker_pipeline.py`)
- Live broker pipeline → Bitso broker: submits order (evidence: `src/pipelines/live_broker_pipeline.py`)
- Bitso broker → Bitso API: places order
- Live broker pipeline → Circuit breakers: checks failures (evidence: `src/pipelines/live_broker_pipeline.py`)
- Live broker pipeline → Incident bundles: records incident (evidence: `src/pipelines/live_broker_pipeline.py`)
- Paper trading → Circuit breakers: checks failures (evidence: `src/pipelines/paper_trading_pipeline.py`)
- Paper trading → Incident bundles: records incident (evidence: `src/pipelines/paper_trading_pipeline.py`)
- Execution divergence → Shadow execution: compares fills
- Execution divergence → Live broker pipeline: compares fills
- Backtest ledger → Trading strategy: uses signals