How erikstream1/reproducible-ml-trading-pipeline works
This repository is a reproducible ML trading research and execution system. Its principal runtime flow collects market quotes and historical data, checks quote quality, builds features, loads a trained model, and turns predictions into a strategy decision. That decision can feed real-time simulation, shadow or paper execution, or a live Bitso order path; separate workflows support training, walk-forward validation, backtesting, and composite-signal research. The README documents several capabilities whose implementations were not sampled; those are represented at their known directory paths, with uncertain connections omitted.
Main entry points: realtime_simulation_pipeline.py Real-time decision step · live_broker_pipeline.py Live broker pipeline · paper_trading_pipeline.py Paper trading · collect_quotes_pipeline.py Quote collection
Components 28
Research workflows
- Walk-forward validation
src/validation - Validation scoring
src/validation/scoring.py - Composite signals
src/strategy/composite_signals.py - Signal research
src/pipelines/signal_research_pipeline.py - Backtest ledger
src/backtest/ledger.py
Market data
- Data pipeline
src/pipelines/data_pipeline.py - Quote collection
src/pipelines/collect_quotes_pipeline.py - Bitso market client
src/data/providers/bitso_client.py - Quote history
src/data/quotes - Quote quality gates
src/data/quotes/quality_gates.py
Modeling
- Feature engineering
src/features/builder.py - Model training
src/pipelines/training_pipeline.py - Model implementations
src/models - Model inference
src/pipelines/inference_pipeline.py
Strategy and execution
- Real-time decision step
src/pipelines/realtime_simulation_pipeline.py - Trading strategy
src/strategy - Shadow execution
src/pipelines/end_to_end_execution_pipeline.py - Paper trading
src/pipelines/paper_trading_pipeline.py - Fill simulator
src/execution/simulator.py - Pre-trade risk
src/execution/risk_limits.py - Live broker pipeline
src/pipelines/live_broker_pipeline.py - Bitso broker
src/execution/brokers/bitso_brokers.py
Monitoring and artifacts
- Circuit breakers
src/execution/circuit_breakers.py - Incident bundles
src/execution/incident_artifacts.py - Execution divergence
src/execution/divergence_monitor.py - Experiment artifacts
src/utils/experiments_artifacts.py
Other components
- Pipeline operator
- Bitso API
Connections 40
- Pipeline operator → Quote collection: runs
- Pipeline operator → Model training: runs
- Pipeline operator → Walk-forward validation: runs
- Pipeline operator → Real-time decision step: runs
- Quote collection → Bitso market client: fetches quotes
- Quote collection → Quote history: stores snapshots
- Real-time decision step → Quote history: loads history
src/pipelines/realtime_simulation_pipeline.py - Real-time decision step → Quote quality gates: checks quality
src/pipelines/realtime_simulation_pipeline.py - Real-time decision step → Data pipeline: loads market data
src/pipelines/realtime_simulation_pipeline.py - Real-time decision step → Feature engineering: builds features
src/pipelines/realtime_simulation_pipeline.py - Real-time decision step → Model implementations: loads model
src/pipelines/realtime_simulation_pipeline.py - Real-time decision step → Trading strategy: generates position
src/pipelines/realtime_simulation_pipeline.py - Real-time decision step → Backtest ledger: persists step
src/pipelines/realtime_simulation_pipeline.py - Model training → Model implementations: builds model
src/pipelines/training_pipeline.py - Model training → Experiment artifacts: saves runs
- Walk-forward validation → Model implementations: validates models
src/validation/cv.py - Walk-forward validation → Validation scoring: scores folds
src/validation/cv.py - Signal research → Data pipeline: loads data
src/pipelines/signal_research_pipeline.py - Signal research → Feature engineering: builds features
src/pipelines/signal_research_pipeline.py - Signal research → Composite signals: builds signals
src/pipelines/signal_research_pipeline.py - Signal research → Validation scoring: evaluates results
src/pipelines/signal_research_pipeline.py - Shadow execution → Quote collection: optionally collects
src/pipelines/end_to_end_execution_pipeline.py - Shadow execution → Real-time decision step: runs decision step
src/pipelines/end_to_end_execution_pipeline.py - Shadow execution → Fill simulator: simulates fills
src/pipelines/end_to_end_execution_pipeline.py - Paper trading → Quote collection: optionally collects
src/pipelines/paper_trading_pipeline.py - Paper trading → Real-time decision step: runs decision step
src/pipelines/paper_trading_pipeline.py - Paper trading → Pre-trade risk: checks trade risk
src/pipelines/paper_trading_pipeline.py - Paper trading → Fill simulator: simulates fills
src/pipelines/paper_trading_pipeline.py - Live broker pipeline → Quote collection: optionally collects
src/pipelines/live_broker_pipeline.py - Live broker pipeline → Real-time decision step: runs decision step
src/pipelines/live_broker_pipeline.py - Live broker pipeline → Pre-trade risk: checks trade risk
src/pipelines/live_broker_pipeline.py - Live broker pipeline → Bitso broker: submits order
src/pipelines/live_broker_pipeline.py - Bitso broker → Bitso API: places order
- Live broker pipeline → Circuit breakers: checks failures
src/pipelines/live_broker_pipeline.py - Live broker pipeline → Incident bundles: records incident
src/pipelines/live_broker_pipeline.py - Paper trading → Circuit breakers: checks failures
src/pipelines/paper_trading_pipeline.py - Paper trading → Incident bundles: records incident
src/pipelines/paper_trading_pipeline.py - Execution divergence → Shadow execution: compares fills
- Execution divergence → Live broker pipeline: compares fills
- Backtest ledger → Trading strategy: uses signals