How ddoqag/binance-trading-system works
1. Overall shape
This repo is not a single app; it is a multi-generation Binance quant-trading platform. The main architectural boundary is between: legacy Python/Node trading and research code, margin/live execution code, and the newer low-latency “new/” stack that adds Go/Rust/Python integration, shared memory, and more advanced agent-based orchestration.
2. External dependencies and system inputs
The system primarily integrates with Binance Spot/Margin/Futures APIs and a PostgreSQL database. Market data can also be stored and replayed from CSV/JSON files under data/. Redis appears as an optional coordination layer in demos/tests, while monitoring targets are exposed via Prometheus/Grafana in the newer Go engine.
3. Data ingestion and persistence
The Node.js side handles much of the data plumbing: Binance REST/WebSocket clients, fetch scripts, DB initialization/migration, and indicator table setup. This feeds PostgreSQL tables such as klines, ticker, order book, and technical indicators. The repository also has Python loaders and generators that can fall back between DB and local files.
4. Research and feature-generation pipeline
The core research path is data_generator/, factors/, indicators/, training_system/, and rl/. Raw market data is cleaned, labeled, and split with leakage/lookahead prevention before being turned into features, factors, and training sets. This is the offline “data → features → labels → models” boundary.
5. ML / factor / RL modeling
The modeling layer spans classical alpha factors, supervised ML, and reinforcement learning. training_system/ defines dataset/model/train/evaluate flows; rl/ contains DQN/PPO environments and trainers; new/brain_py/qlib_models/ and new/brain_py/moe/ add more advanced model ensembles, mixture-of-experts, and regime-aware feature sets. These components produce signals or model artifacts consumed by trading/orchestration code.
6. Strategy layer
There are several strategy families: simple rule-based strategies under strategy/, higher-level trading logic in trading_system/, and plugin-driven strategies under plugins/ and plugin_examples/. This boundary is important: strategies generate intent or signals, but they do not execute orders directly.
7. Risk and portfolio control
Risk is enforced in multiple places. The legacy stack uses risk/ for position sizing, stop-loss, and manager logic; the margin stack adds margin_trading/* risk controllers; the newer stack has new/hedge_fund_os/risk_kernel.py, new/core_go/risk_*, and portfolio allocation modules such as mean-variance, risk parity, and Black-Litterman. The architecture should show risk as a gate between signal generation and execution.
8. Execution and order management
Order execution is split into several runtimes. Legacy Python trading/ supports spot/margin executors and a Rust bridge; margin_trading/ and live_trading_* scripts orchestrate real/paper trading flows. The newer execution path is much more explicit: new/execution_core/, new/core/, new/core_go/, and rust_execution/ provide order state machines, cancel/reprice logic, queue tracking, slippage/fill models, and low-latency exchange adapters.
9. Real-time system boundary
The new/ tree introduces a hard boundary between a fast execution engine and slower Python decision logic. Shared memory, protocol.h, shared/, new/shared/, and Go/Python bridge code are used to move market state, signals, and order updates across process/language boundaries with minimal latency. This is the architectural center of the “HFT / latency-sensitive” path.
10. Orchestration and lifecycle management
There are multiple orchestrators, depending on which generation is in use: main_trading_system.py, live_trading_*, margin_trading/orchestrator.py, trading_system/trader.py, new/hedge_fund_os/orchestrator.py, and new/brain_py/live_integrator.py. These coordinate the flow from market data ingestion, through strategy/risk decisions, into execution, then back through fill and account updates.
11. Monitoring, alerts, and recovery
Operational support is a first-class subsystem. monitoring/ handles structured logs and alerts in the legacy stack; new/core_go/metrics.go, dashboards, Prometheus config, alert rules, and recovery/reconciliation code provide production observability for the newer engine. This layer closes the loop after execution by reconciling positions, fills, and failures.
12. Plugin and extension boundary
The repo supports extension via plugins rather than hardcoding every strategy or factor. plugins/ contains event buses, managers, versioning, and specialized plugin packages; plugin_examples/ demonstrates strategies, factors, and executors. Diagrammatically, this is the “extensible strategy/risk/execution” boundary that sits alongside the core engines.
13. Repository structure to reflect in the diagram
A good diagram should show three major lanes:
• Legacy research/backtest/live stack: data, factors, training, strategies, risk, backtest, execution.
• Margin/live orchestration stack: margin managers, account/position/risk controllers, live trading entrypoints.
• New low-latency stack: Go core engine, shared memory bridge, Rust execution, advanced Python agents, hedge-fund-style orchestration, monitoring.
These lanes share the same external exchange and database, but are otherwise separate implementation paths.
14. Key runtime mix
The repo is deliberately polyglot: Python for research, model training, orchestration, and analytics; Node.js for Binance/DB tooling and some integration utilities; Go for the low-latency engine and observability; Rust for execution performance work. That language split is an architectural feature, not just an implementation detail.
Main entry points: core_go Go core engine · main_trading_system.py Live orchestrators · orchestrator.py New orchestrator · strategy Legacy strategies
Components 21
External systems
Binance, databases, files, and observability targets used by the trading platform.
- Binance APIs · Spot, margin, futures market and account connectivity.
- PostgreSQL · Persistent storage for klines, order book, indicators, and account/trading data.
- Market files · Local CSV/JSON data used for loading, replay, and fallback workflows.
data - Prometheus/Grafana · Metrics, dashboards, alerts, and operational visibility for the newer engine.
new/core_go
Legacy stack
Older Python/Node trading, research, and live orchestration paths.
- Node data layer · REST/WebSocket clients, DB setup, migrations, and market data plumbing.
core - Research pipeline · Cleaning, loading, labeling, leakage-free splitting, and feature generation.
data_generator - Factors & models · Classical factors, supervised training, and reinforcement learning components.
factors - Legacy strategies · Rule-based and ML strategy logic that emits signals but does not execute orders.
strategy - Legacy risk · Position sizing, stop-loss, and trade gating for older trading paths.
risk - Backtest engine · Historical simulation and evaluation for strategy and system behavior.
backtest - Legacy execution · Python and bridge-based order execution for spot, margin, and leverage flows.
trading - Live orchestrators · Entry points that coordinate research, strategy, risk, and execution in production or paper flows.
main_trading_system.py - Margin stack · Dedicated margin account, position, signal, and risk management path.
margin_trading
New low-latency stack
The newer multi-runtime execution and agent-based architecture under new/.
- Go core engine · Fast execution, metrics, reconciliation, recovery, and exchange client handling.
new/core_go - Shared memory bridge · Protocol headers, shared-memory readers/writers, and cross-language state transfer.
new/shared - Rust execution · Performance-oriented execution engine and benchmarks for latency-sensitive order handling.
rust_execution/src - Python agents · Agent registry, expert agents, MoE, regime features, and live integration.
new/brain_py - Portfolio & risk · Risk kernel and portfolio construction modules such as mean-variance, risk parity, and Black-Litterman.
new/hedge_fund_os - Execution core · Order lifecycle, cancel/reprice, queue tracking, slippage, fill, and position state management.
new/execution_core - New orchestrator · Coordinates agents, risk, execution, monitoring, and recovery in the new stack.
new/hedge_fund_os/orchestrator.py
Plugins
Extension boundary for strategies, factors, and executors.
- Plugin layer · Event buses, managers, versioning, and example plugins for extensible behaviors.
plugins
Connections 28
- Node data layer → Binance APIs: market/account data · Fetches live market and account state from Binance.
- Node data layer → PostgreSQL: store/reload · Initializes and populates trading data tables.
- Market files → Research pipeline: replay input · Supplies file-based market history when DB is absent or for offline workflows.
- PostgreSQL → Research pipeline: historical data · Feeds loaders and generators for cleaning and feature creation.
- Research pipeline → Factors & models: features/labels · Produces model-ready datasets and training splits.
- Factors & models → Legacy strategies: signals · Turns modeled outputs into strategy inputs or predictions.
- Legacy strategies → Legacy risk: signal gate · Strategy intent is filtered by risk controls before orders are created.
- Legacy risk → Legacy execution: approve orders · Allows only risk-compliant orders to reach execution.
- Legacy execution → Binance APIs: orders · Submits spot, margin, and leverage orders to exchange APIs.
- Live orchestrators → Node data layer: feeds · Coordinates live market-data acquisition and system startup.
- Live orchestrators → Legacy strategies: runs strategies · Executes the legacy trading loop.
- Live orchestrators → Legacy execution: routes orders · Hands approved actions to execution.
- Live orchestrators → Margin stack: margin mode · Selects the margin-oriented live path when required.
- Margin stack → Binance APIs: margin trading · Interacts with Binance margin accounts and positions.
- Plugin layer → Legacy strategies: strategy plugins · Extends strategy selection and behavior.
- Plugin layer → Legacy execution: executor plugins · Allows alternative execution implementations.
- Python agents → Portfolio & risk: allocations · Agent outputs are constrained and transformed into portfolio decisions.
- Python agents → Shared memory bridge: signals · Writes decision and state payloads into the IPC bridge.
- Shared memory bridge → Go core engine: market/state sync · Moves fast-changing state between Python and Go processes.
- Go core engine → Execution core: execution control · Coordinates order state, lifecycle, and exchange interaction.
- Execution core → Binance APIs: low-latency orders · Sends orders and receives fills through exchange adapters.
- New orchestrator → Python agents: agent loop · Runs the slower reasoning and synthesis layer.
- New orchestrator → Portfolio & risk: risk gate · Applies portfolio construction and risk checks before execution.
- New orchestrator → Shared memory bridge: bridge state · Publishes market, signal, and order updates to the fast path.
- Go core engine → Prometheus/Grafana: metrics · Exports metrics and dashboards for monitoring and alerting.
- Go core engine → Rust execution: hot path · Connects the Go engine to the Rust performance execution layer where used.
- Rust execution → Binance APIs: optimized fills · Implements a high-performance exchange execution path.
- New orchestrator → Go core engine: coordinate · Orchestrates the production lifecycle around the low-latency engine.