How robcarver17/pysystemtrade works
Project type and purpose
• This is a Python-based, modular trading engine and backtester for systematic futures trading.
• It provides:
• A data ingestion and storage layer (CSV, MongoDB/Arctic, Parquet)
• Core utilities and domain objects
• Strategy modeling and backtesting (“systems” package)
• Quantitative estimators and optimizers (“sysquant”)
• An order execution engine interfacing with brokers (Interactive Brokers via IB-insync)
• Orchestration and scheduling scripts for production (sysproduction) and process control (syscontrol)
• A lightweight web dashboard (Flask) for monitoring
File-structure analysis (key directories)
• sysdata: data-access abstractions + backends (csv, mongodb, arctic, parquet)
• syscore: shared utilities (cache, dateutils, fileutils, pandas helpers, exceptions)
• sysobjects: in-memory domain objects (contracts, prices, spreads, positions)
• systems: strategy definitions, backtesting logic, P&L calculators
• sysquant: statistical estimators & portfolio optimizers
• sysexecution: trade-generation & order stacks, broker adapters
• sysbrokers: broker-specific client implementations (IB)
• syscontrol: lightweight scripts to spawn, monitor, and report on processes
• sysinit: data-loading and initialization routines (populating databases)
• sysproduction: cron-style “run_…” scripts and reporting pipelines
• syslogging & syslogdiag: logging configuration, runtime diagnostics, email alerts
• dashboard: Flask app (app.py, static/js/css, templates)
README insights
• Fully automated environment for backtesting and live trading via Interactive Brokers
• Emphasizes modularity: separate concerns for data, core logic, execution, and operations
• Uses IB-insync, pandas, and multiple storage backends
Main components
A. External Data Sources (left)
• CSV files (raw historical data)
• IB real-time market data API
• Databases: MongoDB/Arctic, Parquet storage
B. Data Access Layer
• sysdata package with adapters (csv, mongodb, arctic, parquet)
C. Core Services Layer
• syscore (utilities)
• sysobjects (domain objects)
D. Strategy & Quant Layer
• systems (backtesting & system definitions)
• sysquant (estimators & optimizers)
E. Execution Layer
• sysexecution (order stacks, strategy ↔ order translation)
• sysbrokers (broker client adapters to IB)
F. Orchestration & Operations
• syscontrol (process control, monitoring)
• sysinit (initial data load scripts)
• sysproduction (scheduled “run_…” pipelines, reporting)
G. Logging & Diagnostics
• syslogging (logging server, handlers, configs)
• syslogdiag (email alerts, log analysis)
H. Dashboard UI
• Flask app serving status and metrics
Relationships and data flows
1. Raw data sources → sysdata → Core services
2. Core services + sysobjects → systems/sysquant → generate signals
3. systems/sysquant → sysexecution → sysbrokers → IB API → live orders
4. IB execution reports → sysdata (backfill) & sysobjects (positions)
5. sysproduction & syscontrol orchestrate end-to-end pipelines (backtests, live runs, reports)
6. syslogging captures events from all modules → log stores → syslogdiag alerts
7. Dashboard queries sysdata and syscontrol endpoints → renders status
Architectural patterns & principles
• Layered architecture (Data → Core → Strategy → Execution → Ops → UI)
• Hexagonal/ports-and-adapters: each layer depends on interfaces, with interchangeable backends (csv vs mongodb vs arctic)
• Single Responsibility: each package solves one concern (data, compute, execution, control)
• Plugin-style strategies: “systems/provided” holds user-pluggable strategy definitions