# robcarver17/pysystemtrade architecture

> Architecture diagram of the GitHub repository robcarver17/pysystemtrade, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/robcarver17/pysystemtrade
- Repository: https://github.com/robcarver17/pysystemtrade

## Overview

##### Project type and purpose
- This is a Python-based, modular trading engine and backtester for systematic futures trading.
- It provides:
- A data ingestion and storage layer (CSV, MongoDB/Arctic, Parquet)
- Core utilities and domain objects
- Strategy modeling and backtesting (“systems” package)
- Quantitative estimators and optimizers (“sysquant”)
- An order execution engine interfacing with brokers (Interactive Brokers via IB-insync)
- Orchestration and scheduling scripts for production (sysproduction) and process control (syscontrol)
- A lightweight web dashboard (Flask) for monitoring

##### File-structure analysis (key directories)
- sysdata: data-access abstractions + backends (csv, mongodb, arctic, parquet)
- syscore: shared utilities (cache, dateutils, fileutils, pandas helpers, exceptions)
- sysobjects: in-memory domain objects (contracts, prices, spreads, positions)
- systems: strategy definitions, backtesting logic, P&L calculators
- sysquant: statistical estimators & portfolio optimizers
- sysexecution: trade-generation & order stacks, broker adapters
- sysbrokers: broker-specific client implementations (IB)
- syscontrol: lightweight scripts to spawn, monitor, and report on processes
- sysinit: data-loading and initialization routines (populating databases)
- sysproduction: cron-style “run_…” scripts and reporting pipelines
- syslogging & syslogdiag: logging configuration, runtime diagnostics, email alerts
- dashboard: Flask app (app.py, static/js/css, templates)

##### README insights
- Fully automated environment for backtesting and live trading via Interactive Brokers
- Emphasizes modularity: separate concerns for data, core logic, execution, and operations
- Uses IB-insync, pandas, and multiple storage backends

##### Main components
A. External Data Sources (left)
- CSV files (raw historical data)
- IB real-time market data API
- Databases: MongoDB/Arctic, Parquet storage

B. Data Access Layer
- sysdata package with adapters (csv, mongodb, arctic, parquet)

C. Core Services Layer
- syscore (utilities)
- sysobjects (domain objects)

D. Strategy & Quant Layer
- systems (backtesting & system definitions)
- sysquant (estimators & optimizers)

E. Execution Layer
- sysexecution (order stacks, strategy ↔ order translation)
- sysbrokers (broker client adapters to IB)

F. Orchestration & Operations
- syscontrol (process control, monitoring)
- sysinit (initial data load scripts)
- sysproduction (scheduled “run_…” pipelines, reporting)

G. Logging & Diagnostics
- syslogging (logging server, handlers, configs)
- syslogdiag (email alerts, log analysis)

H. Dashboard UI
- Flask app serving status and metrics

##### Relationships and data flows
1. Raw data sources → sysdata → Core services
2. Core services + sysobjects → systems/sysquant → generate signals
3. systems/sysquant → sysexecution → sysbrokers → IB API → live orders
4. IB execution reports → sysdata (backfill) & sysobjects (positions)
5. sysproduction & syscontrol orchestrate end-to-end pipelines (backtests, live runs, reports)
6. syslogging captures events from all modules → log stores → syslogdiag alerts
7. Dashboard queries sysdata and syscontrol endpoints → renders status

##### Architectural patterns & principles
- Layered architecture (Data → Core → Strategy → Execution → Ops → UI)
- Hexagonal/ports-and-adapters: each layer depends on interfaces, with interchangeable backends (csv vs mongodb vs arctic)
- Single Responsibility: each package solves one concern (data, compute, execution, control)
- Plugin-style strategies: “systems/provided” holds user-pluggable strategy definitions

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources Layer
    subgraph "External Data Sources"
        CSV["CSV files"]:::data
        IBAPI["IB Market Data API"]:::data
        DB["MongoDB / Arctic / Parquet"]:::data
    end

    %% Data Access Layer
    subgraph "Data Layer"
        SYS_DATA["sysdata (storage adapters)"]:::data
    end

    %% Core Layer
    subgraph "Core & Domain"
        SYS_CORE["syscore (utilities)"]:::core
        SYS_OBJECTS["sysobjects (domain objects)"]:::core
    end

    %% Strategy & Quant Layer
    subgraph "Strategy & Quant"
        SYSTEMS["systems (backtesting)"]:::strategy
        SYSQUANT["sysquant (estimators & optimizers)"]:::strategy
    end

    %% Execution Layer
    subgraph "Execution & Brokers"
        SYS_EXEC["sysexecution (order engine)"]:::exec
        SYS_BROKERS["sysbrokers (IB adapter)"]:::exec
    end

    %% Operations Layer
    subgraph "Orchestration & Operations"
        SYS_CONTROL["syscontrol (monitor/control)"]:::ops
        SYS_INIT["sysinit (data init)"]:::ops
        SYS_PROD["sysproduction (scheduled pipelines)"]:::ops
    end

    %% Logging & Diagnostics
    subgraph "Logging & Diagnostics"
        SYS_LOG["syslogging (logging)"]:::log
        SYS_LOGDIAG["syslogdiag (alerts)"]:::log
    end

    %% Dashboard UI
    DASH["dashboard (Flask UI)"]:::ui

    %% Data Flows
    CSV -->|"CSV"| SYS_DATA
    IBAPI -->|"REST / IB-insync"| SYS_DATA
    DB -->|"Parquet / Mongo"| SYS_DATA

    SYS_DATA -->|"data access"| SYS_CORE
    SYS_DATA -->|"data objects"| SYS_OBJECTS

    SYS_CORE -->|"core services"| SYSTEMS
    SYS_OBJECTS -->|"domain objects"| SYSTEMS
    SYS_CORE -->|"core services"| SYSQUANT
    SYS_OBJECTS -->|"domain objects"| SYSQUANT

    SYSTEMS -->|"signals"| SYS_EXEC
    SYSQUANT -->|"signals"| SYS_EXEC

    SYS_EXEC -->|"orders"| SYS_BROKERS
    SYS_BROKERS -->|"IB API"| IBAPI

    IBAPI -->|"exec reports"| SYS_DATA
    IBAPI -->|"exec reports"| SYS_OBJECTS

    SYS_PROD -->|"schedule runs"| SYSTEMS
    SYS_PROD -->|"schedule runs"| SYS_EXEC
    SYS_CONTROL -->|"monitor/control"| SYSTEMS
    SYS_CONTROL -->|"monitor/control"| SYS_EXEC
    SYS_INIT -->|"init data"| SYS_DATA

    %% Logging
    SYS_DATA --> SYS_LOG
    SYS_CORE --> SYS_LOG
    SYS_OBJECTS --> SYS_LOG
    SYSTEMS --> SYS_LOG
    SYSQUANT --> SYS_LOG
    SYS_EXEC --> SYS_LOG
    SYS_BROKERS --> SYS_LOG
    SYS_CONTROL --> SYS_LOG
    SYS_PROD --> SYS_LOG
    SYS_INIT --> SYS_LOG

    SYS_LOG --> SYS_LOGDIAG

    %% Dashboard
    DASH -->|"query data"| SYS_DATA
    DASH -->|"status API"| SYS_CONTROL

    %% Click Events
    click CSV "https://github.com/robcarver17/pysystemtrade/tree/develop/data/"
    click SYS_DATA "https://github.com/robcarver17/pysystemtrade/tree/develop/sysdata/"
    click SYS_CORE "https://github.com/robcarver17/pysystemtrade/tree/develop/syscore/"
    click SYS_OBJECTS "https://github.com/robcarver17/pysystemtrade/tree/develop/sysobjects/"
    click SYSTEMS "https://github.com/robcarver17/pysystemtrade/tree/develop/systems/"
    click SYSQUANT "https://github.com/robcarver17/pysystemtrade/tree/develop/sysquant/"
    click SYS_EXEC "https://github.com/robcarver17/pysystemtrade/tree/develop/sysexecution/"
    click SYS_BROKERS "https://github.com/robcarver17/pysystemtrade/tree/develop/sysbrokers/"
    click SYS_CONTROL "https://github.com/robcarver17/pysystemtrade/tree/develop/syscontrol/"
    click SYS_INIT "https://github.com/robcarver17/pysystemtrade/tree/develop/sysinit/"
    click SYS_PROD "https://github.com/robcarver17/pysystemtrade/tree/develop/sysproduction/"
    click SYS_LOG "https://github.com/robcarver17/pysystemtrade/tree/develop/syslogging/"
    click SYS_LOGDIAG "https://github.com/robcarver17/pysystemtrade/tree/develop/syslogdiag/"
    click DASH "https://github.com/robcarver17/pysystemtrade/blob/develop/dashboard/app.py"

    %% Styles
    classDef data fill:#a2f5a2,stroke:#333;
    classDef core fill:#a2d2f5,stroke:#333;
    classDef strategy fill:#f5d0a2,stroke:#333;
    classDef exec fill:#f5a2a2,stroke:#333;
    classDef ops fill:#d2a2f5,stroke:#333;
    classDef log fill:#c0c0c0,stroke:#333;
    classDef ui fill:#f5f5a2,stroke:#333;
```
