# lokhiufung/alchemist architecture

> Architecture diagram of the GitHub repository lokhiufung/alchemist, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lokhiufung/alchemist
- Repository: https://github.com/lokhiufung/alchemist

## Overview

##### Project type and purpose
- Alchemist is a distributed, high-performance automated trading framework built on Ray. It takes quantitative strategies from research to live execution, handling market data ingestion, order management, risk controls, and persistence.

##### File structure
- Top-level modules:
- alchemist/          – core library (data models, pipelines, strategies, managers, gateways, monitors, utilities)
- externals/ibapi     – third-party Interactive Brokers API code
- run_strategies.py   – entry point launching Ray Serve management service
- seed_timescaledb.py – initializes TimescaleDB schema
- strategy_config_example.yaml – sample strategy setup
- Key subpackages:
- data_pipelines/     – transforms raw market data streams into bar or tick objects
- gateways/           – interfaces to external data sources (IBGateway) and mocks
- datas/              – in-memory data representations (tick, bar, resampler)
- managers/           – Ray actors coordinating data, orders, portfolio state
- strategies/         – stateless or stateful strategy logic (e.g., moving average)
- monitors/           – telemetry (Telegram, TimescaleDB)
- timescaledb/        – ORM models for persistence
- zeromq.py           – messaging layer for intra-process communication
- Configuration and environment: strategy_config.yaml, .env, requirements.txt, setup.py

##### README for additional insights
- Core components: Ray Serve, ZeroMQ messaging, IB or mock data gateways, TimescaleDB for monitoring/trades, strategy replicas on Ray.
- Workflow: ingest → transform → strategy → order → execution → monitoring → persistence.
- Emphasis on backtest/live parity, horizontal scaling, modularity.

a. Main components
- Ray Cluster / Ray Serve
- Management Service (Ray Serve endpoint)
- Data Gateways
– IBGateway (live data/orders via IB API)
– MockGateway (simulator)
- Data Pipeline Actors (bar/tick generator, resampler)
- Strategy Actors (n replicas)
- Order Manager Actor
- Portfolio Manager Actor
- Data Manager Actor
- ZeroMQ Messaging Bus (PUB/SUB sockets)
- External Systems
– Interactive Brokers (IB API)
– TimescaleDB (history, orders, signals, trades)
– Telegram (alerts)
- Persistence Layer (TimescaleDB ORM models)

b. Relationships and interactions
1. Management Service bootstraps: reads config → spawns gateway actors and strategy replicas on Ray.
2. Gateways subscribe to market data, push raw events into ZeroMQ publish socket.
3. Data Pipeline actors subscribe to ZeroMQ, normalize into DataCard objects, forward to Data Manager.
4. Strategy actors subscribe to relevant ports, receive DataCard updates, compute signals.
5. Strategies send order requests over ZeroMQ to Order Manager.
6. Order Manager interacts with Gateway to send to IB or mock. It also informs Portfolio Manager.
7. Portfolio Manager tracks positions, sends risk/portfolio updates.
8. Monitors subscribe to ZeroMQ or hook into managers to log metrics/trades into TimescaleDB or send Telegram alerts.
9. Data Manager persists market data snapshots if configured.

c. Architectural patterns & principles
- Actor model (Ray actors for concurrency and distribution)
- Publish/subscribe messaging (ZeroMQ for decoupling producers/consumers)
- Modular pipeline stages (data ingestion → processing → strategy → execution)
- Dependency inversion: gateways implement base_gateway interface
- Strategy isolation: replicas ensure fault tolerance and scalability

d. Key technologies/frameworks
- Ray (distributed execution, actors, Serve)
- ZeroMQ (messaging layer)
- Interactive Brokers API (externals/ibapi)
- TimescaleDB with SQLAlchemy models
- PyYAML for configuration
- pytest for testing

##### Details to capture
- Interactions between Managers (data, order, portfolio) and their responsibilities.
- Scaling aspects: e.g., n_replicas of strategy actors.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Systems
    subgraph "External Systems" 
        direction TB
        IBAPI("Interactive Brokers API"):::external
        TELEGRAM("Telegram"):::external
    end

    %% Persistence Layer
    subgraph "Persistence"
        direction TB
        TIMESCALEDB[(TimescaleDB)]:::db
        SEED["seed_timescaledb.py"]:::infra
    end

    %% Configuration
    CONFIG["strategy_config_example.yaml"]:::infra

    %% Messaging Layer
    ZMQ["ZeroMQ Messaging Bus"]:::infra

    %% Ray Cluster Boundary
    subgraph "Ray Cluster" 
        direction TB
        MS["Ray Serve Management Service"]:::infra
        OG["Order Manager Actor"]:::strategy
        PG["Portfolio Manager Actor"]:::strategy
        DM["Data Manager Actor"]:::data
        subgraph "Data Gateways"
            direction TB
            IBG["IBGateway"]:::data
            MG["MockGateway"]:::data
        end
        subgraph "Data Pipelines"
            direction TB
            DP_BASE["BaseDataPipeline"]:::data
            DP_IB["IBDataPipeline"]:::data
        end
        subgraph "Strategy Actors"
            direction TB
            STR_BASE["BaseStrategy"]:::strategy
            STR_MA["MovingAverageStrategy"]:::strategy
            STR_MAC["MovingAverageCrossoverStrategy"]:::strategy
        end
        subgraph "Monitors"
            direction TB
            MON_BASE["BaseMonitor"]:::infra
            MON_TG["TelegramMonitor"]:::external
            MON_TS["TimescaleDBMonitor"]:::external
        end
    end

    %% Data Models
    subgraph "Data Models"
        direction TB
        DC["DataCard"]:::data
        TICK["Tick"]:::data
        BAR["Bar"]:::data
        RESAMP["Resampler"]:::data
    end

    %% Connections
    CONFIG -->|drives| MS
    MS -->|spawns| IBG
    MS -->|spawns| MG
    MS -->|spawns| DP_BASE
    MS -->|spawns| DP_IB
    MS -->|spawns| STR_BASE
    MS -->|spawns| STR_MA
    MS -->|spawns| STR_MAC
    MS -->|spawns| OG
    MS -->|spawns| PG
    MS -->|spawns| DM
    MS -->|spawns| MON_BASE
    MON_BASE --> MON_TG
    MON_BASE --> MON_TS

    IBG -->|subscribes| ZMQ
    MG -->|subscribes| ZMQ

    ZMQ -->|publishes raw market data| DP_BASE
    ZMQ -->|publishes raw market data| DP_IB
    DP_BASE -->|"produces DataCard"| DC
    DP_IB -->|"produces DataCard"| DC
    DC --> DM
    DC --> STR_BASE
    DC --> STR_MA
    DC --> STR_MAC

    STR_BASE -->|sends orders| ZMQ
    STR_MA -->|sends orders| ZMQ
    STR_MAC -->|sends orders| ZMQ

    ZMQ -->|delivers orders| OG
    OG -->|executes via| IBG
    OG -->|updates| PG

    PG -->|positions update| DM

    MON_TS -->|writes metrics| TIMESCALEDB
    DM -->|persists data| TIMESCALEDB

    MON_TG -->|alerts via| TELEGRAM

    SEED --> TIMESCALEDB

    %% Click Events
    click MS "https://github.com/lokhiufung/alchemist/blob/main/alchemist/management_service.py"
    click MS "https://github.com/lokhiufung/alchemist/blob/main/run_strategies.py"
    click ZMQ "https://github.com/lokhiufung/alchemist/blob/main/alchemist/zeromq.py"
    click IBG "https://github.com/lokhiufung/alchemist/blob/main/alchemist/gateways/ib_gateway/ib_gateway.py"
    click MG "https://github.com/lokhiufung/alchemist/blob/main/alchemist/gateways/mock_gateway/mock_gateway.py"
    click DP_BASE "https://github.com/lokhiufung/alchemist/blob/main/alchemist/data_pipelines/base_data_pipeline.py"
    click DP_IB "https://github.com/lokhiufung/alchemist/blob/main/alchemist/data_pipelines/ib_data_pipeline.py"
    click DC "https://github.com/lokhiufung/alchemist/blob/main/alchemist/data_card.py"
    click TICK "https://github.com/lokhiufung/alchemist/blob/main/alchemist/datas/tick_data.py"
    click BAR "https://github.com/lokhiufung/alchemist/blob/main/alchemist/datas/bar_data.py"
    click RESAMP "https://github.com/lokhiufung/alchemist/blob/main/alchemist/datas/resampler.py"
    click STR_BASE "https://github.com/lokhiufung/alchemist/blob/main/alchemist/strategies/base_strategy.py"
    click STR_MA "https://github.com/lokhiufung/alchemist/blob/main/alchemist/strategies/moving_average_strategy/moving_average_strategy.py"
    click STR_MAC "https://github.com/lokhiufung/alchemist/blob/main/alchemist/strategies/moving_average_crossover_strategy/moving_average_crossover_strategy.py"
    click OG "https://github.com/lokhiufung/alchemist/blob/main/alchemist/managers/order_manager.py"
    click PG "https://github.com/lokhiufung/alchemist/blob/main/alchemist/managers/portfolio_manager.py"
    click DM "https://github.com/lokhiufung/alchemist/blob/main/alchemist/managers/data_manager.py"
    click MON_TG "https://github.com/lokhiufung/alchemist/blob/main/alchemist/monitors/telegram_monitor.py"
    click MON_TS "https://github.com/lokhiufung/alchemist/blob/main/alchemist/monitors/timescaledb_monitor.py"
    click MON_BASE "https://github.com/lokhiufung/alchemist/blob/main/alchemist/monitors/base_monitor.py"
    click TIMESCALEDB "https://github.com/lokhiufung/alchemist/blob/main/alchemist/timescaledb/models/base.py"
    click SEED "https://github.com/lokhiufung/alchemist/blob/main/seed_timescaledb.py"
    click CONFIG "https://github.com/lokhiufung/alchemist/blob/main/strategy_config_example.yaml"

    %% Styles
    classDef data fill:#D6EAF8,stroke:#1F618D,color:#1F618D
    classDef strategy fill:#D5F5E3,stroke:#239B56,color:#239B56
    classDef external fill:#FAD7A0,stroke:#D35400,color:#D35400
    classDef infra fill:#D5D8DC,stroke:#566573,color:#566573
    classDef db fill:#F9E79F,stroke:#B7950B,color:#B7950B
```
