How lokhiufung/alchemist works
Project type and purpose
• Alchemist is a distributed, high-performance automated trading framework built on Ray. It takes quantitative strategies from research to live execution, handling market data ingestion, order management, risk controls, and persistence.
File structure
• Top-level modules:
• alchemist/ – core library (data models, pipelines, strategies, managers, gateways, monitors, utilities)
• externals/ibapi – third-party Interactive Brokers API code
• run_strategies.py – entry point launching Ray Serve management service
• seed_timescaledb.py – initializes TimescaleDB schema
• strategy_config_example.yaml – sample strategy setup
• Key subpackages:
• data_pipelines/ – transforms raw market data streams into bar or tick objects
• gateways/ – interfaces to external data sources (IBGateway) and mocks
• datas/ – in-memory data representations (tick, bar, resampler)
• managers/ – Ray actors coordinating data, orders, portfolio state
• strategies/ – stateless or stateful strategy logic (e.g., moving average)
• monitors/ – telemetry (Telegram, TimescaleDB)
• timescaledb/ – ORM models for persistence
• zeromq.py – messaging layer for intra-process communication
• Configuration and environment: strategy_config.yaml, .env, requirements.txt, setup.py
README for additional insights
• Core components: Ray Serve, ZeroMQ messaging, IB or mock data gateways, TimescaleDB for monitoring/trades, strategy replicas on Ray.
• Workflow: ingest → transform → strategy → order → execution → monitoring → persistence.
• Emphasis on backtest/live parity, horizontal scaling, modularity.
a. Main components
• Ray Cluster / Ray Serve
• Management Service (Ray Serve endpoint)
• Data Gateways
– IBGateway (live data/orders via IB API)
– MockGateway (simulator)
• Data Pipeline Actors (bar/tick generator, resampler)
• Strategy Actors (n replicas)
• Order Manager Actor
• Portfolio Manager Actor
• Data Manager Actor
• ZeroMQ Messaging Bus (PUB/SUB sockets)
• External Systems
– Interactive Brokers (IB API)
– TimescaleDB (history, orders, signals, trades)
– Telegram (alerts)
• Persistence Layer (TimescaleDB ORM models)
b. Relationships and interactions
1. Management Service bootstraps: reads config → spawns gateway actors and strategy replicas on Ray.
2. Gateways subscribe to market data, push raw events into ZeroMQ publish socket.
3. Data Pipeline actors subscribe to ZeroMQ, normalize into DataCard objects, forward to Data Manager.
4. Strategy actors subscribe to relevant ports, receive DataCard updates, compute signals.
5. Strategies send order requests over ZeroMQ to Order Manager.
6. Order Manager interacts with Gateway to send to IB or mock. It also informs Portfolio Manager.
7. Portfolio Manager tracks positions, sends risk/portfolio updates.
8. Monitors subscribe to ZeroMQ or hook into managers to log metrics/trades into TimescaleDB or send Telegram alerts.
9. Data Manager persists market data snapshots if configured.
c. Architectural patterns & principles
• Actor model (Ray actors for concurrency and distribution)
• Publish/subscribe messaging (ZeroMQ for decoupling producers/consumers)
• Modular pipeline stages (data ingestion → processing → strategy → execution)
• Dependency inversion: gateways implement base_gateway interface
• Strategy isolation: replicas ensure fault tolerance and scalability
d. Key technologies/frameworks
• Ray (distributed execution, actors, Serve)
• ZeroMQ (messaging layer)
• Interactive Brokers API (externals/ibapi)
• TimescaleDB with SQLAlchemy models
• PyYAML for configuration
• pytest for testing
Details to capture
• Interactions between Managers (data, order, portfolio) and their responsibilities.
• Scaling aspects: e.g., n_replicas of strategy actors.