How mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading works
Project Type and Purpose
• This is a full-stack, message-driven backtesting platform for crypto trading.
• It provides a React frontend for users, a Python API backend, a Kafka-based data pipeline, backtesting workers, and a persistent data store.
• It is designed to run multiple backtests at scale, orchestrate data ingestion, and integrate MLOps tools (Kafka, Airflow, MLflow, CML).
Main Components
a. User Interface (Frontend)
• Technology: React with Vite, Tailwind CSS
• Code Location: frontend/ (and updated_frontend/ for the newer version)
• Key files: App.jsx, pages/Backtest.jsx, components/ResultData.jsx
b. API Backend
• Technology: Python (FastAPI or Flask style)
• Code Location: backend/
• Key modules:
• main.py / updated_main.py — HTTP endpoints for login, signup, run-backtest
• auth.py, utils/user_auth.py — JWT/Auth0 integration
• database.py, models.py, schemas.py — ORM & Pydantic schemas
• utils/backtest.py — entry point to backtesting logic
c. Messaging Layer
• Technology: Apache Kafka, Zookeeper
• Code Location: kafka_topic/
• Key files: kafka_config.py, produce_messages.py, consume_messages.py
• Topics defined via YAML (conduktor.yml, full-stack.yml, etc.)
d. Backtesting Worker
• Technology: Python scripts & modules
• Code Location: scripts/backtesting/ or backend/utils/backtest.py
• Implements strategies (rsi_strategy.py, macd_strategy.py, LSTM, etc.), analyzers, and orchestration
e. Data Stores
• Raw CSV/JSON: data/ and backend/data/
• Relational DB: PostgreSQL (schema in db/schema/schema.sql)
• Docker Compose brings up Kafka, ZK, PostgreSQL via docker-compose.yml
f. MLOps & Orchestration (planned/integrated)
• Apache Airflow (not in code but mentioned in README)
• MLflow for experiment tracking
• CML for CI/CD
g. Notebooks & Exploratory Analysis
• Jupyter notebooks in notebooks/
Relationships and Data Flow
1. User (Browser) → React Frontend
2. Frontend (HTTP/REST) → Backend API
3. Backend API
• Authenticated requests → produces a Kafka message onto a “backtest-requests” topic
• Writes/reads metadata to PostgreSQL
4. Backtesting Worker (Kafka Consumer)
• Subscribes to “backtest-requests”
• Fetches historical data (from S3, local CSV, or external API via fetch_binance_data.py / fetch_yahoo_finance_data.py)
• Runs selected strategies and analyzers
• Persists results to PostgreSQL and logs metrics to MLflow
5. Frontend polls or subscribes to results via Backend API → displays results to user
6. Airflow (or CML pipelines) orchestrate daily data ingestion, backtests, and model retraining
Architectural Patterns & Principles
• Microservices / Modular services (frontend, API, workers)
• Event-driven architecture using Kafka as the messaging backbone
• Layered separation: Presentation → API → Processing → Persistence
• MLOps integration for reproducible experiments and CI/CD
Final Checklist
• Clear, distinct labels on every component
• Ancillary MLOps services (Airflow, MLflow) even if they live outside the main codebase
• Call out Docker Compose as the local deployment orchestrator