# mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading architecture

> Architecture diagram of the GitHub repository mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading
- Repository: https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading

## Overview

##### Project Type and Purpose
- This is a full-stack, message-driven backtesting platform for crypto trading.
- It provides a React frontend for users, a Python API backend, a Kafka-based data pipeline, backtesting workers, and a persistent data store.
- It is designed to run multiple backtests at scale, orchestrate data ingestion, and integrate MLOps tools (Kafka, Airflow, MLflow, CML).

##### Main Components
a. User Interface (Frontend)
- Technology: React with Vite, Tailwind CSS
- Code Location: `frontend/` (and `updated_frontend/` for the newer version)
- Key files: `App.jsx`, `pages/Backtest.jsx`, `components/ResultData.jsx`
b. API Backend
- Technology: Python (FastAPI or Flask style)
- Code Location: `backend/`
- Key modules:
- `main.py` / `updated_main.py` — HTTP endpoints for login, signup, run-backtest
- `auth.py`, `utils/user_auth.py` — JWT/Auth0 integration
- `database.py`, `models.py`, `schemas.py` — ORM & Pydantic schemas
- `utils/backtest.py` — entry point to backtesting logic
c. Messaging Layer
- Technology: Apache Kafka, Zookeeper
- Code Location: `kafka_topic/`
- Key files: `kafka_config.py`, `produce_messages.py`, `consume_messages.py`
- Topics defined via YAML (`conduktor.yml`, `full-stack.yml`, etc.)
d. Backtesting Worker
- Technology: Python scripts & modules
- Code Location: `scripts/backtesting/` or `backend/utils/backtest.py`
- Implements strategies (`rsi_strategy.py`, `macd_strategy.py`, LSTM, etc.), analyzers, and orchestration
e. Data Stores
- Raw CSV/JSON: `data/` and `backend/data/`
- Relational DB: PostgreSQL (schema in `db/schema/schema.sql`)
- Docker Compose brings up Kafka, ZK, PostgreSQL via `docker-compose.yml`
f. MLOps & Orchestration (planned/integrated)
- Apache Airflow (not in code but mentioned in README)
- MLflow for experiment tracking
- CML for CI/CD
g. Notebooks & Exploratory Analysis
- Jupyter notebooks in `notebooks/`

##### Relationships and Data Flow
1. User (Browser) → React Frontend
2. Frontend (HTTP/REST) → Backend API
3. Backend API
- Authenticated requests → produces a Kafka message onto a “backtest-requests” topic
- Writes/reads metadata to PostgreSQL
4. Backtesting Worker (Kafka Consumer)
- Subscribes to “backtest-requests”
- Fetches historical data (from S3, local CSV, or external API via `fetch_binance_data.py` / `fetch_yahoo_finance_data.py`)
- Runs selected strategies and analyzers
- Persists results to PostgreSQL and logs metrics to MLflow
5. Frontend polls or subscribes to results via Backend API → displays results to user
6. Airflow (or CML pipelines) orchestrate daily data ingestion, backtests, and model retraining

##### Architectural Patterns & Principles
- Microservices / Modular services (frontend, API, workers)
- Event-driven architecture using Kafka as the messaging backbone
- Layered separation: Presentation → API → Processing → Persistence
- MLOps integration for reproducible experiments and CI/CD

##### Final Checklist
- Clear, distinct labels on every component
- Ancillary MLOps services (Airflow, MLflow) even if they live outside the main codebase
- Call out Docker Compose as the local deployment orchestrator

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "User & Presentation"
        UI_old["React Frontend (Old)\nVite, Tailwind"]:::frontend
        UI_new["React Frontend (Updated)\nVite, Tailwind"]:::frontend
    end

    subgraph "API / Authentication"
        API["API Backend\nFastAPI/Flask"]:::backend
        Auth["Auth Module\nJWT/Auth0"]:::backend
        ORM["ORM & Schemas\nSQLAlchemy & Pydantic"]:::backend
        DataLoader["Initial Data Loader"]:::backend
        BacktestAPI["Backtest Entry Point"]:::backend
    end

    subgraph "Messaging / Orchestration"
        KafkaConfig["Kafka Config"]:::messaging
        Producer["Kafka Producer"]:::messaging
        Consumer["Kafka Consumer"]:::messaging
        KafkaTopics["Kafka Topics Definitions"]:::messaging
    end

    subgraph "Processing / Workers"
        WorkerCLI["Backtesting Worker CLI"]:::worker
        Strategies["Strategies & Analyzers"]:::worker
        UtilsWorker["Worker Utilities"]:::worker
        DataFetch["Historical Data Fetchers"]:::worker
    end

    subgraph "Data Persistence"
        RawData["Raw Data Sources"]:::db
        DB["PostgreSQL"]:::db
        Schema["DB Schema Init"]:::db
        Compose["Docker Compose Orchestrator"]:::db
    end

    subgraph "MLOps / Monitoring"
        Airflow["Apache Airflow"]:::mlops
        MLflow["MLflow"]:::mlops
        CML["CML / CI-CD"]:::mlops
        Notebooks["Jupyter Notebooks"]:::mlops
    end

    UI_old -->|"HTTP/REST"| API
    UI_new -->|"HTTP/REST"| API
    API --> Auth
    API --> BacktestAPI
    BacktestAPI -->|"writes metadata"| DB
    BacktestAPI -->|"produces message"| Producer
    Producer --> KafkaTopics
    Producer -->|"to backtest-requests"| KafkaConfig
    KafkaConfig --> Consumer
    Consumer --> WorkerCLI
    WorkerCLI --> Strategies
    WorkerCLI --> UtilsWorker
    WorkerCLI --> DataFetch
    DataFetch --> RawData
    WorkerCLI -->|"persists results"| DB
    WorkerCLI -->|"logs metrics"| MLflow
    API -->|"reads results"| DB
    API --> UI_old
    API --> UI_new
    Airflow -.-> Producer
    Airflow -.-> WorkerCLI
    CML -.-> API
    Compose --> API
    Compose --> DB
    Compose --> KafkaConfig
    Compose --> Producer
    Compose --> Consumer
    Notebooks --> DB
    Notebooks --> MLflow

    classDef frontend fill:#ADD8E6,stroke:#000;
    classDef backend fill:#90EE90,stroke:#000;
    classDef messaging fill:#FFA500,stroke:#000;
    classDef worker fill:#FFFF99,stroke:#000;
    classDef db fill:#F08080,stroke:#000;
    classDef mlops fill:#D3D3D3,stroke:#000;

    click UI_old "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/frontend/src/App.jsx"
    click UI_old "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/frontend/src/pages/Backtest.jsx"
    click UI_old "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/frontend/src/components/ResultData.jsx"
    click UI_new "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/updated_frontend/src/App.js"
    click UI_new "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/updated_frontend/src/components/BacktestForm.js"
    click API "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/backend/main.py"
    click API "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/backend/updated_main.py"
    click Auth "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/backend/auth.py"
    click Auth "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/backend/utils/user_auth.py"
    click ORM "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/backend/database.py"
    click ORM "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/backend/models.py"
    click ORM "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/backend/schemas.py"
    click BacktestAPI "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/backend/utils/backtest.py"
    click DataLoader "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/backend/utils/init_data.py"
    click KafkaConfig "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/kafka_topic/kafka_config.py"
    click Producer "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/kafka_topic/produce_messages.py"
    click Consumer "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/kafka_topic/consume_messages.py"
    click KafkaTopics "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/kafka_topic/conduktor.yml"
    click KafkaTopics "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/kafka_topic/full-stack.yml"
    click KafkaTopics "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/kafka_topic/zk-single-kafka-single.yml"
    click WorkerCLI "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/scripts/backtest.py"
    click WorkerCLI "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/scripts/backtesting/main.py"
    click Strategies "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/scripts/backtesting/strategies/rsi_strategy.py"
    click Strategies "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/scripts/backtesting/strategies/macd_strategy.py"
    click Strategies "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/scripts/backtesting/strategies/lstm_strategy.py"
    click Strategies "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/scripts/backtesting/strategies/sma_strategy.py"
    click Strategies "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/scripts/backtesting/strategies/bollinger_bands_strategy.py"
    click Strategies "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/scripts/backtesting/analyzers/metrics_analyzer.py"
    click UtilsWorker "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/scripts/backtesting/util/user_input.py"
    click UtilsWorker "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/scripts/backtesting/util/stock_selection.py"
    click DataFetch "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/scripts/fetch_binance_data.py"
    click DataFetch "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/scripts/fetch_yahoo_finance_data.py"
    click RawData "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/data/aapl.csv"
    click RawData "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/data/binance_btc_usdt_candlestick.csv"
    click RawData "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/data/yfinance_btc_usdt_2020-01-01_to_2021-01-01.csv"
    click RawData "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/backend/data/indicators.json"
    click RawData "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/backend/data/stocks.json"
    click Schema "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/db/schema/schema.sql"
    click Compose "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/docker-compose.yml"
    click Notebooks "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/notebooks/backtesting.ipynb"
    click Notebooks "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/notebooks/kafka.ipynb"
    click Notebooks "https://github.com/mikiiiss/scalable_backtesting_infrastructure_for_crypto_trading/blob/main/notebooks/lstm.ipynb"
```
