# krish-rm/market-master-trading-action-prediction architecture

> Architecture diagram of the GitHub repository krish-rm/market-master-trading-action-prediction, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/krish-rm/market-master-trading-action-prediction
- Repository: https://github.com/krish-rm/market-master-trading-action-prediction

## Overview

##### Project Type & Purpose
- This is a full-stack MLOps pipeline (“Market Master”) for hourly index futures trading signals. It collects 1-hour OHLCV data for top NASDAQ-100 constituents, computes features, trains and selects a champion classifier via MLflow, monitors drift, gates promotion, serves predictions through FastAPI, and presents an interactive Streamlit dashboard.

##### High-Level Component Breakdown
A. Data Ingestion & Storage
- External Service: Yahoo Finance API (via `yfinance`)
- Fallback: Built-in CSV data
- Storage:
- Raw & processed CSVs under `data/components/`
- QQQ weights in `data/weights/`
- Drift reports in `data/monitoring/`

B. Feature Engineering
- Module: `src/features.py`
- Utilities: `src/data.py` (data loading, persistence)
- Outputs: DataFrame with rolling returns, RSI, MA, ATR, Bollinger Bands

C. Model Training & Selection
- Candidate Estimators: `src/model_candidates.py` (RF, ET, GB, HGBT, MLP, SVC, LogReg)
- Orchestration:
- Local: Prefect flow in `flows/enhanced_orchestration.py`
- Docker: Script `src/run_pipeline.py`
- Training Logic: `src/train_pooled_compare.py` (pooled dataset, nested MLflow runs)
- Logging & Registry: MLflow tracking server (`mlruns/`) + Model Registry aliases (Staging/Production)

D. Drift Monitoring & Promotion
- Drift Reports: `src/monitor_drift.py` (Evidently)
- Gate & Report: `src/gate_and_report.py` (F1 thresholds, drift checks)
- Registry Management: `src/registry.py` (promote/rollback)

E. Model Serving API
- Framework: FastAPI + Uvicorn (`src/api.py`)
- Endpoints: `/health`, `/predict/component`, `/signal/index`, `/predict`
- Model Load: `models:/market-master-component-classifier@Production` → fallback to `artifacts/model/model.pkl`

F. Interactive Dashboard
- Streamlit App: `dashboard/streamlit_app.py`
- Visualizes live prices, individual stock predictions, Weighted Sentiment Score (WSS), risk metrics

G. CI/CD & Infrastructure
- GitHub Actions: `.github/workflows/ci.yml` (tests, linting, pipeline smoke test)
- Containerization:
- Dockerfiles for API, Dashboard, MLflow
- `docker-compose.yml` orchestrates API, MLflow UI, Streamlit, SQLite backend
- Makefile: one-line commands for local and Docker workflows

##### Relationships & Data Flow
1) Fetch weights & component data from Yahoo Finance → CSV storage
2) Feature engineering & label generation → processed DataFrame
3) Train & evaluate candidates → MLflow tracking → select & register champion
4) Monitor drift & quality → gate criteria → promote to Production
5) FastAPI loads Production model → serves component predictions or WSS index signal
6) Streamlit dashboard consumes API endpoints → real-time UI
7) CI runs tests and smoke tests on each change; Docker Compose ties all services together

##### Architectural Patterns & Principles
- Layered Architecture: Ingestion → Feature → Training → Serving → Presentation
- MLOps Best Practices: Experiment tracking, Model Registry, drift monitoring, gated promotion
- Microservices Containerization: Separate containers for API, MLflow, Dashboard
- Orchestration: Prefect tasks or simple entrypoint script for reproducible pipelines
- CI/CD: Automated code quality, testing, and deployment

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Source
    YF["Yahoo Finance API"]:::external

    %% Data Storage Layer
    subgraph "Data Storage" 
        direction TB
        RawCSV["Raw & Fallback CSVs"]:::data
        ProcessedCSV["Processed CSVs"]:::data
        QQQWeights["QQQ Weights CSV"]:::data
        DriftReports["Drift Reports"]:::data
        MLflowStore["MLflow Artifact Store\n(mlruns/)"]:::data
        SQLiteDB["SQLite Backend"]:::data
    end

    %% Data Ingestion Layer
    subgraph "Data Ingestion" 
        direction TB
        FetchComp["Fetch Components"]:::compute
        FetchSym["Fetch Symbol"]:::compute
        FetchWeights["Fetch QQQ Weights"]:::compute
        DataUtil["Data Utilities"]:::compute
    end

    %% Feature Engineering
    subgraph "Feature Engineering"
        direction TB
        Features["Feature Engine\n(src/features.py)"]:::compute
    end

    %% Model Training & Selection
    subgraph "Model Training & Selection"
        direction TB
        Candidates["Model Candidates\n(src/model_candidates.py)"]:::compute
        Trainer["Pooled Train & Compare\n(src/train_pooled_compare.py)"]:::compute
        Orchestrator["Batch Orchestrator\n(flows/enhanced_orchestration.py)"]:::compute
        RunPipeline["Entrypoint Script\n(src/run_pipeline.py)"]:::compute
        MLflowServer["MLflow Tracking Server"]:::service
    end

    %% Drift Monitoring & Gating
    subgraph "Drift Monitoring & Gating"
        direction TB
        DriftMon["Drift Monitor\n(src/monitor_drift.py)"]:::compute
        GateRpt["Gate & Report\n(src/gate_and_report.py)"]:::compute
        RegistryMgmt["Registry Management\n(src/registry.py)"]:::compute
    end

    %% Serving API
    subgraph "Model Serving API"
        direction TB
        API["FastAPI Server\n(src/api.py)"]:::service
        ModelServe["Model Serving Logic\n(src/model_serving.py)"]:::service
        PredictIdx["Index Signal Endpoint\n(src/predict_index.py)"]:::service
        HourlyPred["Component Predict Endpoint\n(src/hourly_predict.py)"]:::service
    end

    %% Dashboard
    subgraph "Interactive Dashboard"
        direction TB
        Dashboard["Streamlit App\n(dashboard/streamlit_app.py)"]:::service
    end

    %% CI/CD & Infrastructure
    subgraph "CI/CD & Infrastructure"
        direction TB
        GitHub["GitHub Actions\n(.github/workflows/ci.yml)"]:::infra
        DockerAPI["Dockerfile (API)"]:::infra
        DockerDash["Dockerfile.dashboard"]:::infra
        DockerMLflow["Dockerfile.mlflow"]:::infra
        DockerComp["docker-compose.yml"]:::infra
        Makefile["Makefile"]:::infra
    end

    %% Connections: Data Flow
    YF -->|"1h OHLCV data"| FetchComp
    YF --> FetchSym
    YF --> FetchWeights
    FetchComp -->|stores| RawCSV
    FetchSym --> RawCSV
    FetchWeights --> QQQWeights
    RawCSV -->|load| DataUtil
    QQQWeights --> DataUtil
    DataUtil -->|raw→processed| ProcessedCSV

    ProcessedCSV --> Features
    Features -->|features DF| ProcessedCSV

    ProcessedCSV --> Candidates
    Candidates --> Trainer
    Trainer --> MLflowStore
    Trainer --> MLflowServer
    MLflowServer -->|register| MLflowStore

    %% Orchestration
    Orchestrator --> Features
    Orchestrator --> Trainer
    Orchestrator --> DriftMon
    RunPipeline --> Orchestrator

    %% Drift & Gate
    ProcessedCSV --> DriftMon
    DriftMon --> DriftReports
    DriftMon --> GateRpt
    GateRpt --> RegistryMgmt
    RegistryMgmt --> MLflowStore

    %% Serving
    RegistryMgmt -->|Production Model| API
    API --> ModelServe
    API --> PredictIdx
    API --> HourlyPred

    %% Dashboard consumption
    Dashboard -->|REST calls| API

    %% Infra
    GitHub -->|CI| DockerAPI
    GitHub --> DockerDash
    GitHub --> DockerMLflow
    DockerComp --> API
    DockerComp --> MLflowServer
    DockerComp --> Dashboard
    Makefile --> DockerComp

    %% Click Events
    click FetchComp "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/src/fetch_components.py"
    click FetchSym "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/src/fetch_symbol.py"
    click FetchWeights "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/src/fetch_weights_qqq.py"
    click DataUtil "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/src/data.py"
    click Features "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/src/features.py"
    click Candidates "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/src/model_candidates.py"
    click Trainer "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/src/train_pooled_compare.py"
    click Orchestrator "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/flows/enhanced_orchestration.py"
    click RunPipeline "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/src/run_pipeline.py"
    click DriftMon "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/src/monitor_drift.py"
    click GateRpt "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/src/gate_and_report.py"
    click RegistryMgmt "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/src/registry.py"
    click API "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/src/api.py"
    click ModelServe "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/src/model_serving.py"
    click PredictIdx "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/src/predict_index.py"
    click HourlyPred "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/src/hourly_predict.py"
    click Dashboard "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/dashboard/streamlit_app.py"
    click GitHub "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/.github/workflows/ci.yml"
    click DockerAPI "https://github.com/krish-rm/market-master-trading-action-prediction/tree/main/Dockerfile"
    click DockerDash "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/Dockerfile.dashboard"
    click DockerMLflow "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/Dockerfile.mlflow"
    click DockerComp "https://github.com/krish-rm/market-master-trading-action-prediction/blob/main/docker-compose.yml"
    click Makefile "https://github.com/krish-rm/market-master-trading-action-prediction/tree/main/Makefile"

    %% Styles
    classDef external fill:#bbbbbb,stroke:#333,stroke-width:1px
    classDef data fill:#D0E4F5,stroke:#1F77B4,stroke-width:2px
    classDef compute fill:#D5F5E3,stroke:#28A745,stroke-width:2px
    classDef service fill:#FFE5B4,stroke:#FF8C00,stroke-width:2px
    classDef infra fill:#F3E5F5,stroke:#AF7AC5,stroke-width:2px
```
