How krish-rm/market-master-trading-action-prediction works
Project Type & Purpose
• This is a full-stack MLOps pipeline (“Market Master”) for hourly index futures trading signals. It collects 1-hour OHLCV data for top NASDAQ-100 constituents, computes features, trains and selects a champion classifier via MLflow, monitors drift, gates promotion, serves predictions through FastAPI, and presents an interactive Streamlit dashboard.
High-Level Component Breakdown
A. Data Ingestion & Storage
• External Service: Yahoo Finance API (via yfinance)
• Fallback: Built-in CSV data
• Storage:
• Raw & processed CSVs under data/components/
• QQQ weights in data/weights/
• Drift reports in data/monitoring/
B. Feature Engineering
• Module: src/features.py
• Utilities: src/data.py (data loading, persistence)
• Outputs: DataFrame with rolling returns, RSI, MA, ATR, Bollinger Bands
C. Model Training & Selection
• Candidate Estimators: src/model_candidates.py (RF, ET, GB, HGBT, MLP, SVC, LogReg)
• Orchestration:
• Local: Prefect flow in flows/enhanced_orchestration.py
• Docker: Script src/run_pipeline.py
• Training Logic: src/train_pooled_compare.py (pooled dataset, nested MLflow runs)
• Logging & Registry: MLflow tracking server (mlruns/) + Model Registry aliases (Staging/Production)
D. Drift Monitoring & Promotion
• Drift Reports: src/monitor_drift.py (Evidently)
• Gate & Report: src/gate_and_report.py (F1 thresholds, drift checks)
• Registry Management: src/registry.py (promote/rollback)
E. Model Serving API
• Framework: FastAPI + Uvicorn (src/api.py)
• Endpoints: /health, /predict/component, /signal/index, /predict
• Model Load: models:/market-master-component-classifier@Production → fallback to artifacts/model/model.pkl
F. Interactive Dashboard
• Streamlit App: dashboard/streamlit_app.py
• Visualizes live prices, individual stock predictions, Weighted Sentiment Score (WSS), risk metrics
G. CI/CD & Infrastructure
• GitHub Actions: .github/workflows/ci.yml (tests, linting, pipeline smoke test)
• Containerization:
• Dockerfiles for API, Dashboard, MLflow
• docker-compose.yml orchestrates API, MLflow UI, Streamlit, SQLite backend
• Makefile: one-line commands for local and Docker workflows
Relationships & Data Flow
1) Fetch weights & component data from Yahoo Finance → CSV storage
2) Feature engineering & label generation → processed DataFrame
3) Train & evaluate candidates → MLflow tracking → select & register champion
4) Monitor drift & quality → gate criteria → promote to Production
5) FastAPI loads Production model → serves component predictions or WSS index signal
6) Streamlit dashboard consumes API endpoints → real-time UI
7) CI runs tests and smoke tests on each change; Docker Compose ties all services together
Architectural Patterns & Principles
• Layered Architecture: Ingestion → Feature → Training → Serving → Presentation
• MLOps Best Practices: Experiment tracking, Model Registry, drift monitoring, gated promotion
• Microservices Containerization: Separate containers for API, MLflow, Dashboard
• Orchestration: Prefect tasks or simple entrypoint script for reproducible pipelines
• CI/CD: Automated code quality, testing, and deployment