# talha4565/metalmind-smcforge architecture

> Architecture diagram of the GitHub repository talha4565/metalmind-smcforge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/talha4565/metalmind-smcforge
- Repository: https://github.com/talha4565/metalmind-smcforge

## Overview

1. **System shape**
   This is a three-plane trading platform: a **Next.js dashboard**, a **Flask/SocketIO API**, and a **Python ML/data pipeline**. The dashboard is the presentation layer, the API is the runtime control plane, and the pipeline owns ingestion, feature engineering, training, backtesting, and scheduled refreshes.

2. **Frontend boundary**
   `frontend-next/` is a Next.js app-router frontend with separate areas for **auth**, **dashboard**, **backtest**, **risk**, and **watchlist**. It uses reusable UI components plus domain components for **TradingView charts**, **SHAP explanations**, **signal cards**, and **risk metrics**. Client-side hooks in `src/lib/hooks` mediate all data access to the backend and real-time updates.

3. **API boundary**
   `api/app/` is the main backend surface. It exposes endpoints for **health**, **predictions**, **market price**, **SHAP feature importance**, **backtesting**, **auth**, and **watchlist/profile** operations. The API also contains **model loading**, **prediction caching**, **ETL-related routes**, and **error handling middleware**, so it acts as the integration point between UI, models, and stored data.

4. **Real-time and session flow**
   The frontend consumes the API over HTTP and also has a WebSocket path via `useWebSocket` and Flask-SocketIO. That makes the app both a request/response dashboard and a live terminal for price/prediction updates. Authentication is handled through the frontend auth pages plus backend auth services, with NextAuth wiring on the client side.

5. **Data ingestion and refresh**
   The ETL subsystem under `etl/` pulls market data from **Yahoo Finance** and CSV sources using extractor abstractions. Data then moves through transformers for **quality checks** and **feature generation**, and into loaders that append to CSV, load to DB, or populate a feature store. The scheduler coordinates periodic refreshes and alerting.

6. **Feature engineering pipeline**
   `features/` is where domain-specific feature construction happens. It includes **Smart Money Concepts features**, **volume features**, **multi-timeframe aggregation**, and **label generation**. This is the core modeling boundary: raw OHLCV data becomes model-ready sequences with the 90-feature schema referenced in the README.

7. **Model training and retraining**
   `models/` contains training and retraining scripts for the XGBoost classifiers, with separate support for gold and silver workflows. The training pipeline consumes engineered features from `data/features/*.parquet`, writes model artifacts, and records metrics into `reports/training_logs/`. `run_pipeline.py` appears to be the orchestration entry point for backfill, update, retrain, and scheduled modes.

8. **Inference and explainability**
   At runtime, the API loads trained models through `api/app/model_loader.py` and serves predictions to the dashboard. SHAP computation is isolated through `api/app/shap_cache.py` and `explainability/shap_analyzer.py`, so explainability is part of the inference path rather than an offline report-only step. Prediction logs are persisted under `reports/predictions/`.

9. **Backtesting subsystem**
   `backtesting/engine.py` provides a separate evaluation path from live inference. The API exposes backtest run/results endpoints, while the engine likely operates on historical feature/data inputs to simulate trading with slippage, commission, and TP/SL rules. This subsystem is important because it validates model behavior against historical market conditions rather than live predictions.

10. **Persistence and state**
   The project uses a relational database through SQLAlchemy/Alembic migrations, with local SQLite files present in `instance/` and the repo root, while the README also documents PostgreSQL for containerized deployment. The database boundary likely stores **users, auth state, profiles, watchlists, and operational metadata**, while bulk market/history data stays in files and parquet artifacts.

11. **Operational artifacts**
   The repository keeps generated data and results in file-based stores: `data/features/` for model inputs, `reports/` for training logs and prediction history, and `miscellaneous_archive/` for older generated reports. This means the architecture is hybrid: transactional state in the DB, but large analytical artifacts on disk.

12. **Deployment and orchestration**
   `docker-compose.yml`, the `Dockerfile`s, `start_api.py`, `run.py`, and `run_pipeline.py` are the main deployment/run entry points. There is also an Airflow DAG in `airflow/dags/retrain.py`, suggesting the training lifecycle can be automated externally. The overall deployment model is containerized, with the frontend, API, and database exposed as separate services.

13. **External integrations**
   The important external dependencies that affect architecture are **Yahoo Finance** for market data, **SMTP** for email alerts, **TradingView** embedded charts in the frontend, and likely a message channel via SocketIO for live updates. These integrations are not peripheral; they directly shape the runtime flow from ingestion to signal delivery to user notification.

14. **Primary end-to-end flow**
   The main loop is: **fetch market data → build features → train/retrain models → serve predictions with SHAP → log outcomes → backtest/monitor → alert users when confidence thresholds are met**. The dashboard is the consumer of this loop, while the pipeline and API are the producers and state managers.

Main entry points: [`api/app/main.py`](https://github.com/talha4565/metalmind-smcforge/blob/main/api/app/main.py) (Flask API), [`reports/training_logs/gold_latest.json`](https://github.com/talha4565/metalmind-smcforge/blob/main/reports/training_logs/gold_latest.json) (Reports), [`frontend-next/src/lib/hooks/useWebSocket.ts`](https://github.com/talha4565/metalmind-smcforge/blob/main/frontend-next/src/lib/hooks/useWebSocket.ts) (Client hooks), [`api/app/model_loader.py`](https://github.com/talha4565/metalmind-smcforge/blob/main/api/app/model_loader.py) (Model runtime)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Frontend"]
  node_frontend_shell["Next.js app<br/>nextjs app<br/>[layout.tsx]"]
  node_frontend_auth["Auth pages<br/>ui routes<br/>[page.tsx]"]
  node_frontend_dashboard["Dashboard<br/>ui routes<br/>[page.tsx]"]
  node_frontend_backtest["Backtest UI<br/>ui route<br/>[page.tsx]"]
  node_frontend_widgets["Domain widgets<br/>ui components"]
  node_frontend_hooks["Client hooks<br/>data hooks<br/>[useWebSocket.ts]"]
  node_frontend_nextauth["NextAuth<br/>auth wiring<br/>[route.ts]"]
end

subgraph group_backend["API"]
  node_api_main["Flask API<br/>api app<br/>[main.py]"]
  node_api_auth["Auth service<br/>auth endpoints<br/>[auth.py]"]
  node_api_domain["Domain routes<br/>api routes<br/>[watchlist.py]"]
  node_api_model["Model runtime<br/>inference core<br/>[model_loader.py]"]
  node_api_shap["Explainability<br/>shap cache<br/>[shap_cache.py]"]
  node_api_etl["ETL routes<br/>integration routes<br/>[etl_routes.py]"]
end

subgraph group_ml["ML Pipeline"]
  node_ml_etl["ETL pipeline<br/>data pipeline<br/>[pipeline.py]"]
  node_ml_features["Feature engine<br/>[pipeline.py]"]
  node_ml_training["Training jobs<br/>model training<br/>[retrain.py]"]
  node_ml_backtest["Backtest engine<br/>evaluation<br/>[engine.py]"]
  node_ml_scheduler["Scheduler<br/>orchestration<br/>[scheduler.py]"]
end

subgraph group_state["State & Artifacts"]
  node_state_db[("SQL database<br/>relational state<br/>[metalmind_smc.db]")]
  node_state_reports["Reports<br/>artifacts<br/>[gold_latest.json]"]
  node_state_data["Feature store<br/>parquet artifacts"]
end

node_ext_yahoo(("Yahoo Finance<br/>external source"))
node_ext_socketio(("SocketIO<br/>realtime channel"))

node_frontend_shell -->|"routes to"| node_frontend_auth
node_frontend_shell -->|"routes to"| node_frontend_dashboard
node_frontend_shell -->|"routes to"| node_frontend_backtest
node_frontend_dashboard -->|"renders"| node_frontend_widgets
node_frontend_backtest -->|"loads data"| node_frontend_hooks
node_frontend_dashboard -->|"fetches via"| node_frontend_hooks
node_frontend_hooks -->|"HTTP to"| node_api_main
node_frontend_hooks -->|"subscribes to"| node_ext_socketio
node_frontend_nextauth -->|"auth delegates"| node_api_auth
node_api_main -->|"mounts"| node_api_auth
node_api_main -->|"mounts"| node_api_domain
node_api_main -->|"uses"| node_api_model
node_api_main -->|"uses"| node_api_shap
node_api_main -->|"mounts"| node_api_etl
node_api_domain -->|"reads/writes"| node_state_db
node_api_auth -->|"reads/writes"| node_state_db
node_api_model -->|"reads"| node_state_data
node_api_model -->|"logs to"| node_state_reports
node_api_shap -->|"stores"| node_state_reports
node_ml_etl -->|"ingests from"| node_ext_yahoo
node_ml_etl -->|"feeds"| node_ml_features
node_ml_features -->|"writes"| node_state_data
node_ml_features -->|"supplies"| node_ml_training
node_ml_training -->|"writes logs"| node_state_reports
node_ml_training -->|"produces models"| node_api_model
node_ml_backtest -->|"writes results"| node_state_reports
node_ml_scheduler -->|"orchestrates"| node_ml_etl
node_ml_scheduler -->|"triggers"| node_ml_training
node_ml_scheduler -->|"alerts/logs"| node_state_reports
node_ext_socketio -->|"served by"| node_api_main
node_api_etl -->|"bridges to"| node_ml_etl

click node_frontend_shell "https://github.com/talha4565/metalmind-smcforge/blob/main/frontend-next/src/app/layout.tsx"
click node_frontend_auth "https://github.com/talha4565/metalmind-smcforge/blob/main/frontend-next/src/app/auth/login/page.tsx"
click node_frontend_dashboard "https://github.com/talha4565/metalmind-smcforge/blob/main/frontend-next/src/app/dashboard/page.tsx"
click node_frontend_backtest "https://github.com/talha4565/metalmind-smcforge/blob/main/frontend-next/src/app/backtest/page.tsx"
click node_frontend_widgets "https://github.com/talha4565/metalmind-smcforge/blob/main/frontend-next/src/components/Charts/TradingViewChart.tsx"
click node_frontend_hooks "https://github.com/talha4565/metalmind-smcforge/blob/main/frontend-next/src/lib/hooks/useWebSocket.ts"
click node_frontend_nextauth "https://github.com/talha4565/metalmind-smcforge/blob/main/frontend-next/src/app/api/auth/[...nextauth]/route.ts"
click node_api_main "https://github.com/talha4565/metalmind-smcforge/blob/main/api/app/main.py"
click node_api_auth "https://github.com/talha4565/metalmind-smcforge/blob/main/api/app/auth.py"
click node_api_domain "https://github.com/talha4565/metalmind-smcforge/blob/main/api/app/watchlist.py"
click node_api_model "https://github.com/talha4565/metalmind-smcforge/blob/main/api/app/model_loader.py"
click node_api_shap "https://github.com/talha4565/metalmind-smcforge/blob/main/api/app/shap_cache.py"
click node_api_etl "https://github.com/talha4565/metalmind-smcforge/blob/main/api/app/etl_routes.py"
click node_ml_etl "https://github.com/talha4565/metalmind-smcforge/blob/main/etl/pipeline.py"
click node_ml_features "https://github.com/talha4565/metalmind-smcforge/blob/main/features/pipeline.py"
click node_ml_training "https://github.com/talha4565/metalmind-smcforge/blob/main/models/retrain.py"
click node_ml_backtest "https://github.com/talha4565/metalmind-smcforge/blob/main/backtesting/engine.py"
click node_ml_scheduler "https://github.com/talha4565/metalmind-smcforge/blob/main/etl/scheduler.py"
click node_state_db "https://github.com/talha4565/metalmind-smcforge/blob/main/api/instance/metalmind_smc.db"
click node_state_reports "https://github.com/talha4565/metalmind-smcforge/blob/main/reports/training_logs/gold_latest.json"
click node_state_data "https://github.com/talha4565/metalmind-smcforge/blob/main/data/features/XAUUSD_features_latest.parquet"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend_shell,node_frontend_auth,node_frontend_dashboard,node_frontend_backtest,node_frontend_widgets,node_frontend_hooks,node_frontend_nextauth toneBlue
class node_api_main,node_api_auth,node_api_domain,node_api_model,node_api_shap,node_api_etl toneAmber
class node_ml_etl,node_ml_features,node_ml_training,node_ml_backtest,node_ml_scheduler toneMint
class node_state_db,node_state_reports,node_state_data toneRose
class node_ext_yahoo,node_ext_socketio toneNeutral
```

## Components

### Frontend

Next.js dashboard and client-side data access

- [Next.js app](https://github.com/talha4565/metalmind-smcforge/blob/main/frontend-next/src/app/layout.tsx): nextjs app · App-router UI shell and routing entrypoints · `frontend-next/src/app/layout.tsx`
- [Auth pages](https://github.com/talha4565/metalmind-smcforge/blob/main/frontend-next/src/app/auth/login/page.tsx): ui routes · Login, register, and email verification flows · `frontend-next/src/app/auth/login/page.tsx`
- [Dashboard](https://github.com/talha4565/metalmind-smcforge/blob/main/frontend-next/src/app/dashboard/page.tsx): ui routes · Main trading dashboard and profile/risk/watchlist areas · `frontend-next/src/app/dashboard/page.tsx`
- [Backtest UI](https://github.com/talha4565/metalmind-smcforge/blob/main/frontend-next/src/app/backtest/page.tsx): ui route · Backtest workspace and results view · `frontend-next/src/app/backtest/page.tsx`
- [Domain widgets](https://github.com/talha4565/metalmind-smcforge/blob/main/frontend-next/src/components/Charts/TradingViewChart.tsx): ui components · TradingView charts, SHAP explainer, signal cards, risk metrics, watchlist table · `frontend-next/src/components/Charts/TradingViewChart.tsx`
- [Client hooks](https://github.com/talha4565/metalmind-smcforge/blob/main/frontend-next/src/lib/hooks/useWebSocket.ts): data hooks · HTTP and websocket access layer used by pages and widgets · `frontend-next/src/lib/hooks/useWebSocket.ts`
- [NextAuth](https://github.com/talha4565/metalmind-smcforge/blob/main/frontend-next/src/app/api/auth/[...nextauth]/route.ts): auth wiring · Client-side auth/session integration · `frontend-next/src/app/api/auth/[...nextauth]/route.ts`

### API

Flask/SocketIO runtime control plane

- [Flask API](https://github.com/talha4565/metalmind-smcforge/blob/main/api/app/main.py): api app · Main backend surface and request entrypoint · `api/app/main.py`
- [Auth service](https://github.com/talha4565/metalmind-smcforge/blob/main/api/app/auth.py): auth endpoints · Login, registration, password, email, and security flows · `api/app/auth.py`
- [Domain routes](https://github.com/talha4565/metalmind-smcforge/blob/main/api/app/watchlist.py): api routes · Predictions, market price, SHAP, backtesting, profile, and watchlist endpoints · `api/app/watchlist.py`
- [Model runtime](https://github.com/talha4565/metalmind-smcforge/blob/main/api/app/model_loader.py): inference core · Model loading and prediction caching · `api/app/model_loader.py`
- [Explainability](https://github.com/talha4565/metalmind-smcforge/blob/main/api/app/shap_cache.py): shap cache · SHAP caching and explanation serving · `api/app/shap_cache.py`
- [ETL routes](https://github.com/talha4565/metalmind-smcforge/blob/main/api/app/etl_routes.py): integration routes · Backend surface for ETL-related operations · `api/app/etl_routes.py`

### ML Pipeline

Feature engineering, training, backtesting, and refresh jobs

- [ETL pipeline](https://github.com/talha4565/metalmind-smcforge/blob/main/etl/pipeline.py): data pipeline · Extraction, transformation, loading, and scheduling · `etl/pipeline.py`
- [Feature engine](https://github.com/talha4565/metalmind-smcforge/blob/main/features/pipeline.py): feature engineering · SMC, volume, multi-timeframe aggregation, and labels · `features/pipeline.py`
- [Training jobs](https://github.com/talha4565/metalmind-smcforge/blob/main/models/retrain.py): model training · XGBoost training and retraining for gold and silver workflows · `models/retrain.py`
- [Backtest engine](https://github.com/talha4565/metalmind-smcforge/blob/main/backtesting/engine.py): evaluation · Historical simulation of trading behavior · `backtesting/engine.py`
- [Scheduler](https://github.com/talha4565/metalmind-smcforge/blob/main/etl/scheduler.py): orchestration · Periodic refresh, alerts, and pipeline automation · `etl/scheduler.py`

### State & Artifacts

Database, reports, and generated files

- [SQL database](https://github.com/talha4565/metalmind-smcforge/blob/main/api/instance/metalmind_smc.db): relational state · Users, auth state, profiles, watchlists, and operational metadata · `api/instance/metalmind_smc.db`
- [Reports](https://github.com/talha4565/metalmind-smcforge/blob/main/reports/training_logs/gold_latest.json): artifacts · Prediction logs, training logs, and generated analysis outputs · `reports/training_logs/gold_latest.json`
- [Feature store](https://github.com/talha4565/metalmind-smcforge/blob/main/data/features/XAUUSD_features_latest.parquet): parquet artifacts · Model-ready feature files and other file-based analytical state · `data/features/XAUUSD_features_latest.parquet`

### Other components

- **Yahoo Finance**: external source · Market data feed used by ingestion
- **SocketIO**: realtime channel · Live update channel between backend and frontend

## Connections

- Next.js app → Auth pages: routes to · App shell exposes auth screens
- Next.js app → Dashboard: routes to · App shell exposes dashboard areas
- Next.js app → Backtest UI: routes to · App shell exposes backtesting view
- Dashboard → Domain widgets: renders · Dashboard composes charts, signals, SHAP, and risk widgets
- Backtest UI → Client hooks: loads data · Backtest UI fetches results and runs
- Dashboard → Client hooks: fetches via · Dashboard data access is mediated by hooks
- Client hooks → Flask API: HTTP to · Client hooks call backend endpoints
- Client hooks → SocketIO: subscribes to · Live updates flow over websocket
- NextAuth → Auth service: auth delegates · Client auth wiring depends on backend auth services
- Flask API → Auth service: mounts · Main app includes auth surface
- Flask API → Domain routes: mounts · Main app exposes trading domain endpoints
- Flask API → Model runtime: uses · Request handling loads models and caches predictions
- Flask API → Explainability: uses · Explainability is served from runtime cache
- Flask API → ETL routes: mounts · Backend exposes ETL integration routes
- Domain routes → SQL database: reads/writes · Profile and watchlist state persists in database
- Auth service → SQL database: reads/writes · Auth records and user state persist in database
- Model runtime → Feature store: reads · Inference consumes trained feature artifacts
- Model runtime → Reports: logs to · Predictions and runtime outputs are recorded
- Explainability → Reports: stores · Explanation outputs and plots are persisted
- ETL pipeline → Yahoo Finance: ingests from · Primary market data feed
- ETL pipeline → Feature engine: feeds · Extracted data moves into feature construction
- Feature engine → Feature store: writes · Engineered features land in parquet artifacts
- Feature engine → Training jobs: supplies · Training consumes model-ready features
- Training jobs → Reports: writes logs · Training metrics and run summaries are recorded
- Training jobs → Model runtime: produces models · Trained artifacts are loaded by runtime inference
- Backtest engine → Reports: writes results · Historical evaluations produce backtest reports
- Scheduler → ETL pipeline: orchestrates · Scheduled refreshes trigger ingestion and processing
- Scheduler → Training jobs: triggers · Scheduled retraining refreshes models
- Scheduler → Reports: alerts/logs · Operational events are recorded for monitoring
- SocketIO → Flask API: served by · Realtime channel terminates in backend
- ETL routes → ETL pipeline: bridges to · Backend routes can initiate pipeline actions