talha4565/metalmind-smcforge

1. System shape

This is a three-plane trading platform: a Next.js dashboard, a Flask/SocketIO API, and a Python ML/data pipeline. The dashboard is the presentation layer, the API is the runtime control plane, and the pipeline owns ingestion, feature engineering, training, backtesting, and scheduled refreshes.

2. Frontend boundary

frontend-next/ is a Next.js app-router frontend with separate areas for auth, dashboard, backtest, risk, and watchlist. It uses reusable UI components plus domain components for TradingView charts, SHAP explanations, signal cards, and risk metrics. Client-side hooks in src/lib/hooks mediate all data access to the backend and real-time updates.

3. API boundary

api/app/ is the main backend surface. It exposes endpoints for health, predictions, market price, SHAP feature importance, backtesting, auth, and watchlist/profile operations. The API also contains model loading, prediction caching, ETL-related routes, and error handling middleware, so it acts as the integration point between UI, models, and stored data.

4. Real-time and session flow

The frontend consumes the API over HTTP and also has a WebSocket path via useWebSocket and Flask-SocketIO. That makes the app both a request/response dashboard and a live terminal for price/prediction updates. Authentication is handled through the frontend auth pages plus backend auth services, with NextAuth wiring on the client side.

5. Data ingestion and refresh

The ETL subsystem under etl/ pulls market data from Yahoo Finance and CSV sources using extractor abstractions. Data then moves through transformers for quality checks and feature generation, and into loaders that append to CSV, load to DB, or populate a feature store. The scheduler coordinates periodic refreshes and alerting.

6. Feature engineering pipeline

features/ is where domain-specific feature construction happens. It includes Smart Money Concepts features, volume features, multi-timeframe aggregation, and label generation. This is the core modeling boundary: raw OHLCV data becomes model-ready sequences with the 90-feature schema referenced in the README.

7. Model training and retraining

models/ contains training and retraining scripts for the XGBoost classifiers, with separate support for gold and silver workflows. The training pipeline consumes engineered features from data/features/*.parquet, writes model artifacts, and records metrics into reports/training_logs/. run_pipeline.py appears to be the orchestration entry point for backfill, update, retrain, and scheduled modes.

8. Inference and explainability

At runtime, the API loads trained models through api/app/model_loader.py and serves predictions to the dashboard. SHAP computation is isolated through api/app/shap_cache.py and explainability/shap_analyzer.py, so explainability is part of the inference path rather than an offline report-only step. Prediction logs are persisted under reports/predictions/.

9. Backtesting subsystem

backtesting/engine.py provides a separate evaluation path from live inference. The API exposes backtest run/results endpoints, while the engine likely operates on historical feature/data inputs to simulate trading with slippage, commission, and TP/SL rules. This subsystem is important because it validates model behavior against historical market conditions rather than live predictions.

10. Persistence and state

The project uses a relational database through SQLAlchemy/Alembic migrations, with local SQLite files present in instance/ and the repo root, while the README also documents PostgreSQL for containerized deployment. The database boundary likely stores users, auth state, profiles, watchlists, and operational metadata, while bulk market/history data stays in files and parquet artifacts.

11. Operational artifacts

The repository keeps generated data and results in file-based stores: data/features/ for model inputs, reports/ for training logs and prediction history, and miscellaneous_archive/ for older generated reports. This means the architecture is hybrid: transactional state in the DB, but large analytical artifacts on disk.

12. Deployment and orchestration

docker-compose.yml, the Dockerfiles, start_api.py, run.py, and run_pipeline.py are the main deployment/run entry points. There is also an Airflow DAG in airflow/dags/retrain.py, suggesting the training lifecycle can be automated externally. The overall deployment model is containerized, with the frontend, API, and database exposed as separate services.

13. External integrations

The important external dependencies that affect architecture are Yahoo Finance for market data, SMTP for email alerts, TradingView embedded charts in the frontend, and likely a message channel via SocketIO for live updates. These integrations are not peripheral; they directly shape the runtime flow from ingestion to signal delivery to user notification.

14. Primary end-to-end flow

The main loop is: fetch market data → build features → train/retrain models → serve predictions with SHAP → log outcomes → backtest/monitor → alert users when confidence thresholds are met. The dashboard is the consumer of this loop, while the pipeline and API are the producers and state managers.