How akash-kumar5/cryptomarket_regime_classifier works
1. Repository purpose
- This is a crypto market regime intelligence pipeline, not a general trading bot.
- Its core job is to turn Binance OHLCV data into regime labels and next-regime predictions for downstream strategy selection, sizing, and risk control.
2. End-to-end architecture
- The data path is: Binance OHLCV input → feature engineering → PCA-style reduction for regime discovery → HMM for latent regime labeling → LSTM for sequence prediction → exported models/scalers for live use.
- The README describes a two-stage ML design: unsupervised regime discovery first, then supervised temporal prediction.
3. Data acquisition boundary
- src/data_fetcher.py and src/fetch_binance_klines.py are the ingestion layer.
- They likely encapsulate Binance API access and multi-timeframe OHLCV retrieval, with the repository focused on 5m data plus 15m context.
- dashboard/pages/1_Fetch_data.py exposes this step in the UI, suggesting manual or guided data refresh.
4. Data cleaning and normalization
- src/data_cleaner.py is the preprocessing boundary between raw market data and model-ready inputs.
- This layer is important because all downstream modeling depends on time alignment, missing-data handling, and feature consistency across timeframes.
5. Feature engineering subsystem
- src/compute_features.py and dashboard/pages/2_Compute_Features.py are the feature generation stage.
- The README says features cover momentum, trend, and volatility, and they are built from multi-timeframe OHLCV.
- features.txt likely records the selected feature set used for training/inference parity.
6. Regime discovery subsystem
- src/regime_label.py and src/hmm_tuner.py implement the HMM-based latent regime discovery.
- The pipeline uses PCA-reduced features before HMM fitting and selects a 6-state model using lowest BIC, per the README.
- The saved .joblib HMM files in models/ are the persisted regime discovery artifacts.
7. Supervised prediction subsystem
- src/lstm_model.py and src/lstm_tuner.py handle the LSTM that learns temporal patterns over HMM-derived labels.
- The LSTM is the predictive layer that outputs the current regime and likely future regime probabilities.
- models/lstm_regime_model.keras, models/scaler.joblib, and models/lstm_model_metadata.json are the deployable outputs of this stage.
8. Live inference boundary
- src/live_inference.py and dashboard/pages/7_Live_Implementation.py represent the runtime path for using trained artifacts on fresh data.
- This is the integration layer that consumes the exported model/scaler assets from models/ and produces live regime classification results.
9. Dashboard as orchestration and observability
- dashboard/app.py plus the numbered pages create a Streamlit-based workflow UI.
- The page sequence mirrors the pipeline: fetch data, compute features, classify regimes, train models, tune HMM, tune LSTM, and validate live usage.
- plot_regime.py likely supports visualization of regime transitions and model outputs.
10. Training artifacts and experiment tracking
- tuner_logs/ stores Keras Tuner/HMM tuning outputs, including trial configs, checkpoints, and oracle metadata.
- This directory is a clear separation between reproducible experiments and the deployed artifacts in models/.
11. Operational model of the repo
- The repo is organized around a batch research workflow with persistent artifacts, not a single always-on service.
- The key state transitions are: raw Binance data in, engineered features out, trained regime models saved, then reused by live inference and dashboard pages.
12. Architecture diagram hint
- A good diagram should show three layers: data ingestion, model pipeline, and consumption/UI.
- The important storage boundaries are models/ for deployable artifacts and tuner_logs/ for experimentation history.
Main entry points: app.py Dashboard · compute_features.py Feature Engine · regime_label.py Regime Labeling · lstm_tuner.py LSTM Tuner
Components 22
Ingestion
Binance data acquisition and raw-to-clean market data preparation.
- Binance Fetch · Binance OHLCV access and kline retrieval boundary.
src/data_fetcher.py - Kline Loader · Kline download and multi-timeframe retrieval implementation.
src/fetch_binance_klines.py - Cleaner · Alignment, missing-data handling, and model-ready normalization boundary.
src/data_cleaner.py
Modeling
Feature engineering, regime discovery, sequence prediction, and training artifacts.
- Feature Engine · Momentum, trend, and volatility feature construction across timeframes.
src/compute_features.py - Feature Set · Recorded feature list used to keep training and inference consistent.
features.txt - Regime Labeling · PCA-style reduction and HMM-based latent regime discovery.
src/regime_label.py - HMM Tuner · Chooses regime model configuration, including state count and BIC-based fit.
src/hmm_tuner.py - HMM Artifacts · Saved regime discovery models used for downstream labeling and inference.
models/hmm_BTCUSDT_combined_klines_20221106_20251105_features_states6.joblib - LSTM Model · Trains the supervised temporal model over regime labels.
src/lstm_model.py - LSTM Tuner · Tunes sequence model hyperparameters and training setup.
src/lstm_tuner.py - LSTM Artifacts · Exported LSTM model, scaler, and metadata for live use.
models/lstm_regime_model.keras - Tuning Logs · Keras Tuner and HMM trial histories, checkpoints, and oracle state.
tuner_logs/lstm_tuning_BTCUSDT_combined_klines_20221107_20251106_states6_labeled/oracle.json
Serving & UI
Dashboard orchestration, live inference, and visualization.
- Dashboard · Entry point for the numbered workflow pages and model operations.
dashboard/app.py - Fetch Data · Dashboard step for refreshing Binance OHLCV data.
dashboard/pages/1_Fetch_data.py - Compute Features · Dashboard step for generating model features from market data.
dashboard/pages/2_Compute_Features.py - Regime Classifier · Dashboard step for labeling regimes and inspecting classification output.
dashboard/pages/3_Regime_Classifier.py - Model Training · Dashboard step for running model training workflows.
dashboard/pages/4_Model_Training.py - HMM Tuning · Dashboard step for tuning the hidden Markov model.
dashboard/pages/5_HMM_Tuning.py - LSTM Tuning · Dashboard step for tuning the sequence model.
dashboard/pages/6_LSTM_Tuning.py - Live Use · Dashboard step for validating live inference on fresh data.
dashboard/pages/7_Live_Implementation.py - Regime Plot · Visualization helper for regime transitions and outputs.
plot_regime.py - Live Inference · Consumes fresh data and persisted artifacts to produce regime predictions.
src/live_inference.py
Connections 28
- Dashboard → Fetch Data: orchestrates · The UI exposes the data refresh workflow.
- Dashboard → Compute Features: orchestrates · The UI exposes feature generation.
- Dashboard → Regime Classifier: orchestrates · The UI exposes regime labeling.
- Dashboard → Model Training: orchestrates · The UI exposes model training.
- Dashboard → HMM Tuning: orchestrates · The UI exposes HMM tuning.
- Dashboard → LSTM Tuning: orchestrates · The UI exposes LSTM tuning.
- Dashboard → Live Use: orchestrates · The UI exposes live validation.
- Binance Fetch → Kline Loader: retrieves · Raw ingestion is backed by kline retrieval logic.
- Kline Loader → Cleaner: raw data · Downloaded OHLCV is prepared for modeling.
- Cleaner → Feature Engine: cleaned data · Cleaned market data feeds feature generation.
- Feature Engine → Feature Set: defines · The selected feature set documents training/inference parity.
- Feature Engine → Regime Labeling: features · Engineered features are reduced and labeled into regimes.
- Regime Labeling → HMM Tuner: fits · HMM tuning searches regime configurations.
- HMM Tuner → HMM Artifacts: persists · Best regime model is saved for reuse.
- HMM Artifacts → LSTM Model: labels · HMM regime outputs become supervised targets or inputs.
- LSTM Model → LSTM Tuner: tunes · Sequence model hyperparameters are optimized.
- LSTM Tuner → LSTM Artifacts: exports · Final LSTM, scaler, and metadata are stored for deployment.
- Tuning Logs → LSTM Tuner: records · Trial history and checkpoints support repeated tuning runs.
- Live Inference → LSTM Artifacts: loads · Runtime inference consumes the exported LSTM assets.
- Live Inference → HMM Artifacts: loads · Runtime inference also needs the persisted regime model.
- Fetch Data → Binance Fetch: invokes · The dashboard can trigger fresh data acquisition.
- Compute Features → Feature Engine: invokes · The dashboard can trigger feature generation.
- Regime Classifier → Regime Labeling: invokes · The dashboard can run regime classification.
- Model Training → LSTM Model: invokes · The dashboard can run model training.
- HMM Tuning → HMM Tuner: invokes · The dashboard can run HMM tuning.
- LSTM Tuning → LSTM Tuner: invokes · The dashboard can run LSTM tuning.
- Live Use → Live Inference: invokes · The dashboard can run live implementation checks.
- Regime Plot → Regime Labeling: visualizes · Regime plots present labeling and transitions.