# rubenke/indicator-stock-prediction-study architecture

> Architecture diagram of the GitHub repository rubenke/indicator-stock-prediction-study, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rubenke/indicator-stock-prediction-study
- Repository: https://github.com/rubenke/indicator-stock-prediction-study

## Overview

1. **Overall shape**
   - This repo is an offline research harness for algorithmic trading experiments.
   - It has two largely separate experiment tracks: **rule-based strategy backtests** and **supervised ML prediction/backtests**.
   - Both tracks consume the same historical OHLCV market data and share the same config-driven cost/risk assumptions.

2. **Configuration as the control plane**
   - `config/config.yaml` is the main source of truth for instruments, intervals, parameter grids, commissions, slippage, risk sizing, and ML settings.
   - Most scripts read from this file rather than hardcoding symbols or tuning values, which keeps experiments reproducible and diagram-friendly as a single upstream dependency.

3. **Market data ingestion boundary**
   - `utils/data_loader.py` downloads OHLCV data from `yfinance` and stores it as Parquet in `data/raw/`.
   - The loader truncates datasets to a bounded recent window, so the raw-data layer is a curated local cache rather than a full market history warehouse.
   - This raw dataset is the shared input for both the classic backtesting pipeline and ML feature generation.

4. **Classic strategy subsystem**
   - `strategies/` contains the rule-based trading logic: breakout, mean reversion, trend-following, VWAP, RSI, DMAC, SMC-style structure sweep, and risk-managed variants.
   - `run_all.py` orchestrates batch backtests across strategy definitions, instruments, intervals, and parameter grids from config.
   - The output is aggregated experiment rows in `database/results.parquet`, one row per strategy/asset/interval/parameter combination.

5. **Single-strategy execution path**
   - `utils/single_strategy_runner.py` provides a focused entry point for running one strategy with full per-bar outputs.
   - This is a diagnostic boundary: it returns summary stats, equity curve, input price data, and the underlying strategy result object for deeper inspection.
   - It uses the same cost/risk settings as the batch runner unless overridden.

6. **ML feature and dataset subsystem**
   - `ml/features.py` and `ml/datasets.py` build the supervised learning view from raw OHLCV data.
   - The ML target is next-candle direction, and the pipeline keeps per-dataset caches under `data/features/`, including a manifest of usable datasets.
   - This layer is explicitly separate from the rule-based strategies: it transforms market data into model-ready tabular features and labels.

7. **ML model training and evaluation flow**
   - `run_ml.py` is the ML orchestration entry point, supporting feature preparation, single-dataset runs, quick runs, and full leave-one-dataset-out evaluation.
   - `ml/train_eval.py`, `ml/models.py`, `ml/persistence.py`, `ml/utils.py`, and `ml/types.py` together handle model training, scoring, signal generation, artifact saving, and typed experiment metadata.
   - The evaluation design is group-based by dataset: train on all but one dataset, then test on the held-out dataset, so the test boundary is by symbol/interval dataset rather than random candles.

8. **ML artifact and results layout**
   - Trained models and run metadata are stored under `models/ml_registry/{run_id}/...`.
   - Per-candle predictions and signals are written to `analysis/results/ml/{run_id}/...`.
   - Aggregate ML metrics land in `database/ml_results.csv`, making the ML outputs split across three layers: model registry, detailed run outputs, and summary database.

9. **Analysis and reporting subsystem**
   - `analysis/ml_results_analyzer.py` is a CLI reporting tool that reads the ML summary table, ranks runs, exports tables/plots, and produces a markdown report.
   - The notebooks in `analysis/` and `utils/` are exploratory consumers of the same saved artifacts, not part of the production pipeline.
   - This creates a clear downstream analysis boundary: both strategy families write durable results, and analysis tools read those results without rerunning experiments.

10. **Risk optimization loop**
   - `optimize_risk_params.py` is a separate experimental loop that searches over risk-sizing settings.
   - It writes outcomes to `analysis/results/risk_optimization.csv` and can optionally update `config/config.yaml`.
   - In architecture terms, this is a feedback path from analysis back into the central config, affecting both classic and ML backtests.

11. **Persistence and storage boundaries**
   - The repo uses local files as its main persistence mechanism: Parquet for raw and classic results, CSV for summary tables, JSON for manifests/metadata, and joblib-style model artifacts inside the ML registry.
   - `database/` is the result store for aggregated experiment tables, while `analysis/results/` holds exported analysis artifacts and detailed ML predictions.
   - The separation between raw data, feature cache, model registry, and analysis outputs is important for drawing accurate storage boundaries.

12. **Tests and validation**
   - `tests/test_ml_pipeline.py` and `tests/test_ml_results_analyzer.py` validate the ML pipeline and reporting path.
   - These tests suggest the most important contract surfaces are: feature preparation/training correctness and the ability to parse and summarize experiment outputs.

13. **What to show in a diagram**
   - A clean architecture diagram should center on three main flows:
     - `yfinance -> data/raw -> classic strategies -> database/results.parquet`
     - `yfinance -> data/raw -> ml feature cache -> model training/eval -> models/ml_registry + analysis/results/ml + database/ml_results.csv`
     - `results databases -> analysis notebooks/CLI analyzers`
   - `config/config.yaml` should sit above both pipelines as the shared control input, with `optimize_risk_params.py` feeding back into it.

Main entry points: [`ml/train_eval.py`](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/ml/train_eval.py) (Train/eval core), [`config/config.yaml`](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/config/config.yaml) (Config), [`database/ml_results.csv`](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/database/ml_results.csv) (ML summary), [`strategies/atr_volatility_breakout.py`](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/strategies/atr_volatility_breakout.py) (Strategies)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Control plane"]
  node_config["Config<br/>yaml settings<br/>[config.yaml]"]
  node_risk_opt["Risk optimizer<br/>tuning loop"]
end

subgraph group_data["Market data"]
  node_data_loader["Data loader<br/>ingestion<br/>[data_loader.py]"]
  node_raw_data[("Raw cache<br/>parquet cache")]
end

subgraph group_classic["Classic strategies"]
  node_strategy_suite["Strategies<br/>rule-based signals"]
  node_run_all["Batch runner<br/>backtest orchestrator<br/>[run_all.py]"]
  node_single_runner["Single runner<br/>diagnostic runner"]
  node_classic_results[("Strategy results<br/>parquet results<br/>[ml_results.parquet]")]
end

subgraph group_ml["ML pipeline"]
  node_ml_features["Feature builder<br/>tabular features<br/>[features.py]"]
  node_ml_datasets["Dataset cache<br/>feature cache<br/>[datasets.py]"]
  node_run_ml["ML runner<br/>experiment orchestrator<br/>[run_ml.py]"]
  node_ml_core["Train/eval core<br/>training stack<br/>[train_eval.py]"]
  node_ml_registry[("Model registry<br/>artifact store")]
  node_ml_outputs[("ML outputs<br/>predictions store")]
  node_ml_summary["ML summary<br/>csv results<br/>[ml_results.csv]"]
end

subgraph group_analysis["Analysis"]
  node_ml_analyzer["ML analyzer<br/>reporting CLI"]
  node_analysis_notebooks_ml["ML notebooks<br/>exploration"]
  node_analysis_notebooks_nonml["Non-ML analysis<br/>exploration"]
  node_test_ml_pipeline["Pipeline test"]
  node_test_ml_analyzer["Analyzer test"]
end

node_config -->|"controls"| node_run_all
node_config -->|"controls"| node_run_ml
node_config -->|"controls"| node_single_runner
node_config -->|"searches"| node_risk_opt
node_risk_opt -.->|"updates"| node_config
node_data_loader -->|"writes"| node_raw_data
node_raw_data -->|"feeds"| node_strategy_suite
node_raw_data -->|"feeds"| node_ml_features
node_strategy_suite -->|"executes"| node_run_all
node_run_all -->|"writes"| node_classic_results
node_strategy_suite -->|"executes"| node_single_runner
node_single_runner -.->|"inspects"| node_classic_results
node_ml_features -->|"builds"| node_ml_datasets
node_ml_datasets -->|"consumed by"| node_run_ml
node_run_ml -->|"orchestrates"| node_ml_core
node_ml_core -->|"writes"| node_ml_registry
node_ml_core -->|"writes"| node_ml_outputs
node_ml_core -->|"writes"| node_ml_summary
node_ml_summary -->|"reads"| node_ml_analyzer
node_classic_results -->|"reads"| node_analysis_notebooks_nonml
node_ml_outputs -->|"reads"| node_analysis_notebooks_ml
node_ml_summary -->|"reads"| node_analysis_notebooks_ml
node_ml_summary -.->|"validates"| node_test_ml_analyzer
node_ml_core -.->|"validates"| node_test_ml_pipeline

click node_config "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/config/config.yaml"
click node_risk_opt "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/optimize_risk_params.py"
click node_data_loader "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/utils/data_loader.py"
click node_strategy_suite "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/strategies/atr_volatility_breakout.py"
click node_run_all "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/run_all.py"
click node_single_runner "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/utils/single_strategy_runner.py"
click node_classic_results "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/database/ml_results.parquet"
click node_ml_features "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/ml/features.py"
click node_ml_datasets "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/ml/datasets.py"
click node_run_ml "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/run_ml.py"
click node_ml_core "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/ml/train_eval.py"
click node_ml_summary "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/database/ml_results.csv"
click node_ml_analyzer "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/analysis/ml_results_analyzer.py"
click node_analysis_notebooks_ml "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/analysis/ml_results_analysis.ipynb"
click node_analysis_notebooks_nonml "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/analysis/non_ml_full_analysis.ipynb"
click node_test_ml_pipeline "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/tests/test_ml_pipeline.py"
click node_test_ml_analyzer "https://github.com/rubenke/indicator-stock-prediction-study/blob/main/tests/test_ml_results_analyzer.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_config,node_risk_opt toneBlue
class node_data_loader,node_raw_data toneAmber
class node_strategy_suite,node_run_all,node_single_runner,node_classic_results toneMint
class node_ml_features,node_ml_datasets,node_run_ml,node_ml_core,node_ml_registry,node_ml_outputs,node_ml_summary toneRose
class node_ml_analyzer,node_analysis_notebooks_ml,node_analysis_notebooks_nonml,node_test_ml_pipeline,node_test_ml_analyzer toneIndigo
```

## Components

### Control plane

Shared configuration and feedback loops that steer experiments.

- [Config](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/config/config.yaml): yaml settings · Main source of truth for instruments, intervals, costs, risk sizing, and ML settings. · `config/config.yaml`
- [Risk optimizer](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/optimize_risk_params.py): tuning loop · Searches risk-sizing settings and can feed updates back into the config. · `optimize_risk_params.py`

### Market data

Raw data ingestion and cached feature datasets.

- [Data loader](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/utils/data_loader.py): ingestion · Downloads OHLCV data from yfinance and stores bounded recent windows as local Parquet cache. · `utils/data_loader.py`
- **Raw cache**: parquet cache · Curated local OHLCV dataset used by both strategy and ML pipelines.

### Classic strategies

Rule-based strategy backtesting and single-run diagnostics.

- [Strategies](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/strategies/atr_volatility_breakout.py): rule-based signals · Collection of breakout, mean reversion, trend, VWAP, RSI, DMAC, and structure-based strategies. · `strategies/atr_volatility_breakout.py`
- [Batch runner](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/run_all.py): backtest orchestrator · Runs strategy backtests across instruments, intervals, and parameter grids from config. · `run_all.py`
- [Single runner](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/utils/single_strategy_runner.py): diagnostic runner · Executes one strategy and returns detailed per-bar outputs and summary stats. · `utils/single_strategy_runner.py`
- [Strategy results](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/database/ml_results.parquet): parquet results · Aggregated experiment rows for classic strategy runs. · `database/ml_results.parquet`

### ML pipeline

Supervised feature generation, model training, and evaluation.

- [Feature builder](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/ml/features.py): tabular features · Builds supervised features and next-candle labels from raw OHLCV data. · `ml/features.py`
- [Dataset cache](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/ml/datasets.py): feature cache · Stores reusable ML datasets, manifests, and prepared tabular inputs. · `ml/datasets.py`
- [ML runner](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/run_ml.py): experiment orchestrator · Coordinates feature preparation, quick runs, single-dataset runs, and leave-one-dataset-out evaluation. · `run_ml.py`
- [Train/eval core](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/ml/train_eval.py): training stack · Model training, scoring, signal generation, persistence, utilities, and typed metadata. · `ml/train_eval.py`
- **Model registry**: artifact store · Stored models and run metadata under per-run directories.
- **ML outputs**: predictions store · Per-candle predictions and signals written for each ML run.
- [ML summary](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/database/ml_results.csv): csv results · Aggregate ML metrics for ranking and reporting. · `database/ml_results.csv`

### Analysis

Reporting, notebooks, and downstream consumers of saved results.

- [ML analyzer](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/analysis/ml_results_analyzer.py): reporting CLI · Ranks runs, exports tables and plots, and writes markdown reports from ML summary results. · `analysis/ml_results_analyzer.py`
- [ML notebooks](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/analysis/ml_results_analysis.ipynb): exploration · Exploratory consumers of saved ML artifacts and summary tables. · `analysis/ml_results_analysis.ipynb`
- [Non-ML analysis](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/analysis/non_ml_full_analysis.ipynb): exploration · Exploratory consumers of classic strategy outputs and broader offline analysis. · `analysis/non_ml_full_analysis.ipynb`
- [Pipeline test](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/tests/test_ml_pipeline.py): test · Validates feature preparation and training pipeline contracts. · `tests/test_ml_pipeline.py`
- [Analyzer test](https://github.com/rubenke/indicator-stock-prediction-study/blob/main/tests/test_ml_results_analyzer.py): test · Validates parsing and summarization of experiment outputs. · `tests/test_ml_results_analyzer.py`

## Connections

- Config → Batch runner: controls · Strategy universe, grids, and costs are read from config.
- Config → ML runner: controls · ML settings and dataset/evaluation options come from config.
- Config → Single runner: controls · Shares cost and risk assumptions with the focused runner.
- Config → Risk optimizer: searches · Risk optimization evaluates settings against the central config.
- Risk optimizer → Config: updates · Optional feedback path writes improved risk parameters back to config.
- Data loader → Raw cache: writes · Downloads and caches OHLCV data as local Parquet.
- Raw cache → Strategies: feeds · Classic strategies consume the shared historical market data.
- Raw cache → Feature builder: feeds · ML feature generation starts from the same raw OHLCV cache.
- Strategies → Batch runner: executes · Batch runner instantiates strategies across parameter grids.
- Batch runner → Strategy results: writes · Aggregated strategy experiment rows are persisted.
- Strategies → Single runner: executes · Single-run path uses the same strategy implementations for diagnostics.
- Single runner → Strategy results: inspects · Detailed outputs complement the aggregated results store.
- Feature builder → Dataset cache: builds · Prepared features and labels are cached as reusable datasets.
- Dataset cache → ML runner: consumed by · Prepared datasets drive the ML orchestration flow.
- ML runner → Train/eval core: orchestrates · Entry point delegates training, evaluation, and artifact persistence.
- Train/eval core → Model registry: writes · Trained models and run metadata are saved per run.
- Train/eval core → ML outputs: writes · Per-candle predictions and signals are exported.
- Train/eval core → ML summary: writes · Aggregate metrics are appended for reporting.
- ML summary → ML analyzer: reads · Analyzer ranks and summarizes runs from the summary table.
- Strategy results → Non-ML analysis: reads · Notebooks inspect classic experiment outputs.
- ML outputs → ML notebooks: reads · Notebooks explore detailed ML predictions and signals.
- ML summary → ML notebooks: reads · Analysis notebooks also consume summary metrics.
- ML summary → Analyzer test: validates · Tests the reporting contract against stored results.
- Train/eval core → Pipeline test: validates · Tests feature preparation and training behavior.
