# tradytics/surpriver architecture

> Architecture diagram of the GitHub repository tradytics/surpriver, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tradytics/surpriver
- Repository: https://github.com/tradytics/surpriver

## Overview

1. **Overall shape**
   - This is a **single-command, batch analytics pipeline** rather than a service or web app.
   - The main entry point is `detection_engine.py`, which orchestrates data acquisition, feature engineering, anomaly scoring, optional backtesting, and result output.

2. **External data sources**
   - The system pulls market data from **Yahoo Finance** by default, with README support for **Binance** for crypto use cases.
   - Inputs are driven by ticker lists in `stocks/stocks.txt` or `stocks/cryptos.txt`, so the universe of symbols is file-based and configurable.

3. **Data loading boundary**
   - `data_loader.py` is the ingestion layer.
   - Its role is to fetch historical OHLCV-style time series for each symbol at the requested granularity (`1min` through `60min`) and apply basic universe filters such as minimum volume.

4. **Feature engineering boundary**
   - `feature_generator.py` is the transformation layer.
   - It converts raw price/volume bars into model-ready signals such as returns, rolling statistics, and technical indicators, using the `ta`-style indicator ecosystem described in the README.

5. **Anomaly detection core**
   - `detection_engine.py` is the analysis and ranking layer.
   - It scores each stock/bar for “unusual” price + volume behavior, then ranks symbols by anomaly score, where **lower scores mean more anomalous**.
   - The implementation depends on scientific Python libraries, with `scikit-learn` indicating a machine-learning-based scoring approach.

6. **Primary data flow**
   - Stock list file -> data fetch -> feature generation -> anomaly scoring -> ranked output.
   - This is the critical flow to draw: the engine consumes a universe of tickers, generates features from recent history, and produces the “top N anomalous stocks” list.

7. **Caching / persistence layer**
   - The `dictionaries/` folder is a local cache for serialized market data and features, saved as `.npy` dictionaries.
   - This cache supports two modes:
     - **Cold run**: download fresh data and optionally save it.
     - **Warm run**: reload saved dictionaries to skip network fetches and speed up repeated analysis.

8. **Historical testing path**
   - The engine has a second mode that reserves `future_bars` from the most recent history to evaluate whether anomalous symbols later moved more.
   - In this path, the pipeline branches after scoring into a measurement step that compares anomaly rank against future absolute price change and volatility.

9. **Output boundary**
   - Results can be printed to the console (`CLI`) or emitted as a JSON file.
   - For historical testing, the repo also produces a correlation/scatter-style analysis, which is documented in `figures/` and used to validate the scoring heuristic.

10. **Deployment/runtime**
   - The repo is designed for **Python 3** execution with dependencies installed from `requirements.txt`.
   - `dockerfile`, `docker-compose.yml.template`, and `entry_point.sh` provide a containerized execution path, but they do not change the architecture; they just package the same batch pipeline for reproducible runs.

11. **Non-functional boundaries**
   - The code is organized around local files and downloaded market data, not around a database or remote application state.
   - There is no persistent server process: each run is a self-contained analysis job over a selected universe and time window.

12. **Key architectural takeaway**
   - The repository is best diagrammed as a **three-stage analytics pipeline**:
     - **Ingest** market data for a configured symbol universe,
     - **Transform** bars into technical/features,
     - **Score and rank** anomalies, with optional caching and backtesting.

Main entry points: [`data_loader.py`](https://github.com/tradytics/surpriver/blob/master/data_loader.py) (Loader), [`detection_engine.py`](https://github.com/tradytics/surpriver/blob/master/detection_engine.py) (Engine), [`feature_generator.py`](https://github.com/tradytics/surpriver/blob/master/feature_generator.py) (Features), [`dockerfile`](https://github.com/tradytics/surpriver/tree/master/dockerfile) (Container)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Inputs"]
  node_stocks_file["Stock lists<br/>symbol universe<br/>[stocks.txt]"]
  node_cryptos_file["Crypto list<br/>symbol universe<br/>[cryptos.txt]"]
  node_yahoo_finance(("Yahoo Finance<br/>market source"))
  node_binance(("Binance<br/>market source"))
end

subgraph group_pipeline["Pipeline"]
  node_detection_engine{{"Engine<br/>orchestrator"}}
  node_data_loader["Loader<br/>ingestion<br/>[data_loader.py]"]
  node_feature_generator["Features<br/>transforms"]
  node_anomaly_scoring["Scoring<br/>ranker"]
  node_backtest_mode["Backtest<br/>evaluation"]
end

subgraph group_cache["Cache"]
  node_dictionaries_cache[("Dictionaries<br/>local cache")]
end

subgraph group_outputs["Outputs"]
  node_ranked_output["Ranked list<br/>result set"]
  node_json_output["JSON export<br/>report"]
  node_console_cli["CLI printout<br/>report"]
  node_figures["Figures<br/>analysis artifact"]
end

subgraph group_runtime["Runtime"]
  node_requirements["Requirements<br/>dependencies<br/>[requirements.txt]"]
  node_container_setup["Container<br/>runtime packaging"]
end

node_stocks_file -->|"symbols"| node_data_loader
node_cryptos_file -->|"symbols"| node_data_loader
node_yahoo_finance -->|"prices"| node_data_loader
node_binance -.->|"crypto bars"| node_data_loader
node_data_loader -->|"save/load"| node_dictionaries_cache
node_data_loader -->|"bars"| node_feature_generator
node_feature_generator -->|"features"| node_anomaly_scoring
node_detection_engine -->|"orchestrates"| node_data_loader
node_detection_engine -->|"orchestrates"| node_feature_generator
node_detection_engine -->|"orchestrates"| node_anomaly_scoring
node_anomaly_scoring -->|"top N"| node_ranked_output
node_ranked_output -->|"print"| node_console_cli
node_ranked_output -.->|"export"| node_json_output
node_anomaly_scoring -.->|"optional"| node_backtest_mode
node_backtest_mode -->|"validate"| node_figures
node_requirements -->|"install"| node_container_setup
node_container_setup -->|"run"| node_detection_engine

click node_stocks_file "https://github.com/tradytics/surpriver/blob/master/stocks/stocks.txt"
click node_cryptos_file "https://github.com/tradytics/surpriver/blob/master/stocks/cryptos.txt"
click node_detection_engine "https://github.com/tradytics/surpriver/blob/master/detection_engine.py"
click node_data_loader "https://github.com/tradytics/surpriver/blob/master/data_loader.py"
click node_feature_generator "https://github.com/tradytics/surpriver/blob/master/feature_generator.py"
click node_anomaly_scoring "https://github.com/tradytics/surpriver/blob/master/detection_engine.py"
click node_backtest_mode "https://github.com/tradytics/surpriver/blob/master/detection_engine.py"
click node_dictionaries_cache "https://github.com/tradytics/surpriver/tree/master/dictionaries"
click node_figures "https://github.com/tradytics/surpriver/tree/master/figures"
click node_requirements "https://github.com/tradytics/surpriver/blob/master/requirements.txt"
click node_container_setup "https://github.com/tradytics/surpriver/tree/master/dockerfile"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_stocks_file,node_cryptos_file,node_yahoo_finance,node_binance toneBlue
class node_detection_engine,node_data_loader,node_feature_generator,node_anomaly_scoring,node_backtest_mode toneAmber
class node_dictionaries_cache toneMint
class node_ranked_output,node_json_output,node_console_cli,node_figures toneRose
class node_requirements,node_container_setup toneIndigo
```

## Components

### Inputs

Configured symbol universes and external market data sources.

- [Stock lists](https://github.com/tradytics/surpriver/blob/master/stocks/stocks.txt): symbol universe · Ticker universes loaded from configurable text files for stocks or crypto. · `stocks/stocks.txt`
- [Crypto list](https://github.com/tradytics/surpriver/blob/master/stocks/cryptos.txt): symbol universe · Alternative ticker universe for crypto-oriented runs. · `stocks/cryptos.txt`
- **Yahoo Finance**: market source · Default external market data provider for historical OHLCV bars.
- **Binance**: market source · Alternative external source referenced for crypto workflows.

### Pipeline

Core batch analytics stages from ingestion to scoring and optional backtesting.

- [Engine](https://github.com/tradytics/surpriver/blob/master/detection_engine.py): orchestrator · Main command that coordinates loading, feature generation, scoring, ranking, and optional historical testing. · `detection_engine.py`
- [Loader](https://github.com/tradytics/surpriver/blob/master/data_loader.py): ingestion · Fetches historical bars, applies basic filters, and handles warm/cold data access. · `data_loader.py`
- [Features](https://github.com/tradytics/surpriver/blob/master/feature_generator.py): transforms · Converts bars into model-ready indicators, returns, and rolling statistics. · `feature_generator.py`
- [Scoring](https://github.com/tradytics/surpriver/blob/master/detection_engine.py): ranker · Computes anomaly scores and ranks symbols by unusual price and volume behavior. · `detection_engine.py`
- [Backtest](https://github.com/tradytics/surpriver/blob/master/detection_engine.py): evaluation · Reserves future bars and measures whether high-anomaly symbols later move more. · `detection_engine.py`

### Cache

Local serialized dictionaries used to speed warm runs.

- [Dictionaries](https://github.com/tradytics/surpriver/tree/master/dictionaries): local cache · Serialized numpy dictionaries for saved market data and features. · `dictionaries`

### Outputs

Console, JSON, and historical evaluation artifacts.

- **Ranked list**: result set · Top anomalous symbols and their scores produced by the main run.
- **JSON export**: report · Optional machine-readable output file for downstream use.
- **CLI printout**: report · Human-readable console output for interactive runs.
- [Figures](https://github.com/tradytics/surpriver/tree/master/figures): analysis artifact · Scatter and correlation-style validation visuals for historical testing. · `figures`

### Runtime

Packaging and execution entry points for reproducible Python runs.

- [Requirements](https://github.com/tradytics/surpriver/blob/master/requirements.txt): dependencies · Python package set used by the batch analytics pipeline. · `requirements.txt`
- [Container](https://github.com/tradytics/surpriver/tree/master/dockerfile): runtime packaging · Docker and entry scripts used to run the same pipeline reproducibly. · `dockerfile`

## Connections

- Stock lists → Loader: symbols · Ticker universe drives ingestion for stock runs.
- Crypto list → Loader: symbols · Ticker universe drives ingestion for crypto runs.
- Yahoo Finance → Loader: prices · Default external market data source.
- Binance → Loader: crypto bars · Alternative source for crypto-oriented analysis.
- Loader → Dictionaries: save/load · Warm runs reload saved dictionaries; cold runs can persist fetched data.
- Loader → Features: bars · Historical OHLCV data flows into feature construction.
- Features → Scoring: features · Model-ready indicators feed anomaly ranking.
- Engine → Loader: orchestrates · Main command coordinates ingestion.
- Engine → Features: orchestrates · Main command coordinates transformation.
- Engine → Scoring: orchestrates · Main command coordinates scoring and ranking.
- Scoring → Ranked list: top N · Produces ranked anomalous symbols.
- Ranked list → CLI printout: print · Default human-facing output.
- Ranked list → JSON export: export · Optional structured export.
- Scoring → Backtest: optional · Historical mode evaluates whether anomalous symbols later move more.
- Backtest → Figures: validate · Correlation and scatter analyses document scoring behavior.
- Requirements → Container: install · Containerized execution uses the Python dependency set.
- Container → Engine: run · Packaging layer launches the same batch pipeline.