tradytics/surpriver

1. Overall shape

- This is a single-command, batch analytics pipeline rather than a service or web app.

- The main entry point is detection_engine.py, which orchestrates data acquisition, feature engineering, anomaly scoring, optional backtesting, and result output.

2. External data sources

- The system pulls market data from Yahoo Finance by default, with README support for Binance for crypto use cases.

- Inputs are driven by ticker lists in stocks/stocks.txt or stocks/cryptos.txt, so the universe of symbols is file-based and configurable.

3. Data loading boundary

- data_loader.py is the ingestion layer.

- Its role is to fetch historical OHLCV-style time series for each symbol at the requested granularity (1min through 60min) and apply basic universe filters such as minimum volume.

4. Feature engineering boundary

- feature_generator.py is the transformation layer.

- It converts raw price/volume bars into model-ready signals such as returns, rolling statistics, and technical indicators, using the ta-style indicator ecosystem described in the README.

5. Anomaly detection core

- detection_engine.py is the analysis and ranking layer.

- It scores each stock/bar for “unusual” price + volume behavior, then ranks symbols by anomaly score, where lower scores mean more anomalous.

- The implementation depends on scientific Python libraries, with scikit-learn indicating a machine-learning-based scoring approach.

6. Primary data flow

- Stock list file -> data fetch -> feature generation -> anomaly scoring -> ranked output.

- This is the critical flow to draw: the engine consumes a universe of tickers, generates features from recent history, and produces the “top N anomalous stocks” list.

7. Caching / persistence layer

- The dictionaries/ folder is a local cache for serialized market data and features, saved as .npy dictionaries.

- This cache supports two modes:

- Cold run: download fresh data and optionally save it.

- Warm run: reload saved dictionaries to skip network fetches and speed up repeated analysis.

8. Historical testing path

- The engine has a second mode that reserves future_bars from the most recent history to evaluate whether anomalous symbols later moved more.

- In this path, the pipeline branches after scoring into a measurement step that compares anomaly rank against future absolute price change and volatility.

9. Output boundary

- Results can be printed to the console (CLI) or emitted as a JSON file.

- For historical testing, the repo also produces a correlation/scatter-style analysis, which is documented in figures/ and used to validate the scoring heuristic.

10. Deployment/runtime

- The repo is designed for Python 3 execution with dependencies installed from requirements.txt.

- dockerfile, docker-compose.yml.template, and entry_point.sh provide a containerized execution path, but they do not change the architecture; they just package the same batch pipeline for reproducible runs.

11. Non-functional boundaries

- The code is organized around local files and downloaded market data, not around a database or remote application state.

- There is no persistent server process: each run is a self-contained analysis job over a selected universe and time window.

12. Key architectural takeaway

- The repository is best diagrammed as a three-stage analytics pipeline:

- Ingest market data for a configured symbol universe,

- Transform bars into technical/features,

- Score and rank anomalies, with optional caching and backtesting.