# sherlock718/stock-fraud-screener architecture

> Architecture diagram of the GitHub repository sherlock718/stock-fraud-screener, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sherlock718/stock-fraud-screener
- Repository: https://github.com/sherlock718/stock-fraud-screener

## Overview

1. **Overall shape**
   This repo is a research-to-production pipeline for multi-factor equity selection. The architecture is split into: raw data ingestion and feature generation (`pipeline/`), offline analytics and ML/backtesting (`scripts/`), reusable scoring logic (`alpha/` and `src/`), and delivery surfaces (`app_v2.py` Streamlit UI plus `api/` FastAPI).

2. **Data ingestion boundary**
   `pipeline/` is the ETL layer and is explicitly described as “raw data ingestion only.” It pulls company/ticker data and filings across multiple markets, then builds annual company snapshots and derived features. Country-specific step files (`step1_fetch_tickers_*`, `step2_build_snapshots_*`) show the pipeline is market-partitioned rather than a single global feed.

3. **Feature and signal construction**
   Shared feature logic lives in `pipeline/feature_library.py`, which acts as the single source of truth for computed factors. Supporting modules like `fraud_signals.py`, `value_metrics.py`, `market_cap_filter.py`, and the `enrich_*` scripts add domain-specific signals such as fraud risk, market signals, governance, insider activity, and market capitalization filters.

4. **Canonical dataset build**
   The core pipeline output is a cleaned historical dataset, merged from multi-market snapshots into parquet artifacts under `data/`. The README indicates `historical_dataset_clean.parquet` is the main training/scoring input, with related intermediate assets like snapshots, price history, and benchmark returns stored alongside it. This is the handoff boundary between ETL and model/research layers.

5. **Modeling and research layer**
   `scripts/` contains the offline research stack: model training (`train_models.py`, `train_regression_model.py`, `tune_models.py`), feature selection, out-of-fold scoring, factor research, and bias audits. The ML stack is centered on LightGBM, with additional tuning/ensembling via Optuna and CatBoost in some workflows. This layer consumes curated data only; it does not fetch raw filings itself.

6. **Strategy and portfolio construction**
   Backtesting and portfolio formation are separate concerns from model training. `scripts/backtester.py` implements walk-forward evaluation with retraining by year, while `scripts/build_portfolio.py` and `scripts/leverage_strategy.py` encode portfolio construction and sizing logic. Strategy definitions in the README imply multiple ranking regimes, with filtering and weighting applied before portfolio output is produced.

7. **Alpha/scoring domain library**
   The `alpha/` package is the reusable scoring domain: factor composites, factor groups (`value`, `quality`, `momentum`, `growth`, `fraud_risk`), explanation logic, and a `horizon_router` that maps requested investment horizons to the appropriate trained model. This is the abstraction layer used by the app and screener to turn raw features/models into user-facing scores.

8. **Application layer**
   `src/` contains the Streamlit-oriented presentation logic: charts, sidebar, scoring, narrative generation, leverage views, and tab-specific UI modules under `src/ui/`. `app_v2.py` is the cloud-safe dashboard entrypoint, while `app.py` appears to be an older or alternate entrypoint. The app reads from curated artifacts and, when configured, can source them from HuggingFace Hub instead of local files.

9. **API boundary**
   `api/` is a separate FastAPI service with routers for companies, labels, and screener endpoints. This implies a read-only application API sitting on top of the curated dataset/models, distinct from the Streamlit UI. `api/deps.py` suggests shared dependency wiring for whatever datastore or artifact access the service uses.

10. **Artifacts and storage**
   The repo keeps generated assets in `data/`, `models/`, and `reports/`. `models/` holds serialized trained models and feature-set metadata, while `reports/` stores generated analysis outputs such as fraud and overfitting summaries. `data/alpha_registry.json` and related registry files act as metadata layers that bind model/factor outputs to the UI and reports.

11. **Automation and operational flow**
   GitHub Actions drive scheduled maintenance: weekly refresh, drift monitoring, and data regeneration. `scripts/refresh_data.py`, `scripts/monitor_drift.py`, and `scripts/push_to_hf.py` show the main operational loop is: refresh pipeline outputs, rebuild models/reports, then publish to HuggingFace Hub for cloud consumption. This is the primary deployment/data-distribution path.

12. **Infra, testing, and docs**
   `Dockerfile`, `docker-compose.yml`, and `infra/db/init.sql` indicate containerized deployment with optional database initialization. `tests/test_pipeline.py` covers critical data-integrity and leakage checks, reflecting that temporal correctness is a first-class architectural constraint. The extensive `docs/` tree is part of the system design, documenting pipeline integrity, methodology, deployment, and schema changes.

Main entry points: [`data/historical_dataset_clean.parquet`](https://github.com/sherlock718/stock-fraud-screener/blob/main/data/historical_dataset_clean.parquet) (Historical data), [`app_v2.py`](https://github.com/sherlock718/stock-fraud-screener/blob/main/app_v2.py) (Streamlit app), [`pipeline/enrich_market_signals.py`](https://github.com/sherlock718/stock-fraud-screener/blob/main/pipeline/enrich_market_signals.py) (Signal enrich), [`alpha/explain.py`](https://github.com/sherlock718/stock-fraud-screener/blob/main/alpha/explain.py) (Alpha explain)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_pipeline["Pipeline"]
  node_pipeline_orchestrator["Pipeline run<br/>etl orchestrator<br/>[run_pipeline.py]"]
  node_pipeline_ticker_fetch["Ticker fetch<br/>market ingest"]
  node_pipeline_snapshot_build["Snapshot build<br/>market snapshots"]
  node_pipeline_feature_library["Feature lib<br/>feature engine<br/>[feature_library.py]"]
  node_pipeline_signal_enrich["Signal enrich"]
  node_pipeline_fraud_logic["Fraud logic<br/>fraud signals<br/>[fraud_signals.py]"]
  node_pipeline_dataset_build[("Clean build<br/>dataset build")]
end

subgraph group_group_research["Research"]
  node_research_train["Train models<br/>ml training<br/>[train_models.py]"]
  node_research_backtest["Backtester<br/>evaluation<br/>[backtester.py]"]
  node_research_portfolio["Portfolio build<br/>portfolio construction<br/>[build_portfolio.py]"]
  node_research_reports["Report gen<br/>analysis reporting"]
end

subgraph group_group_products["Products"]
  node_alpha_factors{{"Factor set<br/>factor library<br/>[composite.py]"}}
  node_alpha_explain{{"Alpha explain<br/>scoring logic<br/>[explain.py]"}}
  node_alpha_router["Horizon router<br/>model routing<br/>[horizon_router.py]"]
  node_app_ui["Streamlit app<br/>dashboard<br/>[app_v2.py]"]
  node_app_tabs["UI tabs<br/>presentation modules<br/>[tab_screener.py]"]
  node_app_api["FastAPI<br/>read-only api<br/>[main.py]"]
end

subgraph group_group_storage["Artifacts"]
  node_data_historical[("Historical data<br/>curated dataset")]
  node_model_registry["Model registry<br/>model metadata"]
  node_report_outputs["Reports<br/>analysis artifacts<br/>[fraud_report.json]"]
end

node_pipeline_orchestrator -->|"starts"| node_pipeline_ticker_fetch
node_pipeline_ticker_fetch -->|"feeds"| node_pipeline_snapshot_build
node_pipeline_snapshot_build -->|"uses"| node_pipeline_feature_library
node_pipeline_feature_library -->|"drives"| node_pipeline_signal_enrich
node_pipeline_signal_enrich -->|"specializes"| node_pipeline_fraud_logic
node_pipeline_fraud_logic -->|"feeds"| node_pipeline_dataset_build
node_pipeline_signal_enrich -->|"feeds"| node_pipeline_dataset_build
node_pipeline_dataset_build -->|"writes"| node_data_historical
node_data_historical -->|"consumed by"| node_research_train
node_data_historical -->|"consumed by"| node_research_backtest
node_data_historical -->|"consumed by"| node_research_portfolio
node_research_train -->|"writes"| node_model_registry
node_research_backtest -->|"writes"| node_report_outputs
node_research_reports -->|"writes"| node_report_outputs
node_model_registry -->|"binds"| node_alpha_router
node_alpha_factors -->|"composes"| node_alpha_explain
node_alpha_explain -->|"uses"| node_alpha_router
node_alpha_explain -->|"feeds"| node_app_ui
node_alpha_router -->|"feeds"| node_app_ui
node_app_tabs -->|"renders"| node_app_ui
node_data_historical -->|"reads"| node_app_ui
node_data_historical -->|"served by"| node_app_api
node_model_registry -->|"reads"| node_app_api
node_report_outputs -->|"surfaces"| node_app_ui

click node_pipeline_orchestrator "https://github.com/sherlock718/stock-fraud-screener/blob/main/scripts/run_pipeline.py"
click node_pipeline_ticker_fetch "https://github.com/sherlock718/stock-fraud-screener/blob/main/pipeline/step1_fetch_tickers.py"
click node_pipeline_snapshot_build "https://github.com/sherlock718/stock-fraud-screener/blob/main/pipeline/step2_build_snapshots.py"
click node_pipeline_feature_library "https://github.com/sherlock718/stock-fraud-screener/blob/main/pipeline/feature_library.py"
click node_pipeline_signal_enrich "https://github.com/sherlock718/stock-fraud-screener/blob/main/pipeline/enrich_market_signals.py"
click node_pipeline_fraud_logic "https://github.com/sherlock718/stock-fraud-screener/blob/main/pipeline/fraud_signals.py"
click node_pipeline_dataset_build "https://github.com/sherlock718/stock-fraud-screener/blob/main/pipeline/build_historical_dataset.py"
click node_data_historical "https://github.com/sherlock718/stock-fraud-screener/blob/main/data/historical_dataset_clean.parquet"
click node_model_registry "https://github.com/sherlock718/stock-fraud-screener/blob/main/models/model_3y_regression_meta.json"
click node_report_outputs "https://github.com/sherlock718/stock-fraud-screener/blob/main/reports/fraud_report.json"
click node_research_train "https://github.com/sherlock718/stock-fraud-screener/blob/main/scripts/train_models.py"
click node_research_backtest "https://github.com/sherlock718/stock-fraud-screener/blob/main/scripts/backtester.py"
click node_research_portfolio "https://github.com/sherlock718/stock-fraud-screener/blob/main/scripts/build_portfolio.py"
click node_research_reports "https://github.com/sherlock718/stock-fraud-screener/blob/main/scripts/generate_reports.py"
click node_alpha_factors "https://github.com/sherlock718/stock-fraud-screener/blob/main/alpha/factors/composite.py"
click node_alpha_explain "https://github.com/sherlock718/stock-fraud-screener/blob/main/alpha/explain.py"
click node_alpha_router "https://github.com/sherlock718/stock-fraud-screener/blob/main/alpha/horizon_router.py"
click node_app_ui "https://github.com/sherlock718/stock-fraud-screener/blob/main/app_v2.py"
click node_app_tabs "https://github.com/sherlock718/stock-fraud-screener/blob/main/src/ui/tab_screener.py"
click node_app_api "https://github.com/sherlock718/stock-fraud-screener/blob/main/api/main.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline_orchestrator,node_pipeline_ticker_fetch,node_pipeline_snapshot_build,node_pipeline_feature_library,node_pipeline_signal_enrich,node_pipeline_fraud_logic,node_pipeline_dataset_build toneBlue
class node_research_train,node_research_backtest,node_research_portfolio,node_research_reports toneAmber
class node_alpha_factors,node_alpha_explain,node_alpha_router,node_app_ui,node_app_tabs,node_app_api toneMint
class node_data_historical,node_model_registry,node_report_outputs toneRose
```

## Components

### Pipeline

Raw ingestion, enrichment, and canonical dataset build.

- [Pipeline run](https://github.com/sherlock718/stock-fraud-screener/blob/main/scripts/run_pipeline.py): etl orchestrator · Entry point for the multi-market ingestion and feature-building flow. · `scripts/run_pipeline.py`
- [Ticker fetch](https://github.com/sherlock718/stock-fraud-screener/blob/main/pipeline/step1_fetch_tickers.py): market ingest · Country-partitioned ticker and company universe ingestion. · `pipeline/step1_fetch_tickers.py`
- [Snapshot build](https://github.com/sherlock718/stock-fraud-screener/blob/main/pipeline/step2_build_snapshots.py): market snapshots · Builds annual company snapshots across regional feeds. · `pipeline/step2_build_snapshots.py`
- [Feature lib](https://github.com/sherlock718/stock-fraud-screener/blob/main/pipeline/feature_library.py): feature engine · Canonical factor and feature definitions used across the pipeline. · `pipeline/feature_library.py`
- [Signal enrich](https://github.com/sherlock718/stock-fraud-screener/blob/main/pipeline/enrich_market_signals.py): signal enrichment · Fraud, governance, insider, market, and market-cap enrichments. · `pipeline/enrich_market_signals.py`
- [Fraud logic](https://github.com/sherlock718/stock-fraud-screener/blob/main/pipeline/fraud_signals.py): fraud signals · Fraud-specific labels, taxonomy, and risk signal construction. · `pipeline/fraud_signals.py`
- [Clean build](https://github.com/sherlock718/stock-fraud-screener/blob/main/pipeline/build_historical_dataset.py): dataset build · Merges snapshots and features into the canonical historical dataset. · `pipeline/build_historical_dataset.py`

### Research

Offline modeling, backtesting, and report generation.

- [Train models](https://github.com/sherlock718/stock-fraud-screener/blob/main/scripts/train_models.py): ml training · Offline training and tuning for the LightGBM-centered research stack. · `scripts/train_models.py`
- [Backtester](https://github.com/sherlock718/stock-fraud-screener/blob/main/scripts/backtester.py): evaluation · Walk-forward evaluation with retraining by period. · `scripts/backtester.py`
- [Portfolio build](https://github.com/sherlock718/stock-fraud-screener/blob/main/scripts/build_portfolio.py): portfolio construction · Ranking, filtering, weighting, and holdings construction. · `scripts/build_portfolio.py`
- [Report gen](https://github.com/sherlock718/stock-fraud-screener/blob/main/scripts/generate_reports.py): analysis reporting · Bias audits, factor research, and summary generation. · `scripts/generate_reports.py`

### Products

Reusable scoring logic and user-facing surfaces.

- [Factor set](https://github.com/sherlock718/stock-fraud-screener/blob/main/alpha/factors/composite.py): factor library · Reusable factor groups for value, quality, momentum, growth, and fraud risk. · `alpha/factors/composite.py`
- [Alpha explain](https://github.com/sherlock718/stock-fraud-screener/blob/main/alpha/explain.py): scoring logic · Composite scoring and explanation layer for user-facing alpha output. · `alpha/explain.py`
- [Horizon router](https://github.com/sherlock718/stock-fraud-screener/blob/main/alpha/horizon_router.py): model routing · Maps requested investment horizons to the appropriate trained model. · `alpha/horizon_router.py`
- [Streamlit app](https://github.com/sherlock718/stock-fraud-screener/blob/main/app_v2.py): dashboard · Cloud-safe dashboard entrypoint for charts, scoring, and tabs. · `app_v2.py`
- [UI tabs](https://github.com/sherlock718/stock-fraud-screener/blob/main/src/ui/tab_screener.py): presentation modules · Tab-specific views for screener, backtester, strategies, company profile, and benchmarks. · `src/ui/tab_screener.py`
- [FastAPI](https://github.com/sherlock718/stock-fraud-screener/blob/main/api/main.py): read-only api · Service layer exposing companies, labels, and screener endpoints. · `api/main.py`

### Artifacts

Generated datasets, models, registries, and reports.

- [Historical data](https://github.com/sherlock718/stock-fraud-screener/blob/main/data/historical_dataset_clean.parquet): curated dataset · Main training and scoring input plus related market artifacts. · `data/historical_dataset_clean.parquet`
- [Model registry](https://github.com/sherlock718/stock-fraud-screener/blob/main/models/model_3y_regression_meta.json): model metadata · Serialized model metadata and feature-set registries by horizon. · `models/model_3y_regression_meta.json`
- [Reports](https://github.com/sherlock718/stock-fraud-screener/blob/main/reports/fraud_report.json): analysis artifacts · Fraud, overfitting, and backtest outputs. · `reports/fraud_report.json`

## Connections

- Pipeline run → Ticker fetch: starts · Runs the market-partitioned universe fetch stage.
- Ticker fetch → Snapshot build: feeds · Fetched ticker universes are turned into regional snapshots.
- Snapshot build → Feature lib: uses · Snapshot assembly relies on shared feature definitions.
- Feature lib → Signal enrich: drives · Core feature logic feeds domain-specific enrichments.
- Signal enrich → Fraud logic: specializes · Fraud labeling and taxonomy are layered onto the broader enrichment flow.
- Fraud logic → Clean build: feeds · Fraud outputs are merged into the canonical build.
- Signal enrich → Clean build: feeds · Enriched signals are assembled into the clean dataset.
- Clean build → Historical data: writes · The clean historical dataset is materialized as curated artifacts.
- Historical data → Train models: consumed by · Training reads the curated dataset only.
- Historical data → Backtester: consumed by · Backtesting evaluates on curated data and market history.
- Historical data → Portfolio build: consumed by · Portfolio construction uses curated scores and holdings inputs.
- Train models → Model registry: writes · Training produces horizon-specific model metadata and registries.
- Backtester → Reports: writes · Backtesting contributes evaluation outputs and summaries.
- Report gen → Reports: writes · Report generation persists analysis artifacts.
- Model registry → Horizon router: binds · Model metadata is selected by investment horizon.
- Factor set → Alpha explain: composes · Factor groups feed composite alpha scores and explanations.
- Alpha explain → Horizon router: uses · Explanation and scoring logic depend on horizon routing.
- Alpha explain → Streamlit app: feeds · Reusable alpha output powers the Streamlit experience.
- Horizon router → Streamlit app: feeds · The dashboard selects horizon-appropriate models.
- UI tabs → Streamlit app: renders · Tab modules implement the dashboard surfaces.
- Historical data → Streamlit app: reads · The dashboard consumes curated artifacts and cached outputs.
- Historical data → FastAPI: served by · The API reads the curated dataset and registries.
- Model registry → FastAPI: reads · API endpoints can expose model-backed screening results.
- Reports → Streamlit app: surfaces · Analysis outputs are shown in the dashboard.