How sherlock718/stock-fraud-screener works
1. Overall shape
This repo is a research-to-production pipeline for multi-factor equity selection. The architecture is split into: raw data ingestion and feature generation (pipeline/), offline analytics and ML/backtesting (scripts/), reusable scoring logic (alpha/ and src/), and delivery surfaces (app_v2.py Streamlit UI plus api/ FastAPI).
2. Data ingestion boundary
pipeline/ is the ETL layer and is explicitly described as “raw data ingestion only.” It pulls company/ticker data and filings across multiple markets, then builds annual company snapshots and derived features. Country-specific step files (step1_fetch_tickers_*, step2_build_snapshots_*) show the pipeline is market-partitioned rather than a single global feed.
3. Feature and signal construction
Shared feature logic lives in pipeline/feature_library.py, which acts as the single source of truth for computed factors. Supporting modules like fraud_signals.py, value_metrics.py, market_cap_filter.py, and the enrich_* scripts add domain-specific signals such as fraud risk, market signals, governance, insider activity, and market capitalization filters.
4. Canonical dataset build
The core pipeline output is a cleaned historical dataset, merged from multi-market snapshots into parquet artifacts under data/. The README indicates historical_dataset_clean.parquet is the main training/scoring input, with related intermediate assets like snapshots, price history, and benchmark returns stored alongside it. This is the handoff boundary between ETL and model/research layers.
5. Modeling and research layer
scripts/ contains the offline research stack: model training (train_models.py, train_regression_model.py, tune_models.py), feature selection, out-of-fold scoring, factor research, and bias audits. The ML stack is centered on LightGBM, with additional tuning/ensembling via Optuna and CatBoost in some workflows. This layer consumes curated data only; it does not fetch raw filings itself.
6. Strategy and portfolio construction
Backtesting and portfolio formation are separate concerns from model training. scripts/backtester.py implements walk-forward evaluation with retraining by year, while scripts/build_portfolio.py and scripts/leverage_strategy.py encode portfolio construction and sizing logic. Strategy definitions in the README imply multiple ranking regimes, with filtering and weighting applied before portfolio output is produced.
7. Alpha/scoring domain library
The alpha/ package is the reusable scoring domain: factor composites, factor groups (value, quality, momentum, growth, fraud_risk), explanation logic, and a horizon_router that maps requested investment horizons to the appropriate trained model. This is the abstraction layer used by the app and screener to turn raw features/models into user-facing scores.
8. Application layer
src/ contains the Streamlit-oriented presentation logic: charts, sidebar, scoring, narrative generation, leverage views, and tab-specific UI modules under src/ui/. app_v2.py is the cloud-safe dashboard entrypoint, while app.py appears to be an older or alternate entrypoint. The app reads from curated artifacts and, when configured, can source them from HuggingFace Hub instead of local files.
9. API boundary
api/ is a separate FastAPI service with routers for companies, labels, and screener endpoints. This implies a read-only application API sitting on top of the curated dataset/models, distinct from the Streamlit UI. api/deps.py suggests shared dependency wiring for whatever datastore or artifact access the service uses.
10. Artifacts and storage
The repo keeps generated assets in data/, models/, and reports/. models/ holds serialized trained models and feature-set metadata, while reports/ stores generated analysis outputs such as fraud and overfitting summaries. data/alpha_registry.json and related registry files act as metadata layers that bind model/factor outputs to the UI and reports.
11. Automation and operational flow
GitHub Actions drive scheduled maintenance: weekly refresh, drift monitoring, and data regeneration. scripts/refresh_data.py, scripts/monitor_drift.py, and scripts/push_to_hf.py show the main operational loop is: refresh pipeline outputs, rebuild models/reports, then publish to HuggingFace Hub for cloud consumption. This is the primary deployment/data-distribution path.
12. Infra, testing, and docs
Dockerfile, docker-compose.yml, and infra/db/init.sql indicate containerized deployment with optional database initialization. tests/test_pipeline.py covers critical data-integrity and leakage checks, reflecting that temporal correctness is a first-class architectural constraint. The extensive docs/ tree is part of the system design, documenting pipeline integrity, methodology, deployment, and schema changes.
Main entry points: historical_dataset_clean.parquet Historical data · app_v2.py Streamlit app · enrich_market_signals.py Signal enrich · explain.py Alpha explain
Components 20
Pipeline
Raw ingestion, enrichment, and canonical dataset build.
- Pipeline run · Entry point for the multi-market ingestion and feature-building flow.
scripts/run_pipeline.py - Ticker fetch · Country-partitioned ticker and company universe ingestion.
pipeline/step1_fetch_tickers.py - Snapshot build · Builds annual company snapshots across regional feeds.
pipeline/step2_build_snapshots.py - Feature lib · Canonical factor and feature definitions used across the pipeline.
pipeline/feature_library.py - Signal enrich · Fraud, governance, insider, market, and market-cap enrichments.
pipeline/enrich_market_signals.py - Fraud logic · Fraud-specific labels, taxonomy, and risk signal construction.
pipeline/fraud_signals.py - Clean build · Merges snapshots and features into the canonical historical dataset.
pipeline/build_historical_dataset.py
Research
Offline modeling, backtesting, and report generation.
- Train models · Offline training and tuning for the LightGBM-centered research stack.
scripts/train_models.py - Backtester · Walk-forward evaluation with retraining by period.
scripts/backtester.py - Portfolio build · Ranking, filtering, weighting, and holdings construction.
scripts/build_portfolio.py - Report gen · Bias audits, factor research, and summary generation.
scripts/generate_reports.py
Products
Reusable scoring logic and user-facing surfaces.
- Factor set · Reusable factor groups for value, quality, momentum, growth, and fraud risk.
alpha/factors/composite.py - Alpha explain · Composite scoring and explanation layer for user-facing alpha output.
alpha/explain.py - Horizon router · Maps requested investment horizons to the appropriate trained model.
alpha/horizon_router.py - Streamlit app · Cloud-safe dashboard entrypoint for charts, scoring, and tabs.
app_v2.py - UI tabs · Tab-specific views for screener, backtester, strategies, company profile, and benchmarks.
src/ui/tab_screener.py - FastAPI · Service layer exposing companies, labels, and screener endpoints.
api/main.py
Artifacts
Generated datasets, models, registries, and reports.
- Historical data · Main training and scoring input plus related market artifacts.
data/historical_dataset_clean.parquet - Model registry · Serialized model metadata and feature-set registries by horizon.
models/model_3y_regression_meta.json - Reports · Fraud, overfitting, and backtest outputs.
reports/fraud_report.json
Connections 24
- Pipeline run → Ticker fetch: starts · Runs the market-partitioned universe fetch stage.
- Ticker fetch → Snapshot build: feeds · Fetched ticker universes are turned into regional snapshots.
- Snapshot build → Feature lib: uses · Snapshot assembly relies on shared feature definitions.
- Feature lib → Signal enrich: drives · Core feature logic feeds domain-specific enrichments.
- Signal enrich → Fraud logic: specializes · Fraud labeling and taxonomy are layered onto the broader enrichment flow.
- Fraud logic → Clean build: feeds · Fraud outputs are merged into the canonical build.
- Signal enrich → Clean build: feeds · Enriched signals are assembled into the clean dataset.
- Clean build → Historical data: writes · The clean historical dataset is materialized as curated artifacts.
- Historical data → Train models: consumed by · Training reads the curated dataset only.
- Historical data → Backtester: consumed by · Backtesting evaluates on curated data and market history.
- Historical data → Portfolio build: consumed by · Portfolio construction uses curated scores and holdings inputs.
- Train models → Model registry: writes · Training produces horizon-specific model metadata and registries.
- Backtester → Reports: writes · Backtesting contributes evaluation outputs and summaries.
- Report gen → Reports: writes · Report generation persists analysis artifacts.
- Model registry → Horizon router: binds · Model metadata is selected by investment horizon.
- Factor set → Alpha explain: composes · Factor groups feed composite alpha scores and explanations.
- Alpha explain → Horizon router: uses · Explanation and scoring logic depend on horizon routing.
- Alpha explain → Streamlit app: feeds · Reusable alpha output powers the Streamlit experience.
- Horizon router → Streamlit app: feeds · The dashboard selects horizon-appropriate models.
- UI tabs → Streamlit app: renders · Tab modules implement the dashboard surfaces.
- Historical data → Streamlit app: reads · The dashboard consumes curated artifacts and cached outputs.
- Historical data → FastAPI: served by · The API reads the curated dataset and registries.
- Model registry → FastAPI: reads · API endpoints can expose model-backed screening results.
- Reports → Streamlit app: surfaces · Analysis outputs are shown in the dashboard.