# rayensebei/bvmt-market-surveillance architecture

> Architecture diagram of the GitHub repository rayensebei/bvmt-market-surveillance, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rayensebei/bvmt-market-surveillance
- Repository: https://github.com/rayensebei/bvmt-market-surveillance

## Overview

BVMT Market Surveillance screens stored Tunisian market observations for unusual price or volume activity, enriches flags with news and review context, and presents a human-review queue through a read-only Flask API and React Terminal. Separate evaluation tooling measures detector results against sourced events and baselines. The README documents the overall pipeline and product capabilities; sampled source supports selected API, detector, and UI relationships, while unsampled implementation details are not inferred.

Main entry points: [`frontend/src/App.tsx`](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/App.tsx) (Terminal navigation), [`src/evaluation/evaluate.py`](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/evaluation/evaluate.py) (Event evaluation), [`bvmt_data`](https://github.com/rayensebei/bvmt-market-surveillance/tree/master/bvmt_data) (Stored market and news CSVs), [`app.py`](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/app.py) (Read-only Flask API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Data collection"]
  node_market_source(("BVMT observations"))
  node_ilboursa(("ilboursa news"))
  node_scraper["Market scraper<br/>[scrape_ilboursa.py]"]
  node_news_scraper["News scraper<br/>[scrape_news.py]"]
  node_local_data[("Stored market and news CSVs")]
end

subgraph group_screening["Screening and review"]
  node_detector["Rolling anomaly detector"]
  node_decay["Decay detector<br/>[decay_detector.py]"]
  node_candidates["Candidate detectors<br/>[candidates_v2.py]"]
  node_crossref["News and event cross-reference"]
  node_classification["Flag classification"]
  node_watchlist["Watchlist refinement"]
  node_triage["Optional AI triage<br/>[ai_triage_free.py]"]
  node_groq(("Groq service"))
end

subgraph group_evaluation["Evaluation"]
  node_evaluation["Event evaluation<br/>[evaluate.py]"]
  node_baselines["Baselines and sensitivity<br/>[baselines.py]"]
  node_phase2["Phase 2 comparison<br/>[phase2.py]"]
  node_report["Evaluation report<br/>[report.py]"]
end

subgraph group_terminal["Analyst Terminal"]
  node_flask["Read-only Flask API<br/>[app.py]"]
  node_aggregates["Presentation aggregates<br/>[dashboard.py]"]
  node_app_ui["Terminal navigation<br/>[App.tsx]"]
  node_api_client["API client<br/>[api.ts]"]
  node_overview["Overview and market map<br/>[OverviewPage.tsx]"]
  node_feed["Anomaly feed<br/>[FeedPage.tsx]"]
  node_stock["Stock detail<br/>[StockPage.tsx]"]
  node_news["News view<br/>[NewsPage.tsx]"]
  node_watch_ui["Watchlist view<br/>[WatchlistPage.tsx]"]
  node_pipeline_ui["Pipeline view<br/>[PipelinePage.tsx]"]
  node_evaluation_ui["Evaluation view<br/>[EvaluationPage.tsx]"]
end

node_analyst(("Human analyst"))

node_market_source -->|"provides data"| node_scraper
node_ilboursa -->|"provides news"| node_news_scraper
node_scraper -->|"updates files"| node_local_data
node_news_scraper -->|"stores news"| node_local_data
node_local_data -->|"feeds observations"| node_detector
node_detector -->|"flags for context"| node_crossref
node_decay -->|"adds decay flags"| node_watchlist
node_crossref -->|"provides context"| node_classification
node_classification -->|"refines queue"| node_watchlist
node_watchlist -.->|"offers draft triage"| node_triage
node_triage -.->|"requests opinion"| node_groq
node_local_data -->|"supplies evaluation data"| node_evaluation
node_evaluation -->|"compared with"| node_baselines
node_evaluation -->|"evaluates candidates"| node_phase2
node_evaluation -->|"feeds results"| node_report
node_local_data -->|"serves stored files"| node_flask
node_flask -->|"builds read views"| node_aggregates
node_aggregates -->|"returns aggregates"| node_flask
node_analyst -->|"uses Terminal"| node_app_ui
node_app_ui -->|"requests data"| node_api_client
node_api_client -->|"fetches API"| node_flask
node_app_ui -->|"routes views"| node_overview
node_app_ui -->|"routes views"| node_feed
node_app_ui -->|"routes views"| node_stock
node_app_ui -->|"routes views"| node_news
node_app_ui -->|"routes views"| node_watch_ui
node_app_ui -->|"routes views"| node_pipeline_ui
node_app_ui -->|"routes views"| node_evaluation_ui
node_evaluation -.->|"exposes results"| node_evaluation_ui

click node_scraper "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/scraping/scrape_ilboursa.py"
click node_news_scraper "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/scraping/scrape_news.py"
click node_local_data "https://github.com/rayensebei/bvmt-market-surveillance/tree/master/bvmt_data"
click node_detector "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/detection/anomaly_detector.py"
click node_decay "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/detection/decay_detector.py"
click node_candidates "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/detection/candidates_v2.py"
click node_crossref "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/detection/crossref_legal_events.py"
click node_classification "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/validation/classify_anomalies.py"
click node_watchlist "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/validation/refine_watchlist.py"
click node_triage "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/triage/ai_triage_free.py"
click node_evaluation "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/evaluation/evaluate.py"
click node_baselines "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/evaluation/baselines.py"
click node_phase2 "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/evaluation/phase2.py"
click node_report "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/evaluation/report.py"
click node_flask "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/app.py"
click node_aggregates "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/dashboard.py"
click node_app_ui "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/App.tsx"
click node_api_client "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/lib/api.ts"
click node_overview "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/pages/OverviewPage.tsx"
click node_feed "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/pages/FeedPage.tsx"
click node_stock "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/pages/StockPage.tsx"
click node_news "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/pages/NewsPage.tsx"
click node_watch_ui "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/pages/WatchlistPage.tsx"
click node_pipeline_ui "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/pages/PipelinePage.tsx"
click node_evaluation_ui "https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/pages/EvaluationPage.tsx"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_market_source,node_ilboursa,node_scraper,node_news_scraper,node_local_data toneBlue
class node_detector,node_decay,node_candidates,node_crossref,node_classification,node_watchlist,node_triage,node_groq toneAmber
class node_evaluation,node_baselines,node_phase2,node_report toneMint
class node_flask,node_aggregates,node_app_ui,node_api_client,node_overview,node_feed,node_stock,node_news,node_watch_ui,node_pipeline_ui,node_evaluation_ui toneRose
class node_analyst toneIndigo
```

## Components

### Data collection

- **BVMT observations**
- **ilboursa news**
- [Market scraper](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/scraping/scrape_ilboursa.py): `src/scraping/scrape_ilboursa.py`
- [News scraper](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/scraping/scrape_news.py): `src/scraping/scrape_news.py`
- [Stored market and news CSVs](https://github.com/rayensebei/bvmt-market-surveillance/tree/master/bvmt_data): `bvmt_data`

### Screening and review

- [Rolling anomaly detector](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/detection/anomaly_detector.py): `src/detection/anomaly_detector.py`
- [Decay detector](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/detection/decay_detector.py): `src/detection/decay_detector.py`
- [Candidate detectors](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/detection/candidates_v2.py): `src/detection/candidates_v2.py`
- [News and event cross-reference](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/detection/crossref_legal_events.py): `src/detection/crossref_legal_events.py`
- [Flag classification](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/validation/classify_anomalies.py): `src/validation/classify_anomalies.py`
- [Watchlist refinement](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/validation/refine_watchlist.py): `src/validation/refine_watchlist.py`
- [Optional AI triage](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/triage/ai_triage_free.py): `src/triage/ai_triage_free.py`
- **Groq service**

### Evaluation

- [Event evaluation](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/evaluation/evaluate.py): `src/evaluation/evaluate.py`
- [Baselines and sensitivity](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/evaluation/baselines.py): `src/evaluation/baselines.py`
- [Phase 2 comparison](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/evaluation/phase2.py): `src/evaluation/phase2.py`
- [Evaluation report](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/evaluation/report.py): `src/evaluation/report.py`

### Analyst Terminal

- [Read-only Flask API](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/app.py): `app.py`
- [Presentation aggregates](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/src/dashboard.py): `src/dashboard.py`
- [Terminal navigation](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/App.tsx): `frontend/src/App.tsx`
- [API client](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/lib/api.ts): `frontend/src/lib/api.ts`
- [Overview and market map](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/pages/OverviewPage.tsx): `frontend/src/pages/OverviewPage.tsx`
- [Anomaly feed](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/pages/FeedPage.tsx): `frontend/src/pages/FeedPage.tsx`
- [Stock detail](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/pages/StockPage.tsx): `frontend/src/pages/StockPage.tsx`
- [News view](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/pages/NewsPage.tsx): `frontend/src/pages/NewsPage.tsx`
- [Watchlist view](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/pages/WatchlistPage.tsx): `frontend/src/pages/WatchlistPage.tsx`
- [Pipeline view](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/pages/PipelinePage.tsx): `frontend/src/pages/PipelinePage.tsx`
- [Evaluation view](https://github.com/rayensebei/bvmt-market-surveillance/blob/master/frontend/src/pages/EvaluationPage.tsx): `frontend/src/pages/EvaluationPage.tsx`

### Other components

- **Human analyst**

## Connections

- BVMT observations → Market scraper: provides data (evidence: `README.md`)
- ilboursa news → News scraper: provides news (evidence: `README.md`)
- Market scraper → Stored market and news CSVs: updates files (evidence: `src/scraping/scrape_ilboursa.py`)
- News scraper → Stored market and news CSVs: stores news (evidence: `README.md`)
- Stored market and news CSVs → Rolling anomaly detector: feeds observations (evidence: `README.md`)
- Rolling anomaly detector → News and event cross-reference: flags for context (evidence: `README.md`)
- Decay detector → Watchlist refinement: adds decay flags (evidence: `README.md`)
- News and event cross-reference → Flag classification: provides context (evidence: `README.md`)
- Flag classification → Watchlist refinement: refines queue (evidence: `README.md`)
- Watchlist refinement → Optional AI triage: offers draft triage (evidence: `README.md`)
- Optional AI triage → Groq service: requests opinion (evidence: `README.md`)
- Stored market and news CSVs → Event evaluation: supplies evaluation data (evidence: `README.md`)
- Event evaluation → Baselines and sensitivity: compared with (evidence: `src/evaluation/full_universe.py`)
- Event evaluation → Phase 2 comparison: evaluates candidates (evidence: `src/evaluation/phase2.py`)
- Event evaluation → Evaluation report: feeds results (evidence: `src/evaluation/report.py`)
- Stored market and news CSVs → Read-only Flask API: serves stored files (evidence: `README.md`)
- Read-only Flask API → Presentation aggregates: builds read views (evidence: `app.py`)
- Presentation aggregates → Read-only Flask API: returns aggregates (evidence: `app.py`)
- Human analyst → Terminal navigation: uses Terminal (evidence: `README.md`)
- Terminal navigation → API client: requests data (evidence: `frontend/src/App.tsx`)
- API client → Read-only Flask API: fetches API (evidence: `frontend/src/lib/api.ts`)
- Terminal navigation → Overview and market map: routes views (evidence: `frontend/src/App.tsx`)
- Terminal navigation → Anomaly feed: routes views (evidence: `frontend/src/App.tsx`)
- Terminal navigation → Stock detail: routes views (evidence: `frontend/src/App.tsx`)
- Terminal navigation → News view: routes views (evidence: `frontend/src/App.tsx`)
- Terminal navigation → Watchlist view: routes views (evidence: `frontend/src/App.tsx`)
- Terminal navigation → Pipeline view: routes views (evidence: `frontend/src/App.tsx`)
- Terminal navigation → Evaluation view: routes views (evidence: `frontend/src/App.tsx`)
- Event evaluation → Evaluation view: exposes results (evidence: `README.md`)
