# muhammad-ahmad2511/finsight-pk architecture

> Architecture diagram of the GitHub repository muhammad-ahmad2511/finsight-pk, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/muhammad-ahmad2511/finsight-pk
- Repository: https://github.com/muhammad-ahmad2511/finsight-pk

## Overview

FinSight PK is a research pipeline for regime-aware, sentiment-augmented KSE-100 forecasts. The workflow cleans index prices, derives technical indicators and regimes, gathers news and macro features, labels news sentiment, assembles model inputs, and trains a multi-horizon LSTM. Documented product branches include per-regime SHAP explainability and a web app with dashboard, forecasts, sentiment feed, and REST API. Edges reflect explicit script inputs/outputs or README statements; the web and SHAP implementations are not present in the supplied tree, so their nodes point to the README.

Main entry points: [`README.md`](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/README.md) (Web Application), [`Data/Data Scripts/08_build_lstm_dataset.py`](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/08_build_lstm_dataset.py) (Forecast Dataset Builder), [`Data/Data Scripts/07_label_using_distilbert.py`](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/07_label_using_distilbert.py) (News Labeling), [`Data/Data Scripts/01_kse100_clean.py`](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/01_kse100_clean.py) (Price Cleaner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_market["Market Features"]
  node_raw_prices["Raw KSE Prices"]
  node_price_clean["Price Cleaner<br/>[01_kse100_clean.py]"]
  node_indicators["Technical Indicators"]
  node_regimes["Regime Detection"]
  node_epu["EPU Features<br/>[04_epu_clean.py]"]
  node_external["External Features"]
end

subgraph group_news["News Sentiment"]
  node_news_scrape["Dawn Scraper"]
  node_news_corpus["Dawn News Corpus"]
  node_sentiment_train["Sentiment Fine-tuning"]
  node_sentiment_label["News Labeling"]
  node_labeled_news["Labeled News"]
end

subgraph group_forecast["Forecasting"]
  node_dataset["Forecast Dataset Builder"]
  node_lstm["LSTM Training &amp; Forecast<br/>[09_train_lstm.py]"]
end

subgraph group_product["Explainability &amp; App"]
  node_shap["Per-Regime SHAP<br/>[README.md]"]
  node_webapp["Web Application<br/>[README.md]"]
  node_dashboard["Live Dashboard<br/>[README.md]"]
  node_forecast_view["Forecast Views<br/>[README.md]"]
  node_sentiment_feed["Sentiment Feed<br/>[README.md]"]
  node_rest_api["REST API<br/>[README.md]"]
end

node_participant(("Market Participant"))

node_raw_prices -->|"cleans"| node_price_clean
node_price_clean -->|"provides prices"| node_indicators
node_indicators -->|"provides indicators"| node_regimes
node_regimes -->|"provides regimes"| node_dataset
node_news_scrape -->|"writes corpus"| node_news_corpus
node_news_corpus -->|"labels headlines"| node_sentiment_label
node_sentiment_train -->|"supplies model"| node_sentiment_label
node_sentiment_label -->|"writes labels"| node_labeled_news
node_labeled_news -->|"adds sentiment"| node_dataset
node_epu -->|"provides EPU"| node_dataset
node_external -->|"provides features"| node_dataset
node_dataset -->|"provides inputs"| node_lstm
node_lstm -.->|"explained by"| node_shap
node_participant -->|"uses"| node_webapp
node_webapp -->|"serves"| node_dashboard
node_webapp -->|"serves"| node_forecast_view
node_webapp -->|"serves"| node_sentiment_feed
node_webapp -->|"exposes"| node_rest_api
node_shap -.->|"explains forecasts"| node_forecast_view

click node_raw_prices "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Datasets/Karachi%20100%20Historical%20Data.csv"
click node_price_clean "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/01_kse100_clean.py"
click node_indicators "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/02_technical_indicators.py"
click node_regimes "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/03_regime_detection.py"
click node_epu "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/04_epu_clean.py"
click node_external "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/06_fetch_external.py"
click node_news_scrape "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/05_wayback_scraper.py"
click node_news_corpus "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Datasets/dawn_business_2013_to_2026.csv"
click node_sentiment_train "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/model/Distilbert/Script/finetune_distilbert_kse100.py"
click node_sentiment_label "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/07_label_using_distilbert.py"
click node_labeled_news "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Datasets/dawn_business_2013_to_2026_labeled.csv"
click node_dataset "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/08_build_lstm_dataset.py"
click node_lstm "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/09_train_lstm.py"
click node_shap "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/README.md"
click node_webapp "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/README.md"
click node_dashboard "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/README.md"
click node_forecast_view "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/README.md"
click node_sentiment_feed "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/README.md"
click node_rest_api "https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/README.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_prices,node_price_clean,node_indicators,node_regimes,node_epu,node_external toneBlue
class node_news_scrape,node_news_corpus,node_sentiment_train,node_sentiment_label,node_labeled_news toneAmber
class node_dataset,node_lstm toneMint
class node_shap,node_webapp,node_dashboard,node_forecast_view,node_sentiment_feed,node_rest_api toneRose
class node_participant toneIndigo
```

## Components

### Market Features

- [Raw KSE Prices](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Datasets/Karachi%20100%20Historical%20Data.csv): `Data/Datasets/Karachi 100 Historical Data.csv`
- [Price Cleaner](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/01_kse100_clean.py): `Data/Data Scripts/01_kse100_clean.py`
- [Technical Indicators](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/02_technical_indicators.py): `Data/Data Scripts/02_technical_indicators.py`
- [Regime Detection](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/03_regime_detection.py): `Data/Data Scripts/03_regime_detection.py`
- [EPU Features](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/04_epu_clean.py): `Data/Data Scripts/04_epu_clean.py`
- [External Features](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/06_fetch_external.py): `Data/Data Scripts/06_fetch_external.py`

### News Sentiment

- [Dawn Scraper](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/05_wayback_scraper.py): `Data/Data Scripts/05_wayback_scraper.py`
- [Dawn News Corpus](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Datasets/dawn_business_2013_to_2026.csv): `Data/Datasets/dawn_business_2013_to_2026.csv`
- [Sentiment Fine-tuning](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/model/Distilbert/Script/finetune_distilbert_kse100.py): `model/Distilbert/Script/finetune_distilbert_kse100.py`
- [News Labeling](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/07_label_using_distilbert.py): `Data/Data Scripts/07_label_using_distilbert.py`
- [Labeled News](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Datasets/dawn_business_2013_to_2026_labeled.csv): `Data/Datasets/dawn_business_2013_to_2026_labeled.csv`

### Forecasting

- [Forecast Dataset Builder](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/08_build_lstm_dataset.py): `Data/Data Scripts/08_build_lstm_dataset.py`
- [LSTM Training & Forecast](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/Data/Data%20Scripts/09_train_lstm.py): `Data/Data Scripts/09_train_lstm.py`

### Explainability & App

- [Per-Regime SHAP](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/README.md): `README.md`
- [Web Application](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/README.md): `README.md`
- [Live Dashboard](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/README.md): `README.md`
- [Forecast Views](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/README.md): `README.md`
- [Sentiment Feed](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/README.md): `README.md`
- [REST API](https://github.com/muhammad-ahmad2511/finsight-pk/blob/main/README.md): `README.md`

### Other components

- **Market Participant**

## Connections

- Raw KSE Prices → Price Cleaner: cleans (evidence: `Data/Data Scripts/01_kse100_clean.py`)
- Price Cleaner → Technical Indicators: provides prices (evidence: `Data/Data Scripts/02_technical_indicators.py`)
- Technical Indicators → Regime Detection: provides indicators (evidence: `Data/Data Scripts/03_regime_detection.py`)
- Regime Detection → Forecast Dataset Builder: provides regimes (evidence: `Data/Data Scripts/08_build_lstm_dataset.py`)
- Dawn Scraper → Dawn News Corpus: writes corpus (evidence: `README.md`)
- Dawn News Corpus → News Labeling: labels headlines (evidence: `Data/Data Scripts/07_label_using_distilbert.py`)
- Sentiment Fine-tuning → News Labeling: supplies model (evidence: `Data/Data Scripts/07_label_using_distilbert.py`)
- News Labeling → Labeled News: writes labels (evidence: `Data/Data Scripts/07_label_using_distilbert.py`)
- Labeled News → Forecast Dataset Builder: adds sentiment (evidence: `Data/Data Scripts/08_build_lstm_dataset.py`)
- EPU Features → Forecast Dataset Builder: provides EPU (evidence: `Data/Data Scripts/08_build_lstm_dataset.py`)
- External Features → Forecast Dataset Builder: provides features (evidence: `Data/Data Scripts/08_build_lstm_dataset.py`)
- Forecast Dataset Builder → LSTM Training & Forecast: provides inputs (evidence: `Data/Data Scripts/09_train_lstm.py`)
- LSTM Training & Forecast → Per-Regime SHAP: explained by (evidence: `README.md`)
- Market Participant → Web Application: uses (evidence: `README.md`)
- Web Application → Live Dashboard: serves (evidence: `README.md`)
- Web Application → Forecast Views: serves (evidence: `README.md`)
- Web Application → Sentiment Feed: serves (evidence: `README.md`)
- Web Application → REST API: exposes (evidence: `README.md`)
- Per-Regime SHAP → Forecast Views: explains forecasts (evidence: `README.md`)