# rayan-02/news-sentinel architecture

> Architecture diagram of the GitHub repository rayan-02/news-sentinel, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rayan-02/news-sentinel
- Repository: https://github.com/rayan-02/news-sentinel

## Overview

News Sentinel is a news-collection and analysis pipeline: standalone source scrapers append articles to a shared CSV, NLP cleans and enriches those articles and creates embeddings, and ML produces clusters, similarity results, topic trends, and emerging-signal reports. The README is empty; scraper and NLP-to-ML handoffs are supported by the inspected source, while the ML excerpt documents its output artifacts and main-stage calls. `main.py` only prints “hi” and is not part of this workflow.

Main entry points: [`ml/ml_analysis.py`](https://github.com/rayan-02/news-sentinel/blob/main/ml/ml_analysis.py) (Cluster articles), [`data/articles.csv`](https://github.com/rayan-02/news-sentinel/blob/main/data/articles.csv) (Raw articles), [`scrapers/dawn.py`](https://github.com/rayan-02/news-sentinel/blob/main/scrapers/dawn.py) (Dawn scraper), [`scrapers/geo.py`](https://github.com/rayan-02/news-sentinel/blob/main/scrapers/geo.py) (Geo scraper)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["News collection"]
  node_dawn["Dawn scraper<br/>[dawn.py]"]
  node_geo["Geo scraper<br/>[geo.py]"]
  node_tribune["Tribune scraper<br/>[tribune.py]"]
  node_business["Business Recorder scraper"]
end

subgraph group_nlp["Article enrichment"]
  node_nlp_clean["Clean articles<br/>[nlp.py]"]
  node_nlp_features["Extract NLP features<br/>[nlp.py]"]
end

subgraph group_analysis["Topic analysis"]
  node_ml_cluster["Cluster articles<br/>[ml_analysis.py]"]
  node_ml_similarity["Find similar articles<br/>[ml_analysis.py]"]
  node_ml_trends["Calculate topic trends<br/>[ml_analysis.py]"]
  node_ml_signals["Detect emerging topics<br/>[ml_analysis.py]"]
end

subgraph group_artifacts["Data artifacts"]
  node_articles[("Raw articles<br/>[articles.csv]")]
  node_nlp_articles[("NLP articles<br/>[articles_nlp.csv]")]
  node_embeddings[("Article embeddings")]
  node_ml_articles[("ML articles<br/>[articles_ml.csv]")]
  node_clusters[("Cluster summary")]
  node_similar_pairs[("Similar pairs")]
  node_trends[("Topic trends<br/>[topic_trends.csv]")]
  node_signals[("Emerging signals")]
end

node_operator(("Operator"))
node_dawn_site["Dawn"]
node_geo_site["Geo News"]
node_tribune_site["Express Tribune"]
node_business_site["Business Recorder"]

node_operator -->|"runs"| node_dawn
node_operator -->|"runs"| node_geo
node_operator -->|"runs"| node_tribune
node_operator -->|"runs"| node_business
node_operator -->|"runs"| node_nlp_clean
node_operator -->|"runs"| node_ml_cluster
node_dawn -->|"fetches"| node_dawn_site
node_geo -->|"fetches"| node_geo_site
node_tribune -->|"fetches"| node_tribune_site
node_business -->|"fetches"| node_business_site
node_dawn -->|"appends articles"| node_articles
node_geo -->|"appends articles"| node_articles
node_tribune -->|"appends articles"| node_articles
node_business -->|"appends articles"| node_articles
node_articles -->|"reads"| node_nlp_clean
node_nlp_clean -->|"passes cleaned data"| node_nlp_features
node_nlp_features -->|"writes"| node_nlp_articles
node_nlp_features -->|"writes"| node_embeddings
node_nlp_articles -->|"reads"| node_ml_cluster
node_embeddings -->|"reads"| node_ml_cluster
node_ml_cluster -->|"passes clustered data"| node_ml_similarity
node_ml_similarity -->|"passes enriched data"| node_ml_trends
node_ml_trends -->|"passes trends"| node_ml_signals
node_ml_cluster -->|"writes"| node_ml_articles
node_ml_cluster -->|"writes"| node_clusters
node_ml_similarity -->|"writes"| node_similar_pairs
node_ml_trends -->|"writes"| node_trends
node_ml_signals -->|"writes"| node_signals

click node_dawn "https://github.com/rayan-02/news-sentinel/blob/main/scrapers/dawn.py"
click node_geo "https://github.com/rayan-02/news-sentinel/blob/main/scrapers/geo.py"
click node_tribune "https://github.com/rayan-02/news-sentinel/blob/main/scrapers/tribune.py"
click node_business "https://github.com/rayan-02/news-sentinel/blob/main/scrapers/business_recorder.py"
click node_nlp_clean "https://github.com/rayan-02/news-sentinel/blob/main/nlp/nlp.py"
click node_nlp_features "https://github.com/rayan-02/news-sentinel/blob/main/nlp/nlp.py"
click node_ml_cluster "https://github.com/rayan-02/news-sentinel/blob/main/ml/ml_analysis.py"
click node_ml_similarity "https://github.com/rayan-02/news-sentinel/blob/main/ml/ml_analysis.py"
click node_ml_trends "https://github.com/rayan-02/news-sentinel/blob/main/ml/ml_analysis.py"
click node_ml_signals "https://github.com/rayan-02/news-sentinel/blob/main/ml/ml_analysis.py"
click node_articles "https://github.com/rayan-02/news-sentinel/blob/main/data/articles.csv"
click node_nlp_articles "https://github.com/rayan-02/news-sentinel/blob/main/data/articles_nlp.csv"
click node_embeddings "https://github.com/rayan-02/news-sentinel/blob/main/data/article_embeddings.npy"
click node_ml_articles "https://github.com/rayan-02/news-sentinel/blob/main/data/articles_ml.csv"
click node_clusters "https://github.com/rayan-02/news-sentinel/blob/main/data/clusters_summary.csv"
click node_similar_pairs "https://github.com/rayan-02/news-sentinel/blob/main/data/similarity_top_pairs.csv"
click node_trends "https://github.com/rayan-02/news-sentinel/blob/main/data/topic_trends.csv"
click node_signals "https://github.com/rayan-02/news-sentinel/blob/main/data/emerging_signals.csv"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dawn,node_geo,node_tribune,node_business toneBlue
class node_nlp_clean,node_nlp_features toneAmber
class node_ml_cluster,node_ml_similarity,node_ml_trends,node_ml_signals toneMint
class node_articles,node_nlp_articles,node_embeddings,node_ml_articles,node_clusters,node_similar_pairs,node_trends,node_signals toneRose
class node_operator,node_dawn_site,node_geo_site,node_tribune_site,node_business_site toneIndigo
```

## Components

### News collection

- [Dawn scraper](https://github.com/rayan-02/news-sentinel/blob/main/scrapers/dawn.py): `scrapers/dawn.py`
- [Geo scraper](https://github.com/rayan-02/news-sentinel/blob/main/scrapers/geo.py): `scrapers/geo.py`
- [Tribune scraper](https://github.com/rayan-02/news-sentinel/blob/main/scrapers/tribune.py): `scrapers/tribune.py`
- [Business Recorder scraper](https://github.com/rayan-02/news-sentinel/blob/main/scrapers/business_recorder.py): `scrapers/business_recorder.py`

### Article enrichment

- [Clean articles](https://github.com/rayan-02/news-sentinel/blob/main/nlp/nlp.py): `nlp/nlp.py`
- [Extract NLP features](https://github.com/rayan-02/news-sentinel/blob/main/nlp/nlp.py): `nlp/nlp.py`

### Topic analysis

- [Cluster articles](https://github.com/rayan-02/news-sentinel/blob/main/ml/ml_analysis.py): `ml/ml_analysis.py`
- [Find similar articles](https://github.com/rayan-02/news-sentinel/blob/main/ml/ml_analysis.py): `ml/ml_analysis.py`
- [Calculate topic trends](https://github.com/rayan-02/news-sentinel/blob/main/ml/ml_analysis.py): `ml/ml_analysis.py`
- [Detect emerging topics](https://github.com/rayan-02/news-sentinel/blob/main/ml/ml_analysis.py): `ml/ml_analysis.py`

### Data artifacts

- [Raw articles](https://github.com/rayan-02/news-sentinel/blob/main/data/articles.csv): `data/articles.csv`
- [NLP articles](https://github.com/rayan-02/news-sentinel/blob/main/data/articles_nlp.csv): `data/articles_nlp.csv`
- [Article embeddings](https://github.com/rayan-02/news-sentinel/blob/main/data/article_embeddings.npy): `data/article_embeddings.npy`
- [ML articles](https://github.com/rayan-02/news-sentinel/blob/main/data/articles_ml.csv): `data/articles_ml.csv`
- [Cluster summary](https://github.com/rayan-02/news-sentinel/blob/main/data/clusters_summary.csv): `data/clusters_summary.csv`
- [Similar pairs](https://github.com/rayan-02/news-sentinel/blob/main/data/similarity_top_pairs.csv): `data/similarity_top_pairs.csv`
- [Topic trends](https://github.com/rayan-02/news-sentinel/blob/main/data/topic_trends.csv): `data/topic_trends.csv`
- [Emerging signals](https://github.com/rayan-02/news-sentinel/blob/main/data/emerging_signals.csv): `data/emerging_signals.csv`

### Other components

- **Operator**
- **Dawn**
- **Geo News**
- **Express Tribune**
- **Business Recorder**

## Connections

- Operator → Dawn scraper: runs
- Operator → Geo scraper: runs
- Operator → Tribune scraper: runs
- Operator → Business Recorder scraper: runs
- Operator → Clean articles: runs
- Operator → Cluster articles: runs
- Dawn scraper → Dawn: fetches
- Geo scraper → Geo News: fetches
- Tribune scraper → Express Tribune: fetches
- Business Recorder scraper → Business Recorder: fetches
- Dawn scraper → Raw articles: appends articles
- Geo scraper → Raw articles: appends articles
- Tribune scraper → Raw articles: appends articles
- Business Recorder scraper → Raw articles: appends articles
- Raw articles → Clean articles: reads
- Clean articles → Extract NLP features: passes cleaned data
- Extract NLP features → NLP articles: writes
- Extract NLP features → Article embeddings: writes
- NLP articles → Cluster articles: reads
- Article embeddings → Cluster articles: reads
- Cluster articles → Find similar articles: passes clustered data
- Find similar articles → Calculate topic trends: passes enriched data
- Calculate topic trends → Detect emerging topics: passes trends
- Cluster articles → ML articles: writes
- Cluster articles → Cluster summary: writes
- Find similar articles → Similar pairs: writes
- Calculate topic trends → Topic trends: writes
- Detect emerging topics → Emerging signals: writes