How rayan-02/news-sentinel works
News Sentinel is a news-collection and analysis pipeline: standalone source scrapers append articles to a shared CSV, NLP cleans and enriches those articles and creates embeddings, and ML produces clusters, similarity results, topic trends, and emerging-signal reports. The README is empty; scraper and NLP-to-ML handoffs are supported by the inspected source, while the ML excerpt documents its output artifacts and main-stage calls. main.py only prints “hi” and is not part of this workflow.
Main entry points: ml_analysis.py Cluster articles · articles.csv Raw articles · dawn.py Dawn scraper · geo.py Geo scraper
Components 23
News collection
- Dawn scraper
scrapers/dawn.py - Geo scraper
scrapers/geo.py - Tribune scraper
scrapers/tribune.py - Business Recorder scraper
scrapers/business_recorder.py
Article enrichment
- Clean articles
nlp/nlp.py - Extract NLP features
nlp/nlp.py
Topic analysis
- Cluster articles
ml/ml_analysis.py - Find similar articles
ml/ml_analysis.py - Calculate topic trends
ml/ml_analysis.py - Detect emerging topics
ml/ml_analysis.py
Data artifacts
- Raw articles
data/articles.csv - NLP articles
data/articles_nlp.csv - Article embeddings
data/article_embeddings.npy - ML articles
data/articles_ml.csv - Cluster summary
data/clusters_summary.csv - Similar pairs
data/similarity_top_pairs.csv - Topic trends
data/topic_trends.csv - Emerging signals
data/emerging_signals.csv
Other components
- Operator
- Dawn
- Geo News
- Express Tribune
- Business Recorder
Connections 28
- Operator → Dawn scraper: runs
- Operator → Geo scraper: runs
- Operator → Tribune scraper: runs
- Operator → Business Recorder scraper: runs
- Operator → Clean articles: runs
- Operator → Cluster articles: runs
- Dawn scraper → Dawn: fetches
- Geo scraper → Geo News: fetches
- Tribune scraper → Express Tribune: fetches
- Business Recorder scraper → Business Recorder: fetches
- Dawn scraper → Raw articles: appends articles
- Geo scraper → Raw articles: appends articles
- Tribune scraper → Raw articles: appends articles
- Business Recorder scraper → Raw articles: appends articles
- Raw articles → Clean articles: reads
- Clean articles → Extract NLP features: passes cleaned data
- Extract NLP features → NLP articles: writes
- Extract NLP features → Article embeddings: writes
- NLP articles → Cluster articles: reads
- Article embeddings → Cluster articles: reads
- Cluster articles → Find similar articles: passes clustered data
- Find similar articles → Calculate topic trends: passes enriched data
- Calculate topic trends → Detect emerging topics: passes trends
- Cluster articles → ML articles: writes
- Cluster articles → Cluster summary: writes
- Find similar articles → Similar pairs: writes
- Calculate topic trends → Topic trends: writes
- Detect emerging topics → Emerging signals: writes