# vaibhav-prasad707/newsage architecture

> Architecture diagram of the GitHub repository vaibhav-prasad707/newsage, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vaibhav-prasad707/newsage
- Repository: https://github.com/vaibhav-prasad707/newsage

## Overview

Newsage combines a financial-news ingestion and AI-enrichment pipeline with a separate Streamlit app for scraping and headline/market exploration. The documented ingestion path discovers sources, fetches and cleans articles, deduplicates them into PostgreSQL, and supports AI analysis and vector storage. The Streamlit app instead lets a user scrape a URL, parse content with Ollama, and explore sentiment and stock-price predictions. Some AI-processing and database wiring is documented or indexed but not shown in full source excerpts; those links are limited accordingly.

Main entry points: [`app/ingestion/pipeline.py`](https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/pipeline.py) (Ingestion orchestrator), [`scripts/process_articles.py`](https://github.com/vaibhav-prasad707/newsage/blob/main/scripts/process_articles.py) (AI batch processor), [`main.py`](https://github.com/vaibhav-prasad707/newsage/blob/main/main.py) (Newsage explorer), [`app/ai/entities/service.py`](https://github.com/vaibhav-prasad707/newsage/blob/main/app/ai/entities/service.py) (Entity extraction)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["News ingestion"]
  node_ingestion_pipeline["Ingestion orchestrator<br/>[pipeline.py]"]
  node_source_registry["Source registry<br/>[config.py]"]
  node_scraper_strategy["Scraper selection<br/>[base.py]"]
  node_rss_scraper["RSS scraper<br/>[rss_scraper.py]"]
  node_requests_scraper["Requests scraper"]
  node_playwright_scraper["Playwright scraper"]
  node_cloudflare_scraper["Cloudflare scraper"]
  node_html_parser["Link parser<br/>[html_parser.py]"]
  node_content_cleaner["Content cleaner<br/>[content_cleaner.py]"]
end

subgraph group_intelligence["AI intelligence"]
  node_article_processor["AI batch processor"]
  node_sentiment["Financial sentiment<br/>[service.py]"]
  node_entity_extraction["Entity extraction<br/>[service.py]"]
  node_summarization["Article summaries<br/>[service.py]"]
  node_embeddings["Semantic embeddings<br/>[service.py]"]
end

subgraph group_storage["Persistence"]
  node_article_repository["Article repository<br/>[news_repository.py]"]
  node_article_store[("Articles and entities<br/>[news.py]")]
  node_pipeline_runs[("Pipeline run records<br/>[pipeline.py]")]
  node_market_models[("Market models<br/>[market.py]")]
  node_ml_models[("ML models<br/>[ml.py]")]
  node_db_connection["Database sessions<br/>[connection.py]"]
end

subgraph group_exploration["Interactive exploration"]
  node_streamlit_app["Newsage explorer<br/>[main.py]"]
  node_scrape_helpers["Web scraping helpers<br/>[scrape.py]"]
  node_parse_helpers["Ollama parsing<br/>[parse.py]"]
end

subgraph group_runtime["Runtime configuration"]
  node_app_config["Application config<br/>[config.py]"]
end

node_user(("User"))
node_news_sources(("News sources"))
node_ollama["Ollama service"]
node_market_data["Market data provider"]

node_ingestion_pipeline -->|"reads sources"| node_source_registry
node_ingestion_pipeline -->|"selects scraper"| node_scraper_strategy
node_ingestion_pipeline -->|"fetches feed"| node_rss_scraper
node_ingestion_pipeline -->|"fetches pages"| node_requests_scraper
node_ingestion_pipeline -->|"fetches pages"| node_playwright_scraper
node_ingestion_pipeline -->|"fetches pages"| node_cloudflare_scraper
node_ingestion_pipeline -->|"parses links"| node_html_parser
node_ingestion_pipeline -->|"cleans content"| node_content_cleaner
node_ingestion_pipeline -->|"stores articles"| node_article_repository
node_ingestion_pipeline -->|"records runs"| node_pipeline_runs
node_news_sources -.->|"provides news"| node_ingestion_pipeline
node_article_repository -->|"queries and writes"| node_article_store
node_article_repository -->|"uses source config"| node_source_registry
node_ingestion_pipeline -->|"opens session"| node_db_connection
node_db_connection -->|"reads database URL"| node_app_config
node_article_processor -->|"analyzes articles"| node_sentiment
node_article_processor -->|"extracts entities"| node_entity_extraction
node_article_processor -->|"summarizes articles"| node_summarization
node_article_processor -->|"encodes articles"| node_embeddings
node_article_processor -->|"loads and updates"| node_article_store
node_article_processor -->|"opens session"| node_db_connection
node_entity_extraction -->|"reads model config"| node_app_config
node_summarization -->|"reads model config"| node_app_config
node_entity_extraction -->|"invokes LLM"| node_ollama
node_summarization -->|"invokes LLM"| node_ollama
node_user -->|"uses explorer"| node_streamlit_app
node_streamlit_app -->|"scrapes URL"| node_scrape_helpers
node_streamlit_app -->|"parses content"| node_parse_helpers
node_streamlit_app -.->|"requests price history"| node_market_data
node_parse_helpers -->|"parses with LLM"| node_ollama
node_streamlit_app -.->|"market capability"| node_market_models
node_streamlit_app -.->|"forecast capability"| node_ml_models

click node_ingestion_pipeline "https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/pipeline.py"
click node_source_registry "https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/sources/config.py"
click node_scraper_strategy "https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/scrapers/base.py"
click node_rss_scraper "https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/scrapers/rss_scraper.py"
click node_requests_scraper "https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/scrapers/requests_scraper.py"
click node_playwright_scraper "https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/scrapers/playwright_scraper.py"
click node_cloudflare_scraper "https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/scrapers/cloudflare_scraper.py"
click node_html_parser "https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/parsers/html_parser.py"
click node_content_cleaner "https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/cleaners/content_cleaner.py"
click node_article_repository "https://github.com/vaibhav-prasad707/newsage/blob/main/app/database/repositories/news_repository.py"
click node_article_store "https://github.com/vaibhav-prasad707/newsage/blob/main/app/database/models/news.py"
click node_pipeline_runs "https://github.com/vaibhav-prasad707/newsage/blob/main/app/database/models/pipeline.py"
click node_market_models "https://github.com/vaibhav-prasad707/newsage/blob/main/app/database/models/market.py"
click node_ml_models "https://github.com/vaibhav-prasad707/newsage/blob/main/app/database/models/ml.py"
click node_db_connection "https://github.com/vaibhav-prasad707/newsage/blob/main/app/database/connection.py"
click node_article_processor "https://github.com/vaibhav-prasad707/newsage/blob/main/scripts/process_articles.py"
click node_sentiment "https://github.com/vaibhav-prasad707/newsage/blob/main/app/ai/sentiment/service.py"
click node_entity_extraction "https://github.com/vaibhav-prasad707/newsage/blob/main/app/ai/entities/service.py"
click node_summarization "https://github.com/vaibhav-prasad707/newsage/blob/main/app/ai/summarization/service.py"
click node_embeddings "https://github.com/vaibhav-prasad707/newsage/blob/main/app/ai/embeddings/service.py"
click node_app_config "https://github.com/vaibhav-prasad707/newsage/blob/main/app/config.py"
click node_streamlit_app "https://github.com/vaibhav-prasad707/newsage/blob/main/main.py"
click node_scrape_helpers "https://github.com/vaibhav-prasad707/newsage/blob/main/src/scrape.py"
click node_parse_helpers "https://github.com/vaibhav-prasad707/newsage/blob/main/src/parse.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingestion_pipeline,node_source_registry,node_scraper_strategy,node_rss_scraper,node_requests_scraper,node_playwright_scraper,node_cloudflare_scraper,node_html_parser,node_content_cleaner,node_user toneBlue
class node_article_processor,node_sentiment,node_entity_extraction,node_summarization,node_embeddings toneAmber
class node_article_repository,node_article_store,node_pipeline_runs,node_market_models,node_ml_models,node_db_connection toneMint
class node_streamlit_app,node_scrape_helpers,node_parse_helpers toneRose
class node_app_config,node_news_sources,node_ollama,node_market_data toneIndigo
```

## Components

### News ingestion

- [Ingestion orchestrator](https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/pipeline.py): `app/ingestion/pipeline.py`
- [Source registry](https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/sources/config.py): `app/ingestion/sources/config.py`
- [Scraper selection](https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/scrapers/base.py): `app/ingestion/scrapers/base.py`
- [RSS scraper](https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/scrapers/rss_scraper.py): `app/ingestion/scrapers/rss_scraper.py`
- [Requests scraper](https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/scrapers/requests_scraper.py): `app/ingestion/scrapers/requests_scraper.py`
- [Playwright scraper](https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/scrapers/playwright_scraper.py): `app/ingestion/scrapers/playwright_scraper.py`
- [Cloudflare scraper](https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/scrapers/cloudflare_scraper.py): `app/ingestion/scrapers/cloudflare_scraper.py`
- [Link parser](https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/parsers/html_parser.py): `app/ingestion/parsers/html_parser.py`
- [Content cleaner](https://github.com/vaibhav-prasad707/newsage/blob/main/app/ingestion/cleaners/content_cleaner.py): `app/ingestion/cleaners/content_cleaner.py`

### AI intelligence

- [AI batch processor](https://github.com/vaibhav-prasad707/newsage/blob/main/scripts/process_articles.py): `scripts/process_articles.py`
- [Financial sentiment](https://github.com/vaibhav-prasad707/newsage/blob/main/app/ai/sentiment/service.py): `app/ai/sentiment/service.py`
- [Entity extraction](https://github.com/vaibhav-prasad707/newsage/blob/main/app/ai/entities/service.py): `app/ai/entities/service.py`
- [Article summaries](https://github.com/vaibhav-prasad707/newsage/blob/main/app/ai/summarization/service.py): `app/ai/summarization/service.py`
- [Semantic embeddings](https://github.com/vaibhav-prasad707/newsage/blob/main/app/ai/embeddings/service.py): `app/ai/embeddings/service.py`

### Persistence

- [Article repository](https://github.com/vaibhav-prasad707/newsage/blob/main/app/database/repositories/news_repository.py): `app/database/repositories/news_repository.py`
- [Articles and entities](https://github.com/vaibhav-prasad707/newsage/blob/main/app/database/models/news.py): `app/database/models/news.py`
- [Pipeline run records](https://github.com/vaibhav-prasad707/newsage/blob/main/app/database/models/pipeline.py): `app/database/models/pipeline.py`
- [Market models](https://github.com/vaibhav-prasad707/newsage/blob/main/app/database/models/market.py): `app/database/models/market.py`
- [ML models](https://github.com/vaibhav-prasad707/newsage/blob/main/app/database/models/ml.py): `app/database/models/ml.py`
- [Database sessions](https://github.com/vaibhav-prasad707/newsage/blob/main/app/database/connection.py): `app/database/connection.py`

### Interactive exploration

- [Newsage explorer](https://github.com/vaibhav-prasad707/newsage/blob/main/main.py): `main.py`
- [Web scraping helpers](https://github.com/vaibhav-prasad707/newsage/blob/main/src/scrape.py): `src/scrape.py`
- [Ollama parsing](https://github.com/vaibhav-prasad707/newsage/blob/main/src/parse.py): `src/parse.py`

### Runtime configuration

- [Application config](https://github.com/vaibhav-prasad707/newsage/blob/main/app/config.py): `app/config.py`

### Other components

- **User**
- **News sources**
- **Ollama service**
- **Market data provider**

## Connections

- Ingestion orchestrator → Source registry: reads sources (evidence: `app/ingestion/pipeline.py`)
- Ingestion orchestrator → Scraper selection: selects scraper (evidence: `app/ingestion/pipeline.py`)
- Ingestion orchestrator → RSS scraper: fetches feed (evidence: `app/ingestion/pipeline.py`)
- Ingestion orchestrator → Requests scraper: fetches pages (evidence: `app/ingestion/pipeline.py`)
- Ingestion orchestrator → Playwright scraper: fetches pages (evidence: `app/ingestion/pipeline.py`)
- Ingestion orchestrator → Cloudflare scraper: fetches pages (evidence: `app/ingestion/pipeline.py`)
- Ingestion orchestrator → Link parser: parses links (evidence: `app/ingestion/pipeline.py`)
- Ingestion orchestrator → Content cleaner: cleans content (evidence: `app/ingestion/pipeline.py`)
- Ingestion orchestrator → Article repository: stores articles (evidence: `app/ingestion/pipeline.py`)
- Ingestion orchestrator → Pipeline run records: records runs (evidence: `app/ingestion/pipeline.py`)
- News sources → Ingestion orchestrator: provides news (evidence: `README.md`)
- Article repository → Articles and entities: queries and writes (evidence: `app/database/repositories/news_repository.py`)
- Article repository → Source registry: uses source config (evidence: `app/database/repositories/news_repository.py`)
- Ingestion orchestrator → Database sessions: opens session (evidence: `app/ingestion/pipeline.py`)
- Database sessions → Application config: reads database URL (evidence: `app/database/connection.py`)
- AI batch processor → Financial sentiment: analyzes articles (evidence: `scripts/process_articles.py`)
- AI batch processor → Entity extraction: extracts entities (evidence: `scripts/process_articles.py`)
- AI batch processor → Article summaries: summarizes articles (evidence: `scripts/process_articles.py`)
- AI batch processor → Semantic embeddings: encodes articles (evidence: `scripts/process_articles.py`)
- AI batch processor → Articles and entities: loads and updates (evidence: `scripts/process_articles.py`)
- AI batch processor → Database sessions: opens session (evidence: `scripts/process_articles.py`)
- Entity extraction → Application config: reads model config (evidence: `app/ai/entities/service.py`)
- Article summaries → Application config: reads model config (evidence: `app/ai/summarization/service.py`)
- Entity extraction → Ollama service: invokes LLM (evidence: `app/ai/entities/service.py`)
- Article summaries → Ollama service: invokes LLM (evidence: `app/ai/summarization/service.py`)
- User → Newsage explorer: uses explorer (evidence: `main.py`)
- Newsage explorer → Web scraping helpers: scrapes URL (evidence: `main.py`)
- Newsage explorer → Ollama parsing: parses content (evidence: `main.py`)
- Newsage explorer → Market data provider: requests price history (evidence: `main.py`)
- Ollama parsing → Ollama service: parses with LLM (evidence: `README.md`)
- Newsage explorer → Market models: market capability
- Newsage explorer → ML models: forecast capability
