How vaibhav-prasad707/newsage works
Newsage combines a financial-news ingestion and AI-enrichment pipeline with a separate Streamlit app for scraping and headline/market exploration. The documented ingestion path discovers sources, fetches and cleans articles, deduplicates them into PostgreSQL, and supports AI analysis and vector storage. The Streamlit app instead lets a user scrape a URL, parse content with Ollama, and explore sentiment and stock-price predictions. Some AI-processing and database wiring is documented or indexed but not shown in full source excerpts; those links are limited accordingly.
Main entry points: pipeline.py Ingestion orchestrator · process_articles.py AI batch processor · main.py Newsage explorer · service.py Entity extraction
Components 28
News ingestion
- Ingestion orchestrator
app/ingestion/pipeline.py - Source registry
app/ingestion/sources/config.py - Scraper selection
app/ingestion/scrapers/base.py - RSS scraper
app/ingestion/scrapers/rss_scraper.py - Requests scraper
app/ingestion/scrapers/requests_scraper.py - Playwright scraper
app/ingestion/scrapers/playwright_scraper.py - Cloudflare scraper
app/ingestion/scrapers/cloudflare_scraper.py - Link parser
app/ingestion/parsers/html_parser.py - Content cleaner
app/ingestion/cleaners/content_cleaner.py
AI intelligence
- AI batch processor
scripts/process_articles.py - Financial sentiment
app/ai/sentiment/service.py - Entity extraction
app/ai/entities/service.py - Article summaries
app/ai/summarization/service.py - Semantic embeddings
app/ai/embeddings/service.py
Persistence
- Article repository
app/database/repositories/news_repository.py - Articles and entities
app/database/models/news.py - Pipeline run records
app/database/models/pipeline.py - Market models
app/database/models/market.py - ML models
app/database/models/ml.py - Database sessions
app/database/connection.py
Interactive exploration
- Newsage explorer
main.py - Web scraping helpers
src/scrape.py - Ollama parsing
src/parse.py
Runtime configuration
- Application config
app/config.py
Other components
- User
- News sources
- Ollama service
- Market data provider
Connections 32
- Ingestion orchestrator → Source registry: reads sources
app/ingestion/pipeline.py - Ingestion orchestrator → Scraper selection: selects scraper
app/ingestion/pipeline.py - Ingestion orchestrator → RSS scraper: fetches feed
app/ingestion/pipeline.py - Ingestion orchestrator → Requests scraper: fetches pages
app/ingestion/pipeline.py - Ingestion orchestrator → Playwright scraper: fetches pages
app/ingestion/pipeline.py - Ingestion orchestrator → Cloudflare scraper: fetches pages
app/ingestion/pipeline.py - Ingestion orchestrator → Link parser: parses links
app/ingestion/pipeline.py - Ingestion orchestrator → Content cleaner: cleans content
app/ingestion/pipeline.py - Ingestion orchestrator → Article repository: stores articles
app/ingestion/pipeline.py - Ingestion orchestrator → Pipeline run records: records runs
app/ingestion/pipeline.py - News sources → Ingestion orchestrator: provides news
README.md - Article repository → Articles and entities: queries and writes
app/database/repositories/news_repository.py - Article repository → Source registry: uses source config
app/database/repositories/news_repository.py - Ingestion orchestrator → Database sessions: opens session
app/ingestion/pipeline.py - Database sessions → Application config: reads database URL
app/database/connection.py - AI batch processor → Financial sentiment: analyzes articles
scripts/process_articles.py - AI batch processor → Entity extraction: extracts entities
scripts/process_articles.py - AI batch processor → Article summaries: summarizes articles
scripts/process_articles.py - AI batch processor → Semantic embeddings: encodes articles
scripts/process_articles.py - AI batch processor → Articles and entities: loads and updates
scripts/process_articles.py - AI batch processor → Database sessions: opens session
scripts/process_articles.py - Entity extraction → Application config: reads model config
app/ai/entities/service.py - Article summaries → Application config: reads model config
app/ai/summarization/service.py - Entity extraction → Ollama service: invokes LLM
app/ai/entities/service.py - Article summaries → Ollama service: invokes LLM
app/ai/summarization/service.py - User → Newsage explorer: uses explorer
main.py - Newsage explorer → Web scraping helpers: scrapes URL
main.py - Newsage explorer → Ollama parsing: parses content
main.py - Newsage explorer → Market data provider: requests price history
main.py - Ollama parsing → Ollama service: parses with LLM
README.md - Newsage explorer → Market models: market capability
- Newsage explorer → ML models: forecast capability