How multiverse-of-projects/newsai works
Project type and purpose
• NewsAI is an open‑source, full‑stack data pipeline + dashboard application.
• It ingests breaking news from external APIs (NewsAPI, Reddit/PRAW), processes it (summarization, keyword extraction, sentiment analysis), stores it in MongoDB, and serves insights via a Streamlit dashboard (and a FastAPI backend as described in the README).
File structure
• Top‑level:
• pipeline.py – orchestrates the end‑to‑end ETL pipeline.
• src/ingestion – modules that fetch raw articles from NewsAPI and PRAW.
• src/preprocessing – text processing: keyword_extraction.py, summarization.py.
• src/sentiment_analysis – BERT‑based sentiment_model, classify, wordcloud.
• src/dashboard/app.py – Streamlit UI for visualization.
• src/utils – dbconnector (MongoDB), logger.
• Config & metadata: .env.example (API keys), requirements.txt, docs/ (Sphinx), GitHub workflows.
Extract insights from the README
• Core stack: FastAPI (backend), Streamlit (dashboard), Gemini AI & BERT (ML models), MongoDB (NoSQL store).
• Use cases: real‑time news aggregation, AI‑driven insights, visual exploration.
a. Main components
1. External Data Sources: NewsAPI, Reddit via PRAW (and potentially Gemini AI’s endpoints).
2. Ingestion Layer: src/ingestion/fetch_articles.py orchestrated by pipeline.py.
3. Pipeline Orchestrator: pipeline.py coordinates ingestion → preprocessing → sentiment.
4. Preprocessing Layer: keyword_extraction, summarization.
5. Sentiment Analysis Layer: BERT model, classification, wordcloud generation.
6. Persistence Layer: MongoDB via src/utils/dbconnector.py.
7. Backend API: FastAPI endpoints (if implemented in pipeline or a dedicated module).
8. Presentation Layer: Streamlit dashboard (src/dashboard/app.py).
9. Utility Services: Logging (src/utils/logger.py), Configuration (.env).
b. Relationships & data flow
Preprocessing → Sentiment Analysis → MongoDB.
• API calls: Frontend (Streamlit) ↔ Backend (FastAPI) ↔ MongoDB.
• ML calls: Preprocessing & Sentiment layers invoke Gemini AI / BERT endpoints.
• Logger captures events at each stage; .env provides credentials/config.
c. Architectural patterns & principles
• Layered (ETL) architecture: separate ingestion, transformation, analysis, presentation.
• Modular code organization: each domain (ingestion/preprocessing/sentiment/dashboard) is its own package.
• Externalization of config (.env), decoupled persistence (dbconnector).
• Asynchronous or batch pipelines (pipeline.py can be scheduled or triggered).
d. Key technologies, frameworks, libraries
• Python packages: FastAPI, Streamlit, PRAW, NewsAPI client, transformers (BERT), pymongo.
• MongoDB (NoSQL store).
• Logging via custom logger.
• CI/CD via GitHub Actions, documentation via Sphinx/ReadTheDocs.