# multiverse-of-projects/newsai architecture

> Architecture diagram of the GitHub repository multiverse-of-projects/newsai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/multiverse-of-projects/newsai
- Repository: https://github.com/multiverse-of-projects/newsai

## Overview

##### Project type and purpose
- NewsAI is an open‑source, full‑stack data pipeline + dashboard application.
- It ingests breaking news from external APIs (NewsAPI, Reddit/PRAW), processes it (summarization, keyword extraction, sentiment analysis), stores it in MongoDB, and serves insights via a Streamlit dashboard (and a FastAPI backend as described in the README).

##### File structure
- Top‑level:
- pipeline.py – orchestrates the end‑to‑end ETL pipeline.
- src/ingestion – modules that fetch raw articles from NewsAPI and PRAW.
- src/preprocessing – text processing: keyword_extraction.py, summarization.py.
- src/sentiment_analysis – BERT‑based sentiment_model, classify, wordcloud.
- src/dashboard/app.py – Streamlit UI for visualization.
- src/utils – dbconnector (MongoDB), logger.
- Config & metadata: .env.example (API keys), requirements.txt, docs/ (Sphinx), GitHub workflows.

##### Extract insights from the README
- Core stack: FastAPI (backend), Streamlit (dashboard), Gemini AI & BERT (ML models), MongoDB (NoSQL store).
- Use cases: real‑time news aggregation, AI‑driven insights, visual exploration.

a. Main components
1. External Data Sources: NewsAPI, Reddit via PRAW (and potentially Gemini AI’s endpoints).
2. Ingestion Layer: src/ingestion/fetch_articles.py orchestrated by pipeline.py.
3. Pipeline Orchestrator: pipeline.py coordinates ingestion → preprocessing → sentiment.
4. Preprocessing Layer: keyword_extraction, summarization.
5. Sentiment Analysis Layer: BERT model, classification, wordcloud generation.
6. Persistence Layer: MongoDB via src/utils/dbconnector.py.
7. Backend API: FastAPI endpoints (if implemented in pipeline or a dedicated module).
8. Presentation Layer: Streamlit dashboard (src/dashboard/app.py).
9. Utility Services: Logging (src/utils/logger.py), Configuration (.env).

b. Relationships & data flow
Preprocessing → Sentiment Analysis → MongoDB.
- API calls: Frontend (Streamlit) ↔ Backend (FastAPI) ↔ MongoDB.
- ML calls: Preprocessing & Sentiment layers invoke Gemini AI / BERT endpoints.
- Logger captures events at each stage; .env provides credentials/config.

c. Architectural patterns & principles
- Layered (ETL) architecture: separate ingestion, transformation, analysis, presentation.
- Modular code organization: each domain (ingestion/preprocessing/sentiment/dashboard) is its own package.
- Externalization of config (.env), decoupled persistence (dbconnector).
- Asynchronous or batch pipelines (pipeline.py can be scheduled or triggered).

d. Key technologies, frameworks, libraries
- Python packages: FastAPI, Streamlit, PRAW, NewsAPI client, transformers (BERT), pymongo.
- MongoDB (NoSQL store).
- Logging via custom logger.
- CI/CD via GitHub Actions, documentation via Sphinx/ReadTheDocs.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    NewsAPI["NewsAPI"]:::external
    PRAW["Reddit/PRAW"]:::external

    %% Ingestion Layer
    Fetch["Ingestion: fetch_articles.py"]:::etl

    %% Pipeline Orchestrator
    Pipeline["Pipeline Orchestrator: pipeline.py"]:::etl

    %% Preprocessing Layer
    Keyword["Preprocessing: keyword_extraction.py"]:::etl
    Summarization["Preprocessing: summarization.py"]:::etl

    %% Sentiment Analysis Layer
    SentModel["Sentiment Analysis: sentiment_model.py"]:::etl
    Classify["Sentiment Analysis: classify.py"]:::etl
    Wordcloud["Sentiment Analysis: wordcloud.py"]:::etl

    %% Persistence Layer
    DBConn["Persistence: dbconnector.py"]:::storage
    MongoDB["MongoDB"]:::storage

    %% Backend & Frontend
    FastAPI["Backend API: FastAPI"]:::backend
    Dashboard["Presentation: Streamlit UI"]:::ui

    %% Utility Services
    Logger["Logging Service: logger.py"]:::util
    Config["Configuration: .env.example"]:::util
    Docs["Documentation: docs/"]:::util
    CI["CI/CD: .github/workflows/"]:::util

    %% Data Flow
    NewsAPI -->|raw articles| Fetch
    PRAW -->|raw articles| Fetch
    Fetch -->|fetched data| Pipeline
    Pipeline -->|text data| Keyword
    Pipeline -->|text data| Summarization
    Keyword -->|keywords| Classify
    Summarization -->|summary| Classify
    Classify -->|sentiment scores| Wordcloud
    Wordcloud -->|visuals| DBConn
    Classify -->|results| DBConn
    DBConn -->|stores data| MongoDB
    MongoDB -->|data query| FastAPI
    FastAPI -->|API responses| Dashboard
    Dashboard -->|user actions| FastAPI

    %% Config and Logging
    Config --> Fetch
    Config --> Pipeline
    Config --> FastAPI
    Logger --> Fetch
    Logger --> Classify
    Logger --> FastAPI

    %% Supporting Components
    Docs -.-> Pipeline
    CI -.-> Pipeline

    %% Click Events
    click NewsAPI "https://github.com/multiverse-of-projects/newsai/blob/main/src/ingestion/newsapi.py"
    click PRAW "https://github.com/multiverse-of-projects/newsai/blob/main/src/ingestion/prawapi.py"
    click Fetch "https://github.com/multiverse-of-projects/newsai/blob/main/src/ingestion/fetch_articles.py"
    click Pipeline "https://github.com/multiverse-of-projects/newsai/blob/main/src/pipeline.py"
    click Keyword "https://github.com/multiverse-of-projects/newsai/blob/main/src/preprocessing/keyword_extraction.py"
    click Summarization "https://github.com/multiverse-of-projects/newsai/blob/main/src/preprocessing/summarization.py"
    click SentModel "https://github.com/multiverse-of-projects/newsai/blob/main/src/sentiment_analysis/sentiment_model.py"
    click Classify "https://github.com/multiverse-of-projects/newsai/blob/main/src/sentiment_analysis/classify.py"
    click Wordcloud "https://github.com/multiverse-of-projects/newsai/blob/main/src/sentiment_analysis/wordcloud.py"
    click DBConn "https://github.com/multiverse-of-projects/newsai/blob/main/src/utils/dbconnector.py"
    click Dashboard "https://github.com/multiverse-of-projects/newsai/blob/main/src/dashboard/app.py"
    click Logger "https://github.com/multiverse-of-projects/newsai/blob/main/src/utils/logger.py"
    click Config "https://github.com/multiverse-of-projects/newsai/blob/main/.env.example"
    click Docs "https://github.com/multiverse-of-projects/newsai/tree/main/docs/"
    click CI "https://github.com/multiverse-of-projects/newsai/tree/main/.github/workflows/"

    %% Styles
    classDef etl fill:#AED6F1,stroke:#2980B9,color:#1B2631
    classDef storage fill:#ABEBC6,stroke:#27AE60,color:#1B2631
    classDef backend fill:#F5B7B1,stroke:#C0392B,color:#1B2631
    classDef ui fill:#F9E79F,stroke:#B7950B,color:#1B2631
    classDef external fill:#D7DBDD,stroke:#7F8C8D,color:#1B2631
    classDef util fill:#D2B4DE,stroke:#8E44AD,color:#1B2631
```
