# news-scrape-agent/flipr_hackathon architecture

> Architecture diagram of the GitHub repository news-scrape-agent/flipr_hackathon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/news-scrape-agent/flipr_hackathon
- Repository: https://github.com/news-scrape-agent/flipr_hackathon

## Overview

##### Project Type & Purpose
- This is a data-pipeline-style, full-stack Python application that:
- Accepts user queries (via Chainlit/CLI)
- Scrapes news from multiple sources (latest, location, topic)
- Clusters articles with BERT
- Summarizes & generates blogs via a self-hosted LLM (Ollama Llama3.2)
- Optionally translates via MyMemory API
- Publishes to WordPress via OAuth2

##### File Structure for Components
- Top-level Scripts & Entry Points:
- app.py (Chainlit/Flask interface)
- process_user_query.py (fallback parameter extractor)
- Scraper Modules (scrapers/…):
- scrapers/latest_news_scrapers/
- scrapers/location_news_scrapers/
- scrapers/topic_news_scrapers/
- scrapers_call.py (orchestrates scraper selection)
- NLP & Model Components (LangTransModel/ + bert_labelling.py + language_translate_model.py):
- processor.py & collator.py (data prep)
- evaluator.py (model eval)
- version information
- Blog & Translation Components:
- generate_blog.py (LLM summarization & blog structure)
- language_translate_api.py (MyMemory integration)
- image.py (optional media handling)
- wordpress_blog_publish.py (OAuth2 WordPress client)
- Config & Utilities:
- tools_config.py (constants, endpoints)
- requirements.txt
- .env (externalized credentials)
- Supporting Artifacts:
- Jupyter notebooks (inference.ipynb, news-classification-using-bert.ipynb)
- model/ directory for BERT checkpoints
- CSV data for city/state mapping

##### Extract Architecture Notes from README
- Pipeline stages described under “🔄 Workflow.”
- External dependencies: Ollama (Llama3.2), MyMemory API, WordPress OAuth2.
- Emphasizes four query parameters (LatestNews, Topics, Language, Location).
- Fallback to manual extraction if LLM output is malformed.

1. User Interface (Chainlit/CLI)
2. Query Processor
– Primary: self-hosted LLM service (Ollama)
– Fallback: process_user_query.py
3. Scraper Orchestrator
– LatestNewsScraper(s)
– LocationNewsScraper(s)
– TopicNewsScraper(s)
4. Data Store (optional caching/logging – could be ephemeral or flat-files/CSV)
5. Classification Module (BERT model in LangTransModel + bert_labelling.py)
6. Summarization & Blog Generator (generate_blog.py via LLM)
7. Translation Module (language_translate_api.py → MyMemory API)
8. Publisher Module (wordpress_blog_publish.py → WordPress REST API)
User → UI → Query Processor → Scraper Orchestrator → Scraped Articles →
Classification → Summarized Articles → [Translation if needed] → Blog Content →
Publisher → WordPress Site
c. Architectural Patterns & Principles:
– Modular pipeline (each stage a separate module)
– Separation of concerns (scraping vs. NLP vs. publishing)
– Externalize config (tools_config.py + .env)
– Plug-and-play scrapers (folder per category)
– Fallback logic for robustness
d. Key Technologies & Libraries (annotate alongside components):
– Python 3.x, Chainlit (for UI), requests/beautifulsoup (scrapers)
– Transformers/BERT (bert_labelling)
– Ollama Llama3.2 (self-hosted LLM service)
– MyMemory REST API (translation)
– WordPress OAuth2 & REST API (wp-publish)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface
    UI["User Interface\n(app.py)"]:::core
    click UI "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/app.py"

    %% Query Processor
    subgraph "Query Processor"
        direction TB
        LLMService["Ollama Llama3.2\n(External LLM)"]:::external
        Fallback["Fallback Processor\n(process_user_query.py)"]:::core
        click Fallback "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/process_user_query.py"
    end

    UI -->|"JSON query"| LLMService
    UI -->|"JSON query"| Fallback

    %% Scraper Orchestrator
    ScraperOrch["Scraper Orchestrator\n(scrapers_call.py)"]:::core
    click ScraperOrch "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/scrapers_call.py"

    LLMService -->|"Parsed Params"| ScraperOrch
    Fallback -->|"Parsed Params"| ScraperOrch

    subgraph "Scraper Modules"
        direction TB
        Latest["LatestNewsScraper\n(scrapers/latest_news_scrapers/)"]:::core
        click Latest "https://github.com/news-scrape-agent/flipr_hackathon/tree/main/scrapers/latest_news_scrapers/"
        Location["LocationNewsScraper\n(scrapers/location_news_scrapers/)"]:::core
        click Location "https://github.com/news-scrape-agent/flipr_hackathon/tree/main/scrapers/location_news_scrapers/"
        Topic["TopicNewsScraper\n(scrapers/topic_news_scrapers/)"]:::core
        click Topic "https://github.com/news-scrape-agent/flipr_hackathon/tree/main/scrapers/topic_news_scrapers/"
    end

    ScraperOrch -->|"Scrape Jobs"| Latest
    ScraperOrch -->|"Scrape Jobs"| Location
    ScraperOrch -->|"Scrape Jobs"| Topic

    %% Data Store (optional)
    DS[(Data Store\nCSV / logs)]:::util

    Latest -->|"Articles JSON"| DS
    Location -->|"Articles JSON"| DS
    Topic -->|"Articles JSON"| DS

    %% Classification Module
    subgraph "Classification Module"
        direction TB
        Processor["Data Processor\n(LangTransModel/processor.py)"]:::core
        click Processor "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/LangTransModel/processor.py"
        Collator["Data Collator\n(LangTransModel/collator.py)"]:::core
        click Collator "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/LangTransModel/collator.py"
        Bert["BERT Labelling\n(bert_labelling.py)"]:::core
        click Bert "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/bert_labelling.py"
        Evaluator["Evaluator\n(LangTransModel/evaluator.py)"]:::core
        click Evaluator "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/LangTransModel/evaluator.py"
        Version["Version Info\n(LangTransModel/version.py)"]:::core
        click Version "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/LangTransModel/version.py"
    end

    DS --> Processor --> Collator --> Bert --> Evaluator
    Version --> Bert

    %% Summarization & Blog Generation
    Summ["Summarization & Blog Generator\n(generate_blog.py)"]:::core
    click Summ "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/generate_blog.py"

    Evaluator -->|"Clustered Articles"| Summ

    %% Translation Module
    subgraph "Translation Module"
        direction TB
        TranslateModel["Translation Model\n(language_translate_model.py)"]:::core
        click TranslateModel "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/language_translate_model.py"
        TranslateAPI["MyMemory API\n(language_translate_api.py)"]:::external
        click TranslateAPI "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/language_translate_api.py"
    end

    Summ -->|"Blog Draft"| TranslateModel -->|"Translation Request"| TranslateAPI
    TranslateAPI -->|"Translated Content"| Summ

    %% Publisher Module
    Publisher["WordPress Publisher\n(wordpress_blog_publish.py)"]:::core
    click Publisher "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/wordpress_blog_publish.py"
    WPService["WordPress REST API\n(External)"]:::external

    Summ -->|"Final Blog Markdown"| Publisher -->|"OAuth2 Publish"| WPService

    %% Configuration & Utilities
    subgraph "Config & Utilities"
        direction TB
        Tools["Tools Config\n(tools_config.py)"]:::util
        click Tools "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/tools_config.py"
        Req["requirements.txt"]:::util
        click Req "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/requirements.txt"
        Readme["README.md"]:::util
        click Readme "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/readme.md"
        Env[".env\n(Environment)"]:::util
    end

    Env --> Tools
    Tools --> UI
    Tools --> ScraperOrch
    Tools --> Processor
    Tools --> Summ
    Tools --> TranslateModel
    Tools --> Publisher

    %% Data & Model Artifacts
    subgraph "Data & Model Artifacts"
        direction TB
        Model["Model Artifacts\n(model/)"]:::util
        click Model "https://github.com/news-scrape-agent/flipr_hackathon/tree/main/model/"
        CSV["City-State CSV\n(indian_cities_and_states.csv)"]:::util
        click CSV "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/indian_cities_and_states.csv"
        CityStates["City-States Script\n(scrapers/cityandstates.py)"]:::util
        click CityStates "https://github.com/news-scrape-agent/flipr_hackathon/blob/main/scrapers/cityandstates.py"
    end

    CSV --> ScraperOrch
    Model --> Processor

    %% Styles
    classDef core fill:#D0E8FF,stroke:#0366d6,color:#000
    classDef external fill:#D5F5E3,stroke:#27ae60,color:#000
    classDef util fill:#E5E7E9,stroke:#7f8c8d,color:#000
```
