How news-scrape-agent/flipr_hackathon works
Project Type & Purpose
• This is a data-pipeline-style, full-stack Python application that:
• Accepts user queries (via Chainlit/CLI)
• Scrapes news from multiple sources (latest, location, topic)
• Clusters articles with BERT
• Summarizes & generates blogs via a self-hosted LLM (Ollama Llama3.2)
• Optionally translates via MyMemory API
• Publishes to WordPress via OAuth2
File Structure for Components
• Top-level Scripts & Entry Points:
• app.py (Chainlit/Flask interface)
• process_user_query.py (fallback parameter extractor)
• Scraper Modules (scrapers/…):
• scrapers/latest_news_scrapers/
• scrapers/location_news_scrapers/
• scrapers/topic_news_scrapers/
• scrapers_call.py (orchestrates scraper selection)
• NLP & Model Components (LangTransModel/ + bert_labelling.py + language_translate_model.py):
• processor.py & collator.py (data prep)
• evaluator.py (model eval)
• version information
• Blog & Translation Components:
• generate_blog.py (LLM summarization & blog structure)
• language_translate_api.py (MyMemory integration)
• image.py (optional media handling)
• wordpress_blog_publish.py (OAuth2 WordPress client)
• Config & Utilities:
• tools_config.py (constants, endpoints)
• requirements.txt
• .env (externalized credentials)
• Supporting Artifacts:
• Jupyter notebooks (inference.ipynb, news-classification-using-bert.ipynb)
• model/ directory for BERT checkpoints
• CSV data for city/state mapping
Extract Architecture Notes from README
• Pipeline stages described under “🔄 Workflow.”
• External dependencies: Ollama (Llama3.2), MyMemory API, WordPress OAuth2.
• Emphasizes four query parameters (LatestNews, Topics, Language, Location).
• Fallback to manual extraction if LLM output is malformed.
1. User Interface (Chainlit/CLI)
2. Query Processor
– Primary: self-hosted LLM service (Ollama)
– Fallback: process_user_query.py
3. Scraper Orchestrator
– LatestNewsScraper(s)
– LocationNewsScraper(s)
– TopicNewsScraper(s)
4. Data Store (optional caching/logging – could be ephemeral or flat-files/CSV)
5. Classification Module (BERT model in LangTransModel + bert_labelling.py)
6. Summarization & Blog Generator (generate_blog.py via LLM)
7. Translation Module (language_translate_api.py → MyMemory API)
8. Publisher Module (wordpress_blog_publish.py → WordPress REST API)
User → UI → Query Processor → Scraper Orchestrator → Scraped Articles →
Classification → Summarized Articles → [Translation if needed] → Blog Content →
Publisher → WordPress Site
c. Architectural Patterns & Principles:
– Modular pipeline (each stage a separate module)
– Separation of concerns (scraping vs. NLP vs. publishing)
– Externalize config (tools_config.py + .env)
– Plug-and-play scrapers (folder per category)
– Fallback logic for robustness
d. Key Technologies & Libraries (annotate alongside components):
– Python 3.x, Chainlit (for UI), requests/beautifulsoup (scrapers)
– Transformers/BERT (bert_labelling)
– Ollama Llama3.2 (self-hosted LLM service)
– MyMemory REST API (translation)
– WordPress OAuth2 & REST API (wp-publish)