# terbinm/news_scraper_web architecture

> Architecture diagram of the GitHub repository terbinm/news_scraper_web, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/terbinm/news_scraper_web
- Repository: https://github.com/terbinm/news_scraper_web

## Overview

##### Project type and purpose
- This is a full‐stack Django web application: a “新聞爬蟲與全文檢索關聯分析系統” (News Scraper & Text‐Analysis Platform).
- It crawls articles from China Times, performs Chinese NLP (CKIP, NER, keyword extraction, sentiment), provides full‐text search, multi‐dimension analyses, and visualizes results in the browser.

##### File structure
- Top‐level Django project:
- news_scraper_web/ (settings.py, URLs, ASGI/WSGI)
- manage.py
- Django app “scraper/” with subfolders:
- services/ (analysis_service.py, scraper_service.py, search_service.py, sentiment_service.py, task_service.py)
- tasks.py (background tasks)
- utils/ (chart_utils, scraper_utils, sentiment_analyzer)
- models.py, views.py, forms.py, urls.py, admin.py
- Static assets (css, js, images) and templates for front end.

##### README for additional insights
- System modules explicitly defined:
1. 爬蟲模組 (CTSimpleScraper via Selenium & undetected-chromedriver)
2. 分析模組 (CKIP Transformers for tokenization/NER/keywords)
3. 檢索模組 (SearchAnalysisService using Django ORM + custom filters)
4. 情感分析模組 (RoBERTa-based Transformer)
5. 視覺化模組 (Chart.js, D3.js)
6. 用戶管理模組 (Django auth)

##### Details
- ER relationships in a side inset (ScrapeJob → Article → SentimentAnalysis, KeywordAnalysis, NamedEntityAnalysis, CategorySentimentSummary)
- Caching layer between SearchService and DB
- Celery broker & worker separation
- Production specifics: Gunicorn, Nginx, Supervisor if desired

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface Layer
    subgraph "User Interface" 
        Browser["User Browser"]:::external
    end

    %% Web Application Layer
    subgraph "Web Layer" 
        DjangoServer["Django Server (ASGI/WSGI)"]:::web
        URLRouter["URL Router"]:::web
        Views["Views (scraper/views.py)"]:::web
        Templates["Templates"]:::web
        StaticAssets["Static Assets (CSS/JS)"]:::web
        Browser -->|"HTTP Request"| DjangoServer
        DjangoServer -->|"routes to"| URLRouter
        URLRouter -->|"calls"| Views
        Views -->|"renders"| Templates
        Views -->|"serves"| StaticAssets
    end

    %% Service Layer
    subgraph "Services Layer" 
        ScraperService["ScraperService"]:::service
        AnalysisService["AnalysisService"]:::service
        SentimentService["SentimentService"]:::service
        SearchService["SearchService"]:::service
        TaskService["TaskService"]:::service
    end

    %% Background Task Layer
    subgraph "Background Layer" 
        CeleryWorker["Celery Worker"]:::background
        RedisBroker["Redis (Broker & Cache)"]:::data
        Views -->|"trigger task"| TaskService
        TaskService -->|"enqueue"| CeleryWorker
        CeleryWorker -->|"uses"| ScraperService
        CeleryWorker -->|"uses"| AnalysisService
        CeleryWorker -->|"uses"| SentimentService
        CeleryWorker -->|"uses"| SearchService
        CeleryWorker -->|"publishes"| RedisBroker
    end

    %% Data Layer
    subgraph "Data Layer" 
        Database["Database (SQLite/PostgreSQL)"]:::data
        Media["Media Storage"]:::data
        Cache["Redis Cache"]:::data
        ScraperService -->|"writes Article, ScrapeJob"| Database
        AnalysisService -->|"writes Analysis tables"| Database
        SentimentService -->|"writes Sentiment tables"| Database
        SearchService -->|"queries"| Database
        SearchService -->|"reads/writes"| Cache
    end

    %% External Dependencies
    subgraph "External Services" 
        ChinaTimes["China Times Website"]:::external
        CKIP["CKIP Transformers"]:::external
        HFModels["HuggingFace Models"]:::external
        ScraperService -->|"crawls"| ChinaTimes
        AnalysisService -->|"calls"| CKIP
        SentimentService -->|"loads"| HFModels
    end

    %% Search flow back to UI
    SearchService -->|"returns data"| Views
    Views -->|"JSON/HTML"| Browser

    %% Click Events
    click DjangoServer "https://github.com/terbinm/news_scraper_web/blob/master/news_scraper_web/asgi.py"
    click DjangoServer "https://github.com/terbinm/news_scraper_web/blob/master/news_scraper_web/wsgi.py"
    click URLRouter "https://github.com/terbinm/news_scraper_web/blob/master/news_scraper_web/urls.py"
    click Views "https://github.com/terbinm/news_scraper_web/blob/master/scraper/views.py"
    click Templates "https://github.com/terbinm/news_scraper_web/blob/master/templates/base.html"
    click Templates "https://github.com/terbinm/news_scraper_web/blob/master/templates/scraper/*.html"
    click StaticAssets "https://github.com/terbinm/news_scraper_web/blob/master/static/css/main.css"
    click StaticAssets "https://github.com/terbinm/news_scraper_web/blob/master/static/js/main.js"
    click ScraperService "https://github.com/terbinm/news_scraper_web/blob/master/scraper/services/scraper_service.py"
    click AnalysisService "https://github.com/terbinm/news_scraper_web/blob/master/scraper/services/analysis_service.py"
    click SentimentService "https://github.com/terbinm/news_scraper_web/blob/master/scraper/services/sentiment_service.py"
    click SearchService "https://github.com/terbinm/news_scraper_web/blob/master/scraper/services/search_service.py"
    click TaskService "https://github.com/terbinm/news_scraper_web/blob/master/scraper/services/task_service.py"
    click RedisBroker "https://github.com/terbinm/news_scraper_web/blob/master/requirements.txt"
    click Database "https://github.com/terbinm/news_scraper_web/blob/master/database-erd.mermaid"
    click Views "https://github.com/terbinm/news_scraper_web/blob/master/scraper/urls.py"
    click Views "https://github.com/terbinm/news_scraper_web/blob/master/scraper/forms.py"
    click Views "https://github.com/terbinm/news_scraper_web/blob/master/scraper/api.py"
    click Views "https://github.com/terbinm/news_scraper_web/blob/master/scraper/admin.py"
    click CeleryWorker "https://github.com/terbinm/news_scraper_web/blob/master/scraper/tasks.py"

    %% Styles
    classDef web fill:#b3d7ff,stroke:#036,stroke-width:1px
    classDef service fill:#c2f0c2,stroke:#070,stroke-width:1px
    classDef background fill:#ffd9b3,stroke:#f60,stroke-width:1px
    classDef data fill:#e0e0e0,stroke:#666,stroke-width:1px
    classDef external fill:#e6ccff,stroke:#704,stroke-width:1px
```
