How terbinm/news_scraper_web works
Project type and purpose
• This is a full‐stack Django web application: a “新聞爬蟲與全文檢索關聯分析系統” (News Scraper & Text‐Analysis Platform).
• It crawls articles from China Times, performs Chinese NLP (CKIP, NER, keyword extraction, sentiment), provides full‐text search, multi‐dimension analyses, and visualizes results in the browser.
File structure
• Top‐level Django project:
• news_scraper_web/ (settings.py, URLs, ASGI/WSGI)
• manage.py
• Django app “scraper/” with subfolders:
• services/ (analysis_service.py, scraper_service.py, search_service.py, sentiment_service.py, task_service.py)
• tasks.py (background tasks)
• utils/ (chart_utils, scraper_utils, sentiment_analyzer)
• models.py, views.py, forms.py, urls.py, admin.py
• Static assets (css, js, images) and templates for front end.
README for additional insights
• System modules explicitly defined:
1. 爬蟲模組 (CTSimpleScraper via Selenium & undetected-chromedriver)
2. 分析模組 (CKIP Transformers for tokenization/NER/keywords)
3. 檢索模組 (SearchAnalysisService using Django ORM + custom filters)
4. 情感分析模組 (RoBERTa-based Transformer)
5. 視覺化模組 (Chart.js, D3.js)
6. 用戶管理模組 (Django auth)
Details
• ER relationships in a side inset (ScrapeJob → Article → SentimentAnalysis, KeywordAnalysis, NamedEntityAnalysis, CategorySentimentSummary)
• Caching layer between SearchService and DB
• Celery broker & worker separation
• Production specifics: Gunicorn, Nginx, Supervisor if desired