How codelucas/newspaper works
Project type and purpose
• This is an open-source Python3 library (not a web app) whose sole purpose is to scrape, parse, and curate news articles from arbitrary websites. It provides an API (Article, build/Paper) that your own code calls, a multi-threaded download framework, an HTML-to-text extraction pipeline, and optional NLP (keywords, summary) stages.
File structure
• Top-level code:
• newspaper/api.py – public API (Article class, build() function)
• download_corpora.py – CLI to fetch NLP corpora
• setup.py, requirements.txt – build/dependency definitions
• Core modules (in newspaper/):
• network.py – HTTP fetch (wrapping requests, uses user-agents list)
• mthreading.py – download thread pool & task queue
• parsers.py / cleaners.py – HTML parsing and cleaning (lxml)
• extractors.py – main content extraction (text, title, authors, metadata)
• nlp.py – keyword & summary extraction (NLTK, stopwords)
• images.py / videos/ – extract media URLs
• outputformatters.py – final output formats (plaintext, JSON)
• configuration.py / settings.py – global settings (defaults, timeouts)
• Resources:
• newspaper/resources/misc/*.txt – popular sources, user agents, google sources
• newspaper/resources/text/stopwords-*.txt – per-language stopwords
• Tests:
• tests/unit_tests.py and tests/data/ – mocks and sample HTML/text files for each site
• Documentation:
• docs/ – Sphinx source for user guide, API reference
• Client Code (your application)
• Public API Layer (Article, Paper builder)
• Download Framework (mthreading + network)
• Parsing & Cleaning Layer (parsers + cleaners)
• Extraction Layer (extractors, images, videos)
• NLP Layer (nlp module + stopwords resources)
• Output Formatter (JSON, plain text)
• Resource Store (stopwords, user-agents, google_sources)
• External Libraries & Services (requests, lxml, nltk, Pillow)
1. Client → calls API (Article.download / build)
2. API → schedules fetch tasks to Download Framework
3. Download Framework → issues HTTP GETs via network.py (requests + user-agents)
4. Raw HTML → Parsing & Cleaning Layer → cleaned DOM/tree
5. Cleaned tree → Extraction Layer → article.text, authors, top_image, videos
6. article.text → NLP Layer → keywords, summary (uses stopwords resources + nltk)
7. Final objects → Output Formatter → JSON/plain-text delivered back to Client
8. build() → also traverses site to collect category & article URLs
c. Architectural patterns & principles:
• Pipeline pattern: clearly separated stages (Download → Parse/Clean → Extract → NLP → Format)
• Modular design: each responsibility in its own module
• Resource-driven: uses external text files for stopwords and user-agents for easy extensibility
• Thread-safe / multi-threaded download: thread pool managing concurrent fetches
d. Key technologies & libraries:
• Python3
• requests (HTTP client)
• lxml (fast HTML parsing)
• nltk (keyword & summary extraction)
• Pillow (image handling, optional)
• Sphinx (documentation generation)