# codelucas/newspaper architecture

> Architecture diagram of the GitHub repository codelucas/newspaper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/codelucas/newspaper
- Repository: https://github.com/codelucas/newspaper

## Overview

##### Project type and purpose
- This is an open-source Python3 library (not a web app) whose sole purpose is to scrape, parse, and curate news articles from arbitrary websites. It provides an API (Article, build/Paper) that your own code calls, a multi-threaded download framework, an HTML-to-text extraction pipeline, and optional NLP (keywords, summary) stages.

##### File structure
- Top-level code:
- newspaper/api.py – public API (Article class, build() function)
- download_corpora.py – CLI to fetch NLP corpora
- setup.py, requirements.txt – build/dependency definitions
- Core modules (in newspaper/):
- network.py – HTTP fetch (wrapping requests, uses user-agents list)
- mthreading.py – download thread pool & task queue
- parsers.py / cleaners.py – HTML parsing and cleaning (lxml)
- extractors.py – main content extraction (text, title, authors, metadata)
- nlp.py – keyword & summary extraction (NLTK, stopwords)
- images.py / videos/ – extract media URLs
- outputformatters.py – final output formats (plaintext, JSON)
- configuration.py / settings.py – global settings (defaults, timeouts)
- Resources:
- newspaper/resources/misc/*.txt – popular sources, user agents, google sources
- newspaper/resources/text/stopwords-*.txt – per-language stopwords
- Tests:
- tests/unit_tests.py and tests/data/ – mocks and sample HTML/text files for each site
- Documentation:
- docs/ – Sphinx source for user guide, API reference

- Client Code (your application)
- Public API Layer (Article, Paper builder)
- Download Framework (mthreading + network)
- Parsing & Cleaning Layer (parsers + cleaners)
- Extraction Layer (extractors, images, videos)
- NLP Layer (nlp module + stopwords resources)
- Output Formatter (JSON, plain text)
- Resource Store (stopwords, user-agents, google_sources)
- External Libraries & Services (requests, lxml, nltk, Pillow)

1. Client → calls API (Article.download / build)
2. API → schedules fetch tasks to Download Framework
3. Download Framework → issues HTTP GETs via network.py (requests + user-agents)
4. Raw HTML → Parsing & Cleaning Layer → cleaned DOM/tree
5. Cleaned tree → Extraction Layer → article.text, authors, top_image, videos
6. article.text → NLP Layer → keywords, summary (uses stopwords resources + nltk)
7. Final objects → Output Formatter → JSON/plain-text delivered back to Client
8. build() → also traverses site to collect category & article URLs

c. Architectural patterns & principles:
- Pipeline pattern: clearly separated stages (Download → Parse/Clean → Extract → NLP → Format)
- Modular design: each responsibility in its own module
- Resource-driven: uses external text files for stopwords and user-agents for easy extensibility
- Thread-safe / multi-threaded download: thread pool managing concurrent fetches

d. Key technologies & libraries:
- Python3
- requests (HTTP client)
- lxml (fast HTML parsing)
- nltk (keyword & summary extraction)
- Pillow (image handling, optional)
- Sphinx (documentation generation)

## Diagram (Mermaid)

```mermaid
flowchart TB
    Client["Client Application"]:::client

    subgraph "Public API Layer"
        Article["Article Class"]:::module
        Build["build() / Paper Builder"]:::module
    end

    subgraph "Download Framework"
        MThread["mthreading.py"]:::module
        Network["network.py"]:::module
    end

    subgraph "Parsing & Cleaning Layer"
        Parsers["parsers.py"]:::module
        Cleaners["cleaners.py"]:::module
    end

    subgraph "Extraction Layer"
        Extractors["extractors.py"]:::module
        Images["images.py"]:::module
        VideosExtractor["videos/extractors.py"]:::module
        VideosCore["videos/videos.py"]:::module
    end

    subgraph "NLP Layer"
        NLP["nlp.py"]:::module
        Stopwords["Stopwords Resources"]:::resource
    end

    subgraph "Output Formatter"
        Output["outputformatters.py"]:::module
    end

    subgraph "Configuration & Settings"
        Config["configuration.py"]:::module
        Settings["settings.py"]:::module
    end

    subgraph "Resource Store"
        MiscRes["Misc Resources"]:::resource
    end

    CLI["download_corpora.py"]:::module
    Tests["Unit Tests & Data"]:::tests
    Docs["Documentation (Sphinx)"]:::docs

    subgraph "External Dependencies"
        Requests["requests"]:::external
        LXML["lxml"]:::external
        NLTK["nltk"]:::external
        Pillow["Pillow"]:::external
    end

    Client -->|imports| Article
    Client -->|calls| Build

    Article -->|schedules fetch| MThread
    Build -->|traverses site| MThread

    MThread -->|dispatches tasks| Network
    Network -->|fetch HTML| Parsers
    Parsers -->|clean HTML| Cleaners
    Cleaners -->|DOM tree| Extractors
    Cleaners -->|media URLs| Images
    Cleaners -->|media URLs| VideosExtractor
    VideosExtractor --> VideosCore

    Extractors -->|text| NLP
    Images -->|images| NLP
    VideosCore -->|videos| NLP

    NLP -->|keywords & summary| Output
    Output -->|formatted article| Client

    Network --> Requests
    Parsers --> LXML
    NLP --> NLTK
    Images --> Pillow

    CLI -->|download corpora| Stopwords
    MiscRes -->|user-agents, sources| Network
    Stopwords -->|stopwords lists| NLP

    Config -.-> Article
    Config -.-> MThread
    Config -.-> Parsers
    Config -.-> Extractors
    Settings -.-> Article
    Settings -.-> Network

    Tests -->|test modules| Article
    Docs -->|docs source| Article

    click Article "https://github.com/codelucas/newspaper/blob/master/newspaper/article.py"
    click Build "https://github.com/codelucas/newspaper/blob/master/newspaper/api.py"
    click MThread "https://github.com/codelucas/newspaper/blob/master/newspaper/mthreading.py"
    click Network "https://github.com/codelucas/newspaper/blob/master/newspaper/network.py"
    click Parsers "https://github.com/codelucas/newspaper/blob/master/newspaper/parsers.py"
    click Cleaners "https://github.com/codelucas/newspaper/blob/master/newspaper/cleaners.py"
    click Extractors "https://github.com/codelucas/newspaper/blob/master/newspaper/extractors.py"
    click Images "https://github.com/codelucas/newspaper/blob/master/newspaper/images.py"
    click VideosExtractor "https://github.com/codelucas/newspaper/blob/master/newspaper/videos/extractors.py"
    click VideosCore "https://github.com/codelucas/newspaper/blob/master/newspaper/videos/videos.py"
    click NLP "https://github.com/codelucas/newspaper/blob/master/newspaper/nlp.py"
    click Stopwords "https://github.com/codelucas/newspaper/blob/master/newspaper/resources/text/stopwords-*.txt"
    click Output "https://github.com/codelucas/newspaper/blob/master/newspaper/outputformatters.py"
    click Config "https://github.com/codelucas/newspaper/blob/master/newspaper/configuration.py"
    click Settings "https://github.com/codelucas/newspaper/blob/master/newspaper/settings.py"
    click MiscRes "https://github.com/codelucas/newspaper/blob/master/newspaper/resources/misc/*.txt"
    click CLI "https://github.com/codelucas/newspaper/blob/master/download_corpora.py"
    click Tests "https://github.com/codelucas/newspaper/blob/master/tests/unit_tests.py"
    click Docs "https://github.com/codelucas/newspaper/tree/master/docs/"

    classDef module fill:#f9f,stroke:#333,stroke-width:1px;
    classDef resource fill:#ccf,stroke:#333,stroke-width:1px;
    classDef external fill:#cfc,stroke:#333,stroke-width:1px;
    classDef tests fill:#fcf,stroke:#333,stroke-width:1px;
    classDef docs fill:#cff,stroke:#333,stroke-width:1px;
    classDef client fill:#ff9,stroke:#333,stroke-width:1px;
```
