# skylaweber/newspaper4k architecture

> Architecture diagram of the GitHub repository skylaweber/newspaper4k, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/skylaweber/newspaper4k
- Repository: https://github.com/skylaweber/newspaper4k

## Overview

##### Project type and purpose
- Newspaper4k is an open-source Python library and CLI tool for scraping, parsing, and curating news articles from arbitrary websites (and Google News).
- It provides a Python API (Article, Source, GoogleNewsSource classes), a command-line interface, multi-language support, multi-threaded downloads, HTML cleaning/parsing, NLP (keyword extraction, summarization), and output formatters (JSON, CSV, text).

##### File structure
- Top-level packages:
- newspaper/ – core library code (API, Article, Source, extractors, cleaners, parsers, network, nlp, utils, configuration)
- newspaper/extractors/ – specialized modules for title, body, images, metadata, etc.
- newspaper/languages/ – per-language tokenizers and stopword lists under resources/text.
- newspaper/resources/ – text resources (stopwords), site-source lists, user-agents.
- CLI entrypoint: newspaper/cli.py and newspaper/__main__.py
- Configuration: newspaper/configuration.py
- Concurrency: newspaper/mthreading.py uses ThreadPoolExecutor
- Google News integration: newspaper/google_news.py
- Output formatting: newspaper/outputformatters.py
- Build scripts & packaging: setup.py, pyproject.toml, requirements.txt
- Documentation (docs/) and tests (tests/) directories

##### README for additional insights
- Confirms key features: multi-threaded download, source/category detection, NLP summarization, Google News support, CLI usage, Python API usage
- Describes high-level workflow: build a Source, download articles in threads, parse via Article object, extract fields, output

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Interfaces
    subgraph "Interfaces" 
        CLI["CLI Interface"]:::interface
        API["Python API"]:::interface
    end

    %% Orchestrator and Config
    subgraph "Orchestrator Layer"
        ORCH["Orchestrator / Manager"]:::core
        CONFIG["Configuration Manager"]:::resource
        ERR["Error Handling"]:::core
        CONC["Concurrency Layer"]:::core
    end

    %% Core Pipeline
    subgraph "Core Pipeline"
        NET["Network Fetcher"]:::core
        CLEAN["HTML Cleaners"]:::core
        PARSE["HTML Parsers"]:::core
        subgraph "Extraction Layer"
            TITLE["TitleExtractor"]:::core
            BODY["ArticleBodyExtractor"]:::core
            AUTHORS["AuthorsExtractor"]:::core
            CATEGORIES["CategoriesExtractor"]:::core
            CONTENT["ContentExtractor"]:::core
            IMAGE["ImageExtractor"]:::core
            META["MetadataExtractor"]:::core
            PUBDATE["PubDateExtractor"]:::core
            VIDEOS["VideosExtractor"]:::core
            DEFINES["Extractor Definitions"]:::core
        end
        NLP["NLP & Post-processing"]:::core
        OUT["Output Formatters"]:::core
    end

    %% Resources and Adapters
    subgraph "Resources & Adapters"
        RES["Resource Store"]:::resource
        GNEWS["Google News Adapter"]:::external
    end

    %% Data Flow
    CLI -->|uses| ORCH
    API -->|calls| ORCH
    ORCH -->|loads| CONFIG
    ORCH -->|spawns| CONC
    ORCH -->|manages errors| ERR

    CONC -->|parallel tasks| NET
    NET -->|raw HTML| CLEAN
    CLEAN -->|clean HTML| PARSE
    PARSE -->|DOM fragments| TITLE
    PARSE -->|DOM fragments| BODY
    PARSE -->|DOM fragments| AUTHORS
    PARSE -->|DOM fragments| CATEGORIES
    PARSE -->|DOM fragments| CONTENT
    PARSE -->|DOM fragments| IMAGE
    PARSE -->|DOM fragments| META
    PARSE -->|DOM fragments| PUBDATE
    PARSE -->|DOM fragments| VIDEOS
    PARSE -->|uses definitions| DEFINES
    TITLE --> NLP
    BODY --> NLP
    AUTHORS --> NLP
    CATEGORIES --> NLP
    CONTENT --> NLP
    IMAGE --> NLP
    META --> NLP
    PUBDATE --> NLP
    VIDEOS --> NLP
    NLP --> OUT
    CONFIG -->|provides settings| NET
    CONFIG -->|provides settings| CLEAN
    CONFIG -->|provides settings| PARSE
    CONFIG -->|provides settings| NLP
    RES -->|provides data| NLP
    RES -->|provides data| NET
    GNEWS -->|external source| ORCH
    GNEWS -->|fetch| NET

    %% Click Events
    click CLI "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/cli.py"
    click API "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/api.py"
    click ORCH "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/source.py"
    click ORCH "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/article.py"
    click CONFIG "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/configuration.py"
    click CONC "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/mthreading.py"
    click NET "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/network.py"
    click CLEAN "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/cleaners.py"
    click PARSE "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/parsers.py"
    click TITLE "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/extractors/title_extractor.py"
    click BODY "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/extractors/articlebody_extractor.py"
    click AUTHORS "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/extractors/authors_extractor.py"
    click CATEGORIES "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/extractors/categories_extractor.py"
    click CONTENT "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/extractors/content_extractor.py"
    click IMAGE "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/extractors/image_extractor.py"
    click META "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/extractors/metadata_extractor.py"
    click PUBDATE "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/extractors/pubdate_extractor.py"
    click VIDEOS "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/extractors/videos_extractor.py"
    click DEFINES "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/extractors/defines.py"
    click NLP "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/nlp.py"
    click RES "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/resources/misc/google_sources.txt"
    click RES "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/resources/misc/popular_sources.txt"
    click RES "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/resources/misc/useragents.txt"
    click RES "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/resources/text/stopwords-*.txt"
    click OUT "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/outputformatters.py"
    click GNEWS "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/google_news.py"
    click ERR "https://github.com/skylaweber/newspaper4k/blob/master/newspaper/exceptions.py"

    %% Styles
    classDef interface fill:#f9f,stroke:#333,stroke-width:1px;
    classDef core fill:#bbf,stroke:#333,stroke-width:1px;
    classDef resource fill:#bfb,stroke:#333,stroke-width:1px;
    classDef external fill:#fbb,stroke:#333,stroke-width:1px;
```
