# fhamborg/news-please architecture

> Architecture diagram of the GitHub repository fhamborg/news-please, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fhamborg/news-please
- Repository: https://github.com/fhamborg/news-please

## Overview

news-please crawls news sites or accepts article URLs, HTML, and WARC records, then extracts structured article fields. Its CLI supports site crawling and configurable output; library APIs expose direct extraction, while a separate Common Crawl workflow handles archive records and filtering. The graph preserves these documented modes and the extraction and storage layers. Some relationships are supported only by the README or source index; unsampled implementation details are not inferred.

Main entry points: [`newsplease/pipeline/extractor/article_extractor.py`](https://github.com/fhamborg/news-please/blob/master/newsplease/pipeline/extractor/article_extractor.py) (Extraction coordinator), [`newsplease/__init__.py`](https://github.com/fhamborg/news-please/blob/master/newsplease/__init__.py) (Library API), [`newsplease/single_crawler.py`](https://github.com/fhamborg/news-please/blob/master/newsplease/single_crawler.py) (Site runner), [`newsplease/crawler/spiders`](https://github.com/fhamborg/news-please/tree/master/newsplease/crawler/spiders) (Site spiders)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry modes"]
  node_cli["CLI runner<br/>[__main__.py]"]
  node_library["Library API<br/>[__init__.py]"]
end

subgraph group_crawl["Crawling"]
  node_config["Configuration<br/>[config.py]"]
  node_site_runner["Site runner<br/>[single_crawler.py]"]
  node_spiders["Site spiders"]
  node_url_tools["URL tools<br/>[url_extractor.py]"]
  node_heuristics["Article heuristics"]
  node_commoncrawl["Common Crawl fetcher"]
  node_archive_extract["Archive extraction"]
  node_simple_fetch["URL fetcher<br/>[simple_crawler.py]"]
end

subgraph group_extract["Article extraction"]
  node_item["Crawl item<br/>[items.py]"]
  node_extractor["Extraction coordinator"]
  node_candidate["Article candidate"]
  node_extractors["Field extractors"]
  node_cleaner["Candidate cleaner<br/>[cleaner.py]"]
  node_comparer["Candidate comparer<br/>[comparer.py]"]
end

subgraph group_output["Results and storage"]
  node_article["Article result<br/>[NewsArticle.py]"]
  node_pipelines["Output pipelines<br/>[pipelines.py]"]
  node_storage[("Configured storage<br/>[pipelines.py]")]
end

node_user(("User"))
node_web(("News websites"))
node_cc_service(("Common Crawl"))
node_json_files["JSON files"]
node_external_stores[("External stores")]

node_user -->|"runs"| node_cli
node_user -->|"calls"| node_library
node_user -->|"requests archive crawl"| node_commoncrawl
node_cli -->|"starts crawling"| node_site_runner
node_site_runner -->|"loads settings"| node_config
node_site_runner -->|"selects crawler"| node_spiders
node_spiders -.->|"fetches pages"| node_web
node_spiders -->|"uses URL discovery"| node_url_tools
node_spiders -.->|"tests article pages"| node_heuristics
node_site_runner -->|"creates crawl items"| node_item
node_library -->|"fetches URLs"| node_simple_fetch
node_library -->|"builds extraction items"| node_item
node_library -->|"extracts HTML"| node_extractor
node_commoncrawl -->|"downloads archive data"| node_cc_service
node_commoncrawl -->|"processes WARC records"| node_archive_extract
node_archive_extract -->|"uses extraction API"| node_library
node_item -->|"provides crawl item"| node_extractor
node_extractor -->|"runs field extraction"| node_extractors
node_extractors -->|"produces candidates"| node_candidate
node_extractor -->|"cleans candidates"| node_cleaner
node_extractor -->|"compares candidates"| node_comparer
node_comparer -->|"reads candidates"| node_candidate
node_extractor -->|"sets article fields"| node_item
node_library -.->|"returns article"| node_article
node_cli -->|"configures output"| node_pipelines
node_pipelines -->|"extracts articles"| node_extractor
node_pipelines -->|"loads settings"| node_config
node_pipelines -->|"writes results"| node_storage
node_storage -->|"writes JSON"| node_json_files
node_storage -.->|"writes configured output"| node_external_stores

click node_cli "https://github.com/fhamborg/news-please/blob/master/newsplease/__main__.py"
click node_library "https://github.com/fhamborg/news-please/blob/master/newsplease/__init__.py"
click node_config "https://github.com/fhamborg/news-please/blob/master/newsplease/config.py"
click node_site_runner "https://github.com/fhamborg/news-please/blob/master/newsplease/single_crawler.py"
click node_spiders "https://github.com/fhamborg/news-please/tree/master/newsplease/crawler/spiders"
click node_url_tools "https://github.com/fhamborg/news-please/blob/master/newsplease/helper_classes/url_extractor.py"
click node_heuristics "https://github.com/fhamborg/news-please/blob/master/newsplease/helper_classes/sub_classes/heuristics_manager.py"
click node_commoncrawl "https://github.com/fhamborg/news-please/blob/master/newsplease/crawler/commoncrawl_crawler.py"
click node_archive_extract "https://github.com/fhamborg/news-please/blob/master/newsplease/crawler/commoncrawl_extractor.py"
click node_simple_fetch "https://github.com/fhamborg/news-please/blob/master/newsplease/crawler/simple_crawler.py"
click node_item "https://github.com/fhamborg/news-please/blob/master/newsplease/crawler/items.py"
click node_extractor "https://github.com/fhamborg/news-please/blob/master/newsplease/pipeline/extractor/article_extractor.py"
click node_candidate "https://github.com/fhamborg/news-please/blob/master/newsplease/pipeline/extractor/article_candidate.py"
click node_extractors "https://github.com/fhamborg/news-please/tree/master/newsplease/pipeline/extractor/extractors"
click node_cleaner "https://github.com/fhamborg/news-please/blob/master/newsplease/pipeline/extractor/cleaner.py"
click node_comparer "https://github.com/fhamborg/news-please/blob/master/newsplease/pipeline/extractor/comparer/comparer.py"
click node_article "https://github.com/fhamborg/news-please/blob/master/newsplease/NewsArticle.py"
click node_pipelines "https://github.com/fhamborg/news-please/blob/master/newsplease/pipeline/pipelines.py"
click node_storage "https://github.com/fhamborg/news-please/blob/master/newsplease/pipeline/pipelines.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_library,node_user toneBlue
class node_config,node_site_runner,node_spiders,node_url_tools,node_heuristics,node_commoncrawl,node_archive_extract,node_simple_fetch,node_external_stores toneAmber
class node_item,node_extractor,node_candidate,node_extractors,node_cleaner,node_comparer toneMint
class node_article,node_pipelines,node_storage toneRose
class node_web,node_cc_service,node_json_files toneIndigo
```

## Components

### Entry modes

- [CLI runner](https://github.com/fhamborg/news-please/blob/master/newsplease/__main__.py): `newsplease/__main__.py`
- [Library API](https://github.com/fhamborg/news-please/blob/master/newsplease/__init__.py): `newsplease/__init__.py`

### Crawling

- [Configuration](https://github.com/fhamborg/news-please/blob/master/newsplease/config.py): `newsplease/config.py`
- [Site runner](https://github.com/fhamborg/news-please/blob/master/newsplease/single_crawler.py): `newsplease/single_crawler.py`
- [Site spiders](https://github.com/fhamborg/news-please/tree/master/newsplease/crawler/spiders): `newsplease/crawler/spiders`
- [URL tools](https://github.com/fhamborg/news-please/blob/master/newsplease/helper_classes/url_extractor.py): `newsplease/helper_classes/url_extractor.py`
- [Article heuristics](https://github.com/fhamborg/news-please/blob/master/newsplease/helper_classes/sub_classes/heuristics_manager.py): `newsplease/helper_classes/sub_classes/heuristics_manager.py`
- [Common Crawl fetcher](https://github.com/fhamborg/news-please/blob/master/newsplease/crawler/commoncrawl_crawler.py): `newsplease/crawler/commoncrawl_crawler.py`
- [Archive extraction](https://github.com/fhamborg/news-please/blob/master/newsplease/crawler/commoncrawl_extractor.py): `newsplease/crawler/commoncrawl_extractor.py`
- [URL fetcher](https://github.com/fhamborg/news-please/blob/master/newsplease/crawler/simple_crawler.py): `newsplease/crawler/simple_crawler.py`

### Article extraction

- [Crawl item](https://github.com/fhamborg/news-please/blob/master/newsplease/crawler/items.py): `newsplease/crawler/items.py`
- [Extraction coordinator](https://github.com/fhamborg/news-please/blob/master/newsplease/pipeline/extractor/article_extractor.py): `newsplease/pipeline/extractor/article_extractor.py`
- [Article candidate](https://github.com/fhamborg/news-please/blob/master/newsplease/pipeline/extractor/article_candidate.py): `newsplease/pipeline/extractor/article_candidate.py`
- [Field extractors](https://github.com/fhamborg/news-please/tree/master/newsplease/pipeline/extractor/extractors): `newsplease/pipeline/extractor/extractors`
- [Candidate cleaner](https://github.com/fhamborg/news-please/blob/master/newsplease/pipeline/extractor/cleaner.py): `newsplease/pipeline/extractor/cleaner.py`
- [Candidate comparer](https://github.com/fhamborg/news-please/blob/master/newsplease/pipeline/extractor/comparer/comparer.py): `newsplease/pipeline/extractor/comparer/comparer.py`

### Results and storage

- [Article result](https://github.com/fhamborg/news-please/blob/master/newsplease/NewsArticle.py): `newsplease/NewsArticle.py`
- [Output pipelines](https://github.com/fhamborg/news-please/blob/master/newsplease/pipeline/pipelines.py): `newsplease/pipeline/pipelines.py`
- [Configured storage](https://github.com/fhamborg/news-please/blob/master/newsplease/pipeline/pipelines.py): `newsplease/pipeline/pipelines.py`

### Other components

- **User**
- **News websites**
- **Common Crawl**
- **JSON files**
- **External stores**

## Connections

- User → CLI runner: runs
- User → Library API: calls
- User → Common Crawl fetcher: requests archive crawl
- CLI runner → Site runner: starts crawling (evidence: `newsplease/__main__.py`)
- Site runner → Configuration: loads settings (evidence: `newsplease/single_crawler.py`)
- Site runner → Site spiders: selects crawler (evidence: `newsplease/single_crawler.py`)
- Site spiders → News websites: fetches pages
- Site spiders → URL tools: uses URL discovery (evidence: `newsplease/crawler/spiders/rss_crawler.py`)
- Site spiders → Article heuristics: tests article pages
- Site runner → Crawl item: creates crawl items (evidence: `newsplease/single_crawler.py`)
- Library API → URL fetcher: fetches URLs (evidence: `newsplease/__init__.py`)
- Library API → Crawl item: builds extraction items (evidence: `newsplease/__init__.py`)
- Library API → Extraction coordinator: extracts HTML (evidence: `newsplease/__init__.py`)
- Common Crawl fetcher → Common Crawl: downloads archive data (evidence: `newsplease/crawler/commoncrawl_crawler.py`)
- Common Crawl fetcher → Archive extraction: processes WARC records (evidence: `newsplease/crawler/commoncrawl_crawler.py`)
- Archive extraction → Library API: uses extraction API (evidence: `newsplease/crawler/commoncrawl_extractor.py`)
- Crawl item → Extraction coordinator: provides crawl item (evidence: `newsplease/pipeline/extractor/article_extractor.py`)
- Extraction coordinator → Field extractors: runs field extraction (evidence: `newsplease/pipeline/extractor/article_extractor.py`)
- Field extractors → Article candidate: produces candidates (evidence: `newsplease/pipeline/extractor/extractors/abstract_extractor.py`)
- Extraction coordinator → Candidate cleaner: cleans candidates (evidence: `newsplease/pipeline/extractor/article_extractor.py`)
- Extraction coordinator → Candidate comparer: compares candidates (evidence: `newsplease/pipeline/extractor/article_extractor.py`)
- Candidate comparer → Article candidate: reads candidates (evidence: `newsplease/pipeline/extractor/comparer/comparer.py`)
- Extraction coordinator → Crawl item: sets article fields (evidence: `newsplease/pipeline/extractor/article_extractor.py`)
- Library API → Article result: returns article
- CLI runner → Output pipelines: configures output (evidence: `newsplease/__main__.py`)
- Output pipelines → Extraction coordinator: extracts articles (evidence: `newsplease/pipeline/pipelines.py`)
- Output pipelines → Configuration: loads settings (evidence: `newsplease/pipeline/pipelines.py`)
- Output pipelines → Configured storage: writes results (evidence: `newsplease/pipeline/pipelines.py`)
- Configured storage → JSON files: writes JSON
- Configured storage → External stores: writes configured output