How fhamborg/news-please works
news-please crawls news sites or accepts article URLs, HTML, and WARC records, then extracts structured article fields. Its CLI supports site crawling and configurable output; library APIs expose direct extraction, while a separate Common Crawl workflow handles archive records and filtering. The graph preserves these documented modes and the extraction and storage layers. Some relationships are supported only by the README or source index; unsampled implementation details are not inferred.
Main entry points: article_extractor.py Extraction coordinator · __init__.py Library API · single_crawler.py Site runner · spiders Site spiders
Components 24
Entry modes
- CLI runner
newsplease/__main__.py - Library API
newsplease/__init__.py
Crawling
- Configuration
newsplease/config.py - Site runner
newsplease/single_crawler.py - Site spiders
newsplease/crawler/spiders - URL tools
newsplease/helper_classes/url_extractor.py - Article heuristics
newsplease/helper_classes/sub_classes/heuristics_manager.py - Common Crawl fetcher
newsplease/crawler/commoncrawl_crawler.py - Archive extraction
newsplease/crawler/commoncrawl_extractor.py - URL fetcher
newsplease/crawler/simple_crawler.py
Article extraction
- Crawl item
newsplease/crawler/items.py - Extraction coordinator
newsplease/pipeline/extractor/article_extractor.py - Article candidate
newsplease/pipeline/extractor/article_candidate.py - Field extractors
newsplease/pipeline/extractor/extractors - Candidate cleaner
newsplease/pipeline/extractor/cleaner.py - Candidate comparer
newsplease/pipeline/extractor/comparer/comparer.py
Results and storage
- Article result
newsplease/NewsArticle.py - Output pipelines
newsplease/pipeline/pipelines.py - Configured storage
newsplease/pipeline/pipelines.py
Other components
- User
- News websites
- Common Crawl
- JSON files
- External stores
Connections 30
- User → CLI runner: runs
- User → Library API: calls
- User → Common Crawl fetcher: requests archive crawl
- CLI runner → Site runner: starts crawling
newsplease/__main__.py - Site runner → Configuration: loads settings
newsplease/single_crawler.py - Site runner → Site spiders: selects crawler
newsplease/single_crawler.py - Site spiders → News websites: fetches pages
- Site spiders → URL tools: uses URL discovery
newsplease/crawler/spiders/rss_crawler.py - Site spiders → Article heuristics: tests article pages
- Site runner → Crawl item: creates crawl items
newsplease/single_crawler.py - Library API → URL fetcher: fetches URLs
newsplease/__init__.py - Library API → Crawl item: builds extraction items
newsplease/__init__.py - Library API → Extraction coordinator: extracts HTML
newsplease/__init__.py - Common Crawl fetcher → Common Crawl: downloads archive data
newsplease/crawler/commoncrawl_crawler.py - Common Crawl fetcher → Archive extraction: processes WARC records
newsplease/crawler/commoncrawl_crawler.py - Archive extraction → Library API: uses extraction API
newsplease/crawler/commoncrawl_extractor.py - Crawl item → Extraction coordinator: provides crawl item
newsplease/pipeline/extractor/article_extractor.py - Extraction coordinator → Field extractors: runs field extraction
newsplease/pipeline/extractor/article_extractor.py - Field extractors → Article candidate: produces candidates
newsplease/pipeline/extractor/extractors/abstract_extractor.py - Extraction coordinator → Candidate cleaner: cleans candidates
newsplease/pipeline/extractor/article_extractor.py - Extraction coordinator → Candidate comparer: compares candidates
newsplease/pipeline/extractor/article_extractor.py - Candidate comparer → Article candidate: reads candidates
newsplease/pipeline/extractor/comparer/comparer.py - Extraction coordinator → Crawl item: sets article fields
newsplease/pipeline/extractor/article_extractor.py - Library API → Article result: returns article
- CLI runner → Output pipelines: configures output
newsplease/__main__.py - Output pipelines → Extraction coordinator: extracts articles
newsplease/pipeline/pipelines.py - Output pipelines → Configuration: loads settings
newsplease/pipeline/pipelines.py - Output pipelines → Configured storage: writes results
newsplease/pipeline/pipelines.py - Configured storage → JSON files: writes JSON
- Configured storage → External stores: writes configured output