How adbar/trafilatura works
Trafilatura is a Python library and CLI for discovering and downloading web pages, extracting article content and metadata, and emitting structured formats. The main workflow is user input (URL or HTML) → optional discovery/download → HTML cleanup and content extraction → metadata enrichment → serialization. Its README also documents crawling, feeds, sitemaps, comments, and multiple output formats. The tree and sampled files establish the main extraction and download layers; some discovery and interface wiring is inferred only at the documented capability level, so uncertain calls are omitted.
Main entry points: core.py Extraction orchestration · spider.py Crawler · downloads.py Downloads · __init__.py Python API
Components 21
User interfaces
- Python API
trafilatura/__init__.py - Command line
trafilatura/cli.py - CLI utilities
trafilatura/cli_utils.py
Discovery and retrieval
- Crawler
trafilatura/spider.py - Sitemaps
trafilatura/sitemaps.py - Feeds
trafilatura/feeds.py - Downloads
trafilatura/downloads.py - URL deduplication
trafilatura/deduplication.py
Extraction
- Extraction options
trafilatura/settings.py - Extraction orchestration
trafilatura/core.py - HTML preparation
trafilatura/htmlprocessing.py - Main extractor
trafilatura/main_extractor.py - Baseline extraction
trafilatura/baseline.py - Readability fallback
trafilatura/readability_lxml.py - Metadata extraction
trafilatura/metadata.py - JSON-LD metadata
trafilatura/json_metadata.py - Content selectors
trafilatura/xpaths.py
Structured output
- Format serialization
trafilatura/xml.py
Other components
- Caller
- Web sources
- justext fallback
Connections 25
- Caller → Python API: calls
- Caller → Command line: invokes
- Command line → CLI utilities: uses
- Python API → Downloads: fetches URLs
- CLI utilities → Downloads: downloads
- Caller → Crawler: requests crawl
- Crawler → Sitemaps: discovers URLs
- Crawler → Feeds: discovers URLs
- Crawler → Downloads: fetches pages
- Feeds → Downloads: fetches feeds
- Downloads → Web sources: requests pages
- Crawler → URL deduplication: deduplicates URLs
- Python API → Extraction orchestration: extracts input
- Command line → Extraction orchestration: extracts input
- Extraction orchestration → Extraction options: configures extraction
- Extraction orchestration → HTML preparation: prepares HTML
- Extraction orchestration → Metadata extraction: extracts metadata
- Metadata extraction → JSON-LD metadata: parses structured metadata
- Extraction orchestration → Main extractor: extracts content
- Main extractor → Content selectors: selects content
- Extraction orchestration → Baseline extraction: extracts fallback content
- Extraction orchestration → Readability fallback: uses fallback
- Extraction orchestration → justext fallback: uses fallback
- Extraction orchestration → Format serialization: serializes result
- Format serialization → Caller: returns formats