# adbar/trafilatura architecture

> Architecture diagram of the GitHub repository adbar/trafilatura, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/adbar/trafilatura
- Repository: https://github.com/adbar/trafilatura

## Overview

Trafilatura is a Python library and CLI for discovering and downloading web pages, extracting article content and metadata, and emitting structured formats. The main workflow is user input (URL or HTML) → optional discovery/download → HTML cleanup and content extraction → metadata enrichment → serialization. Its README also documents crawling, feeds, sitemaps, comments, and multiple output formats. The tree and sampled files establish the main extraction and download layers; some discovery and interface wiring is inferred only at the documented capability level, so uncertain calls are omitted.

Main entry points: [`trafilatura/core.py`](https://github.com/adbar/trafilatura/blob/master/trafilatura/core.py) (Extraction orchestration), [`trafilatura/spider.py`](https://github.com/adbar/trafilatura/blob/master/trafilatura/spider.py) (Crawler), [`trafilatura/downloads.py`](https://github.com/adbar/trafilatura/blob/master/trafilatura/downloads.py) (Downloads), [`trafilatura/__init__.py`](https://github.com/adbar/trafilatura/blob/master/trafilatura/__init__.py) (Python API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User interfaces"]
  node_api["Python API<br/>[__init__.py]"]
  node_cli["Command line<br/>[cli.py]"]
  node_cli_utils["CLI utilities<br/>[cli_utils.py]"]
end

subgraph group_discovery["Discovery and retrieval"]
  node_spider["Crawler<br/>[spider.py]"]
  node_sitemaps["Sitemaps<br/>[sitemaps.py]"]
  node_feeds["Feeds<br/>[feeds.py]"]
  node_downloads["Downloads<br/>[downloads.py]"]
  node_dedup["URL deduplication<br/>[deduplication.py]"]
end

subgraph group_extraction["Extraction"]
  node_settings["Extraction options<br/>[settings.py]"]
  node_core["Extraction orchestration<br/>[core.py]"]
  node_html["HTML preparation<br/>[htmlprocessing.py]"]
  node_main_extract["Main extractor<br/>[main_extractor.py]"]
  node_baseline["Baseline extraction<br/>[baseline.py]"]
  node_readability["Readability fallback"]
  node_metadata["Metadata extraction<br/>[metadata.py]"]
  node_json_metadata["JSON-LD metadata<br/>[json_metadata.py]"]
  node_xpaths["Content selectors<br/>[xpaths.py]"]
end

subgraph group_output["Structured output"]
  node_xml["Format serialization<br/>[xml.py]"]
end

node_user(("Caller"))
node_discovery_actor(("Web sources"))
node_external_fallback["justext fallback"]

node_user -->|"calls"| node_api
node_user -->|"invokes"| node_cli
node_cli -->|"uses"| node_cli_utils
node_api -->|"fetches URLs"| node_downloads
node_cli_utils -->|"downloads"| node_downloads
node_user -->|"requests crawl"| node_spider
node_spider -->|"discovers URLs"| node_sitemaps
node_spider -->|"discovers URLs"| node_feeds
node_spider -->|"fetches pages"| node_downloads
node_feeds -->|"fetches feeds"| node_downloads
node_downloads -->|"requests pages"| node_discovery_actor
node_spider -->|"deduplicates URLs"| node_dedup
node_api -->|"extracts input"| node_core
node_cli -->|"extracts input"| node_core
node_core -->|"configures extraction"| node_settings
node_core -->|"prepares HTML"| node_html
node_core -->|"extracts metadata"| node_metadata
node_metadata -->|"parses structured metadata"| node_json_metadata
node_core -->|"extracts content"| node_main_extract
node_main_extract -->|"selects content"| node_xpaths
node_core -->|"extracts fallback content"| node_baseline
node_core -.->|"uses fallback"| node_readability
node_core -.->|"uses fallback"| node_external_fallback
node_core -->|"serializes result"| node_xml
node_xml -->|"returns formats"| node_user

click node_api "https://github.com/adbar/trafilatura/blob/master/trafilatura/__init__.py"
click node_cli "https://github.com/adbar/trafilatura/blob/master/trafilatura/cli.py"
click node_cli_utils "https://github.com/adbar/trafilatura/blob/master/trafilatura/cli_utils.py"
click node_spider "https://github.com/adbar/trafilatura/blob/master/trafilatura/spider.py"
click node_sitemaps "https://github.com/adbar/trafilatura/blob/master/trafilatura/sitemaps.py"
click node_feeds "https://github.com/adbar/trafilatura/blob/master/trafilatura/feeds.py"
click node_downloads "https://github.com/adbar/trafilatura/blob/master/trafilatura/downloads.py"
click node_dedup "https://github.com/adbar/trafilatura/blob/master/trafilatura/deduplication.py"
click node_settings "https://github.com/adbar/trafilatura/blob/master/trafilatura/settings.py"
click node_core "https://github.com/adbar/trafilatura/blob/master/trafilatura/core.py"
click node_html "https://github.com/adbar/trafilatura/blob/master/trafilatura/htmlprocessing.py"
click node_main_extract "https://github.com/adbar/trafilatura/blob/master/trafilatura/main_extractor.py"
click node_baseline "https://github.com/adbar/trafilatura/blob/master/trafilatura/baseline.py"
click node_readability "https://github.com/adbar/trafilatura/blob/master/trafilatura/readability_lxml.py"
click node_metadata "https://github.com/adbar/trafilatura/blob/master/trafilatura/metadata.py"
click node_json_metadata "https://github.com/adbar/trafilatura/blob/master/trafilatura/json_metadata.py"
click node_xpaths "https://github.com/adbar/trafilatura/blob/master/trafilatura/xpaths.py"
click node_xml "https://github.com/adbar/trafilatura/blob/master/trafilatura/xml.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_cli,node_cli_utils toneBlue
class node_spider,node_sitemaps,node_feeds,node_downloads,node_dedup toneAmber
class node_settings,node_core,node_html,node_main_extract,node_baseline,node_readability,node_metadata,node_json_metadata,node_xpaths toneMint
class node_xml toneRose
class node_user,node_discovery_actor,node_external_fallback toneIndigo
```

## Components

### User interfaces

- [Python API](https://github.com/adbar/trafilatura/blob/master/trafilatura/__init__.py): `trafilatura/__init__.py`
- [Command line](https://github.com/adbar/trafilatura/blob/master/trafilatura/cli.py): `trafilatura/cli.py`
- [CLI utilities](https://github.com/adbar/trafilatura/blob/master/trafilatura/cli_utils.py): `trafilatura/cli_utils.py`

### Discovery and retrieval

- [Crawler](https://github.com/adbar/trafilatura/blob/master/trafilatura/spider.py): `trafilatura/spider.py`
- [Sitemaps](https://github.com/adbar/trafilatura/blob/master/trafilatura/sitemaps.py): `trafilatura/sitemaps.py`
- [Feeds](https://github.com/adbar/trafilatura/blob/master/trafilatura/feeds.py): `trafilatura/feeds.py`
- [Downloads](https://github.com/adbar/trafilatura/blob/master/trafilatura/downloads.py): `trafilatura/downloads.py`
- [URL deduplication](https://github.com/adbar/trafilatura/blob/master/trafilatura/deduplication.py): `trafilatura/deduplication.py`

### Extraction

- [Extraction options](https://github.com/adbar/trafilatura/blob/master/trafilatura/settings.py): `trafilatura/settings.py`
- [Extraction orchestration](https://github.com/adbar/trafilatura/blob/master/trafilatura/core.py): `trafilatura/core.py`
- [HTML preparation](https://github.com/adbar/trafilatura/blob/master/trafilatura/htmlprocessing.py): `trafilatura/htmlprocessing.py`
- [Main extractor](https://github.com/adbar/trafilatura/blob/master/trafilatura/main_extractor.py): `trafilatura/main_extractor.py`
- [Baseline extraction](https://github.com/adbar/trafilatura/blob/master/trafilatura/baseline.py): `trafilatura/baseline.py`
- [Readability fallback](https://github.com/adbar/trafilatura/blob/master/trafilatura/readability_lxml.py): `trafilatura/readability_lxml.py`
- [Metadata extraction](https://github.com/adbar/trafilatura/blob/master/trafilatura/metadata.py): `trafilatura/metadata.py`
- [JSON-LD metadata](https://github.com/adbar/trafilatura/blob/master/trafilatura/json_metadata.py): `trafilatura/json_metadata.py`
- [Content selectors](https://github.com/adbar/trafilatura/blob/master/trafilatura/xpaths.py): `trafilatura/xpaths.py`

### Structured output

- [Format serialization](https://github.com/adbar/trafilatura/blob/master/trafilatura/xml.py): `trafilatura/xml.py`

### Other components

- **Caller**
- **Web sources**
- **justext fallback**

## Connections

- Caller → Python API: calls
- Caller → Command line: invokes
- Command line → CLI utilities: uses
- Python API → Downloads: fetches URLs
- CLI utilities → Downloads: downloads
- Caller → Crawler: requests crawl
- Crawler → Sitemaps: discovers URLs
- Crawler → Feeds: discovers URLs
- Crawler → Downloads: fetches pages
- Feeds → Downloads: fetches feeds
- Downloads → Web sources: requests pages
- Crawler → URL deduplication: deduplicates URLs
- Python API → Extraction orchestration: extracts input
- Command line → Extraction orchestration: extracts input
- Extraction orchestration → Extraction options: configures extraction
- Extraction orchestration → HTML preparation: prepares HTML
- Extraction orchestration → Metadata extraction: extracts metadata
- Metadata extraction → JSON-LD metadata: parses structured metadata
- Extraction orchestration → Main extractor: extracts content
- Main extractor → Content selectors: selects content
- Extraction orchestration → Baseline extraction: extracts fallback content
- Extraction orchestration → Readability fallback: uses fallback
- Extraction orchestration → justext fallback: uses fallback
- Extraction orchestration → Format serialization: serializes result
- Format serialization → Caller: returns formats