# xenova/chat-downloader architecture

> Architecture diagram of the GitHub repository xenova/chat-downloader, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xenova/chat-downloader
- Repository: https://github.com/xenova/chat-downloader

## Overview

1. **Project shape**
   - `chat_downloader` is a Python package that serves both as a **library** and a **CLI tool**.
   - The top-level behavior is: take a chat/video URL, identify the supported platform, stream parsed chat messages, optionally format/write them, and expose them to Python callers or the command line.

2. **Primary entry points**
   - `chat_downloader/__main__.py` and `chat_downloader/cli.py` are the command-line entry path.
   - `chat_downloader/__init__.py` exposes the public API, centered around `ChatDownloader`.
   - `chat_downloader/chat_downloader.py` is the main orchestration layer that ties together site detection, retrieval, filtering, and output.

3. **Core orchestration layer**
   - `ChatDownloader` is the central coordinator.
   - It accepts a URL and options such as time bounds, message filtering, retry behavior, proxies/cookies, output formatting, and chat type.
   - It produces an iterator/generator of chat items rather than loading everything at once, which is important for long livestreams and large archives.

4. **Site adapter boundary**
   - `chat_downloader/sites/` contains platform-specific extractors for **YouTube, Twitch, Zoom, and Facebook**.
   - `sites/common.py` likely defines the shared interface/base logic used by all site modules.
   - This is the key extensibility boundary: the rest of the system works on normalized chat items, while each site module knows how to talk to a particular external service and transform its response into the common format.

5. **Data acquisition flow**
   - The downloader starts from a URL, selects the matching site module, and fetches raw chat-related data from the target service.
   - The site adapter converts site-specific payloads into standardized message dictionaries containing fields like `message_id`, `message`, `message_type`, timestamps, and author metadata.
   - The README’s “Chat Items” section reflects this normalized contract.

6. **Message filtering and control**
   - Message selection is controlled by options such as `start_time`, `end_time`, `message_types`, `message_groups`, `max_messages`, `timeout`, and inactivity/retry limits.
   - This filtering happens in the core flow before output, so consumers can trim the stream without needing to process every raw message themselves.

7. **Formatting subsystem**
   - `chat_downloader/formatting/` handles turning normalized chat items into human-readable or structured output.
   - `format.py` and `custom_formats.json` suggest both built-in format definitions and user-extensible formatting rules.
   - This subsystem sits between the normalized message stream and any console/file presentation.

8. **Output subsystem**
   - `chat_downloader/output/continuous_write.py` indicates support for incremental writing as messages arrive.
   - This is the file/stream persistence boundary: messages can be emitted continuously to an output target, rather than buffered until the end.
   - The CLI’s `--output`, `--overwrite`, `--sort_keys`, and `--indent` flags fit this layer.

9. **Shared utilities and timing**
   - `chat_downloader/utils/` contains reusable helpers, especially `core.py` and `timed_utils.py`.
   - These likely support parsing, retries, time conversions, rate handling, and other cross-cutting logic used by both site modules and the core downloader.

10. **Errors and debugging**
   - `chat_downloader/errors.py` defines the package’s domain-specific failure modes.
   - `chat_downloader/debugging.py` and CLI flags like `--pause_on_debug`, `--exit_on_debug`, and logging levels show that debugging is a first-class concern.
   - This boundary is important for surfacing site failures, network issues, and unsupported behaviors cleanly.

11. **Metadata and package support**
   - `chat_downloader/metadata.py` likely centralizes versioning or package metadata used by the CLI and distribution.
   - `setup.py`, `setup.cfg`, `MANIFEST.in`, `tox.ini`, and CI workflows show this is a distributable package with automated testing and publishing.

12. **Testing and verification**
   - `tests/` covers the major architecture layers: downloader behavior, CLI, formatting, generators, init parameters, sites, utils, and writers.
   - The presence of site-specific tests and a `youtube_actions.json` fixture suggests integration-style validation against real or recorded platform behavior.

13. **Documentation and generated artifacts**
   - `docs/` contains the user and developer documentation, including CLI docs, site docs, and scripts that generate parts of the published docs/readme.
   - The repository is structured so the docs mirror the same architectural boundaries as the code: CLI, items, options, and supported sites.

14. **External dependency boundary**
   - The only clearly material external dependencies are the supported content platforms themselves: YouTube, Twitch, Zoom, and Facebook.
   - The package is intentionally designed to avoid requiring user authentication for the supported flows, which affects the architecture by emphasizing public-page parsing and platform-specific retrieval logic.

Main entry points: [`chat_downloader/chat_downloader.py`](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/chat_downloader.py) (Downloader), [`chat_downloader/sites/common.py`](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/sites/common.py) (Site base), [`chat_downloader/cli.py`](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/cli.py) (CLI parser), [`chat_downloader/utils/core.py`](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/utils/core.py) (Core utils)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_entry["Entry points"]
  node_node_init["Public API<br/>package export<br/>[__init__.py]"]
  node_node_main["Module entry<br/>cli entry<br/>[__main__.py]"]
  node_node_cli["CLI parser<br/>command interface<br/>[cli.py]"]
end

subgraph group_group_core["Core runtime"]
  node_node_downloader["Downloader<br/>orchestrator<br/>[chat_downloader.py]"]
  node_node_formatting["Formatting<br/>rendering layer<br/>[format.py]"]
  node_node_custom_formats["Format rules<br/>format config"]
  node_node_output["Continuous write<br/>output sink"]
end

subgraph group_group_sites["Site adapters"]
  node_node_sites_common["Site base<br/>adapter base<br/>[common.py]"]
  node_node_youtube["YouTube<br/>site adapter<br/>[youtube.py]"]
  node_node_twitch["Twitch<br/>site adapter<br/>[twitch.py]"]
  node_node_zoom["Zoom<br/>site adapter<br/>[zoom.py]"]
  node_node_facebook["Facebook<br/>site adapter<br/>[facebook.py]"]
end

subgraph group_group_support["Support"]
  node_node_utils_core["Core utils<br/>shared helpers<br/>[core.py]"]
  node_node_timed_utils["Timing utils<br/>time helpers<br/>[timed_utils.py]"]
  node_node_errors["Errors<br/>exception layer<br/>[errors.py]"]
  node_node_debugging["Debugging<br/>debug support<br/>[debugging.py]"]
  node_node_metadata["Metadata<br/>[metadata.py]"]
end

node_node_main -->|"invokes"| node_node_cli
node_node_cli -->|"drives"| node_node_downloader
node_node_init -->|"exports"| node_node_downloader
node_node_downloader -->|"selects via"| node_node_sites_common
node_node_sites_common -->|"implements"| node_node_youtube
node_node_sites_common -->|"implements"| node_node_twitch
node_node_sites_common -->|"implements"| node_node_zoom
node_node_sites_common -->|"implements"| node_node_facebook
node_node_youtube -->|"uses"| node_node_utils_core
node_node_twitch -->|"uses"| node_node_utils_core
node_node_zoom -->|"uses"| node_node_utils_core
node_node_facebook -->|"uses"| node_node_utils_core
node_node_youtube -->|"uses"| node_node_timed_utils
node_node_downloader -->|"formats"| node_node_formatting
node_node_downloader -->|"writes"| node_node_output
node_node_formatting -->|"loads"| node_node_custom_formats
node_node_cli -->|"controls"| node_node_debugging
node_node_downloader -->|"raises"| node_node_errors
node_node_cli -->|"reports"| node_node_metadata
node_node_output -->|"handles"| node_node_errors

click node_node_init "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/__init__.py"
click node_node_main "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/__main__.py"
click node_node_cli "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/cli.py"
click node_node_downloader "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/chat_downloader.py"
click node_node_formatting "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/formatting/format.py"
click node_node_custom_formats "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/formatting/custom_formats.json"
click node_node_output "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/output/continuous_write.py"
click node_node_sites_common "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/sites/common.py"
click node_node_youtube "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/sites/youtube.py"
click node_node_twitch "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/sites/twitch.py"
click node_node_zoom "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/sites/zoom.py"
click node_node_facebook "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/sites/facebook.py"
click node_node_utils_core "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/utils/core.py"
click node_node_timed_utils "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/utils/timed_utils.py"
click node_node_errors "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/errors.py"
click node_node_debugging "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/debugging.py"
click node_node_metadata "https://github.com/xenova/chat-downloader/blob/master/chat_downloader/metadata.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_init,node_node_main,node_node_cli toneBlue
class node_node_downloader,node_node_formatting,node_node_custom_formats,node_node_output toneAmber
class node_node_sites_common,node_node_youtube,node_node_twitch,node_node_zoom,node_node_facebook toneMint
class node_node_utils_core,node_node_timed_utils,node_node_errors,node_node_debugging,node_node_metadata toneRose
```

## Components

### Entry points

CLI and public package entry layers

- [Public API](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/__init__.py): package export · Exports the library-facing ChatDownloader API · `chat_downloader/__init__.py`
- [Module entry](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/__main__.py): cli entry · Python module entrypoint for command-line execution · `chat_downloader/__main__.py`
- [CLI parser](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/cli.py): command interface · Parses command-line options and invokes the downloader · `chat_downloader/cli.py`

### Core runtime

Download orchestration, filtering, formatting, and output

- [Downloader](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/chat_downloader.py): orchestrator · Coordinates site selection, retrieval, filtering, and emission · `chat_downloader/chat_downloader.py`
- [Formatting](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/formatting/format.py): rendering layer · Converts normalized chat items into output formats · `chat_downloader/formatting/format.py`
- [Format rules](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/formatting/custom_formats.json): format config · Built-in and extensible format definitions · `chat_downloader/formatting/custom_formats.json`
- [Continuous write](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/output/continuous_write.py): output sink · Streams chat items to files or other targets as they arrive · `chat_downloader/output/continuous_write.py`

### Site adapters

Platform-specific chat extractors

- [Site base](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/sites/common.py): adapter base · Shared site interface and common adapter logic · `chat_downloader/sites/common.py`
- [YouTube](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/sites/youtube.py): site adapter · Extracts and normalizes YouTube chat data · `chat_downloader/sites/youtube.py`
- [Twitch](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/sites/twitch.py): site adapter · Extracts and normalizes Twitch chat data · `chat_downloader/sites/twitch.py`
- [Zoom](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/sites/zoom.py): site adapter · Extracts and normalizes Zoom chat data · `chat_downloader/sites/zoom.py`
- [Facebook](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/sites/facebook.py): site adapter · Extracts and normalizes Facebook chat data · `chat_downloader/sites/facebook.py`

### Support

Shared utilities, errors, docs, and packaging

- [Core utils](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/utils/core.py): shared helpers · General-purpose parsing and helper logic used across the package · `chat_downloader/utils/core.py`
- [Timing utils](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/utils/timed_utils.py): time helpers · Time conversion and timing-related helpers · `chat_downloader/utils/timed_utils.py`
- [Errors](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/errors.py): exception layer · Package-specific failure modes and error types · `chat_downloader/errors.py`
- [Debugging](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/debugging.py): debug support · Debug and diagnostic behavior used by the CLI and runtime · `chat_downloader/debugging.py`
- [Metadata](https://github.com/xenova/chat-downloader/blob/master/chat_downloader/metadata.py): package metadata · Version and package metadata support · `chat_downloader/metadata.py`

## Connections

- Module entry → CLI parser: invokes · Module entry hands off to the CLI implementation
- CLI parser → Downloader: drives · CLI constructs downloader calls from parsed options
- Public API → Downloader: exports · Public package API exposes ChatDownloader
- Downloader → Site base: selects via · Core runtime uses the shared site adapter boundary
- Site base → YouTube: implements · YouTube adapter extends the common site layer
- Site base → Twitch: implements · Twitch adapter extends the common site layer
- Site base → Zoom: implements · Zoom adapter extends the common site layer
- Site base → Facebook: implements · Facebook adapter extends the common site layer
- YouTube → Core utils: uses · Site logic shares common helper routines
- Twitch → Core utils: uses · Site logic shares common helper routines
- Zoom → Core utils: uses · Site logic shares common helper routines
- Facebook → Core utils: uses · Site logic shares common helper routines
- YouTube → Timing utils: uses · Site logic relies on timing helpers
- Downloader → Formatting: formats · Normalized chat items are rendered for humans or structured output
- Downloader → Continuous write: writes · Downloader can stream results to an output target
- Formatting → Format rules: loads · Formatting engine is driven by built-in format definitions
- CLI parser → Debugging: controls · CLI exposes debug and pause/exit behavior
- Downloader → Errors: raises · Domain errors surface through the orchestration layer
- CLI parser → Metadata: reports · CLI can present package/version metadata
- Continuous write → Errors: handles · Output layer participates in failure reporting
