How xenova/chat-downloader works
1. Project shape
- chat_downloader is a Python package that serves both as a library and a CLI tool.
- The top-level behavior is: take a chat/video URL, identify the supported platform, stream parsed chat messages, optionally format/write them, and expose them to Python callers or the command line.
2. Primary entry points
- chat_downloader/__main__.py and chat_downloader/cli.py are the command-line entry path.
- chat_downloader/__init__.py exposes the public API, centered around ChatDownloader.
- chat_downloader/chat_downloader.py is the main orchestration layer that ties together site detection, retrieval, filtering, and output.
3. Core orchestration layer
- ChatDownloader is the central coordinator.
- It accepts a URL and options such as time bounds, message filtering, retry behavior, proxies/cookies, output formatting, and chat type.
- It produces an iterator/generator of chat items rather than loading everything at once, which is important for long livestreams and large archives.
4. Site adapter boundary
- chat_downloader/sites/ contains platform-specific extractors for YouTube, Twitch, Zoom, and Facebook.
- sites/common.py likely defines the shared interface/base logic used by all site modules.
- This is the key extensibility boundary: the rest of the system works on normalized chat items, while each site module knows how to talk to a particular external service and transform its response into the common format.
5. Data acquisition flow
- The downloader starts from a URL, selects the matching site module, and fetches raw chat-related data from the target service.
- The site adapter converts site-specific payloads into standardized message dictionaries containing fields like message_id, message, message_type, timestamps, and author metadata.
- The README’s “Chat Items” section reflects this normalized contract.
6. Message filtering and control
- Message selection is controlled by options such as start_time, end_time, message_types, message_groups, max_messages, timeout, and inactivity/retry limits.
- This filtering happens in the core flow before output, so consumers can trim the stream without needing to process every raw message themselves.
7. Formatting subsystem
- chat_downloader/formatting/ handles turning normalized chat items into human-readable or structured output.
- format.py and custom_formats.json suggest both built-in format definitions and user-extensible formatting rules.
- This subsystem sits between the normalized message stream and any console/file presentation.
8. Output subsystem
- chat_downloader/output/continuous_write.py indicates support for incremental writing as messages arrive.
- This is the file/stream persistence boundary: messages can be emitted continuously to an output target, rather than buffered until the end.
- The CLI’s --output, --overwrite, --sort_keys, and --indent flags fit this layer.
9. Shared utilities and timing
- chat_downloader/utils/ contains reusable helpers, especially core.py and timed_utils.py.
- These likely support parsing, retries, time conversions, rate handling, and other cross-cutting logic used by both site modules and the core downloader.
10. Errors and debugging
- chat_downloader/errors.py defines the package’s domain-specific failure modes.
- chat_downloader/debugging.py and CLI flags like --pause_on_debug, --exit_on_debug, and logging levels show that debugging is a first-class concern.
- This boundary is important for surfacing site failures, network issues, and unsupported behaviors cleanly.
11. Metadata and package support
- chat_downloader/metadata.py likely centralizes versioning or package metadata used by the CLI and distribution.
- setup.py, setup.cfg, MANIFEST.in, tox.ini, and CI workflows show this is a distributable package with automated testing and publishing.
12. Testing and verification
- tests/ covers the major architecture layers: downloader behavior, CLI, formatting, generators, init parameters, sites, utils, and writers.
- The presence of site-specific tests and a youtube_actions.json fixture suggests integration-style validation against real or recorded platform behavior.
13. Documentation and generated artifacts
- docs/ contains the user and developer documentation, including CLI docs, site docs, and scripts that generate parts of the published docs/readme.
- The repository is structured so the docs mirror the same architectural boundaries as the code: CLI, items, options, and supported sites.
14. External dependency boundary
- The only clearly material external dependencies are the supported content platforms themselves: YouTube, Twitch, Zoom, and Facebook.
- The package is intentionally designed to avoid requiring user authentication for the supported flows, which affects the architecture by emphasizing public-page parsing and platform-specific retrieval logic.
Main entry points: chat_downloader.py Downloader · common.py Site base · cli.py CLI parser · core.py Core utils
Components 17
Entry points
CLI and public package entry layers
- Public API · Exports the library-facing ChatDownloader API
chat_downloader/__init__.py - Module entry · Python module entrypoint for command-line execution
chat_downloader/__main__.py - CLI parser · Parses command-line options and invokes the downloader
chat_downloader/cli.py
Core runtime
Download orchestration, filtering, formatting, and output
- Downloader · Coordinates site selection, retrieval, filtering, and emission
chat_downloader/chat_downloader.py - Formatting · Converts normalized chat items into output formats
chat_downloader/formatting/format.py - Format rules · Built-in and extensible format definitions
chat_downloader/formatting/custom_formats.json - Continuous write · Streams chat items to files or other targets as they arrive
chat_downloader/output/continuous_write.py
Site adapters
Platform-specific chat extractors
- Site base · Shared site interface and common adapter logic
chat_downloader/sites/common.py - YouTube · Extracts and normalizes YouTube chat data
chat_downloader/sites/youtube.py - Twitch · Extracts and normalizes Twitch chat data
chat_downloader/sites/twitch.py - Zoom · Extracts and normalizes Zoom chat data
chat_downloader/sites/zoom.py - Facebook · Extracts and normalizes Facebook chat data
chat_downloader/sites/facebook.py
Support
Shared utilities, errors, docs, and packaging
- Core utils · General-purpose parsing and helper logic used across the package
chat_downloader/utils/core.py - Timing utils · Time conversion and timing-related helpers
chat_downloader/utils/timed_utils.py - Errors · Package-specific failure modes and error types
chat_downloader/errors.py - Debugging · Debug and diagnostic behavior used by the CLI and runtime
chat_downloader/debugging.py - Metadata · Version and package metadata support
chat_downloader/metadata.py
Connections 20
- Module entry → CLI parser: invokes · Module entry hands off to the CLI implementation
- CLI parser → Downloader: drives · CLI constructs downloader calls from parsed options
- Public API → Downloader: exports · Public package API exposes ChatDownloader
- Downloader → Site base: selects via · Core runtime uses the shared site adapter boundary
- Site base → YouTube: implements · YouTube adapter extends the common site layer
- Site base → Twitch: implements · Twitch adapter extends the common site layer
- Site base → Zoom: implements · Zoom adapter extends the common site layer
- Site base → Facebook: implements · Facebook adapter extends the common site layer
- YouTube → Core utils: uses · Site logic shares common helper routines
- Twitch → Core utils: uses · Site logic shares common helper routines
- Zoom → Core utils: uses · Site logic shares common helper routines
- Facebook → Core utils: uses · Site logic shares common helper routines
- YouTube → Timing utils: uses · Site logic relies on timing helpers
- Downloader → Formatting: formats · Normalized chat items are rendered for humans or structured output
- Downloader → Continuous write: writes · Downloader can stream results to an output target
- Formatting → Format rules: loads · Formatting engine is driven by built-in format definitions
- CLI parser → Debugging: controls · CLI exposes debug and pause/exit behavior
- Downloader → Errors: raises · Domain errors surface through the orchestration layer
- CLI parser → Metadata: reports · CLI can present package/version metadata
- Continuous write → Errors: handles · Output layer participates in failure reporting