How leekichang/kbo_video_collection works
1. What this repo is
- kbo_video_collection is a Python-based downloader and cataloging tool for YouTube KBO playlists.
- Its core job is to pull videos locally, preserve the accompanying *.info.json metadata, and keep enough bookkeeping to avoid re-downloading the same video.
2. Entry points and runtime shape
- The project is packaged as a Python application with a console command (kbo-videos) and a module entrypoint in src/kbo_video_collection/__main__.py.
- cli.py is the user-facing control layer: it exposes commands like listing configured playlists and running downloads with options such as dry-run and cookie/auth overrides.
3. Configuration boundary
- playlists.toml is the primary configuration source.
- config.py is the boundary that reads playlist definitions and YouTube authentication settings, turning TOML into structured application state.
- The config file drives which playlists are active, where they are stored, and what metadata tags or collection labels are attached.
4. Core data model
- models.py likely defines the repository’s internal objects for playlists, manifests, and download records.
- These models are the glue between configuration, downloader behavior, and the generated output files under data/.
5. Download subsystem
- downloader.py is the main integration layer around yt-dlp.
- It is responsible for fetching videos, writing the media files, and saving the source metadata JSON next to each video.
- This is also where format merging and postprocessing depend on ffmpeg, which is called out in the README as an external requirement.
6. External system boundary
- The only major external service is YouTube, accessed indirectly through yt-dlp.
- Authentication is optional but supported through browser cookies or a cookies.txt file when YouTube requires login.
- ffmpeg is a local system dependency, not a Python dependency, but it materially affects whether merged video downloads succeed.
7. Storage layout and persistence
- The repository persists three kinds of local state:
- data/raw/<playlist-name>/<year>/... for the actual media files and their *.info.json companions
- data/manifests/<playlist-name>.json for playlist-level collection metadata
- data/.archives/<playlist-name>.txt for the set of YouTube video IDs already collected
- This layout separates raw artifacts from indexing/state files, which is important for understanding the pipeline.
8. Idempotency and replay behavior
- The downloader is designed to be rerun safely.
- It uses a yt-dlp archive file to skip already-downloaded video IDs.
- If the archive file is missing, it can reconstruct that state by scanning existing data/raw outputs and matching each media file with its *.info.json.
9. Repository-specific artifact boundary
- The src/kbo_video_collection/NA/…info.json file in the tree looks like a captured YouTube metadata artifact, not source code.
- It shows that metadata is treated as a first-class output and may be used later for event-level classification or other downstream analysis.
10. Operational flow
- A typical run is: load playlists.toml → select enabled playlists or a specific playlist → resolve auth options → call yt-dlp for each playlist item → write media plus info.json → update archive/manifests.
- The dry-run mode lets the CLI validate that flow without writing downloads, which is useful for checking configuration and playlist reachability.
11. Dependency and environment management
- pyproject.toml handles packaging/installability, while environment.yml supports a Conda-based setup.
- The README explicitly recommends ffmpeg availability, so the repo straddles both Python packaging and system-level media tooling.
12. Architectural takeaway
- This is not a general-purpose media app; it is a narrow ingestion-and-metadata preservation pipeline.
- The key diagram boundary is between: configuration/CLI on one side, YouTube/yt-dlp/ffmpeg on the other, and a local filesystem-based persistence layer in the middle that preserves both raw video and replay state.
Main entry points: downloader.py Downloader · cli.py CLI · config.py Config loader · README.md README
Components 16
Application
User-facing entrypoints and project packaging
- CLI · User-facing commands for listing playlists and running downloads
src/kbo_video_collection/cli.py - Module entry · Package entrypoint for running the tool as a module
src/kbo_video_collection/__main__.py - Packaging · Python packaging and console-script definition
pyproject.toml - Env spec · Conda environment specification for local development and execution
environment.yml - README · Operational overview and setup guidance
README.md
Configuration
Playlist definitions and runtime settings
- Config loader · Loads playlist definitions and auth settings from TOML
src/kbo_video_collection/config.py - Playlists · Primary repository configuration for active playlists and storage targets
playlists.toml - Models · Internal objects for playlists, manifests, and download records
src/kbo_video_collection/models.py
Download pipeline
Fetch, postprocess, and local persistence flow
- Downloader · Coordinates playlist downloads and local artifact creation
src/kbo_video_collection/downloader.py
Local state
Repository artifacts and replay bookkeeping
- Raw video · Downloaded media files and companion metadata preserved locally
- Manifests · Playlist-level collection metadata stored as local JSON
- Archive · Collected video ID ledger used to skip already-downloaded items
- Info JSON · Saved YouTube metadata companion file preserved next to each video
src/kbo_video_collection/NA/⚾ 2024 KBO 리그 [PLQSdHthiYH-8FiPjoGS0YbFtsn_QxzinI].info.json
External systems
Third-party and system dependencies
- yt-dlp · Third-party download engine used to retrieve videos and metadata
- ffmpeg · System dependency for merging and postprocessing video downloads
- YouTube · Source platform for playlist videos and metadata
Connections 19
- Module entry → CLI: starts · Module entrypoint dispatches into the CLI layer
- Packaging → CLI: exposes · Defines the installable console command
- CLI → Config loader: loads · Reads playlist and auth configuration before executing commands
- CLI → Downloader: runs · Invokes the download workflow for selected playlists
- Config loader → Playlists: parses · Turns TOML playlist definitions into structured state
- Config loader → Models: builds · Creates domain objects used throughout the app
- Downloader → Models: uses · Consumes playlist and record models during orchestration
- Downloader → yt-dlp: delegates · Hands off fetch and metadata retrieval to yt-dlp
- yt-dlp → YouTube: fetches from · Downloads videos and metadata from YouTube
- Downloader → ffmpeg: depends on · Relies on ffmpeg for merging and postprocessing
- Downloader → Raw video: writes · Stores downloaded media files and companion metadata locally
- Downloader → Info JSON: writes · Persists source metadata JSON next to each media file
- Downloader → Manifests: updates · Records playlist-level manifest output
- Downloader → Archive: updates · Maintains replay state for already-collected video IDs
- Downloader → Raw video: scans · Can reconstruct archive state by inspecting existing downloads
- Downloader → Archive: rebuilds · Regenerates archive entries when the archive file is missing
- Env spec → ffmpeg: requires · Supports a local runtime that also needs ffmpeg installed
- README → Config loader: documents · Explains configuration, authentication, and usage flow
- README → Downloader: documents · Describes download behavior and operational requirements