How mxpv/podsync works
Project Type and Purpose
• Podsync is a backend service (Go-based CLI + HTTP server) that generates podcast RSS feeds from YouTube, Vimeo, SoundCloud, Twitch and serves them via HTTP.
• It has two main roles: feed/updater service (scheduler + media builder + storage) and web service (HTTP server to serve RSS + media).
File Structure
• cmd/podsync: entrypoint with configuration loading (config.go) and main.go launching services.
• pkg/builder: modules that, given a feed URL, fetch video lists and build media (youtube, vimeo, soundcloud, twitch).
• pkg/ytdl: wrapper around yt-dlp + temp file handling.
• pkg/db: BadgerDB storage for feed metadata and state.
• pkg/fs: abstraction for file storage (local filesystem or S3).
• pkg/feed: RSS feed generation, OPML export, hooks, filters.
• pkg/model: domain models (Feed, Link, defaults, errors).
• services/update: scheduler (cron), feed updater (matcher/updater).
• services/web: HTTP server exposing RSS and static assets.
• config.toml.example + environment variables define service behavior, API keys, storage, server port.
• docker-compose.yml, Dockerfile, cloud_formation.yml: deployment artifacts for containerized environments and AWS.
Key Architectural Patterns
• Modular service layers: “builder” for media retrieval, “feed” for RSS construction, “db” for state, “fs” for storage, “web” for HTTP server, “update” for scheduling.
• Dependency injection via pkg/feed/deps.go for mocking/testing.
• Adapter pattern in pkg/fs to switch storage backends.
• Cron-driven scheduler to periodically update feeds.
• Single-binary deployment via Go, Docker, or AWS CloudFormation.
Main Components
A. Client Configuration
• config.toml or environment variables
B. Scheduler (services/update)
• cron expressions trigger Updater
C. Feed Updater
• matcher.go (decide which episodes are new)
• updater.go (invoke builder modules)
D. Builder Modules (pkg/builder + pkg/ytdl)
• fetch metadata via APIs (YouTube, Vimeo, etc.)
• download media via yt-dlp, transcode with ffmpeg (outside Go but invoked)
E. Storage
• Metadata: BadgerDB (pkg/db)
• Media files: local FS or S3 (pkg/fs)
F. Feed Generator (pkg/feed)
• assemble RSS XML, apply filters, hooks, OPML export
G. Web Server (services/web)
• HTTP endpoints: /{feedID}/rss.xml, /{feedID}/{episodeID}.mp3
• Serve static HTML (html/index.html)
H. External Services
• YouTube/Vimeo/SoundCloud/Twitch APIs
• yt-dlp and ffmpeg command-line tools
I. Deployment / Infrastructure
• Docker container, Docker Compose, AWS CloudFormation
• CI/CD workflows (.github/workflows)
Relationships and Data Flows
1. At startup, main.go reads configuration → initializes DB, FS, scheduler, web server.
2. Scheduler triggers feed updater per cron → feed updater invokes builder modules.
3. Builder modules call external APIs → download media via yt-dlp → transcode if needed → write media files to FS.
4. Updater writes metadata/state to BadgerDB.
5. Web server handles HTTP requests:
• For RSS: reads metadata from DB + builds RSS via pkg/feed → returns XML.
• For media: serves files from FS.
6. Clients (podcast apps) poll RSS feeds → download episodes.
7. OPML export provided via feed generator for bulk imports.
Technologies and Frameworks to Annotate
• Go modules, Go standard library (net/http, database/sql, etc.)
• BadgerDB (embedded key-value store)
• S3 SDK (AWS)
• yt-dlp, ffmpeg (external)
• Cron expressions (robfig/cron or built-in)
• Docker, Docker Compose, AWS CloudFormation
• CI/CD: GitHub Actions, goreleaser