How dir01/mediary works
Project Type and Purpose
• Project Type: A backend service written in Go that handles media downloads (from torrents and YouTube), media processing (concatenation and encoding), and uploads to an S3-compatible storage.
• Purpose: To expose RESTful endpoints (e.g., /metadata and /jobs) that allow clients to request metadata, start media processing jobs, and monitor their status. The service coordinates multiple internal components (downloaders, media processor, uploader, storage management) which are triggered via a job queue.
File Structure
• cmd/server/main.go – Contains the entrypoint for the service.
• http/ – Hosts the API layer and routing for endpoints such as /metadata and /jobs.
• service/ – Contains core business logic for processing jobs; includes different flows (concatenation, encoding) and a jobs queue module.
• downloader/ – Implements downloading functionality for different sources. Notice separate subdirectories for torrent and YouTube (ytdl) downloaders.
• media_processor/ – Contains logic for media transformation.
• uploader/ – Manages uploading processed media to S3.
• storage/ – Provides storage implementations (in-memory, Redis) for caching metadata.
• tests/ – Contains both end-to-end and unit tests.
• Docker-related files (Dockerfile, docker-compose.yml, .env) – Indicate containerization and deployment settings.
• CI/CD configuration in .github – Shows integration with workflows for tests and builds.
Main Components
A. API Layer (HTTP Interface):
• Shows endpoints:
• /metadata (GET/POST) for retrieving media metadata.
• /jobs (POST) for job creation and /jobs/:id (GET) for status checks.
B. Job Management and Workflow:
• The service layer (service/) houses business logic.
• The jobs_queue module should be highlighted as it oversees job scheduling and transitions (statuses such as created, downloading, processing, uploading, complete).
• This as a “Job Dispatcher/Manager” component.
C. Download Module:
• Separated into two specialized modules:
• Torrent Downloader in downloader/torrent.
• YouTube Downloader in downloader/ytdl.
• As distinct blocks under a broader “Downloader” component.
D. Media Processing:
• The media_processor component applies transformations (e.g., concatenation or audio encoding).
• This should be drawn as a “Media Processor” block.
E. Upload Module:
• Handles file uploads to S3; found in uploader/.
F. Storage/Caching:
• Modules under storage (in-memory and Redis) are used to cache metadata or job-related data.
• “Storage” with its two potential backends (Memory, Redis).
G. External Dependencies:
• S3 or pre-signed URL destinations for uploads.
• Any external services (e.g., torrent trackers, YouTube APIs) used by the downloader modules.
Relationships and Data Flow
• Client Applications make HTTP requests to Mediary’s API Gateway.
• The HTTP API layer routes requests to the corresponding service logic:
• A GET/POST to /metadata channels into the downloader component to fetch or cache metadata.
• A POST to /jobs creates a job in the jobs queue which triggers a sequence of operations:
1. Downloader module fetches the media content.
2. Media processor transforms/concatenates or converts the media.
3. Uploader pushes the processed media to the S3 storage endpoint.
• Feedback loops, like caching metadata or querying job status, should be clearly connected to the API layer and storage.
Architectural Patterns & Design Principles
• Layered Architecture: Distinct separation between API, service/business logic, and external integration (downloaders, processors, uploaders).
• Modularity: The project divides responsibilities into clearly defined modules (downloader, media processor, uploader) that can be extended or substituted.
• Asynchronous Processing: Job queue handling enables asynchronous processing of long-running tasks.