# dir01/mediary architecture

> Architecture diagram of the GitHub repository dir01/mediary, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dir01/mediary
- Repository: https://github.com/dir01/mediary

## Overview

##### Project Type and Purpose
- Project Type: A backend service written in Go that handles media downloads (from torrents and YouTube), media processing (concatenation and encoding), and uploads to an S3-compatible storage.
- Purpose: To expose RESTful endpoints (e.g., /metadata and /jobs) that allow clients to request metadata, start media processing jobs, and monitor their status. The service coordinates multiple internal components (downloaders, media processor, uploader, storage management) which are triggered via a job queue.

##### File Structure
- cmd/server/main.go – Contains the entrypoint for the service.
- http/ – Hosts the API layer and routing for endpoints such as /metadata and /jobs.
- service/ – Contains core business logic for processing jobs; includes different flows (concatenation, encoding) and a jobs queue module.
- downloader/ – Implements downloading functionality for different sources. Notice separate subdirectories for torrent and YouTube (ytdl) downloaders.
- media_processor/ – Contains logic for media transformation.
- uploader/ – Manages uploading processed media to S3.
- storage/ – Provides storage implementations (in-memory, Redis) for caching metadata.
- tests/ – Contains both end-to-end and unit tests.
- Docker-related files (Dockerfile, docker-compose.yml, .env) – Indicate containerization and deployment settings.
- CI/CD configuration in .github – Shows integration with workflows for tests and builds.

##### Main Components
A. API Layer (HTTP Interface):
- Shows endpoints:
- /metadata (GET/POST) for retrieving media metadata.
- /jobs (POST) for job creation and /jobs/:id (GET) for status checks.

B. Job Management and Workflow:
- The service layer (service/) houses business logic.
- The jobs_queue module should be highlighted as it oversees job scheduling and transitions (statuses such as created, downloading, processing, uploading, complete).
- This as a “Job Dispatcher/Manager” component.

C. Download Module:
- Separated into two specialized modules:
- Torrent Downloader in downloader/torrent.
- YouTube Downloader in downloader/ytdl.
- As distinct blocks under a broader “Downloader” component.

D. Media Processing:
- The media_processor component applies transformations (e.g., concatenation or audio encoding).
- This should be drawn as a “Media Processor” block.

E. Upload Module:
- Handles file uploads to S3; found in uploader/.

F. Storage/Caching:
- Modules under storage (in-memory and Redis) are used to cache metadata or job-related data.
- “Storage” with its two potential backends (Memory, Redis).

G. External Dependencies:
- S3 or pre-signed URL destinations for uploads.
- Any external services (e.g., torrent trackers, YouTube APIs) used by the downloader modules.

##### Relationships and Data Flow
- Client Applications make HTTP requests to Mediary’s API Gateway.
- The HTTP API layer routes requests to the corresponding service logic:
- A GET/POST to /metadata channels into the downloader component to fetch or cache metadata.
- A POST to /jobs creates a job in the jobs queue which triggers a sequence of operations:
1. Downloader module fetches the media content.
2. Media processor transforms/concatenates or converts the media.
3. Uploader pushes the processed media to the S3 storage endpoint.
- Feedback loops, like caching metadata or querying job status, should be clearly connected to the API layer and storage.

##### Architectural Patterns & Design Principles
- Layered Architecture: Distinct separation between API, service/business logic, and external integration (downloaders, processors, uploaders).
- Modularity: The project divides responsibilities into clearly defined modules (downloader, media processor, uploader) that can be extended or substituted.
- Asynchronous Processing: Job queue handling enables asynchronous processing of long-running tasks.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Global Entities
    Client("Client Applications"):::external

    %% API Layer
    subgraph "API Layer"
        APIGateway("HTTP API Gateway"):::api
    end

    %% Service Layer
    subgraph "Service Layer"
        JobManager("Job Dispatcher/Manager"):::internal
        subgraph "Downloaders"
            TorrentDownloader("Torrent Downloader"):::internal
            YouTubeDownloader("YouTube Downloader"):::internal
        end
        MediaProcessor("Media Processor"):::internal
        Uploader("Uploader"):::internal
    end

    %% Storage Layer
    subgraph "Storage Layer"
        MemoryStorage("Memory Storage"):::storage
        RedisStorage("Redis Storage"):::storage
    end

    %% External Dependencies
    subgraph "External Systems"
        S3Storage("S3 Storage"):::external
    end

    %% Connections and Data Flow
    Client -->|"requests"| APIGateway
    APIGateway -->|"jobRequest"| JobManager
    JobManager -->|"initiateDownload"| TorrentDownloader
    JobManager -->|"initiateDownload"| YouTubeDownloader
    TorrentDownloader -->|"sendsMedia"| MediaProcessor
    YouTubeDownloader -->|"sendsMedia"| MediaProcessor
    MediaProcessor -->|"processedMedia"| Uploader
    Uploader -->|"uploadsMedia"| S3Storage
    APIGateway -->|"cacheMetadata"| MemoryStorage
    APIGateway -->|"cacheMetadata"| RedisStorage
    MemoryStorage -->|"metadataReady"| APIGateway
    RedisStorage -->|"metadataReady"| APIGateway

    %% Click Events for Component Mapping
    click APIGateway "https://github.com/dir01/mediary/blob/master/http/http.go"
    click APIGateway "https://github.com/dir01/mediary/blob/master/http/jobs.go"
    click APIGateway "https://github.com/dir01/mediary/blob/master/http/metadata.go"
    click JobManager "https://github.com/dir01/mediary/blob/master/service/jobs.go"
    click JobManager "https://github.com/dir01/mediary/blob/master/service/jobs_queue/jobs_queue.go"
    click TorrentDownloader "https://github.com/dir01/mediary/blob/master/downloader/torrent/torrent.go"
    click YouTubeDownloader "https://github.com/dir01/mediary/blob/master/downloader/ytdl/ytdl.go"
    click YouTubeDownloader "https://github.com/dir01/mediary/blob/master/downloader/ytdl/structs.go"
    click MediaProcessor "https://github.com/dir01/mediary/blob/master/media_processor/media_processor.go"
    click Uploader "https://github.com/dir01/mediary/blob/master/uploader/uploader.go"
    click MemoryStorage "https://github.com/dir01/mediary/blob/master/storage/memory.go"
    click RedisStorage "https://github.com/dir01/mediary/blob/master/storage/redis.go"

    %% Styles
    classDef internal fill:#bbf,stroke:#000,stroke-width:2px;
    classDef api fill:#fcb,stroke:#000,stroke-width:2px;
    classDef storage fill:#bfb,stroke:#000,stroke-width:2px;
    classDef external fill:#fcc,stroke:#000,stroke-width:2px;
```
