# segin/distconv architecture

> Architecture diagram of the GitHub repository segin/distconv, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/segin/distconv
- Repository: https://github.com/segin/distconv

## Overview

##### Project type and purpose
- DistConv is a distributed video transcoding platform.
- It consists of three main services:
- A central Dispatch Server (C++17) exposing a REST API for job submission, scheduling, state persistence, and engine management.
- Multiple Transcoding Engine workers (C++17) that register via heartbeat, fetch jobs, run FFmpeg subprocesses, update local SQLite queues, and report results.
- A Qt-based Submission Client (desktop GUI) that talks to the Dispatch Server over HTTP.

##### File structure highlights
- dispatch_server_cpp/ → core server logic, API handlers, job scheduler, state persistence, systemd scripts, 150+ tests.
- transcoding_engine/ → engine core + interface/implementation split (HTTP client, DB, subprocess), mocks, 37 tests, Docker/SystemD support.
- submission_client_desktop/ → Qt GUI, drag-drop UI, CMake build, tests.
- deployment/ & config/ → installer scripts, Docker Compose, SystemD unit files, example configs.
- third_party/ → cpr (HTTP client), nlohmann-json, trompeloeil (mocking).

##### Key architecture notes from README
- Clients → Dispatch (HTTP REST with X-API-Key).
- Dispatch stores state in JSON file, recovers on restart.
- Engines send heartbeat to Dispatch, receive job assignments.
- Engines maintain local SQLite queue, invoke FFmpeg via secure subprocess, then notify Dispatch of completion or failure.
- System supports Docker Compose, individual containers, and Kubernetes, plus SystemD production services.

a. Components
- Submission Clients (desktop GUI / CLI)
- Dispatch Server
- Persistent State Storage (server_state.json)
- Transcoding Engines
- Engine Local Storage (SQLite DB)
- Subprocess Layer (FFmpeg)
- External Libraries (cpr, nlohmann-json)
- Client → Dispatch: POST /jobs, GET /jobs/{id}, GET /jobs?status
- Dispatch → State Storage: read/write JSON
- Engine ↔ Dispatch: POST /engines/heartbeat → server schedules jobs → engine polls via API or receives assignment
- Engine → SQLite: enqueue/dequeue job records
- Engine → Subprocess: spawn FFmpeg for actual transcoding
- Engine → Dispatch: POST /jobs/{id}/complete or /fail
c. Patterns & principles
- Service separation (dispatch vs. engine)
- Dependency injection in engine (interfaces vs. implementations)
- Thread-safe scheduling and persistent state for fault tolerance
- Docker/SystemD for deployment
d. Technologies
- C++17, CMake, httplib (dispatch), CPR HTTP client (engine), SQLite, FFmpeg subprocess, nlohmann-json, Qt (client), Docker, SystemD, Kubernetes.

##### Be comprehensive
- Every major block: client, dispatch, storage, engine, subprocess.
- Optionally overlay deployment boundary (Docker or SystemD).
- Tech labels (C++17, Qt, SQLite, FFmpeg, Docker).
- Dependency injection layers inside engine.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Submission Clients
    subgraph "Submission Clients" 
        direction TB
        SubmissionClient["Submission Client\n(Qt GUI / CLI)\nC++, Qt"]:::client
    end

    %% Backend Services
    subgraph "Backend"
        direction TB
        DispatchServer["Dispatch Server\nREST API, Scheduler\nC++17, httplib, CMake"]:::server
        StateStorage["Server State Storage\nserver_state.json"]:::storage
        subgraph "Transcoding Engines"
            direction TB
            EngineWorker["Transcoding Engine Worker\nC++17, CPR, SQLite, FFmpeg"]:::engine
        end
    end

    %% Third-Party Libraries
    subgraph "Third-Party Libraries"
        direction TB
        CPR["cpr HTTP Client"]:::third
        JSONLib["nlohmann-json"]:::third
        MockLib["trompeloeil (Mocking)"]:::third
    end

    %% Deployment
    subgraph "Deployment / Config"
        direction TB
        DispatchService["distconv-dispatch.service\nSystemD"]:::storage
        ConfigExamples["config/*.conf\nserver.conf.example"]:::third
    end

    %% Interactions
    SubmissionClient -->|"HTTP REST\nPOST/GET /jobs\nX-API-Key"| DispatchServer
    DispatchServer -->|"File I/O\nread/write JSON"| StateStorage
    EngineWorker -->|"HTTP REST\n/engines/heartbeat\n/assign_job"| DispatchServer
    EngineWorker -->|"State I/O\nSQLite queue"| EngineWorker
    EngineWorker -->|"Subprocess call\nFFmpeg"| EngineWorker
    EngineWorker -->|"HTTP REST\nPOST /jobs/{id}/complete|/fail"| DispatchServer

    %% Dependencies on Third-Party
    DispatchServer --> CPR
    DispatchServer --> JSONLib
    EngineWorker --> CPR
    EngineWorker --> JSONLib
    EngineWorker --> MockLib

    %% Deployment Links
    DispatchServer --> DispatchService
    DispatchService --> ConfigExamples

    %% Click Events
    click SubmissionClient "https://github.com/segin/distconv/blob/main/submission_client_desktop/app_main.cpp"
    click DispatchServer "https://github.com/segin/distconv/blob/main/dispatch_server_cpp/app_main.cpp"
    click DispatchServer "https://github.com/segin/distconv/blob/main/dispatch_server_cpp/dispatch_server_core.cpp"
    click DispatchServer "https://github.com/segin/distconv/blob/main/dispatch_server_cpp/job_scheduler.cpp"
    click DispatchServer "https://github.com/segin/distconv/blob/main/dispatch_server_cpp/repositories.cpp"
    click DispatchServer "https://github.com/segin/distconv/blob/main/dispatch_server_cpp/api_middleware.cpp"
    click DispatchServer "https://github.com/segin/distconv/blob/main/dispatch_server_cpp/enhanced_endpoints.cpp"
    click DispatchServer "https://github.com/segin/distconv/blob/main/dispatch_server_cpp/response_utils.cpp"
    click StateStorage "https://github.com/segin/distconv/blob/main/dispatch_server_state.json"
    click DispatchService "https://github.com/segin/distconv/blob/main/deployment/distconv-dispatch.service"
    click EngineWorker "https://github.com/segin/distconv/blob/main/transcoding_engine/app_main.cpp"
    click EngineWorker "https://github.com/segin/distconv/blob/main/transcoding_engine/src/core/transcoding_engine.cpp"
    click EngineWorker "https://github.com/segin/distconv/blob/main/transcoding_engine/transcoding_engine_core.cpp"
    click EngineWorker "https://github.com/segin/distconv/blob/main/transcoding_engine/src/interfaces/http_client_interface.h"
    click EngineWorker "https://github.com/segin/distconv/blob/main/transcoding_engine/src/interfaces/database_interface.h"
    click EngineWorker "https://github.com/segin/distconv/blob/main/transcoding_engine/src/interfaces/subprocess_interface.h"
    click EngineWorker "https://github.com/segin/distconv/blob/main/transcoding_engine/src/implementations/cpr_http_client.cpp"
    click EngineWorker "https://github.com/segin/distconv/blob/main/transcoding_engine/src/implementations/sqlite_database.cpp"
    click EngineWorker "https://github.com/segin/distconv/blob/main/transcoding_engine/src/implementations/secure_subprocess.cpp"
    click EngineWorker "https://github.com/segin/distconv/blob/main/transcoding_engine/CMakeLists.txt"
    click CPR "https://github.com/segin/distconv/tree/main/third_party/cpr"
    click JSONLib "https://github.com/segin/distconv/tree/main/third_party/json"
    click MockLib "https://github.com/segin/distconv/tree/main/third_party/trompeloeil"

    %% Styles
    classDef client fill:#a2f5a2,stroke:#333,stroke-width:1px
    classDef server fill:#a2c4f5,stroke:#333,stroke-width:1px
    classDef engine fill:#f5ca7b,stroke:#333,stroke-width:1px
    classDef storage fill:#d3d3d3,stroke:#333,stroke-width:1px
    classDef third fill:#ccccff,stroke:#333,stroke-width:1px
```
