How segin/distconv works
Project type and purpose
• DistConv is a distributed video transcoding platform.
• It consists of three main services:
• A central Dispatch Server (C++17) exposing a REST API for job submission, scheduling, state persistence, and engine management.
• Multiple Transcoding Engine workers (C++17) that register via heartbeat, fetch jobs, run FFmpeg subprocesses, update local SQLite queues, and report results.
• A Qt-based Submission Client (desktop GUI) that talks to the Dispatch Server over HTTP.
File structure highlights
• dispatch_server_cpp/ → core server logic, API handlers, job scheduler, state persistence, systemd scripts, 150+ tests.
• transcoding_engine/ → engine core + interface/implementation split (HTTP client, DB, subprocess), mocks, 37 tests, Docker/SystemD support.
• submission_client_desktop/ → Qt GUI, drag-drop UI, CMake build, tests.
• deployment/ & config/ → installer scripts, Docker Compose, SystemD unit files, example configs.
• third_party/ → cpr (HTTP client), nlohmann-json, trompeloeil (mocking).
Key architecture notes from README
• Clients → Dispatch (HTTP REST with X-API-Key).
• Dispatch stores state in JSON file, recovers on restart.
• Engines send heartbeat to Dispatch, receive job assignments.
• Engines maintain local SQLite queue, invoke FFmpeg via secure subprocess, then notify Dispatch of completion or failure.
• System supports Docker Compose, individual containers, and Kubernetes, plus SystemD production services.
a. Components
• Submission Clients (desktop GUI / CLI)
• Dispatch Server
• Persistent State Storage (server_state.json)
• Transcoding Engines
• Engine Local Storage (SQLite DB)
• Subprocess Layer (FFmpeg)
• External Libraries (cpr, nlohmann-json)
• Client → Dispatch: POST /jobs, GET /jobs/{id}, GET /jobs?status
• Dispatch → State Storage: read/write JSON
• Engine ↔ Dispatch: POST /engines/heartbeat → server schedules jobs → engine polls via API or receives assignment
• Engine → SQLite: enqueue/dequeue job records
• Engine → Subprocess: spawn FFmpeg for actual transcoding
• Engine → Dispatch: POST /jobs/{id}/complete or /fail
c. Patterns & principles
• Service separation (dispatch vs. engine)
• Dependency injection in engine (interfaces vs. implementations)
• Thread-safe scheduling and persistent state for fault tolerance
• Docker/SystemD for deployment
d. Technologies
• C++17, CMake, httplib (dispatch), CPR HTTP client (engine), SQLite, FFmpeg subprocess, nlohmann-json, Qt (client), Docker, SystemD, Kubernetes.
Be comprehensive
• Every major block: client, dispatch, storage, engine, subprocess.
• Optionally overlay deployment boundary (Docker or SystemD).
• Tech labels (C++17, Qt, SQLite, FFmpeg, Docker).
• Dependency injection layers inside engine.