# abraham-ny/vidpipeline architecture

> Architecture diagram of the GitHub repository abraham-ny/vidpipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abraham-ny/vidpipeline
- Repository: https://github.com/abraham-ny/vidpipeline

## Overview

This Go service accepts multipart video uploads, validates and queues them, then processes each job concurrently into video variants, thumbnails, previews, and metadata. The API also exposes job status, an in-memory job list, a processed-video library, and its files. The graph separates upload/validation, queueing, processing stages, and API access to job and artifact state. Processor details are based on partial excerpts; the README documents additional outputs and API capabilities.

Main entry points: [`processor/processor.go`](https://github.com/abraham-ny/vidpipeline/blob/main/processor/processor.go) (Processor), [`api/handler.go`](https://github.com/abraham-ny/vidpipeline/blob/main/api/handler.go) (API handlers), [`worker/pool.go`](https://github.com/abraham-ny/vidpipeline/blob/main/worker/pool.go) (Worker pool), [`main.go`](https://github.com/abraham-ny/vidpipeline/blob/main/main.go) (HTTP server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["HTTP API"]
  node_server["HTTP server<br/>[main.go]"]
  node_handler["API handlers<br/>[handler.go]"]
end

subgraph group_jobs["Job orchestration"]
  node_pool["Worker pool<br/>[pool.go]"]
  node_registry[("Job registry<br/>[job.go]")]
  node_job["Job state<br/>[job.go]"]
end

subgraph group_processing["Video processing"]
  node_processor["Processor<br/>[processor.go]"]
  node_probe["Probe and metadata<br/>[processor.go]"]
  node_transcode["Resolution transcodes<br/>[processor.go]"]
  node_thumbnails["Thumbnail generation<br/>[processor.go]"]
  node_previews["GIF and clip previews<br/>[processor.go]"]
  node_titles["Title parsing<br/>[processor.go]"]
end

subgraph group_artifacts["Processed library"]
  node_filesystem[("Processed files")]
  node_metadata["Video metadata<br/>[job.go]"]
end

node_client(("API client"))
node_ffmpeg(("FFmpeg tools"))

node_client -->|"sends requests"| node_server
node_server -->|"dispatches"| node_handler
node_handler -->|"validates with ffprobe"| node_ffmpeg
node_handler -->|"writes uploads"| node_filesystem
node_handler -->|"enqueues jobs"| node_pool
node_pool -->|"registers jobs"| node_registry
node_pool -->|"dispatches jobs"| node_processor
node_processor -->|"updates progress"| node_job
node_processor -->|"probes source"| node_probe
node_processor -->|"runs transcodes"| node_transcode
node_processor -->|"generates thumbnails"| node_thumbnails
node_processor -->|"generates previews"| node_previews
node_processor -->|"parses title"| node_titles
node_probe -->|"calls ffprobe"| node_ffmpeg
node_transcode -->|"calls ffmpeg"| node_ffmpeg
node_thumbnails -->|"calls ffmpeg"| node_ffmpeg
node_previews -->|"calls ffmpeg"| node_ffmpeg
node_processor -->|"writes artifacts"| node_filesystem
node_processor -->|"builds metadata"| node_metadata
node_metadata -->|"saved as meta.json"| node_filesystem
node_handler -->|"reads job status and list"| node_registry
node_handler -->|"reads library metadata"| node_filesystem
node_handler -->|"serves video files"| node_filesystem

click node_server "https://github.com/abraham-ny/vidpipeline/blob/main/main.go"
click node_handler "https://github.com/abraham-ny/vidpipeline/blob/main/api/handler.go"
click node_pool "https://github.com/abraham-ny/vidpipeline/blob/main/worker/pool.go"
click node_registry "https://github.com/abraham-ny/vidpipeline/blob/main/model/job.go"
click node_job "https://github.com/abraham-ny/vidpipeline/blob/main/model/job.go"
click node_processor "https://github.com/abraham-ny/vidpipeline/blob/main/processor/processor.go"
click node_probe "https://github.com/abraham-ny/vidpipeline/blob/main/processor/processor.go"
click node_transcode "https://github.com/abraham-ny/vidpipeline/blob/main/processor/processor.go"
click node_thumbnails "https://github.com/abraham-ny/vidpipeline/blob/main/processor/processor.go"
click node_previews "https://github.com/abraham-ny/vidpipeline/blob/main/processor/processor.go"
click node_titles "https://github.com/abraham-ny/vidpipeline/blob/main/processor/processor.go"
click node_metadata "https://github.com/abraham-ny/vidpipeline/blob/main/model/job.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_server,node_handler,node_client toneBlue
class node_pool,node_registry,node_job toneAmber
class node_processor,node_probe,node_transcode,node_thumbnails,node_previews,node_titles toneMint
class node_filesystem,node_metadata toneRose
class node_ffmpeg toneIndigo
```

## Components

### HTTP API

- [HTTP server](https://github.com/abraham-ny/vidpipeline/blob/main/main.go): `main.go`
- [API handlers](https://github.com/abraham-ny/vidpipeline/blob/main/api/handler.go): `api/handler.go`

### Job orchestration

- [Worker pool](https://github.com/abraham-ny/vidpipeline/blob/main/worker/pool.go): `worker/pool.go`
- [Job registry](https://github.com/abraham-ny/vidpipeline/blob/main/model/job.go): `model/job.go`
- [Job state](https://github.com/abraham-ny/vidpipeline/blob/main/model/job.go): `model/job.go`

### Video processing

- [Processor](https://github.com/abraham-ny/vidpipeline/blob/main/processor/processor.go): `processor/processor.go`
- [Probe and metadata](https://github.com/abraham-ny/vidpipeline/blob/main/processor/processor.go): `processor/processor.go`
- [Resolution transcodes](https://github.com/abraham-ny/vidpipeline/blob/main/processor/processor.go): `processor/processor.go`
- [Thumbnail generation](https://github.com/abraham-ny/vidpipeline/blob/main/processor/processor.go): `processor/processor.go`
- [GIF and clip previews](https://github.com/abraham-ny/vidpipeline/blob/main/processor/processor.go): `processor/processor.go`
- [Title parsing](https://github.com/abraham-ny/vidpipeline/blob/main/processor/processor.go): `processor/processor.go`

### Processed library

- **Processed files**
- [Video metadata](https://github.com/abraham-ny/vidpipeline/blob/main/model/job.go): `model/job.go`

### Other components

- **API client**
- **FFmpeg tools**

## Connections

- API client → HTTP server: sends requests
- HTTP server → API handlers: dispatches
- API handlers → FFmpeg tools: validates with ffprobe
- API handlers → Processed files: writes uploads
- API handlers → Worker pool: enqueues jobs
- Worker pool → Job registry: registers jobs
- Worker pool → Processor: dispatches jobs
- Processor → Job state: updates progress
- Processor → Probe and metadata: probes source
- Processor → Resolution transcodes: runs transcodes
- Processor → Thumbnail generation: generates thumbnails
- Processor → GIF and clip previews: generates previews
- Processor → Title parsing: parses title
- Probe and metadata → FFmpeg tools: calls ffprobe
- Resolution transcodes → FFmpeg tools: calls ffmpeg
- Thumbnail generation → FFmpeg tools: calls ffmpeg
- GIF and clip previews → FFmpeg tools: calls ffmpeg
- Processor → Processed files: writes artifacts
- Processor → Video metadata: builds metadata
- Video metadata → Processed files: saved as meta.json
- API handlers → Job registry: reads job status and list
- API handlers → Processed files: reads library metadata
- API handlers → Processed files: serves video files