# jellyfin/jellyfin-ffmpeg architecture

> Architecture diagram of the GitHub repository jellyfin/jellyfin-ffmpeg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jellyfin/jellyfin-ffmpeg
- Repository: https://github.com/jellyfin/jellyfin-ffmpeg

## Overview

1. This repository is the FFmpeg multimedia stack: a set of native C libraries plus command-line tools for ingesting, decoding, transforming, and emitting audio/video/subtitle data. The architecture is organized around reusable libraries with thin tool front-ends on top.

2. The core processing pipeline is: input demux/protocol layer (`libavformat`) reads bytes from files, pipes, network streams, or capture devices; codec layer (`libavcodec`) decodes or encodes elementary streams; filtering (`libavfilter`) transforms decoded frames and samples; resampling/scaling (`libswresample`, `libswscale`) handle format conversion; output mux/protocol layer (`libavformat`) packages the results back into containers or streams.

3. `libavformat` is the I/O and container boundary. It owns demuxers, muxers, protocol handlers, timestamp/seek logic, and network transports such as HTTP, RTP/RTSP, HLS/DASH, TCP/UDP/TLS, plus many container-specific readers/writers. This is the main boundary between FFmpeg and the outside world.

4. `libavcodec` is the largest subsystem and contains the codec implementations themselves. It includes decoders, encoders, parsers, bitstream filters, and hardware-accelerated codec paths. The directory structure shows a strong split between codec families, shared codec infrastructure, and many architecture-specific optimized backends.

5. Hardware acceleration is a first-class architectural concern, not an add-on. `libavutil/hwcontext.*` defines device/frame abstractions that connect codecs and filters to GPU or platform APIs such as CUDA, VAAPI, QSV, DXVA2/D3D11VA/D3D12VA, VideoToolbox, Vulkan, OpenCL, MediaCodec, and VDPAU. In the data flow, these paths let frames stay on device memory across decode/filter/encode stages when supported.

6. `libavfilter` is the graph engine for frame and sample processing. It models audio/video transformations as connected filters, with source/sink nodes, graph parsing, frame synchronization, and many CPU/GPU-specific filters. This is the subsystem that sits between decode and encode when the pipeline is doing anything more than pass-through.

7. `libswresample` and `libswscale` are focused conversion subsystems. `libswresample` handles audio sample format changes, channel remixing, and resampling; `libswscale` handles pixel format conversion, colorspace transforms, and resizing. In diagrams, these are the “format adaptation” blocks that often bridge incompatible decoder/filter/encoder requirements.

8. `libavdevice` is the capture/playback edge of the system. It abstracts platform media devices such as ALSA, PulseAudio, DirectShow, AVFoundation, DeckLink, V4L2, X11 grabbers, OpenGL/SDL outputs, and similar device interfaces. It feeds into the same format/codec pipeline as file and network inputs, but with device-specific acquisition and rendering boundaries.

9. The command-line tools in `fftools` are orchestration layers over the libraries rather than independent pipelines. `ffmpeg` drives transcoding and streaming jobs, `ffprobe` performs inspection and metadata analysis, and `ffplay` provides playback/rendering. They coordinate demuxing, decoding, filtering, encoding, muxing, threading, and device output via the shared libraries.

10. `libavutil` underpins all higher layers with common services: memory and buffers, logging, dictionaries, option parsing, time/timestamps, math, pixel/sample descriptors, crypto/hash utilities, CPU feature detection, threading primitives, and hardware context support. It is the base dependency that keeps the rest of the stack portable and modular.

11. Performance is heavily shaped by architecture-specific optimized code. Many directories are split by CPU family (`x86`, `arm`, `aarch64`, `mips`, `ppc`, `riscv`, `loongarch`, etc.) and use handwritten assembly or intrinsics for hot paths in codecs, scaling, resampling, and filtering. The diagram should treat these as backend implementations behind generic APIs, not separate features.

12. The repository also includes a substantial testing and validation layer. `tests/` contains API tests, sample generators, FATE regression definitions, checkasm microbenchmarks for SIMD routines, and golden reference outputs. This is important architecturally because correctness is enforced across many codecs, containers, filters, and platforms.

13. Build and packaging are major cross-cutting concerns. The top-level build system uses Makefiles and configure scripts, with Meson/cross-toolchain artifacts, Docker and CI workflows, Debian packaging, and Windows/MSYS2 build scripts. The `builder/` tree is a dedicated release/build pipeline for reproducible multi-platform binaries and dependency bundling.

14. Overall, the repo is best diagrammed as layered media infrastructure: external inputs/devices/protocols at the edge; format/demux and codec cores in the middle; filter and conversion subsystems as the processing graph; and mux/protocol/device outputs on the other side, all glued together by shared utilities, hardware-context abstractions, platform optimizations, and extensive regression testing.

Main entry points: [`libavfilter`](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavfilter) (avfilter), [`fftools`](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/fftools) (fftools), [`libavutil/hwcontext.c`](https://github.com/jellyfin/jellyfin-ffmpeg/blob/jellyfin/libavutil/hwcontext.c) (hwcontext), [`libavutil`](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavutil) (avutil)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core Media"]
  node_avformat["avformat<br/>containers I/O"]
  node_avcodec["avcodec<br/>codecs"]
  node_avfilter["avfilter<br/>filter graph"]
  node_swresample["swresample<br/>audio convert"]
  node_swscale["swscale<br/>pixel convert"]
  node_avdevice["avdevice<br/>devices"]
  node_avutil["avutil<br/>common runtime"]
  node_postproc["postproc"]
end

subgraph group_tools["Frontends"]
  node_fftools["fftools<br/>cli orchestration"]
end

subgraph group_platform["Platform"]
  node_tools_util["tools<br/>helper utilities"]
  node_hwcontext["hwcontext<br/>hw abstractions<br/>[hwcontext.c]"]
  node_cuda_vaapi_qsv["Accel backends<br/>gpu/platform backends"]
  node_arch_opt["Arch backends<br/>cpu optimized code"]
  node_x86_opt["x86<br/>simd backend"]
  node_arm_opt["arm/arm64<br/>simd backend"]
  node_riscv_opt["riscv<br/>simd backend"]
  node_builder["builder<br/>release builds"]
  node_msys2_pkg["msys2<br/>windows packaging"]
end

subgraph group_quality["Validation"]
  node_tests["tests<br/>regression suite"]
end

node_fftools -->|"demux/mux"| node_avformat
node_fftools -->|"encode/decode"| node_avcodec
node_fftools -->|"filter graphs"| node_avfilter
node_fftools -->|"audio convert"| node_swresample
node_fftools -->|"pixel convert"| node_swscale
node_fftools -->|"devices"| node_avdevice
node_fftools -->|"hw accel"| node_hwcontext
node_avformat -->|"compressed streams"| node_avcodec
node_avcodec -->|"frames/samples"| node_avfilter
node_avfilter -->|"audio path"| node_swresample
node_avfilter -->|"video path"| node_swscale
node_hwcontext -->|"decode/encode"| node_avcodec
node_hwcontext -->|"device frames"| node_avfilter
node_hwcontext -->|"hw formats"| node_swscale
node_hwcontext -->|"audio offload"| node_swresample
node_cuda_vaapi_qsv -->|"implements"| node_hwcontext
node_arch_opt -->|"hot paths"| node_avcodec
node_arch_opt -->|"filter kernels"| node_avfilter
node_arch_opt -->|"scale kernels"| node_swscale
node_arch_opt -->|"resample kernels"| node_swresample
node_x86_opt -->|"backend"| node_arch_opt
node_arm_opt -->|"backend"| node_arch_opt
node_riscv_opt -->|"backend"| node_arch_opt
node_builder -->|"packages"| node_msys2_pkg
node_builder -->|"binaries"| node_fftools
node_builder -->|"builds"| node_avformat
node_builder -->|"builds"| node_avcodec
node_builder -->|"builds"| node_avfilter
node_tests -->|"regresses"| node_avcodec
node_tests -->|"regresses"| node_avformat
node_tests -->|"regresses"| node_avfilter
node_tests -->|"regresses"| node_swscale
node_tests -->|"regresses"| node_swresample
node_tests -->|"regresses"| node_hwcontext
node_tests -->|"helpers"| node_tools_util
node_postproc -->|"legacy path"| node_avfilter
node_avutil -->|"base services"| node_avformat
node_avutil -->|"base services"| node_avcodec
node_avutil -->|"base services"| node_avfilter
node_avutil -->|"base services"| node_swresample
node_avutil -->|"base services"| node_swscale
node_avutil -->|"base services"| node_avdevice
node_avutil -->|"base services"| node_hwcontext

click node_fftools "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/fftools"
click node_tools_util "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/tools"
click node_avformat "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavformat"
click node_avcodec "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavcodec"
click node_avfilter "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavfilter"
click node_swresample "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libswresample"
click node_swscale "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libswscale"
click node_avdevice "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavdevice"
click node_avutil "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavutil"
click node_hwcontext "https://github.com/jellyfin/jellyfin-ffmpeg/blob/jellyfin/libavutil/hwcontext.c"
click node_x86_opt "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavcodec/x86"
click node_arm_opt "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavcodec/arm"
click node_riscv_opt "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavcodec/riscv"
click node_postproc "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libpostproc"
click node_builder "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/builder"
click node_msys2_pkg "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/msys2"
click node_tests "https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/tests"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_avformat,node_avcodec,node_avfilter,node_swresample,node_swscale,node_avdevice,node_avutil,node_postproc toneBlue
class node_fftools toneAmber
class node_tools_util,node_hwcontext,node_cuda_vaapi_qsv,node_arch_opt,node_x86_opt,node_arm_opt,node_riscv_opt,node_builder,node_msys2_pkg toneMint
class node_tests toneRose
```

## Components

### Core Media

Reusable FFmpeg libraries that implement the media pipeline.

- [avformat](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavformat): containers I/O · Demuxers, muxers, protocols, timestamps, seeks, and network transports. · `libavformat`
- [avcodec](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavcodec): codecs · Decoders, encoders, parsers, bitstream filters, and codec plumbing. · `libavcodec`
- [avfilter](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavfilter): filter graph · Audio/video filter graphs, sources, sinks, scheduling, and transforms. · `libavfilter`
- [swresample](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libswresample): audio convert · Audio resampling, channel remixing, and sample format conversion. · `libswresample`
- [swscale](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libswscale): pixel convert · Pixel format conversion, colorspace transforms, and scaling. · `libswscale`
- [avdevice](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavdevice): devices · Capture and playback device abstractions for audio/video inputs and outputs. · `libavdevice`
- [avutil](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavutil): common runtime · Shared utilities for memory, logging, options, timestamps, math, threading, and descriptors. · `libavutil`
- [postproc](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libpostproc): legacy postprocess · Standalone post-processing library for older deblocking and cleanup workflows. · `libpostproc`

### Frontends

Command-line orchestration and utility entry points.

- [fftools](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/fftools): cli orchestration · ffmpeg, ffprobe, and ffplay coordinate jobs across the shared libraries. · `fftools`

### Platform

Build, optimization, and hardware-specific backends.

- [tools](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/tools): helper utilities · Standalone developer and analysis helpers used around the stack. · `tools`
- [hwcontext](https://github.com/jellyfin/jellyfin-ffmpeg/blob/jellyfin/libavutil/hwcontext.c): hw abstractions · Device and frame contexts that bridge CPU code with GPU and platform accelerators. · `libavutil/hwcontext.c`
- **Accel backends**: gpu/platform backends · Representative hardware paths such as CUDA, VAAPI, QSV, DXVA2/D3D11VA/D3D12VA, Vulkan, OpenCL, MediaCodec, VideoToolbox, and VDPAU.
- **Arch backends**: cpu optimized code · Architecture-specific SIMD and assembly implementations for hot paths across the stack.
- [x86](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavcodec/x86): simd backend · x86 assembly and intrinsic optimizations. · `libavcodec/x86`
- [arm/arm64](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavcodec/arm): simd backend · ARM and AArch64 optimized implementations. · `libavcodec/arm`
- [riscv](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/libavcodec/riscv): simd backend · RISC-V vector and scalar optimized implementations. · `libavcodec/riscv`
- [builder](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/builder): release builds · Reproducible build, packaging, and cross-platform binary assembly pipeline. · `builder`
- [msys2](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/msys2): windows packaging · MSYS2 packaging scripts and dependency recipes for Windows builds. · `msys2`

### Validation

Regression and test infrastructure.

- [tests](https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/tests): regression suite · API tests, checkasm SIMD checks, FATE regressions, reference outputs, and fuzz targets. · `tests`

## Connections

- fftools → avformat: demux/mux · Command-line jobs open containers and streams through avformat.
- fftools → avcodec: encode/decode · Transcoding and probing rely on codec APIs.
- fftools → avfilter: filter graphs · Complex jobs build and run filter graphs.
- fftools → swresample: audio convert · Audio pipelines invoke resampling and remixing.
- fftools → swscale: pixel convert · Video pipelines invoke scaling and pixel conversion.
- fftools → avdevice: devices · Playback and capture frontends connect to device inputs and outputs.
- fftools → hwcontext: hw accel · Tooling wires hardware frame/device contexts into pipelines.
- avformat → avcodec: compressed streams · Container demuxing feeds encoded elementary streams to codecs.
- avcodec → avfilter: frames/samples · Decoded media enters the filter graph engine.
- avfilter → swresample: audio path · Filters hand audio through format conversion when needed.
- avfilter → swscale: video path · Filters hand video through scaling and pixel conversion when needed.
- hwcontext → avcodec: decode/encode · Hardware frame contexts attach to codec acceleration paths.
- hwcontext → avfilter: device frames · Filters can keep frames on device memory across stages.
- hwcontext → swscale: hw formats · Scaling/conversion can bridge hardware and software pixel formats.
- hwcontext → swresample: audio offload · Shared hardware abstractions underpin broader accelerated processing.
- Accel backends → hwcontext: implements · Concrete GPU and platform APIs are exposed through hwcontext.
- Arch backends → avcodec: hot paths · Codec implementations dispatch into optimized backends.
- Arch backends → avfilter: filter kernels · Some filters use SIMD backends for performance.
- Arch backends → swscale: scale kernels · Scaling and colorspace code uses optimized implementations.
- Arch backends → swresample: resample kernels · Audio conversion relies on architecture-specific code.
- x86 → Arch backends: backend · Representative x86 optimized implementation tree.
- arm/arm64 → Arch backends: backend · Representative ARM/AArch64 optimized implementation tree.
- riscv → Arch backends: backend · Representative RISC-V optimized implementation tree.
- builder → msys2: packages · Build pipeline feeds platform packaging recipes.
- builder → fftools: binaries · Reproducible build scripts produce the command-line tools.
- builder → avformat: builds · Cross-platform build pipeline compiles core libraries.
- builder → avcodec: builds · Cross-platform build pipeline compiles core libraries.
- builder → avfilter: builds · Cross-platform build pipeline compiles core libraries.
- tests → avcodec: regresses · Codec correctness is enforced through FATE and API tests.
- tests → avformat: regresses · Container and protocol behavior is covered by reference tests.
- tests → avfilter: regresses · Filter graphs and pixel/audio transforms are validated by golden outputs.
- tests → swscale: regresses · Scaling and pixel conversion have dedicated checks.
- tests → swresample: regresses · Audio conversion has dedicated checks.
- tests → hwcontext: regresses · Hardware paths are exercised by API and FATE coverage.
- tests → tools: helpers · Developer utilities support test generation and analysis.
- postproc → avfilter: legacy path · Older post-processing remains adjacent to the modern filter stack.
- avutil → avformat: base services · Core utilities underpin container and protocol code.
- avutil → avcodec: base services · Core utilities underpin codec infrastructure.
- avutil → avfilter: base services · Core utilities underpin filter graph execution.
- avutil → swresample: base services · Core utilities underpin audio conversion.
- avutil → swscale: base services · Core utilities underpin scaling and pixel conversion.
- avutil → avdevice: base services · Core utilities underpin device abstractions.
- avutil → hwcontext: base services · Hardware contexts are defined on top of shared utility primitives.
