How jellyfin/jellyfin-ffmpeg works
1. This repository is the FFmpeg multimedia stack: a set of native C libraries plus command-line tools for ingesting, decoding, transforming, and emitting audio/video/subtitle data. The architecture is organized around reusable libraries with thin tool front-ends on top.
2. The core processing pipeline is: input demux/protocol layer (libavformat) reads bytes from files, pipes, network streams, or capture devices; codec layer (libavcodec) decodes or encodes elementary streams; filtering (libavfilter) transforms decoded frames and samples; resampling/scaling (libswresample, libswscale) handle format conversion; output mux/protocol layer (libavformat) packages the results back into containers or streams.
3. libavformat is the I/O and container boundary. It owns demuxers, muxers, protocol handlers, timestamp/seek logic, and network transports such as HTTP, RTP/RTSP, HLS/DASH, TCP/UDP/TLS, plus many container-specific readers/writers. This is the main boundary between FFmpeg and the outside world.
4. libavcodec is the largest subsystem and contains the codec implementations themselves. It includes decoders, encoders, parsers, bitstream filters, and hardware-accelerated codec paths. The directory structure shows a strong split between codec families, shared codec infrastructure, and many architecture-specific optimized backends.
5. Hardware acceleration is a first-class architectural concern, not an add-on. libavutil/hwcontext.* defines device/frame abstractions that connect codecs and filters to GPU or platform APIs such as CUDA, VAAPI, QSV, DXVA2/D3D11VA/D3D12VA, VideoToolbox, Vulkan, OpenCL, MediaCodec, and VDPAU. In the data flow, these paths let frames stay on device memory across decode/filter/encode stages when supported.
6. libavfilter is the graph engine for frame and sample processing. It models audio/video transformations as connected filters, with source/sink nodes, graph parsing, frame synchronization, and many CPU/GPU-specific filters. This is the subsystem that sits between decode and encode when the pipeline is doing anything more than pass-through.
7. libswresample and libswscale are focused conversion subsystems. libswresample handles audio sample format changes, channel remixing, and resampling; libswscale handles pixel format conversion, colorspace transforms, and resizing. In diagrams, these are the “format adaptation” blocks that often bridge incompatible decoder/filter/encoder requirements.
8. libavdevice is the capture/playback edge of the system. It abstracts platform media devices such as ALSA, PulseAudio, DirectShow, AVFoundation, DeckLink, V4L2, X11 grabbers, OpenGL/SDL outputs, and similar device interfaces. It feeds into the same format/codec pipeline as file and network inputs, but with device-specific acquisition and rendering boundaries.
9. The command-line tools in fftools are orchestration layers over the libraries rather than independent pipelines. ffmpeg drives transcoding and streaming jobs, ffprobe performs inspection and metadata analysis, and ffplay provides playback/rendering. They coordinate demuxing, decoding, filtering, encoding, muxing, threading, and device output via the shared libraries.
10. libavutil underpins all higher layers with common services: memory and buffers, logging, dictionaries, option parsing, time/timestamps, math, pixel/sample descriptors, crypto/hash utilities, CPU feature detection, threading primitives, and hardware context support. It is the base dependency that keeps the rest of the stack portable and modular.
11. Performance is heavily shaped by architecture-specific optimized code. Many directories are split by CPU family (x86, arm, aarch64, mips, ppc, riscv, loongarch, etc.) and use handwritten assembly or intrinsics for hot paths in codecs, scaling, resampling, and filtering. The diagram should treat these as backend implementations behind generic APIs, not separate features.
12. The repository also includes a substantial testing and validation layer. tests/ contains API tests, sample generators, FATE regression definitions, checkasm microbenchmarks for SIMD routines, and golden reference outputs. This is important architecturally because correctness is enforced across many codecs, containers, filters, and platforms.
13. Build and packaging are major cross-cutting concerns. The top-level build system uses Makefiles and configure scripts, with Meson/cross-toolchain artifacts, Docker and CI workflows, Debian packaging, and Windows/MSYS2 build scripts. The builder/ tree is a dedicated release/build pipeline for reproducible multi-platform binaries and dependency bundling.
14. Overall, the repo is best diagrammed as layered media infrastructure: external inputs/devices/protocols at the edge; format/demux and codec cores in the middle; filter and conversion subsystems as the processing graph; and mux/protocol/device outputs on the other side, all glued together by shared utilities, hardware-context abstractions, platform optimizations, and extensive regression testing.
Main entry points: libavfilter avfilter · fftools fftools · hwcontext.c hwcontext · libavutil avutil
Components 19
Core Media
Reusable FFmpeg libraries that implement the media pipeline.
- avformat · Demuxers, muxers, protocols, timestamps, seeks, and network transports.
libavformat - avcodec · Decoders, encoders, parsers, bitstream filters, and codec plumbing.
libavcodec - avfilter · Audio/video filter graphs, sources, sinks, scheduling, and transforms.
libavfilter - swresample · Audio resampling, channel remixing, and sample format conversion.
libswresample - swscale · Pixel format conversion, colorspace transforms, and scaling.
libswscale - avdevice · Capture and playback device abstractions for audio/video inputs and outputs.
libavdevice - avutil · Shared utilities for memory, logging, options, timestamps, math, threading, and descriptors.
libavutil - postproc · Standalone post-processing library for older deblocking and cleanup workflows.
libpostproc
Frontends
Command-line orchestration and utility entry points.
- fftools · ffmpeg, ffprobe, and ffplay coordinate jobs across the shared libraries.
fftools
Platform
Build, optimization, and hardware-specific backends.
- tools · Standalone developer and analysis helpers used around the stack.
tools - hwcontext · Device and frame contexts that bridge CPU code with GPU and platform accelerators.
libavutil/hwcontext.c - Accel backends · Representative hardware paths such as CUDA, VAAPI, QSV, DXVA2/D3D11VA/D3D12VA, Vulkan, OpenCL, MediaCodec, VideoToolbox, and VDPAU.
- Arch backends · Architecture-specific SIMD and assembly implementations for hot paths across the stack.
- x86 · x86 assembly and intrinsic optimizations.
libavcodec/x86 - arm/arm64 · ARM and AArch64 optimized implementations.
libavcodec/arm - riscv · RISC-V vector and scalar optimized implementations.
libavcodec/riscv - builder · Reproducible build, packaging, and cross-platform binary assembly pipeline.
builder - msys2 · MSYS2 packaging scripts and dependency recipes for Windows builds.
msys2
Validation
Regression and test infrastructure.
- tests · API tests, checkasm SIMD checks, FATE regressions, reference outputs, and fuzz targets.
tests
Connections 43
- fftools → avformat: demux/mux · Command-line jobs open containers and streams through avformat.
- fftools → avcodec: encode/decode · Transcoding and probing rely on codec APIs.
- fftools → avfilter: filter graphs · Complex jobs build and run filter graphs.
- fftools → swresample: audio convert · Audio pipelines invoke resampling and remixing.
- fftools → swscale: pixel convert · Video pipelines invoke scaling and pixel conversion.
- fftools → avdevice: devices · Playback and capture frontends connect to device inputs and outputs.
- fftools → hwcontext: hw accel · Tooling wires hardware frame/device contexts into pipelines.
- avformat → avcodec: compressed streams · Container demuxing feeds encoded elementary streams to codecs.
- avcodec → avfilter: frames/samples · Decoded media enters the filter graph engine.
- avfilter → swresample: audio path · Filters hand audio through format conversion when needed.
- avfilter → swscale: video path · Filters hand video through scaling and pixel conversion when needed.
- hwcontext → avcodec: decode/encode · Hardware frame contexts attach to codec acceleration paths.
- hwcontext → avfilter: device frames · Filters can keep frames on device memory across stages.
- hwcontext → swscale: hw formats · Scaling/conversion can bridge hardware and software pixel formats.
- hwcontext → swresample: audio offload · Shared hardware abstractions underpin broader accelerated processing.
- Accel backends → hwcontext: implements · Concrete GPU and platform APIs are exposed through hwcontext.
- Arch backends → avcodec: hot paths · Codec implementations dispatch into optimized backends.
- Arch backends → avfilter: filter kernels · Some filters use SIMD backends for performance.
- Arch backends → swscale: scale kernels · Scaling and colorspace code uses optimized implementations.
- Arch backends → swresample: resample kernels · Audio conversion relies on architecture-specific code.
- x86 → Arch backends: backend · Representative x86 optimized implementation tree.
- arm/arm64 → Arch backends: backend · Representative ARM/AArch64 optimized implementation tree.
- riscv → Arch backends: backend · Representative RISC-V optimized implementation tree.
- builder → msys2: packages · Build pipeline feeds platform packaging recipes.
- builder → fftools: binaries · Reproducible build scripts produce the command-line tools.
- builder → avformat: builds · Cross-platform build pipeline compiles core libraries.
- builder → avcodec: builds · Cross-platform build pipeline compiles core libraries.
- builder → avfilter: builds · Cross-platform build pipeline compiles core libraries.
- tests → avcodec: regresses · Codec correctness is enforced through FATE and API tests.
- tests → avformat: regresses · Container and protocol behavior is covered by reference tests.
- tests → avfilter: regresses · Filter graphs and pixel/audio transforms are validated by golden outputs.
- tests → swscale: regresses · Scaling and pixel conversion have dedicated checks.
- tests → swresample: regresses · Audio conversion has dedicated checks.
- tests → hwcontext: regresses · Hardware paths are exercised by API and FATE coverage.
- tests → tools: helpers · Developer utilities support test generation and analysis.
- postproc → avfilter: legacy path · Older post-processing remains adjacent to the modern filter stack.
- avutil → avformat: base services · Core utilities underpin container and protocol code.
- avutil → avcodec: base services · Core utilities underpin codec infrastructure.
- avutil → avfilter: base services · Core utilities underpin filter graph execution.
- avutil → swresample: base services · Core utilities underpin audio conversion.
- avutil → swscale: base services · Core utilities underpin scaling and pixel conversion.
- avutil → avdevice: base services · Core utilities underpin device abstractions.
- avutil → hwcontext: base services · Hardware contexts are defined on top of shared utility primitives.