How nvpro-samples/vk_video_samples works
Main Components
• Common Libraries & Utilities: The “common” directory contains shared headers, helper modules, codec utilities, and Vulkan-related interfaces (e.g., “VkCodecUtils”, “VkShell”, custom layers). These core libraries provide the backbone for both decoding and encoding pipelines.
• Video Decoder Sample: The “vk_video_decoder” folder includes everything necessary for a complete video decode pipeline—from demuxing with FFMPEG, hardware accelerated decoding using Vulkan, post-processing (YCbCr to RGB conversion via VK_KHR_sampler_ycbcr_conversion), and presentation via Vulkan WSI. Note the demos and additional libraries such as “NvVideoParser” and “VkDecoderUtils.”
• Video Encoder Sample: The “vk_video_encoder” folder contains the encoder sample, which uses Vulkan video encoding extensions, and includes components like encoder configurations and state management from “VkCodecUtils” to manage encoding of various formats (h.264/av1/h265).
• Build and Configuration Infrastructure: The CMakeLists.txt at the top level and subsidiary CMake modules in the “cmake” folder govern the build process. The scripts directory provides several code-generation modules that auto-generate dispatch tables, helper files, and layer validation code.
• External Dependencies: These include external libraries such as FFMPEG, NVIDIA drivers with Vulkan Video Extensions, and other third-party dependencies (e.g., ImageMagick references, and platform-specific configurations for Linux/Windows).
Component Relationships and Data Flow Interactions
• For the Decoder:
• Input Video Stream/File → FFMPEG Demuxer (inside decoder demos and helpers) → HW Accelerated Vulkan Video Decoder (invoking common library functions) → Frame Processing (YCbCr to RGB conversion using Vulkan graphics pipelines) → Presentation via Vulkan WSI.
• For the Encoder:
• Source Image/Frame Data (from a graphics application or a pre-processed frame) → Video Encoding Pipeline (using VkCodecUtils and encoder configuration modules) → Hardware-accelerated encoding via Vulkan Video Extensions → Output compressed video to file.
• Common Libraries: Both pipelines interact with modules in “common” (such as for buffering, thread pool management, and device context handling) which serve as foundational components in the overall architecture.
Architectural Patterns and Design Principles
• Layered Architecture: The system is separated clearly between high-level sample demo applications (decoder/encoder) and lower-level libraries and utilities that handle common functionalities, device resource management, and extended Vulkan functionalities.
• Pipeline/Data Flow Pattern: Data (video frames) flows in a sequential pipeline—starting from demuxing/input to decoding/encoding, then processing (conversion or encapsulation), and finally output (display or file writing).
• Modularity and Extensibility: The design leverages modular components which can be independently compiled and tested. The code-generation scripts further indicate a plug-and-play style for updating dispatch tables and validation layers.
• Cross-Platform Support: Note special build configurations for Linux and Windows and platform-specific wrappers in “VkShell” (e.g., ShellDirect, ShellWayland, ShellWin32).
Relevant Technologies and Libraries
• Vulkan API and Extensions (such as VK_KHR_sampler_ycbcr_conversion and Vulkan video extensions)
• NVIDIA Hardware-accelerated decoding/encoding
• FFMPEG for demuxing
• CMake for build and configuration management
• Python-based code generation scripts that help manage dispatch tables, validation, and thread handling utilities