# nvpro-samples/vk_video_samples architecture

> Architecture diagram of the GitHub repository nvpro-samples/vk_video_samples, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvpro-samples/vk_video_samples
- Repository: https://github.com/nvpro-samples/vk_video_samples

## Overview

##### Main Components
- Common Libraries & Utilities: The “common” directory contains shared headers, helper modules, codec utilities, and Vulkan-related interfaces (e.g., “VkCodecUtils”, “VkShell”, custom layers). These core libraries provide the backbone for both decoding and encoding pipelines.
- Video Decoder Sample: The “vk_video_decoder” folder includes everything necessary for a complete video decode pipeline—from demuxing with FFMPEG, hardware accelerated decoding using Vulkan, post-processing (YCbCr to RGB conversion via VK_KHR_sampler_ycbcr_conversion), and presentation via Vulkan WSI. Note the demos and additional libraries such as “NvVideoParser” and “VkDecoderUtils.”
- Video Encoder Sample: The “vk_video_encoder” folder contains the encoder sample, which uses Vulkan video encoding extensions, and includes components like encoder configurations and state management from “VkCodecUtils” to manage encoding of various formats (h.264/av1/h265).
- Build and Configuration Infrastructure: The CMakeLists.txt at the top level and subsidiary CMake modules in the “cmake” folder govern the build process. The scripts directory provides several code-generation modules that auto-generate dispatch tables, helper files, and layer validation code.
- External Dependencies: These include external libraries such as FFMPEG, NVIDIA drivers with Vulkan Video Extensions, and other third-party dependencies (e.g., ImageMagick references, and platform-specific configurations for Linux/Windows).

##### Component Relationships and Data Flow Interactions
- For the Decoder:
- Input Video Stream/File → FFMPEG Demuxer (inside decoder demos and helpers) → HW Accelerated Vulkan Video Decoder (invoking common library functions) → Frame Processing (YCbCr to RGB conversion using Vulkan graphics pipelines) → Presentation via Vulkan WSI.
- For the Encoder:
- Source Image/Frame Data (from a graphics application or a pre-processed frame) → Video Encoding Pipeline (using VkCodecUtils and encoder configuration modules) → Hardware-accelerated encoding via Vulkan Video Extensions → Output compressed video to file.
- Common Libraries: Both pipelines interact with modules in “common” (such as for buffering, thread pool management, and device context handling) which serve as foundational components in the overall architecture.

##### Architectural Patterns and Design Principles
- Layered Architecture: The system is separated clearly between high-level sample demo applications (decoder/encoder) and lower-level libraries and utilities that handle common functionalities, device resource management, and extended Vulkan functionalities.
- Pipeline/Data Flow Pattern: Data (video frames) flows in a sequential pipeline—starting from demuxing/input to decoding/encoding, then processing (conversion or encapsulation), and finally output (display or file writing).
- Modularity and Extensibility: The design leverages modular components which can be independently compiled and tested. The code-generation scripts further indicate a plug-and-play style for updating dispatch tables and validation layers.
- Cross-Platform Support: Note special build configurations for Linux and Windows and platform-specific wrappers in “VkShell” (e.g., ShellDirect, ShellWayland, ShellWin32).

##### Relevant Technologies and Libraries
- Vulkan API and Extensions (such as VK_KHR_sampler_ycbcr_conversion and Vulkan video extensions)
- NVIDIA Hardware-accelerated decoding/encoding
- FFMPEG for demuxing
- CMake for build and configuration management
- Python-based code generation scripts that help manage dispatch tables, validation, and thread handling utilities

## Diagram (Mermaid)

```mermaid
graph TD
    %% Common Libraries & Utilities
    subgraph "Common Libraries & Utilities"
        comInclude["Headers & Helpers (common/include)"]:::common
        comVkCodec["VkCodecUtils (common/libs/VkCodecUtils)"]:::common
        comVkShell["VkShell (common/libs/VkShell)"]:::common
        commonLib["Common Libraries"]:::common
    end

    %% Video Decoder Sample
    subgraph "Video Decoder Sample"
        vdecoder_Nv["NvVideoParser"]:::decoder
        vdecoder_Vk["VkDecoderUtils"]:::decoder
        vdecoder_demo["Demos"]:::decoder
        vdecoder_tests["Tests"]:::decoder
        videoDecoder["Video Decoder Application"]:::decoder
    end

    %% Video Encoder Sample
    subgraph "Video Encoder Sample"
        vencoder_config["Encoder Config & State"]:::encoder
        vencoder_demo["Demos"]:::encoder
        vencoder_tests["Tests"]:::encoder
        videoEncoder["Video Encoder Application"]:::encoder
    end

    %% Build and Configuration Infrastructure
    subgraph "Build & Configuration"
        cmakeFile["CMakeLists.txt"]:::build
        cmakeModules["CMake Modules (cmake)"]:::build
        scriptsFolder["Code Generation Scripts (scripts)"]:::build
        buildInfra["Build & Config Infrastructure"]:::build
    end

    %% External Dependencies Configuration
    subgraph "External Dependencies"
        ffmpegModule["FFMPEG/FindFFMPEG (cmake/FindFFmpeg.cmake)"]:::external
        nvidiaVulkan["NVIDIA Drivers & Vulkan Extensions"]:::external
        vulkanSDK["Vulkan SDK"]:::external
        extDependencies["External Dependencies"]:::external
    end

    %% Video Decoder Pipeline Flow
    inputStream["Input Video Stream"]:::pipeline
    ffmpegDemuxer["FFMPEG Demuxer"]:::pipeline
    hwDecoder["HW Decoder (Vulkan)"]:::pipeline
    frameProc["Frame Processing (YCbCr->RGB)"]:::pipeline
    presentation["Presentation (Vulkan WSI)"]:::pipeline

    %% Video Encoder Pipeline Flow
    sourceFrame["Source Frame"]:::pipeline
    encPipeline["Encoder Pipeline (VkCodecUtils)"]:::pipeline
    hwEncoder["HW Encoder (Vulkan)"]:::pipeline
    outputVideo["Output Compressed Video"]:::pipeline

    %% Data Flow for Decoder Pipeline
    inputStream -->|"demux"| ffmpegDemuxer
    ffmpegDemuxer -->|"decode"| hwDecoder
    hwDecoder -->|"process"| frameProc
    frameProc -->|"present"| presentation

    %% Data Flow for Encoder Pipeline
    sourceFrame -->|"encode"| encPipeline
    encPipeline -->|"accelerate"| hwEncoder
    hwEncoder -->|"write"| outputVideo

    %% Linking Sample Applications to Pipelines
    videoDecoder -->|"initiates"| inputStream
    presentation -->|"feedback"| videoDecoder
    videoEncoder -->|"supplies"| sourceFrame
    outputVideo -->|"finalizes"| videoEncoder

    %% Common Libraries support pipelines
    commonLib -->|"provides functions"| hwDecoder
    commonLib -->|"supports"| encPipeline

    %% Internal module influences within Video Decoder Sample
    vdecoder_Nv -->|"parses"| ffmpegDemuxer
    vdecoder_Vk -->|"assists"| hwDecoder

    %% Internal module influences within Video Encoder Sample
    vencoder_config -->|"configures"| encPipeline

    %% Build & Configuration integration
    buildInfra -->|"builds"| commonLib
    buildInfra -->|"builds"| videoDecoder
    buildInfra -->|"builds"| videoEncoder

    cmakeModules -->|"configures"| commonLib
    cmakeModules -->|"configures"| videoDecoder
    cmakeModules -->|"configures"| videoEncoder
    scriptsFolder -->|"generates code for"| commonLib
    scriptsFolder -->|"generates code for"| videoDecoder
    scriptsFolder -->|"generates code for"| videoEncoder

    %% External Dependencies connections
    ffmpegModule -->|"configures demuxer"| ffmpegDemuxer
    nvidiaVulkan -->|"accelerates"| hwDecoder
    nvidiaVulkan -->|"accelerates"| hwEncoder
    vulkanSDK -->|"provides API"| hwDecoder
    vulkanSDK -->|"provides API"| hwEncoder

    %% Class Styles
    classDef common fill:#bbf,stroke:#333,stroke-width:2px;
    classDef decoder fill:#cfc,stroke:#333,stroke-width:2px;
    classDef encoder fill:#fcc,stroke:#333,stroke-width:2px;
    classDef build fill:#fcf,stroke:#333,stroke-width:2px;
    classDef external fill:#ff9,stroke:#333,stroke-width:2px;
    classDef pipeline fill:#efe,stroke:#333,stroke-width:2px;

    %% Click Events for Component Mapping
    click commonLib "https://github.com/nvpro-samples/vk_video_samples/tree/main/common"
    click videoDecoder "https://github.com/nvpro-samples/vk_video_samples/tree/main/vk_video_decoder"
    click videoEncoder "https://github.com/nvpro-samples/vk_video_samples/tree/main/vk_video_encoder"
    click cmakeFile "https://github.com/nvpro-samples/vk_video_samples/blob/main/CMakeLists.txt"
    click cmakeModules "https://github.com/nvpro-samples/vk_video_samples/tree/main/cmake"
    click scriptsFolder "https://github.com/nvpro-samples/vk_video_samples/tree/main/scripts"
    click ffmpegModule "https://github.com/nvpro-samples/vk_video_samples/blob/main/cmake/FindFFmpeg.cmake"
```
