# joshdoe/gst-plugins-vision architecture

> Architecture diagram of the GitHub repository joshdoe/gst-plugins-vision, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/joshdoe/gst-plugins-vision
- Repository: https://github.com/joshdoe/gst-plugins-vision

## Overview

This repository is a collection of GStreamer 1.x plugins for machine-vision pipelines. The overall architecture is “many small plugins, one build system”: each plugin is built as its own loadable module and discovered at runtime via `GST_PLUGIN_PATH` rather than being a monolithic app.

The code splits into three main functional areas:

1. **Hardware acquisition plugins (`sys/`)**  
   These are the largest subsystem. Each subdirectory wraps a specific vendor SDK or frame-grabber API and exposes a GStreamer **source** element such as `aptinasrc`, `bitflowsrc`, `gsteuresyssrc`, `gstidsueyesrc`, `gstmatroxsrc`, `gstpylonsrc`, `gstpleorasrc`, and others. The key architectural boundary is that each plugin is isolated around one external camera/frame-grabber ecosystem and depends on that vendor’s C/C++ SDK, license, and driver stack.

2. **Hardware/simulator output plugins (`sys/` and `ext/`)**  
   A smaller set of plugins act as **sinks** or transmitters, not just sources. Examples include `gigesimsink`, `kayasink`, `edtpdvsink`, and `pleorasink`. These are used to feed simulated or SDK-managed outputs into other systems, especially GigE Vision or frame-grabber simulators. This makes the repo useful not only for capture but also for lab/testbed generation.

3. **Generic processing plugins (`gst/`)**  
   These are vendor-neutral elements that operate on image data after capture. Examples include `extractcolor` for channel extraction, `videolevels` for scaling monochrome video to 8-bit with AGC/manual setpoints, `sfx3dnoise` for synthetic noise, and `bayer2gray` for Bayer-pattern handling. These plugins are the main “in-pipeline” image-processing layer between acquisition and downstream consumers.

4. **KLV / MISB metadata path (`gst/klv`, `gst/misb`, `gst-libs/klv`)**  
   This is a distinct metadata subsystem. The repo includes a KLV library and plugins for synchronous KLV handling: injection, inspection, timestamping, and MISB IR packing/unpacking. The README notes KLV is optional and gated by `ENABLE_KLV`, with downstream dependency impact on the Pleora plugin. This is a separate data flow from pixel/video data, but designed to travel alongside it in GStreamer streams.

5. **Image codec / utility plugins (`ext/freeimage`)**  
   This module wraps FreeImage into GStreamer elements for decoding/encoding image formats. It sits at the boundary between file/image codec functionality and the rest of the vision pipeline, typically converting still-image formats into GStreamer buffers or vice versa.

6. **Vendor-independent support code (`common/` and `gst-libs/`)**  
   `common/` holds shared low-level helpers such as pixel-format definitions and a UNIX nanosecond timestamp utility. `gst-libs/klv` is a reusable support library rather than a plugin itself. These shared pieces reduce duplication across the many vendor-specific plugins, especially where buffer metadata or timing semantics are shared.

7. **Build and dependency resolution (`CMakeLists.txt`, `cmake/modules/`)**  
   The repository is heavily CMake-driven and uses many custom `Find*.cmake` modules to detect optional SDKs and libraries. The build is therefore feature-assembled: only the plugins whose vendor SDKs are present will compile. This is an important architectural boundary because the runtime plugin set depends on the build environment, OS, and installed drivers.

8. **Runtime model**  
   At runtime, these plugins are intended to be used inside standard GStreamer pipelines. Typical flow is: hardware source or simulator source → optional processing/metadata plugins → sink or downstream GStreamer consumer. The repository itself does not include an application layer; it supplies the reusable building blocks that other GStreamer-based applications compose.

9. **Platform and SDK coupling**  
   The repo is cross-platform in intent, but many plugins are tightly coupled to Windows-centric vendor SDKs and device drivers, while others can build on Linux. The architecture therefore mixes portable GStreamer plugin logic with highly platform-specific device bindings.

10. **Primary architectural takeaway for a diagram**  
   Draw a central GStreamer plugin layer with three branches: hardware acquisition/simulation, generic image processing, and KLV/MISB metadata handling. Surround that with vendor SDK dependencies as external systems, and show CMake as the build-time selector that includes/excludes plugins based on available SDKs and optional features.

Main entry points: [`sys`](https://github.com/joshdoe/gst-plugins-vision/tree/master/sys) (Camera sources), [`ext/freeimage`](https://github.com/joshdoe/gst-plugins-vision/tree/master/ext/freeimage) (FreeImage codecs), [`gst/klv`](https://github.com/joshdoe/gst-plugins-vision/tree/master/gst/klv) (KLV plugins), [`common`](https://github.com/joshdoe/gst-plugins-vision/tree/master/common) (Common code)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_build["Build system"]
  node_cmake["CMake<br/>build system<br/>[CMakeLists.txt]"]
  node_find_modules["Find modules<br/>dependency probes"]
end

subgraph group_runtime["Plugin runtime"]
  node_plugin_runtime["GStreamer plugins<br/>plugin layer"]
end

subgraph group_acquisition["Acquisition / I/O"]
  node_acq_sources["Camera sources<br/>source plugins"]
  node_acq_sinks["Output sinks<br/>sink plugins"]
  node_gentl["GenTL source<br/>GenTL capture"]
end

subgraph group_processing["Image processing"]
  node_freeimage["FreeImage codecs<br/>codec plugins"]
  node_bayerutils["Bayer utils<br/>pixel utility"]
  node_extractcolor["Extract color<br/>channel extraction"]
  node_videoadjust["Video levels<br/>tone mapping"]
  node_sensorfx["Sensor FX<br/>synthetic effects"]
end

subgraph group_metadata["KLV / MISB"]
  node_klv_plugins["KLV plugins<br/>metadata plugins"]
  node_misb["MISB IR<br/>metadata plugins"]
end

subgraph group_support["Shared support"]
  node_klv_lib["KLV lib<br/>support library"]
  node_common["Common code<br/>shared helpers"]
end

subgraph group_external["External SDKs"]
  node_vendor_sdks[("Vendor SDKs<br/>external dependencies")]
end

node_cmake -->|"configures"| node_find_modules
node_find_modules -.->|"detects"| node_vendor_sdks
node_cmake -->|"builds"| node_plugin_runtime
node_plugin_runtime -->|"includes"| node_acq_sources
node_plugin_runtime -->|"includes"| node_acq_sinks
node_plugin_runtime -->|"includes"| node_gentl
node_plugin_runtime -->|"includes"| node_freeimage
node_plugin_runtime -->|"includes"| node_bayerutils
node_plugin_runtime -->|"includes"| node_extractcolor
node_plugin_runtime -->|"includes"| node_videoadjust
node_plugin_runtime -->|"includes"| node_sensorfx
node_plugin_runtime -->|"includes"| node_klv_plugins
node_plugin_runtime -->|"includes"| node_misb
node_klv_plugins -->|"uses"| node_klv_lib
node_acq_sources -.->|"wraps"| node_vendor_sdks
node_acq_sinks -.->|"targets"| node_vendor_sdks
node_gentl -.->|"depends on"| node_vendor_sdks
node_freeimage -.->|"links to"| node_vendor_sdks
node_bayerutils -->|"uses"| node_common
node_videoadjust -->|"uses"| node_common
node_sensorfx -->|"uses"| node_common
node_klv_plugins -->|"streams with"| node_plugin_runtime
node_acq_sources -->|"feeds"| node_plugin_runtime
node_acq_sinks -->|"consumes"| node_plugin_runtime
node_freeimage -->|"converts"| node_plugin_runtime

click node_cmake "https://github.com/joshdoe/gst-plugins-vision/blob/master/CMakeLists.txt"
click node_find_modules "https://github.com/joshdoe/gst-plugins-vision/tree/master/cmake/modules"
click node_acq_sources "https://github.com/joshdoe/gst-plugins-vision/tree/master/sys"
click node_acq_sinks "https://github.com/joshdoe/gst-plugins-vision/tree/master/sys"
click node_gentl "https://github.com/joshdoe/gst-plugins-vision/tree/master/sys/gentl"
click node_freeimage "https://github.com/joshdoe/gst-plugins-vision/tree/master/ext/freeimage"
click node_bayerutils "https://github.com/joshdoe/gst-plugins-vision/tree/master/gst/bayerutils"
click node_extractcolor "https://github.com/joshdoe/gst-plugins-vision/tree/master/gst/extractcolor"
click node_videoadjust "https://github.com/joshdoe/gst-plugins-vision/tree/master/gst/videoadjust"
click node_sensorfx "https://github.com/joshdoe/gst-plugins-vision/tree/master/gst/sensorfx"
click node_klv_lib "https://github.com/joshdoe/gst-plugins-vision/tree/master/gst-libs/klv"
click node_klv_plugins "https://github.com/joshdoe/gst-plugins-vision/tree/master/gst/klv"
click node_misb "https://github.com/joshdoe/gst-plugins-vision/tree/master/gst/misb"
click node_common "https://github.com/joshdoe/gst-plugins-vision/tree/master/common"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cmake,node_find_modules,node_vendor_sdks toneBlue
class node_plugin_runtime toneAmber
class node_acq_sources,node_acq_sinks,node_gentl toneMint
class node_freeimage,node_bayerutils,node_extractcolor,node_videoadjust,node_sensorfx toneRose
class node_klv_plugins,node_misb toneIndigo
class node_klv_lib,node_common toneTeal
```

## Components

### Build system

CMake-based feature selection and module discovery

- [CMake](https://github.com/joshdoe/gst-plugins-vision/blob/master/CMakeLists.txt): build system · Top-level CMake build that assembles available plugins · `CMakeLists.txt`
- [Find modules](https://github.com/joshdoe/gst-plugins-vision/tree/master/cmake/modules): dependency probes · Custom CMake checks for GStreamer, image libs, and vendor SDKs · `cmake/modules`

### Plugin runtime

GStreamer plugin layer composed of loadable modules

- **GStreamer plugins**: plugin layer · Loadable modules discovered at runtime through the GStreamer plugin path

### Acquisition / I/O

Vendor and simulator source/sink plugins for camera and frame-grabber ecosystems

- [Camera sources](https://github.com/joshdoe/gst-plugins-vision/tree/master/sys): source plugins · Vendor-specific capture elements wrapping camera and frame-grabber APIs · `sys`
- [Output sinks](https://github.com/joshdoe/gst-plugins-vision/tree/master/sys): sink plugins · Simulator and SDK-managed output elements for testbeds and transmit paths · `sys`
- [GenTL source](https://github.com/joshdoe/gst-plugins-vision/tree/master/sys/gentl): GenTL capture · Generic GenTL-based acquisition plugin and bundled support code · `sys/gentl`

### Image processing

Vendor-neutral in-pipeline image transforms

- [FreeImage codecs](https://github.com/joshdoe/gst-plugins-vision/tree/master/ext/freeimage): codec plugins · Image decode/encode wrapper around the FreeImage library · `ext/freeimage`
- [Bayer utils](https://github.com/joshdoe/gst-plugins-vision/tree/master/gst/bayerutils): pixel utility · Bayer-pattern handling and gray conversion utilities · `gst/bayerutils`
- [Extract color](https://github.com/joshdoe/gst-plugins-vision/tree/master/gst/extractcolor): channel extraction · Channel and component extraction from image buffers · `gst/extractcolor`
- [Video levels](https://github.com/joshdoe/gst-plugins-vision/tree/master/gst/videoadjust): tone mapping · Monochrome scaling and AGC/manual level adjustment · `gst/videoadjust`
- [Sensor FX](https://github.com/joshdoe/gst-plugins-vision/tree/master/gst/sensorfx): synthetic effects · Noise and blur generation for synthetic or test pipelines · `gst/sensorfx`

### KLV / MISB

Synchronous metadata and IR payload handling

- [KLV plugins](https://github.com/joshdoe/gst-plugins-vision/tree/master/gst/klv): metadata plugins · KLV injection, inspection, and timestamping elements · `gst/klv`
- [MISB IR](https://github.com/joshdoe/gst-plugins-vision/tree/master/gst/misb): metadata plugins · MISB infrared packing and unpacking elements · `gst/misb`

### Shared support

Reusable helpers and library code

- [KLV lib](https://github.com/joshdoe/gst-plugins-vision/tree/master/gst-libs/klv): support library · Reusable KLV library used by metadata plugins · `gst-libs/klv`
- [Common code](https://github.com/joshdoe/gst-plugins-vision/tree/master/common): shared helpers · Shared pixel-format and timestamp utilities · `common`

### External SDKs

Optional vendor SDKs, drivers, and libraries detected at build time

- **Vendor SDKs**: external dependencies · Optional camera, frame-grabber, and imaging libraries detected by CMake

## Connections

- CMake → Find modules: configures · Top-level build delegates feature detection to custom find modules
- Find modules → Vendor SDKs: detects · Probes for vendor SDKs, drivers, and third-party libraries
- CMake → GStreamer plugins: builds · Compiles only the plugin modules enabled by the environment
- GStreamer plugins → Camera sources: includes · Runtime plugin layer exposes camera acquisition elements
- GStreamer plugins → Output sinks: includes · Runtime plugin layer exposes simulator/output elements
- GStreamer plugins → GenTL source: includes · Generic GenTL acquisition module is part of the plugin set
- GStreamer plugins → FreeImage codecs: includes · Image codec boundary for still-image conversion
- GStreamer plugins → Bayer utils: includes · Vendor-neutral Bayer conversion utilities
- GStreamer plugins → Extract color: includes · Channel extraction element used inside pipelines
- GStreamer plugins → Video levels: includes · Video scaling and level adjustment element
- GStreamer plugins → Sensor FX: includes · Synthetic noise and blur effects for test pipelines
- GStreamer plugins → KLV plugins: includes · Synchronous metadata handling elements
- GStreamer plugins → MISB IR: includes · MISB-specific metadata transforms
- KLV plugins → KLV lib: uses · Metadata plugins share a reusable KLV support library
- Camera sources → Vendor SDKs: wraps · Each source element binds to a specific vendor ecosystem
- Output sinks → Vendor SDKs: targets · Sink plugins integrate with simulator or device SDKs
- GenTL source → Vendor SDKs: depends on · GenTL capture path relies on external transport-layer support
- FreeImage codecs → Vendor SDKs: links to · Codec module depends on the FreeImage library
- Bayer utils → Common code: uses · Shares pixel-format definitions and low-level helpers
- Video levels → Common code: uses · Shares numeric/timing support where needed
- Sensor FX → Common code: uses · Synthetic effects share common utility code
- KLV plugins → GStreamer plugins: streams with · Metadata travels alongside video buffers in GStreamer pipelines
- Camera sources → GStreamer plugins: feeds · Capture elements are upstream entry points into the pipeline
- Output sinks → GStreamer plugins: consumes · Output elements terminate or hand off pipeline data
- FreeImage codecs → GStreamer plugins: converts · Codec elements bridge file/image formats and pipeline buffers