joshdoe/gst-plugins-vision

This repository is a collection of GStreamer 1.x plugins for machine-vision pipelines. The overall architecture is “many small plugins, one build system”: each plugin is built as its own loadable module and discovered at runtime via GST_PLUGIN_PATH rather than being a monolithic app.

The code splits into three main functional areas:

1. Hardware acquisition plugins (sys/)

These are the largest subsystem. Each subdirectory wraps a specific vendor SDK or frame-grabber API and exposes a GStreamer source element such as aptinasrc, bitflowsrc, gsteuresyssrc, gstidsueyesrc, gstmatroxsrc, gstpylonsrc, gstpleorasrc, and others. The key architectural boundary is that each plugin is isolated around one external camera/frame-grabber ecosystem and depends on that vendor’s C/C++ SDK, license, and driver stack.

2. Hardware/simulator output plugins (sys/ and ext/)

A smaller set of plugins act as sinks or transmitters, not just sources. Examples include gigesimsink, kayasink, edtpdvsink, and pleorasink. These are used to feed simulated or SDK-managed outputs into other systems, especially GigE Vision or frame-grabber simulators. This makes the repo useful not only for capture but also for lab/testbed generation.

3. Generic processing plugins (gst/)

These are vendor-neutral elements that operate on image data after capture. Examples include extractcolor for channel extraction, videolevels for scaling monochrome video to 8-bit with AGC/manual setpoints, sfx3dnoise for synthetic noise, and bayer2gray for Bayer-pattern handling. These plugins are the main “in-pipeline” image-processing layer between acquisition and downstream consumers.

4. KLV / MISB metadata path (gst/klv, gst/misb, gst-libs/klv)

This is a distinct metadata subsystem. The repo includes a KLV library and plugins for synchronous KLV handling: injection, inspection, timestamping, and MISB IR packing/unpacking. The README notes KLV is optional and gated by ENABLE_KLV, with downstream dependency impact on the Pleora plugin. This is a separate data flow from pixel/video data, but designed to travel alongside it in GStreamer streams.

5. Image codec / utility plugins (ext/freeimage)

This module wraps FreeImage into GStreamer elements for decoding/encoding image formats. It sits at the boundary between file/image codec functionality and the rest of the vision pipeline, typically converting still-image formats into GStreamer buffers or vice versa.

6. Vendor-independent support code (common/ and gst-libs/)

common/ holds shared low-level helpers such as pixel-format definitions and a UNIX nanosecond timestamp utility. gst-libs/klv is a reusable support library rather than a plugin itself. These shared pieces reduce duplication across the many vendor-specific plugins, especially where buffer metadata or timing semantics are shared.

7. Build and dependency resolution (CMakeLists.txt, cmake/modules/)

The repository is heavily CMake-driven and uses many custom Find*.cmake modules to detect optional SDKs and libraries. The build is therefore feature-assembled: only the plugins whose vendor SDKs are present will compile. This is an important architectural boundary because the runtime plugin set depends on the build environment, OS, and installed drivers.

8. Runtime model

At runtime, these plugins are intended to be used inside standard GStreamer pipelines. Typical flow is: hardware source or simulator source → optional processing/metadata plugins → sink or downstream GStreamer consumer. The repository itself does not include an application layer; it supplies the reusable building blocks that other GStreamer-based applications compose.

9. Platform and SDK coupling

The repo is cross-platform in intent, but many plugins are tightly coupled to Windows-centric vendor SDKs and device drivers, while others can build on Linux. The architecture therefore mixes portable GStreamer plugin logic with highly platform-specific device bindings.

10. Primary architectural takeaway for a diagram

Draw a central GStreamer plugin layer with three branches: hardware acquisition/simulation, generic image processing, and KLV/MISB metadata handling. Surround that with vendor SDK dependencies as external systems, and show CMake as the build-time selector that includes/excludes plugins based on available SDKs and optional features.