How sysprog21/vpipe works
1. What this repo is
- vpipe is a Linux-only V4L2 mem2mem prototype, not a vision stack.
- Its goal is to measure frame-path costs: copies, queueing, context switches, scheduler jitter, cache behavior, and a small deterministic kernel-side image transform.
2. Top-level architecture
- The repo splits into three main areas:
- kmod/ — kernel module implementation
- user/ — userspace capture, benchmark, validation, and reference tools
- scripts/, tests/, docs/ — automation, fixtures, and design/benchmark documentation
- The architecture is intentionally narrow: single-plane V4L2_PIX_FMT_GREY, 640×480, one mem2mem node, and one metadata sideband device.
3. Kernel transport path
- kmod/vpipe-m2m.c implements the V4L2 mem2mem device with OUTPUT and CAPTURE queues.
- Frames arrive on OUTPUT via either V4L2_MEMORY_DMABUF or V4L2_MEMORY_MMAP, are processed in device_run(), then emitted on CAPTURE.
- This is the core transport boundary the repo is measuring.
4. Kernel image-processing path
- kmod/vpipe-cv.c holds the “Tiny CV” logic: a bounded threshold operation over a clamped ROI.
- The algorithm is deliberately simple and deterministic, with no floating point and no hot-path allocation.
- This keeps algorithm cost small and auditable so transport effects remain visible.
5. Kernel metadata sideband
- kmod/vpipe-meta.c exposes /dev/vpipe-meta as a miscdevice carrying per-frame metadata.
- Metadata is stored in a shared ring buffer with per-open reader cursors, so multiple readers can drain it independently.
- This sideband avoids overloading the pixel payload and lets timing/state be correlated frame-by-frame.
6. Shared UAPI boundary
- kmod/vpipe.h defines the user/kernel contract: ioctls, control IDs, and metadata layout.
- Per-buffer controls such as source sequence, algorithm selection, threshold, and ROI are snapshotted at QBUF time.
- That design prevents control races for frames already in flight.
7. Userspace orchestration
- The user/ programs drive the scenarios rather than implement the kernel logic.
- Key binaries include:
- vpipe-capture-* for baseline capture modes (read, mmap, dmabuf, m2m)
- vpipe-bench-fixture for deterministic fixture transport runs
- vpipe-meta-drain to collect metadata CSV
- vpipe-cv-ref and vpipe-pgm-diff for reference comparison
- Userspace owns sequence correlation, CSV logging, and artifact capture.
8. Fixture-driven validation path
- The deterministic path uses fixture PGM files from tests/fixtures/.
- vpipe-fixture-feed and the benchmark scripts load fixture bytes into dma-heap memory, then transport them via DMABUF into vpipe.
- That intentionally includes one explicit userspace memcpy() so copy counts stay measurable and there are no accidental zero-copy claims.
9. Measurement and artifact flow
- Benchmark scripts in scripts/ run capture loops, perf collection, tracing, and summary generation.
- Outputs go into a gitignored bench/ directory:
- per-frame CSV logs
- metadata CSV sidecars
- perf CSVs
- input/reference/kernel/diff PGM sets for visual inspection
- The main validation loop is: fixture → userspace reference → kernel output → diff.
10. Validation and CI boundary
- make builds both kernel and userspace; sudo make check runs the validation suite.
- The suite covers unit tests, vivid enumeration, module load, metadata sanity, capture baselines, and full Tiny CV fixture checks.
- GitHub Actions and the .ci/ scripts enforce formatting, security, newline, and cppcheck gates.
11. External runtime dependencies that matter architecturally
- The repo depends on Linux V4L2, vivid for virtual camera input, dma-heap for deterministic buffer allocation, and perf for measurement.
- The documented reference environment is an Ubuntu lima guest, which matters because some kernel features and PMU counters are absent there.
- Phase 5 explicit-sync validation is deliberately gated on V4L2 UAPI availability.
12. How to draw the system
- Draw a userspace layer that prepares sources, drives V4L2 ioctls, drains metadata, and writes benchmark artifacts.
- Draw a kernel layer with three distinct parts: mem2mem transport, tiny CV processing, and metadata ring buffer.
- Show two distinct data planes: pixel buffers through OUTPUT→processing→CAPTURE, and metadata through /dev/vpipe-meta.
- Highlight the explicit boundary between measurement/orchestration in userspace and deterministic frame handling in the kernel.
Main entry points: vpipe-m2m.c M2M device · vpipe-bench-fixture.c Bench driver · bench_vpipe.sh Bench scripts · validate-tinycv.sh Validation suite
Components 19
Userspace
Programs that orchestrate V4L2 flows, reference comparison, and artifact capture.
- Shared API · Userspace-side common types, ioctls, and buffer/control helpers.
user/vpipe-common.h - Capture modes · Baseline capture programs for read, mmap, dmabuf, userptr, and m2m paths.
user/vpipe-capture-read.c - Fixture feed · Loads deterministic PGM bytes into dma-heap memory and feeds vpipe via DMABUF.
user/vpipe-fixture-feed.c - Bench driver · Runs deterministic transport scenarios and records per-frame results.
user/vpipe-bench-fixture.c - Meta drain · Reads /dev/vpipe-meta and writes per-frame metadata CSV.
user/vpipe-meta-drain.c - CV reference · Reference Tiny CV implementation used to compare kernel output.
user/vpipe-cv-ref.c - PGM diff · Compares input, reference, kernel, and diff images for validation.
user/vpipe-pgm-diff.c
Kernel
V4L2 mem2mem device, Tiny CV transform, and metadata sideband.
- M2M device · Kernel mem2mem transport with OUTPUT and CAPTURE queues.
kmod/vpipe-m2m.c - Tiny CV · Deterministic thresholding over a clamped ROI with no floating point.
kmod/vpipe-cv.c - Meta ring · Per-frame metadata sideband exposed as /dev/vpipe-meta with reader cursors.
kmod/vpipe-meta.c - Kernel UAPI · Shared ioctl, control, and metadata layout definitions for user and kernel.
kmod/vpipe.h - Internal state · Private kernel definitions shared by the transport, CV, and metadata code.
kmod/vpipe-internal.h
Validation
Fixtures, scripts, docs, and CI gates that exercise and measure the system.
- PGM fixtures · Deterministic grayscale source images for transport and Tiny CV validation.
tests/fixtures/edge-block.pgm - Bench scripts · Shell scripts that run capture loops, perf tracing, and benchmark collection.
scripts/bench_vpipe.sh - Trace tools · Helpers for V4L2 tracing, worst-case tracing, and vivid setup.
scripts/trace_v4l2.sh - Summarizer · Post-processes benchmark outputs into summary artifacts.
scripts/summarize-benchmark.py - Validation suite · Tiny CV and integration checks that verify module load, metadata, and capture behavior.
tests/validate-tinycv.sh - Design docs · Architecture and benchmark notes explaining the transport and measurement model.
docs/design.md - CI gates · Formatting, security, newline, and cppcheck enforcement in CI and local scripts.
.github/workflows/main.yml
Connections 21
- Capture modes → M2M device: V4L2 ioctls · Userspace capture tools drive the mem2mem device through V4L2.
- Fixture feed → M2M device: DMABUF input · Fixture bytes are loaded into dma-heap memory and queued to OUTPUT.
- Bench driver → Fixture feed: runs · Benchmark harness invokes deterministic fixture transport scenarios.
- Bench driver → Meta drain: drains · Bench runs collect metadata sideband output as CSV.
- CV reference → PGM diff: compares via · Reference output feeds the image diff tool for validation.
- Shared API → Kernel UAPI: shared contract · Userspace and kernel share ioctl and control definitions.
- M2M device → Tiny CV: device_run · Queued frames are processed by the Tiny CV kernel path.
- M2M device → Meta ring: frame metadata · Per-frame state is emitted to the metadata sideband.
- M2M device → Kernel UAPI: controls · The transport honors snapshotted per-buffer controls at QBUF time.
- Tiny CV → Internal state: uses · The transform uses private kernel state and helpers.
- Meta ring → Internal state: ring state · Metadata storage uses internal ring-buffer bookkeeping.
- Meta drain → Meta ring: reads /dev/vpipe-meta · Userspace drains metadata from the miscdevice.
- PGM fixtures → Fixture feed: source data · Fixture images seed deterministic transport runs.
- Bench scripts → Capture modes: orchestrates · Shell automation runs capture modes and records artifacts.
- Bench scripts → Bench driver: orchestrates · Benchmark scripts drive fixture-based runs.
- Trace tools → M2M device: traces · Tracing scripts observe V4L2 queueing and transport behavior.
- Summarizer → Bench scripts: summarizes · Benchmark summaries are derived from collected outputs.
- Validation suite → CV reference: validates · Validation compares kernel output with the reference path.
- Validation suite → M2M device: loads module · Integration checks exercise module load and capture paths.
- Design docs → M2M device: describes · Docs explain the transport boundary and measurement goals.
- CI gates → Validation suite: gates · CI enforces the repository validation suite and style checks.