# lukasbommes/mv-extractor architecture

> Architecture diagram of the GitHub repository lukasbommes/mv-extractor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lukasbommes/mv-extractor
- Repository: https://github.com/lukasbommes/mv-extractor

## Overview

This is an **offline/streaming video processing tool + Python package** that exposes an **OpenCV-VideoCapture-like API** to extract, per frame:
- frame type (I/P/B)
- motion vectors (from codec/bitstream metadata via FFmpeg)
- optionally the decoded frame (BGR) for preview / dumping

It’s not a distributed “system” in the classic service sense; the “architecture” is primarily:
- **Python user-facing API + CLI**
- backed by a **native C++ core** (FFmpeg/OpenCV integration)
- packaged as a Python distribution (wheels) and also runnable via Docker
- verified by unit + end-to-end tests (with reference outputs, including RTSP)

##### User-facing entrypoints
- **CLI script**: `extract_mvs.py` (invoked as `extract_mvs ...` per README)
- **Python package entry**: `src/mvextractor/__main__.py` (module execution) and `__init__.py`

##### Native core (where “the work” happens)
Located in `src/mvextractor/`:
- `video_cap.cpp` / `video_cap.hpp`: main C++ implementation (capture, grab/retrieve/read, motion vector extraction, optional decode)
- `py_video_cap.cpp`: Python bindings layer (likely pybind11 or Python C-API style; exact mechanism isn’t shown in tree but this file is the bridge)
- `mat_to_ndarray.cpp/.hpp`: conversion between OpenCV `cv::Mat` and NumPy arrays
- `pycompat.hpp`: compatibility helpers for Python versions

##### Packaging / build & distribution
- `pyproject.toml`, `setup.py`, `MANIFEST.in`: Python build configuration (build native extension, publish wheels)
- `Dockerfile`, `run.sh`, `dockerhub.md`: container build/runtime environment with FFmpeg/OpenCV preinstalled
- `.github/workflows/ci.yml`: CI pipeline (build/test)

##### Tests & reference artifacts
- `tests/unit_tests.py`
- `tests/end_to_end_tests.py`
- `tests/reference/*`: golden motion-vector outputs (`.npy`), frame types, etc for H264 / MPEG4 Part2 / RTSP
- `tests/tools/live555MediaServer`: used to stand up an RTSP server during tests
- sample videos at repo root: `vid_h264.mp4`, `vid_mpeg4_part2.mp4`, etc.

##### User / Client Code
- “Python script using `mvextractor.VideoCap`”
- “CLI user running `extract_mvs`”

##### Python Layer (mvextractor package)
- `mvextractor.VideoCap` (Python-facing class)
- CLI/preview/dump logic (`extract_mvs.py`)

##### Native Extension Module
- “C++ VideoCap core” (`video_cap.cpp/.hpp`)
- “Python binding” (`py_video_cap.cpp`)
- “cv::Mat ↔ numpy” (`mat_to_ndarray.*`)

##### External Native Dependencies
- **FFmpeg libs** (demux/decode, access to `AVMotionVector`)
- **OpenCV** (VideoCapture-like patterns; frame decode to BGR; `cv::Mat`)
- OS / drivers as needed for file I/O / networking

##### Input Sources
- “Video file (H.264 / MPEG4 Part2)” (filesystem)
- “RTSP stream” (network)

##### Outputs
- Motion vectors (`numpy int32 (N,10)`)
- Frame type (`"I"|"P"|"B"|"?"`)
- Optional frame (`numpy uint8 (H,W,3)` in BGR) or `None` when decode disabled
- Optional dump artifacts on disk (frames/mvs) if CLI `--dump`

- Input video/RTSP → FFmpeg demux/decode pipeline → C++ extraction
- C++ → (optional) OpenCV decode frame → `cv::Mat`
- `cv::Mat` → `mat_to_ndarray` → NumPy frame
- FFmpeg motion vector structs → converted → NumPy motion vectors
- Python API returns tuple to user; CLI consumes same tuple for preview/dump

Also explicitly show the **mode switch**:
- `set_decode_frames(False)` or CLI `--skip-decoding-frames` → bypass “decode frame to BGR” branch.

##### Component-level view of `VideoCap` (grab/retrieve/read)
Purpose: show internal responsibilities and interactions, closer to code.

- **Open/Init**
- open input (file/rtsp url)
- initialize FFmpeg format context, codec context
- **Grab**
- read packet(s), advance decoder state, collect motion-vector side data
- store current frame metadata + motion vectors
- **Retrieve**
- if decode enabled: decode frame to raw, convert to BGR (OpenCV)
- return cached motion vectors + frame type + optional frame
- **Conversion**
- motion vector struct → dense numpy array (N,10)
- `cv::Mat` → numpy (H,W,3)
- **Release**
- free FFmpeg/OpenCV resources

##### Build/Release/Execution view (CI + packaging + Docker)
Purpose: show “how it ships” and how deps are provided.

- Developer / GitHub
- GitHub Actions (`.github/workflows/ci.yml`)
- builds native extension
- runs unit/e2e tests (including RTSP via live555)
- produces artifacts/wheels (if configured)
- PyPI package `motion-vector-extractor`
- end user `pip install ...` (x86_64 linux wheels)
- DockerHub image (`lubo1994/mv-extractor`)
- contains OS + FFmpeg + OpenCV + package
- End user execution paths:
- local pip install path (uses system libs packaged in wheel or linked—show as “native deps resolved at build time”)
- docker path: `run.sh` → `docker run` → executes `extract_mvs.py`

#### 4) Tailoring guidance specifically for this project (how to be “very detailed” without overthinking)
Because the value is in the **native/video pipeline**, detail these boundaries:
- **Demux vs Decode vs Metadata extraction**
- Motion vectors come from codec side-data (FFmpeg `AVMotionVector`), not from optical flow or post-processing.
- **Two performance modes**
- “Decode frames ON”: returns BGR frames + MVs (preview possible)
- “Decode frames OFF”: returns `None` for frame, but still returns MVs + frame type (high throughput)
- **Two input transports**
- File input (local)
- RTSP input (network stream); optionally mention test RTSP server (live555)

- Motion vectors: `int32 (N,10)` with the 10 columns (source, w, h, src_x, src_y, dst_x, dst_y, motion_x, motion_y, motion_scale)
- Frame type: `"I"/"P"/"B"/"?"`
- Frame: `uint8 (H,W,3)` BGR or `None`

**Client Layer**
- “CLI: extract_mvs (extract_mvs.py)”
- “Python API user code”

**Python Package**
- “mvextractor (src/mvextractor)”
- “VideoCap (Python wrapper)”
- “__main__.py (module entrypoint)”

**Native Extension**
- “py_video_cap.cpp (bindings)”
- “video_cap.cpp/.hpp (core capture/extract)”
- “mat_to_ndarray.* (OpenCV↔NumPy bridge)”
- “pycompat.hpp (py version compatibility)”

**External Dependencies**
- “FFmpeg (libavformat/libavcodec)”
- “OpenCV (frame conversion/BGR mat)”

**Inputs**
- “Video file: H264/MPEG4 Part2”
- “RTSP stream”

**Outputs**
- “NumPy: motion_vectors int32 (N,10)”
- “Frame type (I/P/B/?)”
- “Optional NumPy frame uint8 (H,W,3) BGR / or None”
- “Dump directory (frames/, motion_vectors/)”

**Build/Infra (optional second pane)**
- “GitHub Actions CI”
- “PyPI wheel distribution (x86_64 linux)”
- “Docker image (DockerHub) + run.sh”
- “Tests: unit_tests.py, end_to_end_tests.py + reference .npy + live555 RTSP server”

## Diagram (Mermaid)

```mermaid
flowchart TD

%% =========================
%% View A: Runtime (single-machine)
%% =========================
subgraph "A) Runtime (Context + Container • Single machine)"
direction TB

subgraph "Client Layer"
direction TB
UserCode["Python user code\n(uses mvextractor.VideoCap)"]:::python
CLI["CLI: extract_mvs"]:::python
end

subgraph "Python Package (mvextractor)"
direction TB
Pkg["mvextractor package"]:::python
MainPy["__main__.py\n(module entrypoint)"]:::python
InitPy["__init__.py\n(package init)"]:::python
PyVideoCap["VideoCap (Python-facing API)"]:::python
end

subgraph "Native Extension (C++ / bindings)"
direction TB
Bind["Python↔C++ bindings\n(py_video_cap.cpp)"]:::native
Core["VideoCap C++ core\n(video_cap.cpp/.hpp)"]:::native
MatBridge["cv::Mat ↔ NumPy bridge\n(mat_to_ndarray.*)"]:::native
PyCompat["Python C-API compatibility\n(pycompat.hpp)"]:::native
end

subgraph "External Native Dependencies"
direction TB
FFmpeg["FFmpeg libs\n(libavformat/libavcodec)\nDemux+Decode+MV side-data"]:::external
OpenCV["OpenCV\ncv::Mat + BGR conversion"]:::external
OS["OS/Filesystem/Network stack"]:::infra
end

subgraph "Inputs"
direction TB
VideoFile[("Video file\n(H.264 / MPEG4 Part2)\n(local disk)")]:::artifact
RTSP["RTSP stream\n(network)"]:::cloud
end

subgraph "Outputs (data contract)"
direction TB
MVOut["NumPy motion_vectors\nint32 (N,10)\n[source,w,h,src_x,src_y,dst_x,dst_y,motion_x,motion_y,motion_scale]"]:::data
FTOut["Frame type\nI | P | B | ?"]:::data
FrameOut["Optional NumPy frame\nuint8 (H,W,3) BGR\nor None when decoding disabled"]:::data
DumpDir[("Dump directory\nframes/ + motion_vectors/\n(when CLI --dump)")]:::artifact
end

DecodeSwitch{"Decode frames?\nON: return BGR frame\nOFF: frame=None"}:::control

end

%% Runtime relationships (vertical, detailed)
UserCode -->|"calls open/grab/retrieve/read"| PyVideoCap
CLI -->|"invokes extract_mvs\n(preview/dump logic)"| PyVideoCap
MainPy -->|"python -m mvextractor"| PyVideoCap
InitPy -->|"exports package API"| PyVideoCap
Pkg --> PyVideoCap

PyVideoCap -->|"native calls"| Bind
Bind -->|"invokes"| Core
Core -->|"uses"| PyCompat
Core -->|"demux+decode pipeline\nextract MV side-data (AVMotionVector)"| FFmpeg
Core -->|"optional BGR frame path\n(cv::Mat workflow)"| OpenCV
OpenCV -->|"cv::Mat"| MatBridge
MatBridge -->|"convert Mat→ndarray"| FrameOut

VideoFile -->|"read bytes"| OS
RTSP -->|"stream packets"| OS
OS -->|"I/O"| FFmpeg

Core -->|"convert MV structs→ndarray"| MVOut
Core -->|"derive frame type"| FTOut

Core --> DecodeSwitch
DecodeSwitch -->|"ON"| OpenCV
DecodeSwitch -->|"OFF (bypass decode)"| FrameOut

PyVideoCap -->|"returns tuple\n(mvs, frame_type, frame/None)"| MVOut
PyVideoCap -->|"returns tuple\n(mvs, frame_type, frame/None)"| FTOut
PyVideoCap -->|"returns tuple\n(mvs, frame_type, frame/None)"| FrameOut

CLI -->|"--dump writes"| DumpDir
MVOut -->|"optional dump"| DumpDir
FrameOut -->|"optional dump"| DumpDir

%% =========================
%% View B: Component view of VideoCap (grab/retrieve/read)
%% =========================
subgraph "B) VideoCap internals (methods + responsibilities)"
direction TB

subgraph "Python API surface"
direction TB
PyMethods["Python methods\nopen() / grab() / retrieve() / read() / release()"]:::python
end

subgraph "Bindings boundary"
direction TB
Bind2["Binding layer\n(py_video_cap.cpp)"]:::native
end

subgraph "C++ VideoCap core pipeline"
direction TB
OpenInit["Open/Init\n- open file/rtsp URL\n- init FFmpeg format+codec contexts"]:::native
Grab["Grab\n- read packets\n- advance decoder state\n- collect MV side-data\n- cache frame metadata+MVs"]:::native
Retrieve["Retrieve\n- if decode enabled: decode raw→BGR\n- return cached (MVs, frame type, frame/None)"]:::native
Convert["Conversion\n- MV→NumPy int32 (N,10)\n- cv::Mat→NumPy uint8 (H,W,3)"]:::native
Release["Release\n- free FFmpeg/OpenCV resources"]:::native
end

subgraph "Native deps used by core"
direction TB
FFmpeg2["FFmpeg\n(AVPacket/AVFrame + AVMotionVector side-data)"]:::external
OpenCV2["OpenCV\n(BGR conversion, cv::Mat)"]:::external
MatBridge2["mat_to_ndarray.*"]:::native
end

PyMethods -->|"calls into"| Bind2
Bind2 -->|"dispatches"| OpenInit
Bind2 -->|"dispatches"| Grab
Bind2 -->|"dispatches"| Retrieve
Bind2 -->|"dispatches"| Release

OpenInit -->|"init contexts"| FFmpeg2
Grab -->|"read/demux/decode state\nextract MV side-data"| FFmpeg2
Retrieve -->|"decode frame path (optional)"| OpenCV2
Retrieve -->|"produce arrays"| Convert
Convert -->|"Mat→ndarray"| MatBridge2

end

%% =========================
%% View C: Build / Release / Execution (CI + packaging + Docker)
%% =========================
subgraph "C) Build / Release / Execution (CI + packaging + Docker)"
direction TB

subgraph "Source & Build Config"
direction TB
PyProject["pyproject.toml"]:::infra
SetupPy["setup.py"]:::infra
Manifest["MANIFEST.in"]:::infra
end

subgraph "CI Pipeline"
direction TB
GHA["GitHub Actions CI\n(.github/workflows/ci.yml)"]:::infra
UnitTests["Unit tests\n(tests/unit_tests.py)"]:::test
E2ETests["End-to-end tests\n(tests/end_to_end_tests.py)"]:::test
RefData[("Reference / golden outputs\n(tests/reference/*)\n.npy + frame types + frames")]:::artifact
Live555["RTSP test server tooling\n(live555MediaServer)"]:::test
end

subgraph "Distribution"
direction TB
PyPI["PyPI\nmotion-vector-extractor\n(wheels)"]:::cloud
DockerHub["DockerHub\nlubo1994/mv-extractor"]:::cloud
end

subgraph "Execution paths"
direction TB
EndUser["End user machine"]:::infra
PipInstall["pip install motion-vector-extractor\n(native extension installed)"]:::infra
DockerfileN["Dockerfile\n(FFmpeg/OpenCV + package)"]:::infra
RunSh["run.sh\n(wrapper for docker run)"]:::infra
DockerDoc["dockerhub.md\n(usage notes)"]:::infra
Samples[("Sample videos\nvid_h264.mp4\nvid_mpeg4_part2.mp4\nvid_h264.264")]:::artifact
end

PyProject -->|"build config"| GHA
SetupPy -->|"build native ext"| GHA
Manifest -->|"package include rules"| GHA

GHA -->|"runs"| UnitTests
GHA -->|"runs"| E2ETests
RefData -->|"assert against"| UnitTests
RefData -->|"assert against"| E2ETests
Live555 -->|"used for RTSP scenarios"| E2ETests

GHA -->|"publishes (if configured)"| PyPI
GHA -->|"builds/pushes (if configured)"| DockerHub

EndUser -->|"pip install"| PipInstall
PipInstall -->|"runs CLI / imports package"| CLI
PipInstall -->|"uses samples for demo/tests"| Samples

DockerfileN -->|"build image"| DockerHub
RunSh -->|"docker run image"| DockerHub
DockerDoc -->|"documents"| RunSh
DockerHub -->|"container runtime"| CLI
DockerHub -->|"container runtime"| PyVideoCap

end

%% =========================
%% Click events (component_mapping)
%% =========================
click CLI "https://github.com/lukasbommes/mv-extractor/blob/master/extract_mvs.py"

click MainPy "https://github.com/lukasbommes/mv-extractor/blob/master/src/mvextractor/__main__.py"
click InitPy "https://github.com/lukasbommes/mv-extractor/blob/master/src/mvextractor/__init__.py"

click Core "https://github.com/lukasbommes/mv-extractor/blob/master/src/mvextractor/video_cap.cpp"
click OpenInit "https://github.com/lukasbommes/mv-extractor/blob/master/src/mvextractor/video_cap.hpp"
click Grab "https://github.com/lukasbommes/mv-extractor/blob/master/src/mvextractor/video_cap.cpp"
click Retrieve "https://github.com/lukasbommes/mv-extractor/blob/master/src/mvextractor/video_cap.cpp"
click Release "https://github.com/lukasbommes/mv-extractor/blob/master/src/mvextractor/video_cap.cpp"

click Bind "https://github.com/lukasbommes/mv-extractor/blob/master/src/mvextractor/py_video_cap.cpp"
click Bind2 "https://github.com/lukasbommes/mv-extractor/blob/master/src/mvextractor/py_video_cap.cpp"

click MatBridge "https://github.com/lukasbommes/mv-extractor/blob/master/src/mvextractor/mat_to_ndarray.cpp"
click MatBridge2 "https://github.com/lukasbommes/mv-extractor/blob/master/src/mvextractor/mat_to_ndarray.hpp"

click PyCompat "https://github.com/lukasbommes/mv-extractor/blob/master/src/mvextractor/pycompat.hpp"

click PyProject "https://github.com/lukasbommes/mv-extractor/blob/master/pyproject.toml"
click SetupPy "https://github.com/lukasbommes/mv-extractor/blob/master/setup.py"
click Manifest "https://github.com/lukasbommes/mv-extractor/blob/master/MANIFEST.in"

click DockerfileN "https://github.com/lukasbommes/mv-extractor/tree/master/Dockerfile"
click RunSh "https://github.com/lukasbommes/mv-extractor/blob/master/run.sh"
click DockerDoc "https://github.com/lukasbommes/mv-extractor/blob/master/dockerhub.md"

click GHA "https://github.com/lukasbommes/mv-extractor/blob/master/.github/workflows/ci.yml"

click UnitTests "https://github.com/lukasbommes/mv-extractor/blob/master/tests/unit_tests.py"
click E2ETests "https://github.com/lukasbommes/mv-extractor/blob/master/tests/end_to_end_tests.py"
click RefData "https://github.com/lukasbommes/mv-extractor/tree/master/tests/reference/"
click Live555 "https://github.com/lukasbommes/mv-extractor/tree/master/tests/tools/live555MediaServer"

click Samples "https://github.com/lukasbommes/mv-extractor/blob/master/vid_h264.mp4"
click Samples "https://github.com/lukasbommes/mv-extractor/blob/master/vid_mpeg4_part2.mp4"
click Samples "https://github.com/lukasbommes/mv-extractor/blob/master/vid_h264.264"

%% =========================
%% Styles (color is critical)
%% =========================
classDef python fill:#1f77b4,stroke:#0b3557,color:#ffffff,stroke-width:1.2px
classDef native fill:#ff7f0e,stroke:#8a3f00,color:#ffffff,stroke-width:1.2px
classDef external fill:#2ca02c,stroke:#145214,color:#ffffff,stroke-width:1.2px
classDef infra fill:#7f7f7f,stroke:#3f3f3f,color:#ffffff,stroke-width:1.2px
classDef data fill:#9467bd,stroke:#4a2a66,color:#ffffff,stroke-width:1.2px
classDef artifact fill:#c5b0d5,stroke:#5a4b6a,color:#111111,stroke-width:1.2px
classDef test fill:#d62728,stroke:#6b0f10,color:#ffffff,stroke-width:1.2px
classDef cloud fill:#bdbdbd,stroke:#4d4d4d,color:#111111,stroke-width:1.2px,stroke-dasharray: 4 3
classDef control fill:#f2c14e,stroke:#8a6a00,color:#111111,stroke-width:1.4px
```
