How lukasbommes/mv-extractor works
This is an offline/streaming video processing tool + Python package that exposes an OpenCV-VideoCapture-like API to extract, per frame:
• frame type (I/P/B)
• motion vectors (from codec/bitstream metadata via FFmpeg)
• optionally the decoded frame (BGR) for preview / dumping
It’s not a distributed “system” in the classic service sense; the “architecture” is primarily:
• Python user-facing API + CLI
• backed by a native C++ core (FFmpeg/OpenCV integration)
• packaged as a Python distribution (wheels) and also runnable via Docker
• verified by unit + end-to-end tests (with reference outputs, including RTSP)
User-facing entrypoints
• CLI script: extract_mvs.py (invoked as extract_mvs ... per README)
• Python package entry: src/mvextractor/__main__.py (module execution) and __init__.py
Native core (where “the work” happens)
Located in src/mvextractor/:
• video_cap.cpp / video_cap.hpp: main C++ implementation (capture, grab/retrieve/read, motion vector extraction, optional decode)
• py_video_cap.cpp: Python bindings layer (likely pybind11 or Python C-API style; exact mechanism isn’t shown in tree but this file is the bridge)
• mat_to_ndarray.cpp/.hpp: conversion between OpenCV cv::Mat and NumPy arrays
• pycompat.hpp: compatibility helpers for Python versions
Packaging / build & distribution
• pyproject.toml, setup.py, MANIFEST.in: Python build configuration (build native extension, publish wheels)
• Dockerfile, run.sh, dockerhub.md: container build/runtime environment with FFmpeg/OpenCV preinstalled
• .github/workflows/ci.yml: CI pipeline (build/test)
Tests & reference artifacts
• tests/unit_tests.py
• tests/end_to_end_tests.py
• tests/reference/*: golden motion-vector outputs (.npy), frame types, etc for H264 / MPEG4 Part2 / RTSP
• tests/tools/live555MediaServer: used to stand up an RTSP server during tests
• sample videos at repo root: vid_h264.mp4, vid_mpeg4_part2.mp4, etc.
User / Client Code
• “Python script using mvextractor.VideoCap”
• “CLI user running extract_mvs”
Python Layer (mvextractor package)
• mvextractor.VideoCap (Python-facing class)
• CLI/preview/dump logic (extract_mvs.py)
Native Extension Module
• “C++ VideoCap core” (video_cap.cpp/.hpp)
• “Python binding” (py_video_cap.cpp)
• “cv::Mat ↔ numpy” (mat_to_ndarray.*)
External Native Dependencies
• FFmpeg libs (demux/decode, access to AVMotionVector)
• OpenCV (VideoCapture-like patterns; frame decode to BGR; cv::Mat)
• OS / drivers as needed for file I/O / networking
Input Sources
• “Video file (H.264 / MPEG4 Part2)” (filesystem)
• “RTSP stream” (network)
Outputs
• Motion vectors (numpy int32 (N,10))
• Frame type ("I"|"P"|"B"|"?")
• Optional frame (numpy uint8 (H,W,3) in BGR) or None when decode disabled
• Optional dump artifacts on disk (frames/mvs) if CLI --dump
• Input video/RTSP → FFmpeg demux/decode pipeline → C++ extraction
• C++ → (optional) OpenCV decode frame → cv::Mat
• cv::Mat → mat_to_ndarray → NumPy frame
• FFmpeg motion vector structs → converted → NumPy motion vectors
• Python API returns tuple to user; CLI consumes same tuple for preview/dump
Also explicitly show the mode switch:
• set_decode_frames(False) or CLI --skip-decoding-frames → bypass “decode frame to BGR” branch.
Component-level view of VideoCap (grab/retrieve/read)
Purpose: show internal responsibilities and interactions, closer to code.
• Open/Init
• open input (file/rtsp url)
• initialize FFmpeg format context, codec context
• Grab
• read packet(s), advance decoder state, collect motion-vector side data
• store current frame metadata + motion vectors
• Retrieve
• if decode enabled: decode frame to raw, convert to BGR (OpenCV)
• return cached motion vectors + frame type + optional frame
• Conversion
• motion vector struct → dense numpy array (N,10)
• cv::Mat → numpy (H,W,3)
• Release
• free FFmpeg/OpenCV resources
Build/Release/Execution view (CI + packaging + Docker)
Purpose: show “how it ships” and how deps are provided.
• Developer / GitHub
• GitHub Actions (.github/workflows/ci.yml)
• builds native extension
• runs unit/e2e tests (including RTSP via live555)
• produces artifacts/wheels (if configured)
• PyPI package motion-vector-extractor
• end user pip install ... (x86_64 linux wheels)
• DockerHub image (lubo1994/mv-extractor)
• contains OS + FFmpeg + OpenCV + package
• End user execution paths:
• local pip install path (uses system libs packaged in wheel or linked—show as “native deps resolved at build time”)
• docker path: run.sh → docker run → executes extract_mvs.py
4) Tailoring guidance specifically for this project (how to be “very detailed” without overthinking)
Because the value is in the native/video pipeline, detail these boundaries:
• Demux vs Decode vs Metadata extraction
• Motion vectors come from codec side-data (FFmpeg AVMotionVector), not from optical flow or post-processing.
• Two performance modes
• “Decode frames ON”: returns BGR frames + MVs (preview possible)
• “Decode frames OFF”: returns None for frame, but still returns MVs + frame type (high throughput)
• Two input transports
• File input (local)
• RTSP input (network stream); optionally mention test RTSP server (live555)
• Motion vectors: int32 (N,10) with the 10 columns (source, w, h, src_x, src_y, dst_x, dst_y, motion_x, motion_y, motion_scale)
• Frame type: "I"/"P"/"B"/"?"
• Frame: uint8 (H,W,3) BGR or None
Client Layer
• “CLI: extract_mvs (extract_mvs.py)”
• “Python API user code”
Python Package
• “mvextractor (src/mvextractor)”
• “VideoCap (Python wrapper)”
• “__main__.py (module entrypoint)”
Native Extension
• “py_video_cap.cpp (bindings)”
• “video_cap.cpp/.hpp (core capture/extract)”
• “mat_to_ndarray.* (OpenCV↔NumPy bridge)”
• “pycompat.hpp (py version compatibility)”
External Dependencies
• “FFmpeg (libavformat/libavcodec)”
• “OpenCV (frame conversion/BGR mat)”
Inputs
• “Video file: H264/MPEG4 Part2”
• “RTSP stream”
Outputs
• “NumPy: motion_vectors int32 (N,10)”
• “Frame type (I/P/B/?)”
• “Optional NumPy frame uint8 (H,W,3) BGR / or None”
• “Dump directory (frames/, motion_vectors/)”
Build/Infra (optional second pane)
• “GitHub Actions CI”
• “PyPI wheel distribution (x86_64 linux)”
• “Docker image (DockerHub) + run.sh”
• “Tests: unit_tests.py, end_to_end_tests.py + reference .npy + live555 RTSP server”