How jishnujayakumar/mv-tractus works
1. What this repo is
- MV-Tractus is a local video-processing toolchain for extracting motion vectors from H.264/MPEG streams.
- The system has two main faces: native CLI binaries and a Python package wrapper.
2. Primary runtime boundary
- The core work happens in native code under src/c, which interfaces with FFmpeg APIs.
- The repository does not define a backend service or networked architecture; it is a file-in / file-out desktop-style pipeline.
3. Native extraction subsystem
- src/c/extract_mvs.c is the main motion-vector extractor.
- src/c/extract_mvs_with_frames.c extends extraction to also save decoded frames.
- src/c/e_json.c suggests a small serialization layer for writing motion vectors as JSON.
- These programs produce artifacts in the output directories rather than returning in-memory results to another service.
4. FFmpeg dependency layer
- The large include/ tree is a vendored FFmpeg source/header snapshot, not just a thin header include directory.
- This is the main third-party dependency boundary and likely the build target for the native binaries.
- The repo’s own code sits on top of FFmpeg’s decode/filter APIs to access motion vectors and frame data.
5. CLI build and execution path
- compile and compile_command.txt indicate a scripted native build workflow.
- The compiled tools live in bin/, including extract_mvs, extract_mvs_with_frames, and efmvs.
- first_run.sh, install_ffmpeg.sh, and ffmpeg.conf are setup/support assets for locating or building FFmpeg dependencies.
6. Output artifacts and data flow
- Input: a video file, typically H.264/MPEG.
- Processing: FFmpeg decodes the stream and exposes motion vectors to the extractor.
- Output: JSON motion-vector files under output/mv, and optionally frame images under output/frames.
- The default paths imply a simple local pipeline with convention-based storage.
7. Python package subsystem
- mvtpy/mv_tractus is the packaged Python API.
- wrapper.py exposes classes such as MVTractus and MVTractusWithFrames, mirroring the CLI capabilities.
- This package is the higher-level integration boundary for users who want programmatic access instead of shell commands.
8. Overlay/visualization path
- mvtpy/mv_tractus/overlay.py adds a visualization stage that draws motion vectors on saved frames.
- The README shows this uses Pillow-style image handling and operates on two prior outputs: frame images and JSON motion-vector files.
- That makes overlay a post-processing consumer of extractor outputs, not part of the decode stage.
9. Python package build and distribution
- mvtpy/pyproject.toml, setup.py, and setup.cfg define the Python packaging boundary.
- The repo also contains built distributions in mvtpy/dist, showing the package is intended for installable reuse.
10. Experimental and prototype area
- experiments/, original/, and python-boost/ contain alternate implementations and backups.
- These include C, C++, and Python experiments, plus OpenCV/Boost.Python conversion prototypes.
- For an architecture diagram, these should be treated as non-production R&D branches, not part of the main runtime flow.
11. Technology profile that matters
- Native layer: C/C++ with FFmpeg.
- Python layer: package wrapper for orchestration and visualization.
- Image overlay and conversion experiments reference OpenCV, Boost.Python, and ZMQ headers, but the README emphasizes FFmpeg-based extraction as the core path.
12. Overall architectural shape
- The system is a layered offline processing pipeline:
- video input → FFmpeg-based native extractor → JSON motion vectors and optional frames → Python overlay/analysis tools.
- The key diagram boundaries are: external video file, FFmpeg/native extraction engine, generated artifacts on disk, and Python consumer/visualization tooling.
Main entry points: extract_mvs_with_frames.c Extract + frames · extract_mvs.c Extract MVs · overlay.py Overlay tool · pyproject.toml Package files
Components 19
Native pipeline
C/C++ extraction binaries and their build/bootstrap flow.
- Extract MVs · Main motion-vector extraction binary.
src/c/extract_mvs.c - Extract + frames · Extractor variant that also saves decoded frames.
src/c/extract_mvs_with_frames.c - JSON writer · Motion-vector serialization helper used by native extraction.
src/c/e_json.c - Build script · Scripted native build entrypoint for the CLI binaries.
compile - Compile args · Saved compile command used to reproduce the native build.
compile_command.txt - CLI tools · Compiled executables for extraction and related processing.
bin - Setup assets · FFmpeg setup and first-run support files.
first_run.sh - FFmpeg config · Local configuration used to locate or build FFmpeg dependencies.
ffmpeg.conf
Python package
Packaged wrapper and visualization layer for programmatic use.
- Python API · Packaged Python interface mirroring the native extraction workflows.
mvtpy/mv_tractus/wrapper.py - Overlay tool · Post-processing stage that draws motion vectors onto saved frames.
mvtpy/mv_tractus/overlay.py - Package files · Python packaging boundary for installable distribution.
mvtpy/pyproject.toml - Distributions · Built wheel and source distributions for the Python package.
mvtpy/dist
FFmpeg snapshot
Vendored FFmpeg source tree and APIs used by the native extractor.
- FFmpeg APIs · Vendored FFmpeg headers and sources that provide decode and filter capabilities.
include
Experiments
Backups and prototype implementations outside the main runtime path.
- Experiments · Alternative implementations, backups, and conversion experiments.
experiments - Originals · Earlier native extractor versions and historical variants.
original - Boost prototypes · OpenCV/Boost.Python conversion experiments and alternative wrappers.
python-boost
Other components
- Video file · Local H.264/MPEG input consumed by the extractor.
input/1.h264 - Frame output · Saved decoded frame images produced by the native pipeline.
- MV output · Generated motion-vector JSON output directory.
Connections 22
- Video file → Extract MVs: decode · Video input is decoded by the native motion-vector extractor.
- Video file → Extract + frames: decode · Video input is decoded by the extractor variant that also saves frames.
- Extract MVs → FFmpeg APIs: FFmpeg calls · Extractor uses FFmpeg decode APIs and motion-vector side data.
- Extract + frames → FFmpeg APIs: FFmpeg calls · Frames+vectors variant relies on FFmpeg decoding and frame access.
- Extract MVs → JSON writer: serialize · Extractor writes motion-vector results through the JSON layer.
- Extract + frames → JSON writer: serialize · Frames+vectors variant writes motion-vector JSON output.
- Extract MVs → MV output: write · Motion-vector JSON is emitted to the motion-vector output location.
- Extract + frames → MV output: write · Motion-vector JSON is emitted to the motion-vector output location.
- Extract + frames → Frame output: write · Decoded frames are saved as output artifacts.
- Build script → Compile args: drives · Build script uses the recorded compile command/workflow.
- Build script → CLI tools: produces · Native build produces CLI binaries in bin/.
- Setup assets → FFmpeg config: configures · Bootstrap scripts help locate or prepare FFmpeg dependencies.
- FFmpeg config → FFmpeg APIs: targets · Configuration points the native build/runtime at vendored FFmpeg.
- Python API → CLI tools: wraps · Python API mirrors the compiled command-line tools.
- Overlay tool → Frame output: consumes · Overlay operates on previously saved frame images.
- Overlay tool → MV output: consumes · Overlay consumes motion-vector JSON files.
- Package files → Python API: packages · Packaging boundary exposes the Python wrapper module.
- Package files → Overlay tool: packages · Packaging boundary includes visualization utilities.
- Distributions → Package files: built from · Distribution artifacts are produced from the Python packaging setup.
- Experiments → Extract MVs: prototype of · Experimental code explores alternate extractor implementations.
- Originals → Extract + frames: prototype of · Historical sources capture earlier extractor variants.
- Boost prototypes → Overlay tool: prototype of · Boost/OpenCV experiments explore Python-side visualization and conversion.