How jutanke/mv3dpose works
Project Type and Purpose
• This is an open-source, command-line driven multiple-person, multi-view 3D human pose estimation system. It ingests synchronized video frames and camera calibration data, runs 2D pose detection (via OpenPose), reconstructs 3D joint positions using geometric triangulation and greedy bipartite matching, tracks poses across time, and outputs 3D tracks. It also provides evaluation and visualization utilities.
High-Level File Structure Analysis
• Top‐level scripts and folders:
• docker/ → container build, run, and helper scripts for GPU‐accelerated processing
• mv3dpose/ → core Python package implementing data loading, geometry, matching, tracking, evaluation, and visualization
• preprocessing/ → scripts to convert raw video and calibration data into the expected dataset format
• openpose/ → (submodule likely containing OpenPose code or bindings)
• Inside mv3dpose/:
• data/ → default camera landmarks, limb definitions, OpenPose JSON parsers
• geometry/ → camera models & projection mathematics (camera.py, stereo.py)
• baseline.py → reference multi‐view aggregation strategy
• hypothesis.py → pose hypotheses generation
• mvpose.py → high‐level orchestration of the reconstruction and tracking pipeline
• evaluate.py → performance metrics and benchmark evaluation
• tracking.py → greedy bipartite matching across views and time
README Insights
• Clearly defines input dataset layout (videos + per‐frame JSON for each camera).
• Describes camera JSON schemas (projection‐only or full OpenCV intrinsics/distortion).
• Lists optional parameters for epipolar thresholds, smoothing, tracking delays.
• Shows final output format: one JSON per detected person containing 3D joint locations over frames.
a. Main Components:
1) Data Ingestion & Preprocessing
• Raw video frames + calibration JSON → preprocessing scripts (preprocessing/) → structured dataset folder.
2) 2D Pose Detection
• OpenPose module (openpose/) processes images → per‐frame 2D joint JSON.
3) Core 3D Reconstruction & Tracking
• mv3dpose package:
• Data loader reads 2D detections + camera models
• Geometry module computes ray intersections / triangulation
• Hypothesis & Baseline modules aggregate candidate joints
• Tracking module performs greedy bipartite matching across views and frames
4) Evaluation & Visualization
• evaluate.py runs benchmarks (optional eval.sh)
5) Containerization & Execution Environment
• Dockerfile + build/run scripts → reproducible GPU‐enabled Docker image
b. Relationships & Data Flows:
• Dataset folder → Preprocessor → Structured input
• Structured input → OpenPose → 2D JSON outputs
• 2D JSON + calibration data → mv3dpose.geometry → 3D point candidates
• 3D candidates → mv3dpose.hypothesis & baseline → filtered hypotheses
• Hypotheses → mv3dpose.tracking → temporally linked tracks
c. Architectural Patterns & Principles:
– Modular pipeline: clear separation of concerns (preprocessing, detection, reconstruction, tracking, evaluation)
– Data-driven design: JSON‐based I/O, parameter files, CLI flags
– Containerized reproducibility: Docker ensures consistent CUDA environment
d. Key Technologies & Frameworks:
– Python 3 with NumPy/SciPy for geometry
– OpenPose (C++) for 2D pose estimation
– Docker & NVIDIA‐Docker for GPU acceleration
– JSON for configuration and data interchange
– Shell scripts for orchestration
Detail
• Optional parameters and configuration files (dataset.json) as annotations.
• How mvpose.sh orchestrates Docker → preprocessing → OpenPose → mv3dpose pipeline.
• That calibration handling (projection vs. full intrinsics) is called out in the camera model sub‐component.