# jutanke/mv3dpose architecture

> Architecture diagram of the GitHub repository jutanke/mv3dpose, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jutanke/mv3dpose
- Repository: https://github.com/jutanke/mv3dpose

## Overview

##### Project Type and Purpose
- This is an open-source, command-line driven multiple-person, multi-view 3D human pose estimation system. It ingests synchronized video frames and camera calibration data, runs 2D pose detection (via OpenPose), reconstructs 3D joint positions using geometric triangulation and greedy bipartite matching, tracks poses across time, and outputs 3D tracks. It also provides evaluation and visualization utilities.

##### High-Level File Structure Analysis
- Top‐level scripts and folders:
- docker/        → container build, run, and helper scripts for GPU‐accelerated processing
- mv3dpose/      → core Python package implementing data loading, geometry, matching, tracking, evaluation, and visualization
- preprocessing/ → scripts to convert raw video and calibration data into the expected dataset format
- openpose/      → (submodule likely containing OpenPose code or bindings)
- Inside mv3dpose/:
- data/          → default camera landmarks, limb definitions, OpenPose JSON parsers
- geometry/      → camera models & projection mathematics (camera.py, stereo.py)
- baseline.py    → reference multi‐view aggregation strategy
- hypothesis.py  → pose hypotheses generation
- mvpose.py      → high‐level orchestration of the reconstruction and tracking pipeline
- evaluate.py    → performance metrics and benchmark evaluation
- tracking.py    → greedy bipartite matching across views and time

##### README Insights
- Clearly defines input dataset layout (videos + per‐frame JSON for each camera).
- Describes camera JSON schemas (projection‐only or full OpenCV intrinsics/distortion).
- Lists optional parameters for epipolar thresholds, smoothing, tracking delays.
- Shows final output format: one JSON per detected person containing 3D joint locations over frames.

a. Main Components:
1) Data Ingestion & Preprocessing
- Raw video frames + calibration JSON → preprocessing scripts (preprocessing/) → structured dataset folder.
2) 2D Pose Detection
- OpenPose module (openpose/) processes images → per‐frame 2D joint JSON.
3) Core 3D Reconstruction & Tracking
- mv3dpose package:
- Data loader reads 2D detections + camera models
- Geometry module computes ray intersections / triangulation
- Hypothesis & Baseline modules aggregate candidate joints
- Tracking module performs greedy bipartite matching across views and frames
4) Evaluation & Visualization
- evaluate.py runs benchmarks (optional eval.sh)
5) Containerization & Execution Environment
- Dockerfile + build/run scripts → reproducible GPU‐enabled Docker image

b. Relationships & Data Flows:
- Dataset folder → Preprocessor → Structured input
- Structured input → OpenPose → 2D JSON outputs
- 2D JSON + calibration data → mv3dpose.geometry → 3D point candidates
- 3D candidates → mv3dpose.hypothesis & baseline → filtered hypotheses
- Hypotheses → mv3dpose.tracking → temporally linked tracks

c. Architectural Patterns & Principles:
– Modular pipeline: clear separation of concerns (preprocessing, detection, reconstruction, tracking, evaluation)
– Data-driven design: JSON‐based I/O, parameter files, CLI flags
– Containerized reproducibility: Docker ensures consistent CUDA environment

d. Key Technologies & Frameworks:
– Python 3 with NumPy/SciPy for geometry
– OpenPose (C++) for 2D pose estimation
– Docker & NVIDIA‐Docker for GPU acceleration
– JSON for configuration and data interchange
– Shell scripts for orchestration

##### Detail
- Optional parameters and configuration files (dataset.json) as annotations.
- How mvpose.sh orchestrates Docker → preprocessing → OpenPose → mv3dpose pipeline.
- That calibration handling (projection vs. full intrinsics) is called out in the camera model sub‐component.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Stores
    RawVideos[(Raw Videos)]:::data
    CalibrationData[(Calibration JSON)]:::data

    %% Preprocessing Layer
    subgraph "Data Ingestion & Preprocessing"
        Preprocessor["Preprocessing\n(preprocessing/)"]:::preprocess
        StructuredData[(Structured Dataset)]:::data
    end

    %% 2D Detection Layer
    subgraph "2D Pose Detection"
        OpenPoseLib((OpenPose\nLibrary)):::external
        OpenPoseAdapter["OpenPose Adapter\n(mv3dpose/data/openpose.py)"]:::preprocess
        Pose2DData[(2D JSON Outputs)]:::data
    end

    %% Core Reconstruction & Tracking Layer
    subgraph "3D Reconstruction & Tracking"
        DefaultData["Default Limbs\n(mv3dpose/data/default_limbs.py)"]:::core
        CameraModel["Camera Model\n(mv3dpose/geometry/camera.py)"]:::core
        GeometryUtils["Geometry Utils\n(mv3dpose/geometry/geometry.py)"]:::core
        Stereo["Stereo Triangulation\n(mv3dpose/geometry/stereo.py)"]:::core
        Baseline["Baseline Aggregation\n(mv3dpose/baseline.py)"]:::core
        Hypothesis["Hypothesis Generation\n(mv3dpose/hypothesis.py)"]:::core
        Tracking["Tracking\n(mv3dpose/tracking.py)"]:::core
        PermuteUtil["Permutation Testing\n(mv3dpose/test_permutations.py)"]:::core
        Tracks[(3D Tracks JSON)]:::data
        Orchestrator["Pipeline Orchestrator\n(mv3dpose/mvpose.py)"]:::core
    end

    %% Evaluation & Visualization Layer
    subgraph "Evaluation & Visualization"
        Evaluate["Evaluation Module\n(mv3dpose/evaluate.py)"]:::eval
        EvalScript["eval.sh"]:::infra
        Visualize["Visualization Module\n(mv3dpose/visualize.py)"]:::eval
        VisScript["visualize.sh"]:::infra
    end

    %% Infrastructure Layer
    subgraph "Containerization & Orchestration"
        Dockerfile["Dockerfile\n(docker/Dockerfile)"]:::infra
        BuildScript["build.sh\n(docker/build.sh)"]:::infra
        RunScript["run.sh\n(docker/run.sh)"]:::infra
        ExecScripts["Execution Helpers\n(docker/exec*.sh)"]:::infra
        MVPosescript["mvpose.sh"]:::infra
    end

    %% Data Flow
    RawVideos -->|frames| Preprocessor
    CalibrationData -->|calib.json| Preprocessor
    Preprocessor -->|structured data| StructuredData

    StructuredData -->|images| OpenPoseAdapter
    OpenPoseLib --> OpenPoseAdapter
    OpenPoseAdapter -->|pose2D.json| Pose2DData

    Pose2DData --> Orchestrator
    CalibrationData --> Orchestrator
    Orchestrator --> DefaultData
    Orchestrator --> CameraModel
    CameraModel --> GeometryUtils
    GeometryUtils --> Stereo
    Stereo --> Baseline
    Baseline --> Hypothesis
    Hypothesis --> Tracking
    Tracking --> Tracks
    Tracking --> PermuteUtil

    Tracks --> Evaluate
    EvalScript --> Evaluate
    Tracks --> Visualize
    VisScript --> Visualize

    %% Orchestration
    MVPosescript --> Preprocessor
    MVPosescript --> OpenPoseAdapter
    MVPosescript --> Orchestrator

    Dockerfile --> BuildScript --> RunScript
    RunScript --> ExecScripts
    ExecScripts --> MVPosescript

    %% Click Events
    click Preprocessor "https://github.com/jutanke/mv3dpose/tree/master/preprocessing/"
    click OpenPoseLib "https://github.com/jutanke/mv3dpose/tree/master/openpose/"
    click OpenPoseAdapter "https://github.com/jutanke/mv3dpose/blob/master/mv3dpose/data/openpose.py"
    click DefaultData "https://github.com/jutanke/mv3dpose/blob/master/mv3dpose/data/default_limbs.py"
    click CameraModel "https://github.com/jutanke/mv3dpose/blob/master/mv3dpose/geometry/camera.py"
    click GeometryUtils "https://github.com/jutanke/mv3dpose/blob/master/mv3dpose/geometry/geometry.py"
    click Stereo "https://github.com/jutanke/mv3dpose/blob/master/mv3dpose/geometry/stereo.py"
    click Baseline "https://github.com/jutanke/mv3dpose/blob/master/mv3dpose/baseline.py"
    click Hypothesis "https://github.com/jutanke/mv3dpose/blob/master/mv3dpose/hypothesis.py"
    click Tracking "https://github.com/jutanke/mv3dpose/blob/master/mv3dpose/tracking.py"
    click PermuteUtil "https://github.com/jutanke/mv3dpose/blob/master/mv3dpose/test_permutations.py"
    click Orchestrator "https://github.com/jutanke/mv3dpose/blob/master/mv3dpose/mvpose.py"
    click Evaluate "https://github.com/jutanke/mv3dpose/blob/master/mv3dpose/evaluate.py"
    click EvalScript "https://github.com/jutanke/mv3dpose/blob/master/eval.sh"
    click Visualize "https://github.com/jutanke/mv3dpose/blob/master/mv3dpose/visualize.py"
    click VisScript "https://github.com/jutanke/mv3dpose/blob/master/visualize.sh"
    click Dockerfile "https://github.com/jutanke/mv3dpose/tree/master/docker/Dockerfile"
    click BuildScript "https://github.com/jutanke/mv3dpose/blob/master/docker/build.sh"
    click RunScript "https://github.com/jutanke/mv3dpose/blob/master/docker/run.sh"
    click ExecScripts "https://github.com/jutanke/mv3dpose/blob/master/docker/exec.sh"
    click MVPosescript "https://github.com/jutanke/mv3dpose/blob/master/mvpose.sh"

    %% Styles
    classDef data fill:#BDE0FE,stroke:#333,stroke-width:1px
    classDef preprocess fill:#A2D2FF,stroke:#333,stroke-width:1px
    classDef core fill:#B8E986,stroke:#333,stroke-width:1px
    classDef eval fill:#FFD6A5,stroke:#333,stroke-width:1px
    classDef infra fill:#CCCCCC,stroke:#333,stroke-width:1px
    classDef external fill:#E0AAFF,stroke:#333,stroke-width:1px
```
