How perfanalytics/pose2sim works
Project type and purpose
• Pose2Sim is an open-source Python package and command-line/API toolkit that implements a modular pipeline for 3D markerless motion analysis. It ingests multi-camera video and calibration data, applies 2D pose estimation, synchronizes and associates detections, triangulates and filters 3D keypoints, optionally augments markers, runs OpenSim scaling and inverse kinematics, and produces .trc/.mot outputs and visualizations.
Key architectural elements from the file tree and README
• Top-level “Pose2Sim” Python package:
• Pose2Sim.py – main orchestration class exposing runAll() and stage methods
• Submodules: calibration.py, poseEstimation.py, synchronization.py, personAssociation.py, triangulation.py, filtering.py, markerAugmentation.py, kinematics.py, plus utility scripts.
• Demo folders under Pose2Sim/Demo_*: preconfigured input data (videos, Config.toml, calibration).
• Utilities/ – standalone conversion and plotting scripts (Blazepose_runsave.py, DLC_to_OpenPose.py, trc_plot.py, etc.).
• Configuration: Config.toml files drive each step’s parameters.
• External dependencies:
• 2D pose engines (RTMPose via ONNX Runtime, PyTorch; legacy OpenPose/AlphaPose/Mediapipe)
• OpenCV for calibration and reprojection
• OpenSim Python API or opensim-cmd for scaling and IK
• Blender add-on for 3D visualization
• FFMPEG (video I/O)
• CI/CD (GitHub Actions), packaging (pyproject.toml), demos and website content.
Main components
A. Inputs
• Raw multi-camera video feeds (mp4)
• Calibration data (checkerboard images, existing formats: .qca.txt, .xcp, .pickle, .yml, bioCV)
• Optional precomputed 2D keypoint JSON or DeepLabCut H5/CSV
• Config.toml for parameters
B. Core pipeline modules (layered sequence)
1. Calibration
2. 2D Pose Estimation
3. Synchronization
4. Person Association
5. Triangulation
6. Filtering
7. Marker Augmentation (optional)
8. Kinematics (OpenSim scaling + IK)
C. Data stores / artifacts
• Intermediate: JSON keypoints, synchronized JSON, .trc 3D marker files
• Final outputs: .mot joint angles, scaled .osim models, logs
D. Visualization tools
• OpenSim GUI
• Pose2Sim Blender add-on
E. External libraries/services
• MMPose/RTMPose (ONNX models, PyTorch)
• OpenPose/AlphaPose/MediaPipe (legacy)
• OpenCV
• ONNX Runtime, CUDA backend
• OpenSim Python API or opensim-cmd
• FFMPEG
Relationships and data flows
• Video + Calib.toml → Pose Estimation → 2D JSON
• 2D JSON → Synchronization → time-aligned JSON
• Aligned JSON → Person Association → grouped person keypoints
• Person keypoints + Calib.toml → Triangulation → raw .trc
• .trc → Filtering → filtered .trc
• filtered .trc → Marker Augmentation (conditional) → augmented .trc
• (augmented) .trc + .osim base model → Kinematics (OpenSim) → scaled model + .mot angles
• Outputs → Visualization: OpenSim GUI / Blender add-on
Architectural patterns and principles
• Pipeline/chain-of-responsibility: each stage ingests standardized data and outputs for the next
• Configuration-driven: Config.toml controls behavior; demos override defaults
• Modular and extensible: new pose engines or calibration formats plugged via Utilities submodule
• Separation of concerns: compute stages vs. visualization vs. file conversion utilities
• Batch vs. single-trial processing: same modules, different folder schemas