# perfanalytics/pose2sim architecture

> Architecture diagram of the GitHub repository perfanalytics/pose2sim, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/perfanalytics/pose2sim
- Repository: https://github.com/perfanalytics/pose2sim

## Overview

##### Project type and purpose
- Pose2Sim is an open-source Python package and command-line/​API toolkit that implements a modular pipeline for 3D markerless motion analysis. It ingests multi-camera video and calibration data, applies 2D pose estimation, synchronizes and associates detections, triangulates and filters 3D keypoints, optionally augments markers, runs OpenSim scaling and inverse kinematics, and produces .trc/.mot outputs and visualizations.

##### Key architectural elements from the file tree and README
- Top-level “Pose2Sim” Python package:
- Pose2Sim.py – main orchestration class exposing runAll() and stage methods
- Submodules: calibration.py, poseEstimation.py, synchronization.py, personAssociation.py, triangulation.py, filtering.py, markerAugmentation.py, kinematics.py, plus utility scripts.
- Demo folders under Pose2Sim/Demo_*: preconfigured input data (videos, Config.toml, calibration).
- Utilities/ – standalone conversion and plotting scripts (Blazepose_runsave.py, DLC_to_OpenPose.py, trc_plot.py, etc.).
- Configuration: Config.toml files drive each step’s parameters.
- External dependencies:
- 2D pose engines (RTMPose via ONNX Runtime, PyTorch; legacy OpenPose/AlphaPose/Mediapipe)
- OpenCV for calibration and reprojection
- OpenSim Python API or opensim-cmd for scaling and IK
- Blender add-on for 3D visualization
- FFMPEG (video I/O)
- CI/CD (GitHub Actions), packaging (pyproject.toml), demos and website content.

##### Main components
A. Inputs
- Raw multi-camera video feeds (mp4)
- Calibration data (checkerboard images, existing formats: .qca.txt, .xcp, .pickle, .yml, bioCV)
- Optional precomputed 2D keypoint JSON or DeepLabCut H5/CSV
- Config.toml for parameters
B. Core pipeline modules (layered sequence)
1. Calibration
2. 2D Pose Estimation
3. Synchronization
4. Person Association
5. Triangulation
6. Filtering
7. Marker Augmentation (optional)
8. Kinematics (OpenSim scaling + IK)
C. Data stores / artifacts
- Intermediate: JSON keypoints, synchronized JSON, .trc 3D marker files
- Final outputs: .mot joint angles, scaled .osim models, logs
D. Visualization tools
- OpenSim GUI
- Pose2Sim Blender add-on
E. External libraries/services
- MMPose/RTMPose (ONNX models, PyTorch)
- OpenPose/AlphaPose/MediaPipe (legacy)
- OpenCV
- ONNX Runtime, CUDA backend
- OpenSim Python API or opensim-cmd
- FFMPEG

##### Relationships and data flows
- Video + Calib.toml → Pose Estimation → 2D JSON
- 2D JSON → Synchronization → time-aligned JSON
- Aligned JSON → Person Association → grouped person keypoints
- Person keypoints + Calib.toml → Triangulation → raw .trc
- .trc → Filtering → filtered .trc
- filtered .trc → Marker Augmentation (conditional) → augmented .trc
- (augmented) .trc + .osim base model → Kinematics (OpenSim) → scaled model + .mot angles
- Outputs → Visualization: OpenSim GUI / Blender add-on

##### Architectural patterns and principles
- Pipeline/chain-of-responsibility: each stage ingests standardized data and outputs for the next
- Configuration-driven: Config.toml controls behavior; demos override defaults
- Modular and extensible: new pose engines or calibration formats plugged via Utilities submodule
- Separation of concerns: compute stages vs. visualization vs. file conversion utilities
- Batch vs. single-trial processing: same modules, different folder schemas

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Inputs
    subgraph "Inputs" 
        direction TB
        RawVideo["Raw Multi-Camera Videos (mp4)"]:::inputs
        CalibData["Calibration Data (.qca.txt, .xcp, .pickle, .yml)"]:::inputs
        PreKey["Precomputed 2D Keypoints (JSON/H5/CSV)"]:::inputs
        Config["Config.toml"]:::inputs
    end

    %% Orchestration
    Orchestrator["Main Orchestration\nPose2Sim.py"]:::pipeline

    %% Core Pipeline Modules
    subgraph "Core Pipeline" 
        direction TB
        CalibM["Calibration"]:::pipeline
        PoseEst["2D Pose Estimation"]:::pipeline
        Sync["Synchronization"]:::pipeline
        PersonAssoc["Person Association"]:::pipeline
        Triang["Triangulation"]:::pipeline
        Filtering["Filtering"]:::pipeline
        MarkerAug["Marker Augmentation"]:::pipeline
        Kinematics["Kinematics (Scaling & IK)"]:::pipeline
    end

    %% Data Artifacts
    subgraph "Data Artifacts"
        direction TB
        CalibOut["Calib.toml"]:::data
        JSON2D["2D Keypoints JSON"]:::data
        JSONSync["Time-aligned JSON"]:::data
        TRCRaw["Raw .trc"]:::data
        TRCFilter["Filtered .trc"]:::data
        TRCAug["Augmented .trc"]:::data
        OSIMModel["Scaled .osim Model"]:::data
        MOT[".mot Joint Angles"]:::data
    end

    %% External Dependencies
    subgraph "External Services" 
        direction TB
        OpenCV["OpenCV"]:::external
        ONNX["ONNX Runtime / PyTorch"]:::external
        OPose["OpenPose/AlphaPose/MediaPipe"]:::external
        OSIMAPI["OpenSim API / opensim-cmd"]:::external
        FFmpeg["FFmpeg"]:::external
    end

    %% Visualization
    subgraph "Visualization"
        direction TB
        OpenSimGUI["OpenSim GUI"]:::viz
        BlenderAddOn["Blender Add-On"]:::viz
    end

    %% Utilities & Demos
    subgraph "Utilities & Demos"
        direction TB
        Utilities["Conversion & Plotting Scripts"]:::misc
        Common["common.py"]:::misc
        Skeletons["skeletons.py"]:::misc
        subgraph "Demos"
            direction TB
            DemoSingle["Demo_SinglePerson"]:::misc
            DemoMulti["Demo_MultiPerson"]:::misc
            DemoBatch["Demo_Batch"]:::misc
            DemoConfig["Config.toml"]:::misc
        end
        OpenSimSetup["OpenSim_Setup"]:::misc
        MarkerLSTM["MarkerAugmenter LSTM ONNX"]:::misc
    end

    %% Flows: Inputs to Orchestrator
    RawVideo --> Orchestrator
    CalibData --> Orchestrator
    PreKey --> Orchestrator
    Config --> Orchestrator

    %% Orchestration to Pipeline
    Orchestrator --> CalibM
    Orchestrator --> PoseEst
    Orchestrator --> Sync
    Orchestrator --> PersonAssoc
    Orchestrator --> Triang
    Orchestrator --> Filtering
    Orchestrator --> MarkerAug
    Orchestrator --> Kinematics

    %% Pipeline flows
    CalibM --> CalibOut
    CalibM --> OpenCV
    CalibOut --> PoseEst
    RawVideo --> PoseEst
    PoseEst --> JSON2D
    PoseEst --> ONNX
    PoseEst --> OPose
    JSON2D --> Sync
    Sync --> JSONSync
    JSONSync --> PersonAssoc
    PersonAssoc --> Triang
    PersonAssoc -->|"uses"| CalibOut
    Triang --> TRCRaw
    Triang --> CalibOut
    Triang --> OpenCV
    TRCRaw --> Filtering
    Filtering --> TRCFilter
    TRCFilter --> MarkerAug
    MarkerAug --> TRCAug
    MarkerAug --> MarkerLSTM
    TRCAug --> Kinematics
    Kinematics --> OSIMModel
    Kinematics --> MOT
    Kinematics --> OSIMAPI

    %% Visualization flows
    OSIMModel --> OpenSimGUI
    MOT --> OpenSimGUI
    TRCAug --> BlenderAddOn

    %% Config feeding all modules
    Config --> CalibM
    Config --> PoseEst
    Config --> Sync
    Config --> PersonAssoc
    Config --> Triang
    Config --> Filtering
    Config --> MarkerAug
    Config --> Kinematics

    %% Demos & Utilities usage
    Common --> Orchestrator
    Skeletons --> PoseEst
    Utilities --> PoseEst
    Utilities --> Export["Conversion"] 
    DemoSingle --> RawVideo
    DemoSingle --> CalibData
    DemoMulti --> RawVideo
    DemoMulti --> CalibData
    DemoBatch --> Config
    OpenSimSetup --> Kinematics

    %% Click Events
    click Orchestrator "https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Pose2Sim.py"
    click CalibM "https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/calibration.py"
    click PoseEst "https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/poseEstimation.py"
    click Sync "https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/synchronization.py"
    click PersonAssoc "https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/personAssociation.py"
    click Triang "https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/triangulation.py"
    click Filtering "https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/filtering.py"
    click MarkerAug "https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/markerAugmentation.py"
    click Kinematics "https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/kinematics.py"
    click Common "https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/common.py"
    click Skeletons "https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/skeletons.py"
    click DemoConfig "https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Demo_Batch/Config.toml"
    click DemoSingle "https://github.com/perfanalytics/pose2sim/tree/main/Pose2Sim/Demo_SinglePerson"
    click DemoMulti "https://github.com/perfanalytics/pose2sim/tree/main/Pose2Sim/Demo_MultiPerson"
    click DemoBatch "https://github.com/perfanalytics/pose2sim/tree/main/Pose2Sim/Demo_Batch"
    click Utilities "https://github.com/perfanalytics/pose2sim/tree/main/Pose2Sim/Utilities/"
    click OpenSimSetup "https://github.com/perfanalytics/pose2sim/tree/main/Pose2Sim/OpenSim_Setup/"
    click MarkerLSTM "https://github.com/perfanalytics/pose2sim/tree/main/Pose2Sim/MarkerAugmenter/LSTM/"

    %% Styles
    classDef inputs fill:#a8e6cf,stroke:#34495e,color:#34495e;
    classDef pipeline fill:#81cfe0,stroke:#1b4f72,color:#1b4f72;
    classDef data fill:#be90d4,stroke:#512e5f,color:#512e5f;
    classDef external fill:#f5b041,stroke:#874b0f,color:#874b0f;
    classDef viz fill:#76d7c4,stroke:#117a65,color:#117a65;
    classDef misc fill:#d2d7d3,stroke:#424949,color:#424949;
```
