# stanfordnmbl/opencap-core architecture

> Architecture diagram of the GitHub repository stanfordnmbl/opencap-core, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stanfordnmbl/opencap-core
- Repository: https://github.com/stanfordnmbl/opencap-core

## Overview

##### Project Type and Purpose
- The project is a full-stack data processing and analysis pipeline focused on 3D motion capture using smartphone videos.
- Its purpose is to take input videos, estimate 3D positions using pose estimation (via OpenPose and mmpose), augment marker data (via LSTM-based MarkerAugmenter), and then compute human movement kinematics/dynamics using OpenSim.
- It supports both cloud-based processing and the option to run locally, with scripts, Docker configurations, and utility modules provided.

##### Main Components
a. Data Ingestion & Input:
– Input Videos: Videos captured from iOS devices (as indicated in the README) or other synchronized sources.
– Camera Calibration Module: Represented by the “CameraIntrinsics” directory that contains camera calibration data for many device models.
b. Pose Estimation Engine:
– Two parallel modules using different frameworks:
- “openpose” module (with its loop and requirements files) for pose estimation.
- “mmpose” module (with several utility scripts and configurations) also used for pose estimation.
c. Marker Augmentation:
– The “MarkerAugmenter” component that includes LSTM models for posture/marker augmentation. Different LSTM versions (v0.0, v0.1_lower, v0.1_upper, etc.) show multiple iterations for accuracy.
d. Biomechanical Processing (OpenSim Pipeline):
– The “opensimPipeline” directory houses several modules:
- Inverse Kinematics (IK) setups, scaling configurations, and different model specifications.
– This is where the processed marker data is converted into joint angles and kinematics.
e. API and Utility Modules:
– Various “utils*.py” files (e.g., utilsServer.py, utilsCameraPy3.py, utilsMMpose.py, utilsOpenSim.py) organize helper functions, backend API interaction, and data post-process analysis.
f. Orchestration & Execution:
– Main scripts (app.py, main.py, main_calcIntrinsics.py) tie together the components into an execution workflow.
– Docker configurations (docker/, .github/workflows) and CI/CD integration are provided for containerized deployment and automated testing.
g. External Dependencies and Services:
– Integration with cloud services (via authentication and session management modules found in Examples and the README authentication instructions).
– ffmpeg and external build dependencies (OpenPose build from downloaded dependencies).

##### Architectural Relationships and Data Flow
- Videos (input) and metadata are first fed into the camera calibration module that uses the precomputed “cameraIntrinsics.pickle” files.
- The video data is then passed in parallel (or as a selection option) to the pose estimation engines:
– One branch for OpenPose and one for mmpose, indicating possibility for fallback or comparative analysis.
- Once 2D/3D marker positions are calculated, the MarkerAugmenter module uses LSTM models to refine or augment marker predictions.
- The refined marker data is then forwarded to the OpenSim pipeline modules, where scaling, inverse kinematics (IK), and other biomechanical computations take place.
- Processed results (e.g., joint angles, kinematic curves) can be post-processed (via utilsResultsAnalysis.py and similar modules) and either displayed or fed back into the cloud database.
- API/Server modules (like utilsServer.py) handle any communication with the cloud application, ensuring that results can be visualized at app.opencap.ai.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration & Execution Layer
    OE("Main Orchestration"):::orchestration
    CD("Containerization & CI/CD"):::orchestration

    %% Input Layer
    VI("Video Input"):::input
    CC("Camera Calibration"):::input

    %% Pose Estimation Layer Subgraph
    subgraph "Pose Estimation Engine"
        OP("OpenPose Module"):::processing
        MP("mmpose Module"):::processing
    end

    %% Augmentation Layer
    MA("LSTM Marker Augmenter"):::augmentation

    %% Biomechanical Processing Layer
    OS("OpenSim Pipeline"):::biomech

    %% API & Utility Layer
    API("API & Utility Modules"):::api

    %% Utility Files Subgraph
    subgraph "Utility Files"
        U1("utilsServer.py"):::api
        U2("utilsAPI.py"):::api
        U3("utilsCameraPy3.py"):::api
        U4("utilsMMpose.py"):::api
        U5("utilsOpenSim.py"):::api
        U6("utilsResultsAnalysis.py"):::api
    end

    %% Cloud Integration Node
    CI("Cloud API/Results Integration"):::api

    %% Data Flow
    OE --> VI
    VI --> CC
    CC --> OP
    CC --> MP
    OP --> MA
    MP --> MA
    MA --> OS
    OS --> API
    API --> CI
    OS --> U1
    OS --> U2
    OS --> U3
    OS --> U4
    OS --> U5
    OS --> U6
    OE --> CD

    %% Click Events for Component Mapping
    click CC "https://github.com/stanfordnmbl/opencap-core/tree/main/CameraIntrinsics"
    click OE "https://github.com/stanfordnmbl/opencap-core/blob/main/main.py"
    click OE "https://github.com/stanfordnmbl/opencap-core/blob/main/main_calcIntrinsics.py"
    click OP "https://github.com/stanfordnmbl/opencap-core/blob/main/openpose/loop_openpose.py"
    click OP "https://github.com/stanfordnmbl/opencap-core/blob/main/openpose/requirements.txt"
    click MP "https://github.com/stanfordnmbl/opencap-core/blob/main/mmpose/loop_mmpose.py"
    click MP "https://github.com/stanfordnmbl/opencap-core/blob/main/mmpose/hrnet_w48_coco_wholebody_384x288_dark_plus.py"
    click MA "https://github.com/stanfordnmbl/opencap-core/tree/main/MarkerAugmenter/"
    click OS "https://github.com/stanfordnmbl/opencap-core/tree/main/opensimPipeline/"
    click U1 "https://github.com/stanfordnmbl/opencap-core/blob/main/utilsServer.py"
    click U2 "https://github.com/stanfordnmbl/opencap-core/blob/main/utilsAPI.py"
    click U3 "https://github.com/stanfordnmbl/opencap-core/blob/main/utilsCameraPy3.py"
    click U4 "https://github.com/stanfordnmbl/opencap-core/blob/main/utilsMMpose.py"
    click U5 "https://github.com/stanfordnmbl/opencap-core/blob/main/utilsOpenSim.py"
    click U6 "https://github.com/stanfordnmbl/opencap-core/blob/main/utilsResultsAnalysis.py"
    click CD "https://github.com/stanfordnmbl/opencap-core/tree/main/docker/"
    click CD "https://github.com/stanfordnmbl/opencap-core/tree/main/.github/workflows"

    %% Style Definitions
    classDef input fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef processing fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef augmentation fill:#87CEFA,stroke:#000,stroke-width:1px;
    classDef biomech fill:#98FB98,stroke:#000,stroke-width:1px;
    classDef api fill:#FFA07A,stroke:#000,stroke-width:1px;
    classDef orchestration fill:#FFD700,stroke:#000,stroke-width:1px;
```
