How stanfordnmbl/opencap-core works
Project Type and Purpose
• The project is a full-stack data processing and analysis pipeline focused on 3D motion capture using smartphone videos.
• Its purpose is to take input videos, estimate 3D positions using pose estimation (via OpenPose and mmpose), augment marker data (via LSTM-based MarkerAugmenter), and then compute human movement kinematics/dynamics using OpenSim.
• It supports both cloud-based processing and the option to run locally, with scripts, Docker configurations, and utility modules provided.
Main Components
a. Data Ingestion & Input:
– Input Videos: Videos captured from iOS devices (as indicated in the README) or other synchronized sources.
– Camera Calibration Module: Represented by the “CameraIntrinsics” directory that contains camera calibration data for many device models.
b. Pose Estimation Engine:
– Two parallel modules using different frameworks:
• “openpose” module (with its loop and requirements files) for pose estimation.
• “mmpose” module (with several utility scripts and configurations) also used for pose estimation.
c. Marker Augmentation:
– The “MarkerAugmenter” component that includes LSTM models for posture/marker augmentation. Different LSTM versions (v0.0, v0.1_lower, v0.1_upper, etc.) show multiple iterations for accuracy.
d. Biomechanical Processing (OpenSim Pipeline):
– The “opensimPipeline” directory houses several modules:
• Inverse Kinematics (IK) setups, scaling configurations, and different model specifications.
– This is where the processed marker data is converted into joint angles and kinematics.
e. API and Utility Modules:
– Various “utils*.py” files (e.g., utilsServer.py, utilsCameraPy3.py, utilsMMpose.py, utilsOpenSim.py) organize helper functions, backend API interaction, and data post-process analysis.
f. Orchestration & Execution:
– Main scripts (app.py, main.py, main_calcIntrinsics.py) tie together the components into an execution workflow.
– Docker configurations (docker/, .github/workflows) and CI/CD integration are provided for containerized deployment and automated testing.
g. External Dependencies and Services:
– Integration with cloud services (via authentication and session management modules found in Examples and the README authentication instructions).
– ffmpeg and external build dependencies (OpenPose build from downloaded dependencies).
Architectural Relationships and Data Flow
• Videos (input) and metadata are first fed into the camera calibration module that uses the precomputed “cameraIntrinsics.pickle” files.
• The video data is then passed in parallel (or as a selection option) to the pose estimation engines:
– One branch for OpenPose and one for mmpose, indicating possibility for fallback or comparative analysis.
• Once 2D/3D marker positions are calculated, the MarkerAugmenter module uses LSTM models to refine or augment marker predictions.
• The refined marker data is then forwarded to the OpenSim pipeline modules, where scaling, inverse kinematics (IK), and other biomechanical computations take place.
• Processed results (e.g., joint angles, kinematic curves) can be post-processed (via utilsResultsAnalysis.py and similar modules) and either displayed or fed back into the cloud database.
• API/Server modules (like utilsServer.py) handle any communication with the cloud application, ensuring that results can be visualized at app.opencap.ai.