# facebookresearch/emg2pose architecture

> Architecture diagram of the GitHub repository facebookresearch/emg2pose, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/emg2pose
- Repository: https://github.com/facebookresearch/emg2pose

## Overview

##### Main Components
a. Data Management & Ingestion:
- HDF5 dataset files (full dataset and mini/sanity-check dataset)
- Metadata (CSV file downloads and metadata configuration)
- Data loaders defined in “emg2pose/data.py”, “emg2pose/UmeTrack/dataset”, and utility functions (async, batched dataset, etc. in “lib/data_utils” and “lib/batched_dataset”)
b. Configuration & Experiment Management:
- Numerous YAML config files under the “config” directory that define base settings, data split, experiment parameters, network settings, optimizer configuration, transforms, and specific module configurations (e.g., pose_module with decoders, stateful settings)
c. Model and Neural Network Architecture:
- Model scripts and definitions in “emg2pose/lib/models” (including backbone_resnet, feature_extractor, regressor, skeleton_encoder, temporal processing, and the integrated “umetrack_model”)
- Additional pose modules and transformation pipelines in “emg2pose/pose_modules.py” and “transforms.py”
d. Training, Evaluation, and Testing Pipelines:
- The main entry point in “emg2pose/train.py” facilitates training and evaluation using command-line arguments (e.g., experiment selection, checkpoint usage)
- Test and analysis scripts such as “emg2pose/test_analysis.py” and evaluation scripts for known/unknown skeletons, showcasing a clear separation between training and evaluation phases.
e. Kinematics & Visualization (UmeTrack):
- The “emg2pose/UmeTrack” module includes code for hand mesh skinning, forward kinematics, and additional evaluation utilities
- Visualization tools spread across “emg2pose/visualization.py” and supplemented by Jupyter Notebook “notebooks/getting_started.ipynb”
f. External Dependencies and Environment:
- Environment specifications in “environment.yml” show necessary libraries
- Setup files (setup.py, setup.cfg) indicate the package structure and installation process.

##### Relationships and Interactions
- Data flows start with the dataset file (HDF5 inputs) and metadata, then move through the data loader utilities and data transformation modules.
- The Configuration layer reads YAML files to setup experiments, influencing:
- Data (data splits, dataset transforms)
- Model (network architectures, optimizer settings)
- Experiment specifics (tracking vs. regression experiments)
- The training pipeline invokes the model training modules, which in turn rely on the data loader and configuration settings.
- After training, evaluation components are triggered which use checkpoints saved during training.
- An independent UmeTrack module integrates for post-processing like forward kinematics and visualization, providing results which are then optionally examined in notebooks.

##### Architectural Patterns & Design Principles
- Modularity: Components are separated into dedicated directories (config, data_utils, models, transforms), making extensions or modifications easier.
- Config-Driven Experimentation: The use of YAML files to define experiments, network settings, and data splits promotes reusability and clarity.
- Separation of Concerns: The clear demarcation between data ingestion, model training, experiment execution, and kinematic/visual evaluation simplifies debugging and further development.
- Integration of External Modules: UmeTrack as an integrated sub-module demonstrates design flexibility allowing additional functionalities (e.g., mesh skinning) to plug into the main pipeline.

##### Relevant Technologies, Frameworks, & Libraries
- Python for overall orchestration.
- PyTorch (implied by model structure and “pytorch3d_transforms_so3.py”) for deep learning model implementations.
- YAML for configuration.
- HDF5 for dataset storage.
- Jupyter Notebooks for visualization and exploratory data analysis.
- Conda for environment management.
- CI/CD workflows via the “.github/workflows” (main.yml) for continuous integration.

##### Putting It All Together
- Move sequentially to the data processing block (data loader, data transformation modules).
- Branch out to the training module (which includes model definitions, experiments configured via YAML, training pipelines).
- Finally, illustrate the integration of the UmeTrack module and notebook-based visualization as the final outputs.
- Component includes annotations on key technologies (e.g., “PyTorch Model”, “HDF5 Data Loader”).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data & Configuration"
        hdf5["HDF5 Dataset & Metadata"]:::data
        dataLoader["Data Loader & Utils"]:::data
        yamlConfig["YAML Configs"]:::data
        umeDataset["UmeTrack Dataset Loader"]:::aux
    end

    subgraph "Model & Training"
        nnModels["Neural Network Models"]:::model
        poseModules["Pose Modules"]:::model
        trainer["Trainer"]:::model
    end

    subgraph "Evaluation & Testing"
        evaluator["Evaluator & Testing"]:::evaluate
        testUtils["Test Utilities"]:::evaluate
    end

    subgraph "Kinematics & Visualization"
        kinematics["Kinematics & Skinning"]:::aux
        visualization["Visualization Tools"]:::aux
    end

    subgraph "External"
        environment["Environment & Setup"]:::external
    end

    %% Data flows
    hdf5 -->|"input"| dataLoader
    hdf5 -->|"input"| umeDataset
    dataLoader -->|"trainingData"| trainer
    yamlConfig -->|"configures"| nnModels
    yamlConfig -->|"configures"| trainer
    nnModels -->|"model"| trainer
    poseModules -->|"modules"| trainer
    trainer -->|"checkpoints"| evaluator
    trainer -->|"checkpoints"| testUtils
    evaluator -->|"results"| visualization
    kinematics -->|"visualData"| visualization
    umeDataset -->|"specialData"| kinematics
    environment -->|"dependencies"| yamlConfig

    %% Click Events
    click hdf5 "https://github.com/facebookresearch/emg2pose/blob/main/emg2pose/tests/assets/test_data.hdf5"
    click dataLoader "https://github.com/facebookresearch/emg2pose/blob/main/emg2pose/data.py"
    click umeDataset "https://github.com/facebookresearch/emg2pose/tree/main/emg2pose/UmeTrack/dataset"
    click yamlConfig "https://github.com/facebookresearch/emg2pose/tree/main/config"
    click nnModels "https://github.com/facebookresearch/emg2pose/tree/main/emg2pose/UmeTrack/lib/models"
    click poseModules "https://github.com/facebookresearch/emg2pose/blob/main/emg2pose/pose_modules.py"
    click trainer "https://github.com/facebookresearch/emg2pose/blob/main/emg2pose/train.py"
    click evaluator "https://github.com/facebookresearch/emg2pose/blob/main/emg2pose/test_analysis.py"
    click testUtils "https://github.com/facebookresearch/emg2pose/tree/main/emg2pose/tests"
    click kinematics "https://github.com/facebookresearch/emg2pose/blob/main/emg2pose/kinematics.py"
    click visualization "https://github.com/facebookresearch/emg2pose/blob/main/emg2pose/visualization.py"
    click environment "https://github.com/facebookresearch/emg2pose/blob/main/environment.yml"

    %% Styles
    classDef data fill:#ccffcc,stroke:#333,stroke-width:2px;
    classDef model fill:#cce5ff,stroke:#333,stroke-width:2px;
    classDef evaluate fill:#ffe5cc,stroke:#333,stroke-width:2px;
    classDef aux fill:#ffcccc,stroke:#333,stroke-width:2px;
    classDef external fill:#e0e0e0,stroke:#333,stroke-width:2px;
```
