How facebookresearch/emg2pose works
Main Components
a. Data Management & Ingestion:
• HDF5 dataset files (full dataset and mini/sanity-check dataset)
• Metadata (CSV file downloads and metadata configuration)
• Data loaders defined in “emg2pose/data.py”, “emg2pose/UmeTrack/dataset”, and utility functions (async, batched dataset, etc. in “lib/data_utils” and “lib/batched_dataset”)
b. Configuration & Experiment Management:
• Numerous YAML config files under the “config” directory that define base settings, data split, experiment parameters, network settings, optimizer configuration, transforms, and specific module configurations (e.g., pose_module with decoders, stateful settings)
c. Model and Neural Network Architecture:
• Model scripts and definitions in “emg2pose/lib/models” (including backbone_resnet, feature_extractor, regressor, skeleton_encoder, temporal processing, and the integrated “umetrack_model”)
• Additional pose modules and transformation pipelines in “emg2pose/pose_modules.py” and “transforms.py”
d. Training, Evaluation, and Testing Pipelines:
• The main entry point in “emg2pose/train.py” facilitates training and evaluation using command-line arguments (e.g., experiment selection, checkpoint usage)
• Test and analysis scripts such as “emg2pose/test_analysis.py” and evaluation scripts for known/unknown skeletons, showcasing a clear separation between training and evaluation phases.
e. Kinematics & Visualization (UmeTrack):
• The “emg2pose/UmeTrack” module includes code for hand mesh skinning, forward kinematics, and additional evaluation utilities
• Visualization tools spread across “emg2pose/visualization.py” and supplemented by Jupyter Notebook “notebooks/getting_started.ipynb”
f. External Dependencies and Environment:
• Environment specifications in “environment.yml” show necessary libraries
• Setup files (setup.py, setup.cfg) indicate the package structure and installation process.
Relationships and Interactions
• Data flows start with the dataset file (HDF5 inputs) and metadata, then move through the data loader utilities and data transformation modules.
• The Configuration layer reads YAML files to setup experiments, influencing:
• Data (data splits, dataset transforms)
• Model (network architectures, optimizer settings)
• Experiment specifics (tracking vs. regression experiments)
• The training pipeline invokes the model training modules, which in turn rely on the data loader and configuration settings.
• After training, evaluation components are triggered which use checkpoints saved during training.
• An independent UmeTrack module integrates for post-processing like forward kinematics and visualization, providing results which are then optionally examined in notebooks.
Architectural Patterns & Design Principles
• Modularity: Components are separated into dedicated directories (config, data_utils, models, transforms), making extensions or modifications easier.
• Config-Driven Experimentation: The use of YAML files to define experiments, network settings, and data splits promotes reusability and clarity.
• Separation of Concerns: The clear demarcation between data ingestion, model training, experiment execution, and kinematic/visual evaluation simplifies debugging and further development.
• Integration of External Modules: UmeTrack as an integrated sub-module demonstrates design flexibility allowing additional functionalities (e.g., mesh skinning) to plug into the main pipeline.
Relevant Technologies, Frameworks, & Libraries
• Python for overall orchestration.
• PyTorch (implied by model structure and “pytorch3d_transforms_so3.py”) for deep learning model implementations.
• YAML for configuration.
• HDF5 for dataset storage.
• Jupyter Notebooks for visualization and exploratory data analysis.
• Conda for environment management.
• CI/CD workflows via the “.github/workflows” (main.yml) for continuous integration.
Putting It All Together
• Move sequentially to the data processing block (data loader, data transformation modules).
• Branch out to the training module (which includes model definitions, experiments configured via YAML, training pipelines).
• Finally, illustrate the integration of the UmeTrack module and notebook-based visualization as the final outputs.
• Component includes annotations on key technologies (e.g., “PyTorch Model”, “HDF5 Data Loader”).