How facebookresearch/motion_rolling_prediction works
1. Project purpose
- This repository implements Rolling Prediction Models (RPM) for real-time full-body motion generation from sparse XR inputs.
- The core problem is converting headset + controller or hand-tracking signals into plausible body motion, while handling missing or unreliable input and switching smoothly between tracking mode and synthesis mode.
2. Top-level architecture
- The repo is organized around a standard ML pipeline:
- raw dataset preparation
- processed dataset generation
- training
- evaluation / generation
- visualization
- The main runtime is PyTorch, with model training and inference wrapped by custom utilities rather than a larger framework.
3. Data preparation boundary
- prepare_data/ contains dataset-specific splits and metadata for:
- AMASS Phase 1
- AMASS Phase 2
- GORP
- The scripts prepare_data.py and prepare_data_gorp.py appear to convert raw motion sources into the format consumed by the model.
- This is a clear boundary between raw motion capture / VR recordings and the normalized training representation.
4. Processed dataset layer
- datasets_processed/ stores the model-ready artifacts:
- per-dataset mean/std normalization tensors
- evaluation gap configs describing missing-input patterns
- normalized motion data in new_format_data/
- This layer is what training and evaluation operate on, so the architecture depends on a precomputed preprocessing stage rather than on-the-fly dataset transformation.
5. Input representation and sparsity handling
- data_loaders/dataloader.py likely constructs batches of motion windows plus sparse observation masks.
- The model/maskers.py module indicates explicit modeling of which input signals are present, missing, or partially observed.
- This is central to the system: the model is not only predicting motion, but also reasoning over observation availability.
6. Core model stack
- model/mdm_model.py suggests the system builds on or adapts MDM-style motion generation.
- model/model_wrapper.py likely adapts the base model to RPM-specific input/output conventions and training objectives.
- rolling/rolling_model.py is the distinctive runtime component: it implements the online rolling prediction behavior, i.e., generating motion incrementally over time and maintaining continuity across successive windows.
7. Training subsystem
- train.py is the entry point for fitting the model.
- runner/training_loop.py centralizes the epoch/iteration orchestration, checkpointing, and optimization flow.
- rolling/fp16_util.py and rolling/logger.py suggest support for mixed-precision training and structured logging.
- The training boundary is therefore: processed dataset → loader → model wrapper/rolling model → training loop → checkpoints/logs.
8. Evaluation and generation subsystem
- test.py drives evaluation.
- evaluation/evaluation.py, evaluation/generators.py, evaluation/utils.py, and evaluation/visualization.py form a separate inference pipeline for:
- generating motions from benchmark inputs
- scoring results with metrics
- visualizing predictions
- datasets_processed/*/eval_gap_configs/ indicates evaluation is organized around specific missing-data or gap scenarios, not just generic test splits.
9. Metrics and analysis
- utils/metrics.py and utils/utils_visualize.py support quantitative and qualitative assessment.
- The system appears to compare predictions in terms of motion fidelity, smoothness, and robustness to missing inputs, which aligns with the paper’s focus on seamless transitions between tracking and synthesis.
10. Geometry and rotation utilities
- utils/rotation_conversions.py, utils/utils_transform.py, and related helpers provide the motion-geometry layer needed to move between representation spaces.
- This is important because body motion pipelines often convert among rotations, joint positions, and model parameterizations.
11. Dataset-specific focus: AMASS and GORP
- The repository clearly supports two major data families:
- AMASS for broader motion prior training / synthetic evaluation
- GORP for real VR input with paired ground-truth body motion
- GORP is the practical target domain: sparse, noisy, and temporally missing input from real headset/controller or hand-tracking hardware.
12. External dependencies and rendering stack
- The README requires local copies of human_body_prior, body_visualizer, and the SMPL-X body model.
- These dependencies matter architecturally because they provide:
- the body parameterization and kinematics
- mesh/pose visualization
- human-body model assets used throughout training and rendering
- So the repo is not self-contained; it relies on external body-model tooling.
13. Configuration and reproducibility
- environment.yaml defines the runtime environment.
- utils/parser_util.py, utils/model_util.py, utils/dist_util.py, and utils/constants.py centralize configuration, device/distributed setup, and model construction.
- The architecture therefore separates experiment setup from the model and data logic.
14. Overall data flow
- The high-level flow is:
- raw motion / VR recordings are split and prepared
- processed datasets are normalized and packaged
- loaders feed sparse observations and masks into the RPM model
- the rolling model predicts motion online, handling both present and missing inputs
- evaluation modules generate benchmark outputs and visualizations
- In diagram form, the key boundaries are data preprocessing, model training/inference, and evaluation/visualization.
Main entry points: dataloader.py Dataloader · training_loop.py Training loop · evaluation.py Evaluation core · model_wrapper.py Model wrapper
Components 21
Data prep
Raw dataset splits and conversion into model-ready motion records.
- Prepare data · Main preprocessing entrypoint for raw motion sources.
prepare_data.py - AMASS P1 splits · Phase 1 AMASS split metadata used during preprocessing.
prepare_data/amass_p1 - AMASS P2 splits · Phase 2 AMASS split metadata used during preprocessing.
prepare_data/amass_p2 - GORP splits · Real-input GORP train and test split definitions.
prepare_data/gorp - Prepare GORP · Dataset-specific preprocessing for GORP recordings.
prepare_data_gorp.py
Processed data
Normalized artifacts and evaluation gap configs consumed by training and testing.
- AMASS P1 data · Processed AMASS Phase 1 statistics and serialized motion data.
datasets_processed/amass_p1 - AMASS P2 data · Processed AMASS Phase 2 statistics and serialized motion data.
datasets_processed/amass_p2 - GORP data · Processed GORP statistics and serialized motion data.
datasets_processed/gorp
Core runtime
Training, rolling inference, and model components.
- Dataloader · Builds motion windows, sparse observations, and masks for training and evaluation.
data_loaders/dataloader.py - Maskers · Encodes which inputs are present, missing, or partially observed.
model/maskers.py - MDM model · Core motion generation network adapted from MDM-style modeling.
model/mdm_model.py - Model wrapper · Adapts the base model to RPM inputs, outputs, and training objectives.
model/model_wrapper.py - Rolling model · Implements incremental rolling prediction and continuity across windows.
rolling/rolling_model.py - Train entry · Main training script.
train.py - Training loop · Coordinates epochs, checkpoints, and optimization steps.
runner/training_loop.py
Evaluation
Benchmark generation, scoring, and visualization.
- Test entry · Main evaluation and generation script.
test.py - Evaluation core · Generates benchmark outputs and computes results.
evaluation/evaluation.py - Generators · Constructs evaluation-time motion generation workflows.
evaluation/generators.py - Visualization · Evaluation-time qualitative visualization of predictions.
evaluation/visualization.py
Shared utilities
Configuration, geometry, metrics, and external body-model support.
- Rotation utils · Rotation conversions and motion representation transforms.
utils/rotation_conversions.py - Model utils · Configuration, constants, distributed setup, and model construction helpers.
utils/model_util.py
Connections 22
- Prepare data → AMASS P1 splits: prepares · Processes AMASS Phase 1 sources using split metadata.
- Prepare data → AMASS P2 splits: prepares · Processes AMASS Phase 2 sources using split metadata.
- Prepare GORP → GORP splits: prepares · Converts GORP raw recordings into training-ready format.
- AMASS P1 splits → AMASS P1 data: feeds · Raw split definitions lead to processed AMASS P1 artifacts.
- AMASS P2 splits → AMASS P2 data: feeds · Raw split definitions lead to processed AMASS P2 artifacts.
- GORP splits → GORP data: feeds · GORP splits lead to processed GORP artifacts.
- AMASS P1 data → Dataloader: loads · Training and evaluation read AMASS P1 normalized data.
- AMASS P2 data → Dataloader: loads · Training and evaluation read AMASS P2 normalized data.
- GORP data → Dataloader: loads · Training and evaluation read GORP normalized data.
- Dataloader → Maskers: masks · Loader produces sparse observation patterns for the model.
- Maskers → Model wrapper: conditions · Observation availability shapes wrapper inputs and objectives.
- Model wrapper → MDM model: builds · Wrapper drives the underlying motion generation network.
- MDM model → Rolling model: powers · Core model is used inside rolling online inference.
- Train entry → Training loop: runs · Training entry invokes the training loop.
- Training loop → Dataloader: consumes · The loop iterates over data batches from the loader.
- Training loop → Model wrapper: optimizes · The loop trains the wrapped model.
- Test entry → Evaluation core: drives · Evaluation entry invokes benchmarking and generation.
- Evaluation core → Generators: uses · Evaluation core delegates sample generation.
- Evaluation core → Visualization: renders · Evaluation core produces visualization outputs.
- Generators → Rolling model: samples · Evaluation generation uses the rolling inference model.
- Training loop → Model utils: configures · Training depends on shared model/configuration utilities.
- Evaluation core → Rotation utils: transforms · Evaluation and scoring rely on rotation and transform utilities.