# facebookresearch/motion_rolling_prediction architecture

> Architecture diagram of the GitHub repository facebookresearch/motion_rolling_prediction, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/motion_rolling_prediction
- Repository: https://github.com/facebookresearch/motion_rolling_prediction

## Overview

1. **Project purpose**
   - This repository implements **Rolling Prediction Models (RPM)** for **real-time full-body motion generation** from sparse XR inputs.
   - The core problem is converting **headset + controller or hand-tracking signals** into plausible body motion, while handling **missing or unreliable input** and switching smoothly between **tracking mode** and **synthesis mode**.

2. **Top-level architecture**
   - The repo is organized around a standard ML pipeline:
     - **raw dataset preparation**
     - **processed dataset generation**
     - **training**
     - **evaluation / generation**
     - **visualization**
   - The main runtime is **PyTorch**, with model training and inference wrapped by custom utilities rather than a larger framework.

3. **Data preparation boundary**
   - `prepare_data/` contains dataset-specific splits and metadata for:
     - **AMASS Phase 1**
     - **AMASS Phase 2**
     - **GORP**
   - The scripts `prepare_data.py` and `prepare_data_gorp.py` appear to convert raw motion sources into the format consumed by the model.
   - This is a clear boundary between **raw motion capture / VR recordings** and the **normalized training representation**.

4. **Processed dataset layer**
   - `datasets_processed/` stores the model-ready artifacts:
     - per-dataset **mean/std normalization tensors**
     - **evaluation gap configs** describing missing-input patterns
     - normalized motion data in `new_format_data/`
   - This layer is what training and evaluation operate on, so the architecture depends on a **precomputed preprocessing stage** rather than on-the-fly dataset transformation.

5. **Input representation and sparsity handling**
   - `data_loaders/dataloader.py` likely constructs batches of motion windows plus sparse observation masks.
   - The `model/maskers.py` module indicates explicit modeling of **which input signals are present, missing, or partially observed**.
   - This is central to the system: the model is not only predicting motion, but also reasoning over **observation availability**.

6. **Core model stack**
   - `model/mdm_model.py` suggests the system builds on or adapts **MDM-style motion generation**.
   - `model/model_wrapper.py` likely adapts the base model to RPM-specific input/output conventions and training objectives.
   - `rolling/rolling_model.py` is the distinctive runtime component: it implements the **online rolling prediction behavior**, i.e., generating motion incrementally over time and maintaining continuity across successive windows.

7. **Training subsystem**
   - `train.py` is the entry point for fitting the model.
   - `runner/training_loop.py` centralizes the epoch/iteration orchestration, checkpointing, and optimization flow.
   - `rolling/fp16_util.py` and `rolling/logger.py` suggest support for **mixed-precision training** and structured logging.
   - The training boundary is therefore: **processed dataset → loader → model wrapper/rolling model → training loop → checkpoints/logs**.

8. **Evaluation and generation subsystem**
   - `test.py` drives evaluation.
   - `evaluation/evaluation.py`, `evaluation/generators.py`, `evaluation/utils.py`, and `evaluation/visualization.py` form a separate inference pipeline for:
     - generating motions from benchmark inputs
     - scoring results with metrics
     - visualizing predictions
   - `datasets_processed/*/eval_gap_configs/` indicates evaluation is organized around **specific missing-data or gap scenarios**, not just generic test splits.

9. **Metrics and analysis**
   - `utils/metrics.py` and `utils/utils_visualize.py` support quantitative and qualitative assessment.
   - The system appears to compare predictions in terms of **motion fidelity, smoothness, and robustness to missing inputs**, which aligns with the paper’s focus on seamless transitions between tracking and synthesis.

10. **Geometry and rotation utilities**
   - `utils/rotation_conversions.py`, `utils/utils_transform.py`, and related helpers provide the motion-geometry layer needed to move between representation spaces.
   - This is important because body motion pipelines often convert among **rotations, joint positions, and model parameterizations**.

11. **Dataset-specific focus: AMASS and GORP**
   - The repository clearly supports two major data families:
     - **AMASS** for broader motion prior training / synthetic evaluation
     - **GORP** for real VR input with paired ground-truth body motion
   - GORP is the practical target domain: sparse, noisy, and temporally missing input from real headset/controller or hand-tracking hardware.

12. **External dependencies and rendering stack**
   - The README requires local copies of **human_body_prior**, **body_visualizer**, and the **SMPL-X** body model.
   - These dependencies matter architecturally because they provide:
     - the body parameterization and kinematics
     - mesh/pose visualization
     - human-body model assets used throughout training and rendering
   - So the repo is not self-contained; it relies on external body-model tooling.

13. **Configuration and reproducibility**
   - `environment.yaml` defines the runtime environment.
   - `utils/parser_util.py`, `utils/model_util.py`, `utils/dist_util.py`, and `utils/constants.py` centralize configuration, device/distributed setup, and model construction.
   - The architecture therefore separates **experiment setup** from the model and data logic.

14. **Overall data flow**
   - The high-level flow is:
     - raw motion / VR recordings are split and prepared
     - processed datasets are normalized and packaged
     - loaders feed sparse observations and masks into the RPM model
     - the rolling model predicts motion online, handling both present and missing inputs
     - evaluation modules generate benchmark outputs and visualizations
   - In diagram form, the key boundaries are **data preprocessing**, **model training/inference**, and **evaluation/visualization**.

Main entry points: [`data_loaders/dataloader.py`](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/data_loaders/dataloader.py) (Dataloader), [`runner/training_loop.py`](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/runner/training_loop.py) (Training loop), [`evaluation/evaluation.py`](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/evaluation/evaluation.py) (Evaluation core), [`model/model_wrapper.py`](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/model/model_wrapper.py) (Model wrapper)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g1["Data prep"]
  node_n1["Prepare data<br/>conversion entry<br/>[prepare_data.py]"]
  node_n2["AMASS P1 splits<br/>dataset splits"]
  node_n3["AMASS P2 splits<br/>dataset splits"]
  node_n4["GORP splits<br/>dataset splits"]
  node_n5["Prepare GORP<br/>conversion script"]
end

subgraph group_g2["Processed data"]
  node_n6[("AMASS P1 data<br/>normalized dataset")]
  node_n7[("AMASS P2 data<br/>normalized dataset")]
  node_n8[("GORP data<br/>normalized dataset")]
end

subgraph group_g3["Core runtime"]
  node_n9["Dataloader<br/>batch loader<br/>[dataloader.py]"]
  node_n10["Maskers<br/>sparsity logic<br/>[maskers.py]"]
  node_n11["MDM model<br/>base motion model<br/>[mdm_model.py]"]
  node_n12["Model wrapper<br/>adapter<br/>[model_wrapper.py]"]
  node_n13["Rolling model<br/>online inference<br/>[rolling_model.py]"]
  node_n14["Train entry<br/>training entry<br/>[train.py]"]
  node_n15["Training loop<br/>optimization loop<br/>[training_loop.py]"]
end

subgraph group_g4["Evaluation"]
  node_n16["Test entry<br/>evaluation entry<br/>[test.py]"]
  node_n17["Evaluation core<br/>metrics pipeline<br/>[evaluation.py]"]
  node_n18["Generators<br/>inference drivers<br/>[generators.py]"]
  node_n19["Visualization<br/>rendering<br/>[visualization.py]"]
end

subgraph group_g5["Shared utilities"]
  node_n20["Rotation utils<br/>geometry"]
  node_n21["Model utils<br/>setup<br/>[model_util.py]"]
end

node_n1 -->|"prepares"| node_n2
node_n1 -->|"prepares"| node_n3
node_n5 -->|"prepares"| node_n4
node_n2 -->|"feeds"| node_n6
node_n3 -->|"feeds"| node_n7
node_n4 -->|"feeds"| node_n8
node_n6 -->|"loads"| node_n9
node_n7 -->|"loads"| node_n9
node_n8 -->|"loads"| node_n9
node_n9 -->|"masks"| node_n10
node_n10 -->|"conditions"| node_n12
node_n12 -->|"builds"| node_n11
node_n11 -->|"powers"| node_n13
node_n14 -->|"runs"| node_n15
node_n15 -->|"consumes"| node_n9
node_n15 -->|"optimizes"| node_n12
node_n16 -->|"drives"| node_n17
node_n17 -->|"uses"| node_n18
node_n17 -->|"renders"| node_n19
node_n18 -->|"samples"| node_n13
node_n15 -->|"configures"| node_n21
node_n17 -->|"transforms"| node_n20

click node_n1 "https://github.com/facebookresearch/motion_rolling_prediction/blob/main/prepare_data.py"
click node_n2 "https://github.com/facebookresearch/motion_rolling_prediction/tree/main/prepare_data/amass_p1"
click node_n3 "https://github.com/facebookresearch/motion_rolling_prediction/tree/main/prepare_data/amass_p2"
click node_n4 "https://github.com/facebookresearch/motion_rolling_prediction/tree/main/prepare_data/gorp"
click node_n5 "https://github.com/facebookresearch/motion_rolling_prediction/blob/main/prepare_data_gorp.py"
click node_n6 "https://github.com/facebookresearch/motion_rolling_prediction/tree/main/datasets_processed/amass_p1"
click node_n7 "https://github.com/facebookresearch/motion_rolling_prediction/tree/main/datasets_processed/amass_p2"
click node_n8 "https://github.com/facebookresearch/motion_rolling_prediction/tree/main/datasets_processed/gorp"
click node_n9 "https://github.com/facebookresearch/motion_rolling_prediction/blob/main/data_loaders/dataloader.py"
click node_n10 "https://github.com/facebookresearch/motion_rolling_prediction/blob/main/model/maskers.py"
click node_n11 "https://github.com/facebookresearch/motion_rolling_prediction/blob/main/model/mdm_model.py"
click node_n12 "https://github.com/facebookresearch/motion_rolling_prediction/blob/main/model/model_wrapper.py"
click node_n13 "https://github.com/facebookresearch/motion_rolling_prediction/blob/main/rolling/rolling_model.py"
click node_n14 "https://github.com/facebookresearch/motion_rolling_prediction/blob/main/train.py"
click node_n15 "https://github.com/facebookresearch/motion_rolling_prediction/blob/main/runner/training_loop.py"
click node_n16 "https://github.com/facebookresearch/motion_rolling_prediction/blob/main/test.py"
click node_n17 "https://github.com/facebookresearch/motion_rolling_prediction/blob/main/evaluation/evaluation.py"
click node_n18 "https://github.com/facebookresearch/motion_rolling_prediction/blob/main/evaluation/generators.py"
click node_n19 "https://github.com/facebookresearch/motion_rolling_prediction/blob/main/evaluation/visualization.py"
click node_n20 "https://github.com/facebookresearch/motion_rolling_prediction/blob/main/utils/rotation_conversions.py"
click node_n21 "https://github.com/facebookresearch/motion_rolling_prediction/blob/main/utils/model_util.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n1,node_n2,node_n3,node_n4,node_n5 toneBlue
class node_n6,node_n7,node_n8 toneAmber
class node_n9,node_n10,node_n11,node_n12,node_n13,node_n14,node_n15 toneMint
class node_n16,node_n17,node_n18,node_n19 toneRose
class node_n20,node_n21 toneIndigo
```

## Components

### Data prep

Raw dataset splits and conversion into model-ready motion records.

- [Prepare data](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/prepare_data.py): conversion entry · Main preprocessing entrypoint for raw motion sources. · `prepare_data.py`
- [AMASS P1 splits](https://github.com/facebookresearch/motion_rolling_prediction/tree/main/prepare_data/amass_p1): dataset splits · Phase 1 AMASS split metadata used during preprocessing. · `prepare_data/amass_p1`
- [AMASS P2 splits](https://github.com/facebookresearch/motion_rolling_prediction/tree/main/prepare_data/amass_p2): dataset splits · Phase 2 AMASS split metadata used during preprocessing. · `prepare_data/amass_p2`
- [GORP splits](https://github.com/facebookresearch/motion_rolling_prediction/tree/main/prepare_data/gorp): dataset splits · Real-input GORP train and test split definitions. · `prepare_data/gorp`
- [Prepare GORP](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/prepare_data_gorp.py): conversion script · Dataset-specific preprocessing for GORP recordings. · `prepare_data_gorp.py`

### Processed data

Normalized artifacts and evaluation gap configs consumed by training and testing.

- [AMASS P1 data](https://github.com/facebookresearch/motion_rolling_prediction/tree/main/datasets_processed/amass_p1): normalized dataset · Processed AMASS Phase 1 statistics and serialized motion data. · `datasets_processed/amass_p1`
- [AMASS P2 data](https://github.com/facebookresearch/motion_rolling_prediction/tree/main/datasets_processed/amass_p2): normalized dataset · Processed AMASS Phase 2 statistics and serialized motion data. · `datasets_processed/amass_p2`
- [GORP data](https://github.com/facebookresearch/motion_rolling_prediction/tree/main/datasets_processed/gorp): normalized dataset · Processed GORP statistics and serialized motion data. · `datasets_processed/gorp`

### Core runtime

Training, rolling inference, and model components.

- [Dataloader](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/data_loaders/dataloader.py): batch loader · Builds motion windows, sparse observations, and masks for training and evaluation. · `data_loaders/dataloader.py`
- [Maskers](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/model/maskers.py): sparsity logic · Encodes which inputs are present, missing, or partially observed. · `model/maskers.py`
- [MDM model](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/model/mdm_model.py): base motion model · Core motion generation network adapted from MDM-style modeling. · `model/mdm_model.py`
- [Model wrapper](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/model/model_wrapper.py): adapter · Adapts the base model to RPM inputs, outputs, and training objectives. · `model/model_wrapper.py`
- [Rolling model](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/rolling/rolling_model.py): online inference · Implements incremental rolling prediction and continuity across windows. · `rolling/rolling_model.py`
- [Train entry](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/train.py): training entry · Main training script. · `train.py`
- [Training loop](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/runner/training_loop.py): optimization loop · Coordinates epochs, checkpoints, and optimization steps. · `runner/training_loop.py`

### Evaluation

Benchmark generation, scoring, and visualization.

- [Test entry](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/test.py): evaluation entry · Main evaluation and generation script. · `test.py`
- [Evaluation core](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/evaluation/evaluation.py): metrics pipeline · Generates benchmark outputs and computes results. · `evaluation/evaluation.py`
- [Generators](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/evaluation/generators.py): inference drivers · Constructs evaluation-time motion generation workflows. · `evaluation/generators.py`
- [Visualization](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/evaluation/visualization.py): rendering · Evaluation-time qualitative visualization of predictions. · `evaluation/visualization.py`

### Shared utilities

Configuration, geometry, metrics, and external body-model support.

- [Rotation utils](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/utils/rotation_conversions.py): geometry · Rotation conversions and motion representation transforms. · `utils/rotation_conversions.py`
- [Model utils](https://github.com/facebookresearch/motion_rolling_prediction/blob/main/utils/model_util.py): setup · Configuration, constants, distributed setup, and model construction helpers. · `utils/model_util.py`

## Connections

- Prepare data → AMASS P1 splits: prepares · Processes AMASS Phase 1 sources using split metadata.
- Prepare data → AMASS P2 splits: prepares · Processes AMASS Phase 2 sources using split metadata.
- Prepare GORP → GORP splits: prepares · Converts GORP raw recordings into training-ready format.
- AMASS P1 splits → AMASS P1 data: feeds · Raw split definitions lead to processed AMASS P1 artifacts.
- AMASS P2 splits → AMASS P2 data: feeds · Raw split definitions lead to processed AMASS P2 artifacts.
- GORP splits → GORP data: feeds · GORP splits lead to processed GORP artifacts.
- AMASS P1 data → Dataloader: loads · Training and evaluation read AMASS P1 normalized data.
- AMASS P2 data → Dataloader: loads · Training and evaluation read AMASS P2 normalized data.
- GORP data → Dataloader: loads · Training and evaluation read GORP normalized data.
- Dataloader → Maskers: masks · Loader produces sparse observation patterns for the model.
- Maskers → Model wrapper: conditions · Observation availability shapes wrapper inputs and objectives.
- Model wrapper → MDM model: builds · Wrapper drives the underlying motion generation network.
- MDM model → Rolling model: powers · Core model is used inside rolling online inference.
- Train entry → Training loop: runs · Training entry invokes the training loop.
- Training loop → Dataloader: consumes · The loop iterates over data batches from the loader.
- Training loop → Model wrapper: optimizes · The loop trains the wrapped model.
- Test entry → Evaluation core: drives · Evaluation entry invokes benchmarking and generation.
- Evaluation core → Generators: uses · Evaluation core delegates sample generation.
- Evaluation core → Visualization: renders · Evaluation core produces visualization outputs.
- Generators → Rolling model: samples · Evaluation generation uses the rolling inference model.
- Training loop → Model utils: configures · Training depends on shared model/configuration utilities.
- Evaluation core → Rotation utils: transforms · Evaluation and scoring rely on rotation and transform utilities.
