# nvlabs/protomotions architecture

> Architecture diagram of the GitHub repository nvlabs/protomotions, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvlabs/protomotions
- Repository: https://github.com/nvlabs/protomotions

## Overview

ProtoMotions 3 is a GPU-accelerated framework for training, evaluating, and deploying physically simulated humanoid policies. The main workflow is to prepare or retarget motion data, configure a robot and simulation task, train a policy through environment interaction, then evaluate or export it for sim-to-sim or hardware use. The graph also includes procedural scenes and terrain, motion tracking and generative-policy capabilities, and reusable simulator backends. Several subsystem links are documented only at a high level; those are marked as unverified rather than inferred as direct calls.

Main entry points: [`protomotions/envs/base_env/env.py`](https://github.com/nvlabs/protomotions/blob/main/protomotions/envs/base_env/env.py) (Environment), [`protomotions/train_agent.py`](https://github.com/nvlabs/protomotions/blob/main/protomotions/train_agent.py) (Training entry), [`protomotions/agents/ppo/agent.py`](https://github.com/nvlabs/protomotions/blob/main/protomotions/agents/ppo/agent.py) (PPO agent), [`protomotions/inference_agent.py`](https://github.com/nvlabs/protomotions/blob/main/protomotions/inference_agent.py) (Policy inference)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_motion["Motion and scenes"]
  node_converters["Motion conversion"]
  node_retargeting["PyRoki retargeting"]
  node_motion_lib["Motion library<br/>[motion_lib.py]"]
  node_motion_manager["Motion manager<br/>[motion_manager.py]"]
  node_scenes["Scene library<br/>[scene_lib.py]"]
  node_terrain["Terrain generation<br/>[terrain.py]"]
end

subgraph group_simulation["Simulation environment"]
  node_robot_config["Robot configuration<br/>[base.py]"]
  node_environment["Environment<br/>[env.py]"]
  node_context["Environment context<br/>[context_views.py]"]
  node_component_manager["Component execution"]
  node_control_manager["Control manager<br/>[manager.py]"]
  node_sim_factory["Simulator factory<br/>[factory.py]"]
  node_base_simulator["Simulator interface<br/>[simulator.py]"]
  node_isaacgym["Isaac Gym backend<br/>[simulator.py]"]
  node_newton["Newton backend<br/>[simulator.py]"]
end

subgraph group_learning["Policy learning"]
  node_training["Training entry<br/>[train_agent.py]"]
  node_ppo["PPO agent<br/>[agent.py]"]
  node_base_agent["Base agent<br/>[agent.py]"]
  node_amp["AMP learning<br/>[component.py]"]
  node_generative["Generative policies"]
end

subgraph group_deployment["Evaluation and deployment"]
  node_inference["Policy inference<br/>[inference_agent.py]"]
  node_export["ONNX export"]
end

node_researcher(("Researcher"))
node_motion_data["Motion data"]
node_hardware(("Robot hardware"))

node_researcher -->|"configures and trains"| node_training
node_motion_data -.->|"converts"| node_converters
node_motion_data -->|"retargets"| node_retargeting
node_retargeting -.->|"prepares motions"| node_motion_lib
node_converters -.->|"prepares motions"| node_motion_lib
node_motion_lib -->|"samples motion data"| node_motion_manager
node_terrain -->|"provides terrain"| node_scenes
node_training -.->|"starts training"| node_ppo
node_ppo -->|"interacts with"| node_environment
node_ppo -->|"inherits from"| node_base_agent
node_amp -->|"extends"| node_base_agent
node_generative -.->|"trains policy via"| node_training
node_environment -.->|"uses motion sampling"| node_motion_manager
node_environment -.->|"uses scenes"| node_scenes
node_environment -.->|"uses terrain"| node_terrain
node_environment -.->|"manages controls"| node_control_manager
node_environment -.->|"executes components"| node_component_manager
node_component_manager -->|"resolves context"| node_context
node_control_manager -->|"uses context"| node_context
node_robot_config -->|"configures simulator"| node_sim_factory
node_sim_factory -.->|"creates backend"| node_base_simulator
node_isaacgym -->|"implements interface"| node_base_simulator
node_newton -->|"implements interface"| node_base_simulator
node_researcher -->|"runs evaluation"| node_inference
node_inference -->|"runs policy in"| node_environment
node_inference -->|"loads agent"| node_base_agent
node_training -->|"exports policy"| node_export
node_export -->|"deploys policy"| node_hardware

click node_converters "https://github.com/nvlabs/protomotions/tree/main/data/scripts"
click node_retargeting "https://github.com/nvlabs/protomotions/blob/main/pyroki/batch_retarget_to_g1_from_keypoints.py"
click node_motion_lib "https://github.com/nvlabs/protomotions/blob/main/protomotions/components/motion_lib.py"
click node_motion_manager "https://github.com/nvlabs/protomotions/blob/main/protomotions/envs/motion_manager/motion_manager.py"
click node_scenes "https://github.com/nvlabs/protomotions/blob/main/protomotions/components/scene_lib.py"
click node_terrain "https://github.com/nvlabs/protomotions/blob/main/protomotions/components/terrains/terrain.py"
click node_robot_config "https://github.com/nvlabs/protomotions/blob/main/protomotions/robot_configs/base.py"
click node_environment "https://github.com/nvlabs/protomotions/blob/main/protomotions/envs/base_env/env.py"
click node_context "https://github.com/nvlabs/protomotions/blob/main/protomotions/envs/context_views.py"
click node_component_manager "https://github.com/nvlabs/protomotions/blob/main/protomotions/envs/component_manager.py"
click node_control_manager "https://github.com/nvlabs/protomotions/blob/main/protomotions/envs/control/manager.py"
click node_sim_factory "https://github.com/nvlabs/protomotions/blob/main/protomotions/simulator/factory.py"
click node_base_simulator "https://github.com/nvlabs/protomotions/blob/main/protomotions/simulator/base_simulator/simulator.py"
click node_isaacgym "https://github.com/nvlabs/protomotions/blob/main/protomotions/simulator/isaacgym/simulator.py"
click node_newton "https://github.com/nvlabs/protomotions/blob/main/protomotions/simulator/newton/simulator.py"
click node_training "https://github.com/nvlabs/protomotions/blob/main/protomotions/train_agent.py"
click node_ppo "https://github.com/nvlabs/protomotions/blob/main/protomotions/agents/ppo/agent.py"
click node_base_agent "https://github.com/nvlabs/protomotions/blob/main/protomotions/agents/base_agent/agent.py"
click node_amp "https://github.com/nvlabs/protomotions/blob/main/protomotions/agents/amp/component.py"
click node_generative "https://github.com/nvlabs/protomotions/blob/main/protomotions/agents/supervised/masked_mimic_model.py"
click node_inference "https://github.com/nvlabs/protomotions/blob/main/protomotions/inference_agent.py"
click node_export "https://github.com/nvlabs/protomotions/blob/main/deployment/export_tracker_onnx.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_converters,node_retargeting,node_motion_lib,node_motion_manager,node_scenes,node_terrain toneBlue
class node_robot_config,node_environment,node_context,node_component_manager,node_control_manager,node_sim_factory,node_base_simulator,node_isaacgym,node_newton toneAmber
class node_training,node_ppo,node_base_agent,node_amp,node_generative toneMint
class node_inference,node_export toneRose
class node_researcher,node_motion_data,node_hardware toneIndigo
```

## Components

### Motion and scenes

- [Motion conversion](https://github.com/nvlabs/protomotions/tree/main/data/scripts): `data/scripts`
- [PyRoki retargeting](https://github.com/nvlabs/protomotions/blob/main/pyroki/batch_retarget_to_g1_from_keypoints.py): `pyroki/batch_retarget_to_g1_from_keypoints.py`
- [Motion library](https://github.com/nvlabs/protomotions/blob/main/protomotions/components/motion_lib.py): `protomotions/components/motion_lib.py`
- [Motion manager](https://github.com/nvlabs/protomotions/blob/main/protomotions/envs/motion_manager/motion_manager.py): `protomotions/envs/motion_manager/motion_manager.py`
- [Scene library](https://github.com/nvlabs/protomotions/blob/main/protomotions/components/scene_lib.py): `protomotions/components/scene_lib.py`
- [Terrain generation](https://github.com/nvlabs/protomotions/blob/main/protomotions/components/terrains/terrain.py): `protomotions/components/terrains/terrain.py`

### Simulation environment

- [Robot configuration](https://github.com/nvlabs/protomotions/blob/main/protomotions/robot_configs/base.py): `protomotions/robot_configs/base.py`
- [Environment](https://github.com/nvlabs/protomotions/blob/main/protomotions/envs/base_env/env.py): `protomotions/envs/base_env/env.py`
- [Environment context](https://github.com/nvlabs/protomotions/blob/main/protomotions/envs/context_views.py): `protomotions/envs/context_views.py`
- [Component execution](https://github.com/nvlabs/protomotions/blob/main/protomotions/envs/component_manager.py): `protomotions/envs/component_manager.py`
- [Control manager](https://github.com/nvlabs/protomotions/blob/main/protomotions/envs/control/manager.py): `protomotions/envs/control/manager.py`
- [Simulator factory](https://github.com/nvlabs/protomotions/blob/main/protomotions/simulator/factory.py): `protomotions/simulator/factory.py`
- [Simulator interface](https://github.com/nvlabs/protomotions/blob/main/protomotions/simulator/base_simulator/simulator.py): `protomotions/simulator/base_simulator/simulator.py`
- [Isaac Gym backend](https://github.com/nvlabs/protomotions/blob/main/protomotions/simulator/isaacgym/simulator.py): `protomotions/simulator/isaacgym/simulator.py`
- [Newton backend](https://github.com/nvlabs/protomotions/blob/main/protomotions/simulator/newton/simulator.py): `protomotions/simulator/newton/simulator.py`

### Policy learning

- [Training entry](https://github.com/nvlabs/protomotions/blob/main/protomotions/train_agent.py): `protomotions/train_agent.py`
- [PPO agent](https://github.com/nvlabs/protomotions/blob/main/protomotions/agents/ppo/agent.py): `protomotions/agents/ppo/agent.py`
- [Base agent](https://github.com/nvlabs/protomotions/blob/main/protomotions/agents/base_agent/agent.py): `protomotions/agents/base_agent/agent.py`
- [AMP learning](https://github.com/nvlabs/protomotions/blob/main/protomotions/agents/amp/component.py): `protomotions/agents/amp/component.py`
- [Generative policies](https://github.com/nvlabs/protomotions/blob/main/protomotions/agents/supervised/masked_mimic_model.py): `protomotions/agents/supervised/masked_mimic_model.py`

### Evaluation and deployment

- [Policy inference](https://github.com/nvlabs/protomotions/blob/main/protomotions/inference_agent.py): `protomotions/inference_agent.py`
- [ONNX export](https://github.com/nvlabs/protomotions/blob/main/deployment/export_tracker_onnx.py): `deployment/export_tracker_onnx.py`

### Other components

- **Researcher**
- **Motion data**
- **Robot hardware**

## Connections

- Researcher → Training entry: configures and trains (evidence: `README.md`)
- Motion data → Motion conversion: converts
- Motion data → PyRoki retargeting: retargets (evidence: `README.md`)
- PyRoki retargeting → Motion library: prepares motions
- Motion conversion → Motion library: prepares motions
- Motion library → Motion manager: samples motion data (evidence: `protomotions/envs/motion_manager/motion_manager.py`)
- Terrain generation → Scene library: provides terrain (evidence: `protomotions/components/scene_lib.py`)
- Training entry → PPO agent: starts training
- PPO agent → Environment: interacts with (evidence: `protomotions/agents/ppo/agent.py`)
- PPO agent → Base agent: inherits from (evidence: `protomotions/agents/ppo/agent.py`)
- AMP learning → Base agent: extends (evidence: `protomotions/agents/amp/component.py`)
- Generative policies → Training entry: trains policy via
- Environment → Motion manager: uses motion sampling
- Environment → Scene library: uses scenes
- Environment → Terrain generation: uses terrain
- Environment → Control manager: manages controls (evidence: `protomotions/envs/control/manager.py`)
- Environment → Component execution: executes components
- Component execution → Environment context: resolves context (evidence: `protomotions/envs/component_manager.py`)
- Control manager → Environment context: uses context (evidence: `protomotions/envs/control/manager.py`)
- Robot configuration → Simulator factory: configures simulator (evidence: `protomotions/robot_configs/base.py`)
- Simulator factory → Simulator interface: creates backend
- Isaac Gym backend → Simulator interface: implements interface (evidence: `protomotions/simulator/isaacgym/simulator.py`)
- Newton backend → Simulator interface: implements interface (evidence: `protomotions/simulator/newton/simulator.py`)
- Researcher → Policy inference: runs evaluation (evidence: `README.md`)
- Policy inference → Environment: runs policy in (evidence: `protomotions/inference_agent.py`)
- Policy inference → Base agent: loads agent (evidence: `protomotions/inference_agent.py`)
- Training entry → ONNX export: exports policy (evidence: `README.md`)
- ONNX export → Robot hardware: deploys policy (evidence: `README.md`)
