How nvlabs/protomotions works
ProtoMotions 3 is a GPU-accelerated framework for training, evaluating, and deploying physically simulated humanoid policies. The main workflow is to prepare or retarget motion data, configure a robot and simulation task, train a policy through environment interaction, then evaluate or export it for sim-to-sim or hardware use. The graph also includes procedural scenes and terrain, motion tracking and generative-policy capabilities, and reusable simulator backends. Several subsystem links are documented only at a high level; those are marked as unverified rather than inferred as direct calls.
Main entry points: env.py Environment · train_agent.py Training entry · agent.py PPO agent · inference_agent.py Policy inference
Components 25
Motion and scenes
- Motion conversion
data/scripts - PyRoki retargeting
pyroki/batch_retarget_to_g1_from_keypoints.py - Motion library
protomotions/components/motion_lib.py - Motion manager
protomotions/envs/motion_manager/motion_manager.py - Scene library
protomotions/components/scene_lib.py - Terrain generation
protomotions/components/terrains/terrain.py
Simulation environment
- Robot configuration
protomotions/robot_configs/base.py - Environment
protomotions/envs/base_env/env.py - Environment context
protomotions/envs/context_views.py - Component execution
protomotions/envs/component_manager.py - Control manager
protomotions/envs/control/manager.py - Simulator factory
protomotions/simulator/factory.py - Simulator interface
protomotions/simulator/base_simulator/simulator.py - Isaac Gym backend
protomotions/simulator/isaacgym/simulator.py - Newton backend
protomotions/simulator/newton/simulator.py
Policy learning
- Training entry
protomotions/train_agent.py - PPO agent
protomotions/agents/ppo/agent.py - Base agent
protomotions/agents/base_agent/agent.py - AMP learning
protomotions/agents/amp/component.py - Generative policies
protomotions/agents/supervised/masked_mimic_model.py
Evaluation and deployment
- Policy inference
protomotions/inference_agent.py - ONNX export
deployment/export_tracker_onnx.py
Other components
- Researcher
- Motion data
- Robot hardware
Connections 28
- Researcher → Training entry: configures and trains
README.md - Motion data → Motion conversion: converts
- Motion data → PyRoki retargeting: retargets
README.md - PyRoki retargeting → Motion library: prepares motions
- Motion conversion → Motion library: prepares motions
- Motion library → Motion manager: samples motion data
protomotions/envs/motion_manager/motion_manager.py - Terrain generation → Scene library: provides terrain
protomotions/components/scene_lib.py - Training entry → PPO agent: starts training
- PPO agent → Environment: interacts with
protomotions/agents/ppo/agent.py - PPO agent → Base agent: inherits from
protomotions/agents/ppo/agent.py - AMP learning → Base agent: extends
protomotions/agents/amp/component.py - Generative policies → Training entry: trains policy via
- Environment → Motion manager: uses motion sampling
- Environment → Scene library: uses scenes
- Environment → Terrain generation: uses terrain
- Environment → Control manager: manages controls
protomotions/envs/control/manager.py - Environment → Component execution: executes components
- Component execution → Environment context: resolves context
protomotions/envs/component_manager.py - Control manager → Environment context: uses context
protomotions/envs/control/manager.py - Robot configuration → Simulator factory: configures simulator
protomotions/robot_configs/base.py - Simulator factory → Simulator interface: creates backend
- Isaac Gym backend → Simulator interface: implements interface
protomotions/simulator/isaacgym/simulator.py - Newton backend → Simulator interface: implements interface
protomotions/simulator/newton/simulator.py - Researcher → Policy inference: runs evaluation
README.md - Policy inference → Environment: runs policy in
protomotions/inference_agent.py - Policy inference → Base agent: loads agent
protomotions/inference_agent.py - Training entry → ONNX export: exports policy
README.md - ONNX export → Robot hardware: deploys policy
README.md