How mondo-robotics/pmt works
PMT is a config-driven humanoid motion-tracking research system: users select a task, train or play a policy in simulation, and can evaluate flat tasks through an optional MuJoCo backend. A separate TCRS tool optimizes motion clips for terrain. The sampled code supports task derivation, motion-command behavior, policy components, and training storage; wiring of unsampled subsystems is included only where the README documents it.
Main entry points: tracking_env_cfg.py Tracking environments · mppi_foot_planner_smooth.py TCRS foot planner · on_policy_runner.py Training runners · builder.py Task builder
Components 28
Task configuration
- Task builder
pmt_tasks/builder.py - Task configurations
pmt_tasks/env_cfgs - Agent configurations
pmt_tasks/agent_cfgs - Task registry
pmt_tasks/registry_gym.py
Simulation and tracking
- Tracking environments
pmt_tasks/tracking_env_cfg.py - Motion commands
pmt_tasks/mdp/commands/multi_motion_command.py - Observations
pmt_tasks/mdp/observations.py - Rewards and termination
pmt_tasks/mdp/rewards.py - G1 articulation
pmt_tasks/robots/g1.py - Terrain utilities
pmt_tasks/utils/terrain.py - Isaac Lab
pmt_tasks/backends - MuJoCo evaluation
scripts/mjlab_eval_pmt_ckpt.py
RL training
- Train entrypoint
scripts/train.py - Playback entrypoint
scripts/play.py - Training runners
motion_tracking_rl/runners/on_policy_runner.py - Distillation runner
motion_tracking_rl/runners/distillation_runner.py - RL algorithms
motion_tracking_rl/algorithms - Rollout storage
motion_tracking_rl/storage/rollout_storage.py
Policy networks
- Policy networks
motion_tracking_rl/networks - Motion token tracker
motion_tracking_rl/networks/perceptive_motion/token_tracker.py
Terrain motion optimizer
- TCRS foot planner
TCRS/stair_mppi/mppi_foot_planner_smooth.py - TCRS terrain model
TCRS/stair_mppi/terrain.py
Other components
- Researcher
- Raw motion clips
- Terrain scene
- Optimized clips
- MuJoCo / mjlab
- Policy checkpoint
Connections 32
- Researcher → Train entrypoint: starts training
- Researcher → Playback entrypoint: starts playback
- Train entrypoint → Task builder: builds task
- Task builder → Task configurations: composes
- Task builder → Agent configurations: derives settings
- Task builder → Task registry: registers task
- Train entrypoint → Training runners: dispatches runner
- Training runners → RL algorithms: updates policy
- Training runners → Rollout storage: stores rollouts
- RL algorithms → Rollout storage: samples batches
- RL algorithms → Policy networks: optimizes
- Distillation runner → Policy networks: trains student
- Task configurations → Tracking environments: configures
- Tracking environments → Isaac Lab: runs in
- Tracking environments → Motion commands: uses commands
- Tracking environments → Observations: computes observations
- Tracking environments → Rewards and termination: computes rewards
- Tracking environments → G1 articulation: simulates
- Tracking environments → Terrain utilities: uses terrain
- Motion commands → Raw motion clips: loads clips
- Playback entrypoint → Training runners: loads policy
- Playback entrypoint → Policy checkpoint: reads
- Training runners → Policy checkpoint: writes
- Motion token tracker → Policy networks: provides architecture
- Researcher → TCRS foot planner: runs optimizer
- TCRS foot planner → Raw motion clips: optimizes motion
- TCRS foot planner → TCRS terrain model: queries terrain
- TCRS foot planner → Terrain scene: uses scene
- TCRS foot planner → Optimized clips: generates clips
- Researcher → MuJoCo evaluation: evaluates policy
- MuJoCo evaluation → Policy checkpoint: loads
- MuJoCo evaluation → MuJoCo / mjlab: runs on