How nvlabs/hover works
Main Components
• Data & Motion Library:
• Motion datasets. Include “AMASS Dataset” and retargeted data (e.g., stable_punch.pkl). Note the role of the third_party/human2humanoid scripts that process the raw motion data.
• Simulation Environment:
• Isaac Lab, Mujoco, and their accompanying wrappers. The “neural_wbc/inference_env”, “neural_wbc/mujoco_wrapper”, and “neural_wbc/isaac_lab_wrapper” modules indicate distinct simulation and evaluation pipelines.
• Core Control and Evaluation:
• The “neural_wbc/core” folder contains functionalities like body state, environment wrapper, evaluator, mask definitions, and math utilities. This core module implements the underlying control/physics logic.
• Policy Modules:
• Teacher Policy (training script, model checkpoint storage, configuration in “scripts/rsl_rl/train_teacher_policy.py”).
• Student Policy (training and policy trainer, with associated configuration and storage).
• Hardware and Real-World Integration:
• Hardware Wrappers (“neural_wbc/hw_wrappers”) that interface with the Unitree H1 robot.
• External Libraries and Third-Party Tools:
• The third_party directory shows inclusion of customized RSL RL algorithms (PPO, actor-critic modules) and additional viewers that integrate with Mujoco.
• Utilities and Testing:
• Incorporate testing modules, build scripts (e.g., install_deps.sh), and configuration overwrite capabilities. These support development and unit testing.
Relationships and Data Flow
• Data Flow Sequence:
• The process begins with raw motion capture data (from AMASS), which undergoes retargeting via the provided scripts (retarget_h1.sh and associated YAML configuration).
• This processed data feeds into the simulation environments where policies are trained.
• Training Pipeline:
• In one branch, the Teacher Policy is trained first, storing checkpoints (displayed in neural_wbc/data/data/policy/h1:teacher/).
• Then the Student Policy is trained using the Teacher Policy’s checkpoint (“neural_wbc/student_policy”) representing a policy distillation flow.
• Simulation & Evaluation:
• Training and evaluation flow through Isaac Lab and Mujoco simulation environments.
• Inference modules (e.g., neural_wbc/inference_env) provide means for both sim-to-sim and sim-to-real testing.
• Hardware Integration:
• The hw_wrappers module (within neural_wbc/hw_wrappers) directly interfaces with the real robot, converting simulation commands to hardware commands.
Architectural Patterns and Design Principles
• Modularity and Separation of Concerns:
• The project is split into dedicated modules (core, data, policy, wrappers, and inference), each encapsulating a logical unit of functionality.
• Extensibility:
• The architecture supports adding new motion modes (generalist vs. specialist), experimenting with new simulation environments (Mujoco, Isaac Lab), and deploying across different hardware platforms.
• Testing and Deployment:
• Unit tests and integration tests are in place, as seen in the tests directories in each module, ensuring reliable integration of components.