How liangpan99/tokenhsi works
Project Type and Purpose
• TokenHSI is a physics-based human-scene interaction system that uses task tokenization to unify various motion skills (e.g., carrying, sitting, climbing, trajectory following) under a single transformer-based policy.
• At its core, the project ingests motion data (often pre-processed into numpy formats such as npy height maps), processes this data, and converts it into simulation-ready representations for IsaacGym. It also builds policies based on these conversions through reinforcement and imitation learning.
Analyzing the File Structure
• Key directories include:
• “lpanlib/isaacgym_utils”: Contains modules for IsaacGym-related conversions and visualization (e.g., joint_monkey.py, vis/api.py, vis/utils for body building, animation, and XML parsing).
• “lpanlib/poselib”: Offers foundational utilities for rotations, tensor operations, and skeleton handling essential for mapping motion data.
• “tokenhsi/data”: Includes datasets and configuration files (with .yaml and .npy files) that represent motion (and object) data. These .npy files (e.g., height maps, possibly other motion arrays) are essential for converting raw motion data into simulation states.
• “tokenhsi/env”: Defines the simulation environment tasks that use IsaacGym to simulate interactions.
• “tokenhsi/learning”: Contains the learning models that transform the converted motions into unified policy outputs.
• The provided README makes it clear that external dependencies such as IsaacGym and optionally PyTorch3D are integrated to take the processed motion data and render / simulate the physical interactions.
Key Components
a. Data Preprocessing & Asset Management:
• Motion and Object Data: Show an input block that represents raw motion data (from datasets like AMASS, SAMP, OMOMO) as well as object configuration data. Highlight the preprocessing step where these datasets are converted into structured formats (.yaml, .npy).
• Motion Conversion – Numpy Interface: Emphasize that the .npy files (for example, height maps, intermediate motion states) are generated by scripts under tokenhsi/data and custom modules that likely convert raw motion into simulation-consumable representations.
b. Motion-to-Code Conversion Modules for IsaacGym:
• lpanlib/isaacgym_utils: Illustrate this module as the “conversion engine.” It encapsulates utilities to interpret motion (and skeleton) data, modifying joint configurations (e.g., joint_monkey.py) and building visual representations (via vis/api.py, vis/utils/body_builder.py, sp_animation.py, xml_parser.py). Mark these components to show how motion representations (numpy arrays from .npy files) are parsed and converted into simulation-ready formats.
• poselib: This layer adds the necessary mathematical and skeleton transformation functions (rotation3d, tensor_utils, etc.) required to perform accurate spatial conversions. Connect this module to the IsaacGym conversion utilities.
c. Simulation Environment Integration:
d. Policy Learning and Adaptation:
e. Configuration and Support Utilities:
• tokenhsi/utils: Represents configuration utilities (e.g., config.py, parse_task.py) that pass hyperparameters and dataset paths into the conversion and simulation pipelines.
• Additional scripts (in tokenhsi/scripts and run.py) serve as orchestration points for training/testing different skills and adaptations.
Relationships and Data Flow
• Conversion output being fed into the simulation environments (tokenhsi/env), which are powered by IsaacGym.
• Subsequent interactions with the learning modules (tokenhsi/learning) that create and refine the unified policy.
• Control flow from configuration utilities (tokenhsi/utils) to all major components.
Architectural Patterns and Design Principles
• Separation of Concerns: Differentiate clearly between the data preprocessing, motion conversion, simulation, and learning layers.
• Modular Integration: Highlight that each module (e.g., motion conversion versus simulation tasks) is a plug-and-play component, linked by well-defined interfaces.
• Data-Driven Pipeline: Emphasize the conversion of static data (npy files) to in-simulation dynamics, underlining the efficient transition from pre-processed motion data to code running in IsaacGym.