# liangpan99/tokenhsi architecture

> Architecture diagram of the GitHub repository liangpan99/tokenhsi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/liangpan99/tokenhsi
- Repository: https://github.com/liangpan99/tokenhsi

## Overview

##### Project Type and Purpose
- TokenHSI is a physics-based human-scene interaction system that uses task tokenization to unify various motion skills (e.g., carrying, sitting, climbing, trajectory following) under a single transformer-based policy.
- At its core, the project ingests motion data (often pre-processed into numpy formats such as npy height maps), processes this data, and converts it into simulation-ready representations for IsaacGym. It also builds policies based on these conversions through reinforcement and imitation learning.

##### Analyzing the File Structure
- Key directories include:
- “lpanlib/isaacgym_utils”: Contains modules for IsaacGym-related conversions and visualization (e.g., joint_monkey.py, vis/api.py, vis/utils for body building, animation, and XML parsing).
- “lpanlib/poselib”: Offers foundational utilities for rotations, tensor operations, and skeleton handling essential for mapping motion data.
- “tokenhsi/data”: Includes datasets and configuration files (with .yaml and .npy files) that represent motion (and object) data. These .npy files (e.g., height maps, possibly other motion arrays) are essential for converting raw motion data into simulation states.
- “tokenhsi/env”: Defines the simulation environment tasks that use IsaacGym to simulate interactions.
- “tokenhsi/learning”: Contains the learning models that transform the converted motions into unified policy outputs.
- The provided README makes it clear that external dependencies such as IsaacGym and optionally PyTorch3D are integrated to take the processed motion data and render / simulate the physical interactions.

##### Key Components
a. Data Preprocessing & Asset Management:
- Motion and Object Data: Show an input block that represents raw motion data (from datasets like AMASS, SAMP, OMOMO) as well as object configuration data. Highlight the preprocessing step where these datasets are converted into structured formats (.yaml, .npy).
- Motion Conversion – Numpy Interface: Emphasize that the .npy files (for example, height maps, intermediate motion states) are generated by scripts under tokenhsi/data and custom modules that likely convert raw motion into simulation-consumable representations.

b. Motion-to-Code Conversion Modules for IsaacGym:
- lpanlib/isaacgym_utils: Illustrate this module as the “conversion engine.” It encapsulates utilities to interpret motion (and skeleton) data, modifying joint configurations (e.g., joint_monkey.py) and building visual representations (via vis/api.py, vis/utils/body_builder.py, sp_animation.py, xml_parser.py). Mark these components to show how motion representations (numpy arrays from .npy files) are parsed and converted into simulation-ready formats.
- poselib: This layer adds the necessary mathematical and skeleton transformation functions (rotation3d, tensor_utils, etc.) required to perform accurate spatial conversions. Connect this module to the IsaacGym conversion utilities.

c. Simulation Environment Integration:

d. Policy Learning and Adaptation:

e. Configuration and Support Utilities:
- tokenhsi/utils: Represents configuration utilities (e.g., config.py, parse_task.py) that pass hyperparameters and dataset paths into the conversion and simulation pipelines.
- Additional scripts (in tokenhsi/scripts and run.py) serve as orchestration points for training/testing different skills and adaptations.

##### Relationships and Data Flow
- Conversion output being fed into the simulation environments (tokenhsi/env), which are powered by IsaacGym.
- Subsequent interactions with the learning modules (tokenhsi/learning) that create and refine the unified policy.
- Control flow from configuration utilities (tokenhsi/utils) to all major components.

##### Architectural Patterns and Design Principles
- Separation of Concerns: Differentiate clearly between the data preprocessing, motion conversion, simulation, and learning layers.
- Modular Integration: Highlight that each module (e.g., motion conversion versus simulation tasks) is a plug-and-play component, linked by well-defined interfaces.
- Data-Driven Pipeline: Emphasize the conversion of static data (npy files) to in-simulation dynamics, underlining the efficient transition from pre-processed motion data to code running in IsaacGym.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Configuration Layer
    subgraph "Data & Configuration Layer"
        Data["Motion & Object Data (.npy, YAML)"]:::data
        Config["Configuration & Support Utilities"]:::config
        Scripts["Orchestration Scripts"]:::config
    end

    %% Motion Conversion Layer
    subgraph "Motion Conversion Layer"
        Conversion["Motion Conversion Engine (IsaacGym Utils)"]:::conversion
        Pose["Pose & Skeleton Conversions (poselib)"]:::conversion
    end

    %% Simulation Environment Layer
    subgraph "Simulation Environment Layer"
        Simulation["Simulation Environment (tokenhsi/env)"]:::simulation
    end

    %% Learning & Policy Layer
    subgraph "Learning & Policy Layer"
        Learning["Policy Learning & Adaptation"]:::learning
        Transformer["Transformer Models"]:::learning
    end

    %% Main Data Flow
    Data -->|"preprocessed"| Conversion
    Data -->|"preprocessed"| Pose
    Conversion -->|"npy-based output"| Simulation
    Pose -->|"spatial transforms"| Simulation
    Simulation -->|"simulation outcomes"| Learning
    Learning -->|"policy feedback"| Transformer

    %% Configuration influences all layers
    Config ---|"configures"| Data
    Config ---|"configures"| Conversion
    Config ---|"configures"| Pose
    Config ---|"configures"| Simulation
    Config ---|"configures"| Learning
    Scripts ---|"orchestrates"| Conversion
    Scripts ---|"orchestrates"| Simulation
    Scripts ---|"orchestrates"| Learning

    %% Click Events
    click Data "https://github.com/liangpan99/tokenhsi/tree/main/tokenhsi/data"
    click Conversion "https://github.com/liangpan99/tokenhsi/tree/main/lpanlib/isaacgym_utils"
    click Pose "https://github.com/liangpan99/tokenhsi/tree/main/lpanlib/poselib"
    click Simulation "https://github.com/liangpan99/tokenhsi/tree/main/tokenhsi/env"
    click Learning "https://github.com/liangpan99/tokenhsi/tree/main/tokenhsi/learning"
    click Transformer "https://github.com/liangpan99/tokenhsi/tree/main/tokenhsi/learning/transformer"
    click Config "https://github.com/liangpan99/tokenhsi/tree/main/tokenhsi/utils"
    click Scripts "https://github.com/liangpan99/tokenhsi/tree/main/tokenhsi/scripts"

    %% Styles
    classDef data fill:#aee1f9,stroke:#333,stroke-width:2px;
    classDef conversion fill:#ffcccc,stroke:#aa3333,stroke-width:2px;
    classDef simulation fill:#d0f0c0,stroke:#336633,stroke-width:2px;
    classDef learning fill:#fff2b2,stroke:#996600,stroke-width:2px;
    classDef config fill:#e6e6fa,stroke:#6666cc,stroke-width:2px;
```
