# yanjieze/twist architecture

> Architecture diagram of the GitHub repository yanjieze/twist, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yanjieze/twist
- Repository: https://github.com/yanjieze/twist

## Overview

##### Project Type and Purpose
- TWIST is a full-stack robotics imitation‐learning project. It includes:
- Offline training pipelines for “teacher” and “student” policies using reinforcement learning (RL) and behavior cloning.
- A simulation environment powered by NVIDIA Isaac Gym (via the `legged_gym` package).
- A pose‐retargeting library (`pose/poselib`).
- Deployment scripts to run the trained student policy in sim2sim (simulation) and sim2real (Unitree G1 robot) modes.
- A two‐layer control architecture: high‐level motion server (teleop or prerecorded motion) and low‐level controller server (the RL policy).
- Inter‐process communication via Redis.

##### File‐Structure Analysis
- Top‐level modules:
- assets/ — URDFs, XML robot descriptions, meshes, pretrained motion tracker.
- deploy_real/ — scripts for sim2sim and sim2real (high‐level and low‐level servers).
- legged_gym/ — the Isaac Gym RL environment and training scripts.
- pose/ — pose‐processing and retargeting library.
- rsl_rl/ — core RL algorithms (PPO, DAgger, RMA, etc.), network modules, runners, storage.
- shell scripts: training (`train_teacher.sh`, `train_student.sh`), export (`to_jit.sh`), play (`play_student.sh`, `play_teacher.sh`).
- unitree_g1.md — instructions specific to real‐robot deployment.

##### Key README Insights
- Two‐stage training: teacher (RL), then student (RL+BC).
- Two‐layer runtime:
a. High‐Level Motion Server (`server_high_level_motion_lib.py`) publishes motion targets via Redis.
b. Low‐Level Controller Server (`server_low_level_g1_sim.py` / `server_low_level_g1_real.py`) subscribes to Redis, runs the policy (JIT‐exported PyTorch model) in sim or real hardware.
- External dependencies: Redis[hiredis], Isaac Gym, MuJoCo, Unitree SDK, pyttsx3, flask, hydra-core, etc.

a. Main Components
1. Data & Assets
- Motion dataset (root path set in `legged_gym/motion_data_configs/twist_dataset.yaml`)
- URDF/XML & meshes (assets/)
- Pretrained motion tracker (assets/twist_general_motion_tracker.pt)
2. Training Pipeline
- RL Trainer (rsl_rl)
- Teacher RL
- Student RL+BC
- LeggedGym environments (legged_gym/envs/g1, base/legged_robot, etc.)
- Output: student policy checkpoint & JIT‐traced model
3. Pose Retargeting Library
- pose/poselib (core backend, skeleton, visualization)
- GMR as optional real‐time retargeting front end
4. Deployment Services (Microservices)
- High‐Level Motion Server
- Reads motion file or MOCAP stream (legacy OptiTrack or GMR)
- Publishes joint‐target commands to Redis
- Redis broker (pub/sub)
- Low‐Level Controller Server
- Subscribes to high‐level commands
- Loads JIT‐exported student policy (PyTorch)
- Drives either:
– Sim2sim: Isaac Gym simulator
– Sim2real: Unitree G1 via unitree_sdk2py
5. External Tools & Infrastructure
- NVIDIA GPU for training & simulation
- Redis server
- Unitree G1 hardware (Ethernet link, remote controller)
- Optional MoCap / GMR service for real‐time teleop

b. Relationships & Data Flows
- Offline: Motion Dataset → LeggedGym Env → RL Trainer → Policy Checkpoint → `to_jit.sh` → JIT Model
- Online Simulation: JIT Model + Isaac Gym → simulator → visualization
- Online Teleop/Real: MOCAP/GMR or prerecorded motion → High‐Level Server → Redis → Low‐Level Server → Simulator or Robot hardware

c. Architectural Patterns & Principles
- Layered Architecture: clear separation of offline training vs. runtime inference
- Microservices: high‐level and low‐level controllers as separate processes communicating via Redis
- Modular codebase: distinct packages for RL (`rsl_rl`), simulation (`legged_gym`), pose retarget (`pose`), deployment (`deploy_real`)
- Pluggable backends: supports both sim and real robot with minimal code changes

d. Key Technologies & Frameworks
- rsl_rl (PyTorch-based RL algorithms, on-policy runners)
- legged_gym (Isaac Gym wrapper, Hydra configs)
- pose/poselib (tensor utils, skeleton, visualization)
- Redis (pub/sub messaging)
- MuJoCo / mujoco-python-viewer
- Unitree SDK (`unitree_sdk2py`)
- Bash scripts & Hydra for experiment management

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data & Assets
    subgraph "Data & Assets"
        Dataset["Motion Dataset Config"]:::data
        URDF["Robot Descriptions & Meshes"]:::data
        Tracker["Pretrained Motion Tracker"]:::data
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        subgraph "RL Trainer (rsl_rl)"
            Alg["Algorithms"]:::train
            Run["Runners"]:::train
            Mod["Network Modules"]:::train
            Sto["Storage"]:::train
        end
        Entry["Training Entry Point (train.py)"]:::train
        TeacherShell["train_teacher.sh"]:::train
        StudentShell["train_student.sh"]:::train
        JITShell["to_jit.sh"]:::train
        SaveBC["save_jit_stu_bc.py"]:::train
        SaveRL["save_jit_stu_rlbc.py"]:::train
        JITModel["JIT Model"]:::train
    end

    %% Simulation Environment
    subgraph "Simulation Environment"
        BaseEnv["Core Env Definitions"]:::sim
        G1Env["G1-Specific Env & Config"]:::sim
        Utils["Utility Modules"]:::sim
    end

    %% Pose Retargeting Library
    subgraph "Pose Retargeting Library"
        Core["Core Backend & Tensor Utils"]:::lib
        Skeleton["Skeleton & 3D Models"]:::lib
        Viz["Visualization & Plotting"]:::lib
        Helpers["Utility Functions & Helpers"]:::lib
    end

    %% Deployment Services
    subgraph "Deployment Services"
        High["High-Level Motion Server"]:::deploy
        Redis["Redis Broker"]:::deploy
        LowSim["Low-Level Controller Server (Sim)"]:::deploy
        LowReal["Low-Level Controller Server (Real)"]:::deploy
        DeployUtils["Deployment Utilities & Configs"]:::deploy
    end

    %% Execution Scripts
    subgraph "Runtime Inference Scripts"
        PlayT["play_teacher.sh"]:::run
        PlayS["play_student.sh"]:::run
    end

    %% External Systems
    subgraph "External Systems"
        GPU["NVIDIA GPU"]:::external
        MoCap["MoCap/GMR Service"]:::external
        Robot["Unitree G1 Hardware"]:::external
    end

    %% Data Flows & Dependencies
    Dataset -->|"dataset config"| Entry
    URDF -->|assets| BaseEnv
    URDF -->|assets| G1Env
    Tracker -->|tracker model| Entry

    Entry -->|invoke trainer| Alg
    Entry -->|invoke trainer| Run
    Entry -->|invoke trainer| Mod
    Entry -->|invoke trainer| Sto

    Alg -->|output checkpoint| SaveBC
    Alg -->|output checkpoint| SaveRL
    Run -->|output checkpoint| SaveBC
    Run -->|output checkpoint| SaveRL
    SaveBC -->|"traces student BC"| JITModel
    SaveRL -->|"traces student RL+BC"| JITModel
    TeacherShell -->|wraps| Entry
    StudentShell -->|wraps| Entry
    JITShell -->|wraps| SaveBC
    JITShell -->|wraps| SaveRL

    JITModel -->|loads model| LowSim
    JITModel -->|loads model| LowReal

    LowSim -->|controls simulation| BaseEnv
    BaseEnv -->|sim state| Utils

    %% Online Teleop/Sim
    PlayT -->|launches motion server| High
    PlayS -->|launches motion server| High
    MoCap -->|motion stream| High
    High -->|"motion targets"| Redis
    Redis -->|"motion targets"| LowSim
    Redis -->|"motion targets"| LowReal
    LowReal -->|"robot commands"| Robot

    %% Dependencies on Pose Library
    Entry -->|uses| Core
    Entry -->|uses| Skeleton
    Entry -->|uses| Viz
    Entry -->|uses| Helpers

    %% External Resources
    Entry -->|runs on| GPU

    %% Click Events
    click Dataset "https://github.com/yanjieze/twist/blob/master/legged_gym/motion_data_configs/twist_dataset.yaml"
    click URDF "https://github.com/yanjieze/twist/tree/master/assets/g1/"
    click Tracker "https://github.com/yanjieze/twist/blob/master/assets/twist_general_motion_tracker.pt"
    click Alg "https://github.com/yanjieze/twist/tree/master/rsl_rl/rsl_rl/algorithms/"
    click Run "https://github.com/yanjieze/twist/tree/master/rsl_rl/rsl_rl/runners/"
    click Mod "https://github.com/yanjieze/twist/tree/master/rsl_rl/rsl_rl/modules/"
    click Sto "https://github.com/yanjieze/twist/tree/master/rsl_rl/rsl_rl/storage/"
    click Entry "https://github.com/yanjieze/twist/blob/master/legged_gym/legged_gym/scripts/train.py"
    click SaveBC "https://github.com/yanjieze/twist/blob/master/legged_gym/legged_gym/scripts/save_jit_stu_bc.py"
    click SaveRL "https://github.com/yanjieze/twist/blob/master/legged_gym/legged_gym/scripts/save_jit_stu_rlbc.py"
    click TeacherShell "https://github.com/yanjieze/twist/blob/master/train_teacher.sh"
    click StudentShell "https://github.com/yanjieze/twist/blob/master/train_student.sh"
    click JITShell "https://github.com/yanjieze/twist/blob/master/to_jit.sh"
    click BaseEnv "https://github.com/yanjieze/twist/tree/master/legged_gym/legged_gym/envs/base/"
    click G1Env "https://github.com/yanjieze/twist/blob/master/legged_gym/legged_gym/envs/g1/g1_mimic_distill.py"
    click Utils "https://github.com/yanjieze/twist/tree/master/legged_gym/legged_gym/gym_utils/"
    click Core "https://github.com/yanjieze/twist/tree/master/pose/pose/poselib/poselib/core/"
    click Skeleton "https://github.com/yanjieze/twist/tree/master/pose/pose/poselib/poselib/skeleton/"
    click Viz "https://github.com/yanjieze/twist/tree/master/pose/pose/poselib/poselib/visualization/"
    click Helpers "https://github.com/yanjieze/twist/tree/master/pose/pose/util_funcs/"
    click Helpers "https://github.com/yanjieze/twist/tree/master/pose/pose/utils/"
    click High "https://github.com/yanjieze/twist/blob/master/deploy_real/server_high_level_motion_lib.py"
    click LowSim "https://github.com/yanjieze/twist/blob/master/deploy_real/server_low_level_g1_sim.py"
    click LowReal "https://github.com/yanjieze/twist/blob/master/deploy_real/server_low_level_g1_real.py"
    click DeployUtils "https://github.com/yanjieze/twist/tree/master/deploy_real/data_utils/"
    click DeployUtils "https://github.com/yanjieze/twist/tree/master/deploy_real/robot_control/"
    click PlayT "https://github.com/yanjieze/twist/blob/master/play_teacher.sh"
    click PlayS "https://github.com/yanjieze/twist/blob/master/play_student.sh"

    %% Legend
    subgraph "Legend"
        D["Data & Assets"]:::data
        T["Training Components"]:::train
        S["Simulation Components"]:::sim
        L["Pose Library"]:::lib
        P["Deployment Services"]:::deploy
        R["Runtime Scripts"]:::run
        E["External Systems"]:::external
    end

    classDef data fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef train fill:#AED6F1,stroke:#333,stroke-width:1px
    classDef sim fill:#A9DFBF,stroke:#333,stroke-width:1px
    classDef lib fill:#D5D8DC,stroke:#333,stroke-width:1px
    classDef deploy fill:#F5B7B1,stroke:#333,stroke-width:1px
    classDef run fill:#F9E79F,stroke:#333,stroke-width:1px
    classDef external fill:#E5E7E9,stroke:#333,stroke-width:1px
```
