# nguyen-hao-khang/vla-robot-sorting architecture

> Architecture diagram of the GitHub repository nguyen-hao-khang/vla-robot-sorting, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nguyen-hao-khang/vla-robot-sorting
- Repository: https://github.com/nguyen-hao-khang/vla-robot-sorting

## Overview

VLA Robot Sorting is a research prototype for language-conditioned pick-and-place on the Panthera-HT, developed and evaluated in a RoboTwin digital twin before intended real-arm use. The operator workflow is command and observations → policy inference → safety filtering → simulated or physical execution. A parallel workflow records scripted demonstrations, converts them to LeRobot format, and fine-tunes a VLA; seeded evaluation measures task outcomes. The README documents the broad pipeline; sampled source shows selected twin helpers and safety contracts, not the complete runtime wiring.

Main entry points: [`tools/twin/twin_console.py`](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_console.py) (Twin Console), [`tools/twin/twin_eval.py`](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_eval.py) (Seeded Evaluation), [`src/vla_robot_sorting/safety/action_filter.py`](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/src/vla_robot_sorting/safety/action_filter.py) (Action Filter), [`tools/twin/sorting_task.py`](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/sorting_task.py) (Sorting Scenes)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_twin["Digital Twin"]
  node_sorting_task["Sorting Scenes<br/>[sorting_task.py]"]
  node_scene["Twin Scene<br/>[twin_scene.py]"]
  node_assets["Twin Assets<br/>[twin_assets.py]"]
  node_console["Twin Console<br/>[twin_console.py]"]
  node_perception["VLM Perception<br/>[twin_perception.py]"]
  node_geometry["Camera Geometry<br/>[twin_geometry.py]"]
  node_grasp["Grasp Planning<br/>[twin_grasp.py]"]
  node_kinematics["Arm Kinematics<br/>[twin_kinematics.py]"]
end

subgraph group_policy["Policy and Safety"]
  node_policy_server["Policy Server<br/>[policy_bridge.py]"]
  node_vla["Fine-tuned VLA"]
  node_safety_filter["Action Filter<br/>[action_filter.py]"]
  node_joint_limits["Trajectory Limits<br/>[joint_limits.py]"]
end

subgraph group_data["Demonstration Data"]
  node_recorder["Demo Recorder<br/>[twin_record.py]"]
  node_dataset[("LeRobot Dataset")]
  node_converter["Dataset Converter<br/>[lerobot_convert.py]"]
  node_training["VLA Fine-tuning"]
  node_command_drafts["Test Commands"]
end

subgraph group_eval["Evaluation"]
  node_evaluation["Seeded Evaluation<br/>[twin_eval.py]"]
  node_metrics["Evaluation Metrics<br/>[eval_metrics.py]"]
end

subgraph group_contracts["Control Contracts"]
  node_skill_contracts["Semantic Contracts<br/>[models.py]"]
  node_motion_contracts["Motion Contracts<br/>[motion.py]"]
end

node_operator(("Operator"))
node_simulator["RoboTwin"]
node_robot["Panthera-HT Arm"]

node_operator -->|"enters command"| node_console
node_sorting_task -->|"uses scene model"| node_scene
node_scene -->|"builds assets"| node_assets
node_console -->|"runs twin"| node_simulator
node_console -->|"requests policy"| node_policy_server
node_policy_server -->|"runs checkpoint"| node_vla
node_console -->|"uses detections"| node_perception
node_perception -->|"projects detections"| node_geometry
node_console -->|"plans grasps"| node_grasp
node_console -->|"uses arm model"| node_kinematics
node_console -->|"filters actions"| node_safety_filter
node_safety_filter -->|"guards execution"| node_robot
node_safety_filter -->|"guards execution"| node_simulator
node_joint_limits -->|"validates trajectories"| node_motion_contracts
node_recorder -->|"records demonstrations"| node_simulator
node_recorder -->|"writes episodes"| node_dataset
node_converter -->|"converts recordings"| node_dataset
node_training -->|"produces checkpoint"| node_vla
node_command_drafts -->|"uses task vocabulary"| node_sorting_task
node_evaluation -->|"runs episodes"| node_console
node_evaluation -->|"summarizes results"| node_metrics
node_evaluation -->|"plans scenes"| node_sorting_task
node_evaluation -->|"evaluates policy"| node_policy_server
node_skill_contracts -->|"defines shared types"| node_motion_contracts

click node_sorting_task "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/sorting_task.py"
click node_scene "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_scene.py"
click node_assets "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_assets.py"
click node_console "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_console.py"
click node_perception "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_perception.py"
click node_geometry "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_geometry.py"
click node_grasp "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_grasp.py"
click node_kinematics "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_kinematics.py"
click node_policy_server "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/policy_bridge.py"
click node_safety_filter "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/src/vla_robot_sorting/safety/action_filter.py"
click node_joint_limits "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/src/vla_robot_sorting/safety/joint_limits.py"
click node_recorder "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_record.py"
click node_converter "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/lerobot_convert.py"
click node_command_drafts "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/data/draft_test_commands.py"
click node_evaluation "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_eval.py"
click node_metrics "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/eval_metrics.py"
click node_skill_contracts "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/src/vla_robot_sorting/contracts/models.py"
click node_motion_contracts "https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/src/vla_robot_sorting/contracts/motion.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sorting_task,node_scene,node_assets,node_console,node_perception,node_geometry,node_grasp,node_kinematics toneBlue
class node_policy_server,node_vla,node_safety_filter,node_joint_limits toneAmber
class node_recorder,node_dataset,node_converter,node_training,node_command_drafts toneMint
class node_evaluation,node_metrics toneRose
class node_skill_contracts,node_motion_contracts,node_operator,node_simulator,node_robot toneIndigo
```

## Components

### Digital Twin

- [Sorting Scenes](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/sorting_task.py): `tools/twin/sorting_task.py`
- [Twin Scene](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_scene.py): `tools/twin/twin_scene.py`
- [Twin Assets](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_assets.py): `tools/twin/twin_assets.py`
- [Twin Console](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_console.py): `tools/twin/twin_console.py`
- [VLM Perception](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_perception.py): `tools/twin/twin_perception.py`
- [Camera Geometry](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_geometry.py): `tools/twin/twin_geometry.py`
- [Grasp Planning](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_grasp.py): `tools/twin/twin_grasp.py`
- [Arm Kinematics](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_kinematics.py): `tools/twin/twin_kinematics.py`

### Policy and Safety

- [Policy Server](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/policy_bridge.py): `tools/twin/policy_bridge.py`
- **Fine-tuned VLA**
- [Action Filter](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/src/vla_robot_sorting/safety/action_filter.py): `src/vla_robot_sorting/safety/action_filter.py`
- [Trajectory Limits](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/src/vla_robot_sorting/safety/joint_limits.py): `src/vla_robot_sorting/safety/joint_limits.py`

### Demonstration Data

- [Demo Recorder](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_record.py): `tools/twin/twin_record.py`
- **LeRobot Dataset**
- [Dataset Converter](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/lerobot_convert.py): `tools/twin/lerobot_convert.py`
- **VLA Fine-tuning**
- [Test Commands](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/data/draft_test_commands.py): `tools/data/draft_test_commands.py`

### Evaluation

- [Seeded Evaluation](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/twin_eval.py): `tools/twin/twin_eval.py`
- [Evaluation Metrics](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/tools/twin/eval_metrics.py): `tools/twin/eval_metrics.py`

### Control Contracts

- [Semantic Contracts](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/src/vla_robot_sorting/contracts/models.py): `src/vla_robot_sorting/contracts/models.py`
- [Motion Contracts](https://github.com/nguyen-hao-khang/vla-robot-sorting/blob/main/src/vla_robot_sorting/contracts/motion.py): `src/vla_robot_sorting/contracts/motion.py`

### Other components

- **Operator**
- **RoboTwin**
- **Panthera-HT Arm**

## Connections

- Operator → Twin Console: enters command (evidence: `README.md`)
- Sorting Scenes → Twin Scene: uses scene model (evidence: `tools/twin/sorting_task.py`)
- Twin Scene → Twin Assets: builds assets (evidence: `tools/twin/twin_scene.py`)
- Twin Console → RoboTwin: runs twin (evidence: `README.md`)
- Twin Console → Policy Server: requests policy (evidence: `README.md`)
- Policy Server → Fine-tuned VLA: runs checkpoint (evidence: `README.md`)
- Twin Console → VLM Perception: uses detections (evidence: `tools/twin/twin_console.py`)
- VLM Perception → Camera Geometry: projects detections (evidence: `tools/twin/twin_perception.py`)
- Twin Console → Grasp Planning: plans grasps (evidence: `tools/twin/twin_console.py`)
- Twin Console → Arm Kinematics: uses arm model (evidence: `tools/twin/twin_console.py`)
- Twin Console → Action Filter: filters actions (evidence: `README.md`)
- Action Filter → Panthera-HT Arm: guards execution (evidence: `README.md`)
- Action Filter → RoboTwin: guards execution (evidence: `README.md`)
- Trajectory Limits → Motion Contracts: validates trajectories (evidence: `src/vla_robot_sorting/safety/joint_limits.py`)
- Demo Recorder → RoboTwin: records demonstrations (evidence: `README.md`)
- Demo Recorder → LeRobot Dataset: writes episodes (evidence: `README.md`)
- Dataset Converter → LeRobot Dataset: converts recordings (evidence: `README.md`)
- VLA Fine-tuning → Fine-tuned VLA: produces checkpoint (evidence: `README.md`)
- Test Commands → Sorting Scenes: uses task vocabulary (evidence: `tools/data/draft_test_commands.py`)
- Seeded Evaluation → Twin Console: runs episodes (evidence: `tools/twin/twin_eval.py`)
- Seeded Evaluation → Evaluation Metrics: summarizes results (evidence: `tools/twin/twin_eval.py`)
- Seeded Evaluation → Sorting Scenes: plans scenes (evidence: `tools/twin/twin_eval.py`)
- Seeded Evaluation → Policy Server: evaluates policy (evidence: `tools/twin/twin_eval.py`)
- Semantic Contracts → Motion Contracts: defines shared types (evidence: `src/vla_robot_sorting/contracts/motion.py`)