How nguyen-hao-khang/vla-robot-sorting works
VLA Robot Sorting is a research prototype for language-conditioned pick-and-place on the Panthera-HT, developed and evaluated in a RoboTwin digital twin before intended real-arm use. The operator workflow is command and observations → policy inference → safety filtering → simulated or physical execution. A parallel workflow records scripted demonstrations, converts them to LeRobot format, and fine-tunes a VLA; seeded evaluation measures task outcomes. The README documents the broad pipeline; sampled source shows selected twin helpers and safety contracts, not the complete runtime wiring.
Main entry points: twin_console.py Twin Console · twin_eval.py Seeded Evaluation · action_filter.py Action Filter · sorting_task.py Sorting Scenes
Components 24
Digital Twin
- Sorting Scenes
tools/twin/sorting_task.py - Twin Scene
tools/twin/twin_scene.py - Twin Assets
tools/twin/twin_assets.py - Twin Console
tools/twin/twin_console.py - VLM Perception
tools/twin/twin_perception.py - Camera Geometry
tools/twin/twin_geometry.py - Grasp Planning
tools/twin/twin_grasp.py - Arm Kinematics
tools/twin/twin_kinematics.py
Policy and Safety
- Policy Server
tools/twin/policy_bridge.py - Fine-tuned VLA
- Action Filter
src/vla_robot_sorting/safety/action_filter.py - Trajectory Limits
src/vla_robot_sorting/safety/joint_limits.py
Demonstration Data
- Demo Recorder
tools/twin/twin_record.py - LeRobot Dataset
- Dataset Converter
tools/twin/lerobot_convert.py - VLA Fine-tuning
- Test Commands
tools/data/draft_test_commands.py
Evaluation
- Seeded Evaluation
tools/twin/twin_eval.py - Evaluation Metrics
tools/twin/eval_metrics.py
Control Contracts
- Semantic Contracts
src/vla_robot_sorting/contracts/models.py - Motion Contracts
src/vla_robot_sorting/contracts/motion.py
Other components
- Operator
- RoboTwin
- Panthera-HT Arm
Connections 24
- Operator → Twin Console: enters command
README.md - Sorting Scenes → Twin Scene: uses scene model
tools/twin/sorting_task.py - Twin Scene → Twin Assets: builds assets
tools/twin/twin_scene.py - Twin Console → RoboTwin: runs twin
README.md - Twin Console → Policy Server: requests policy
README.md - Policy Server → Fine-tuned VLA: runs checkpoint
README.md - Twin Console → VLM Perception: uses detections
tools/twin/twin_console.py - VLM Perception → Camera Geometry: projects detections
tools/twin/twin_perception.py - Twin Console → Grasp Planning: plans grasps
tools/twin/twin_console.py - Twin Console → Arm Kinematics: uses arm model
tools/twin/twin_console.py - Twin Console → Action Filter: filters actions
README.md - Action Filter → Panthera-HT Arm: guards execution
README.md - Action Filter → RoboTwin: guards execution
README.md - Trajectory Limits → Motion Contracts: validates trajectories
src/vla_robot_sorting/safety/joint_limits.py - Demo Recorder → RoboTwin: records demonstrations
README.md - Demo Recorder → LeRobot Dataset: writes episodes
README.md - Dataset Converter → LeRobot Dataset: converts recordings
README.md - VLA Fine-tuning → Fine-tuned VLA: produces checkpoint
README.md - Test Commands → Sorting Scenes: uses task vocabulary
tools/data/draft_test_commands.py - Seeded Evaluation → Twin Console: runs episodes
tools/twin/twin_eval.py - Seeded Evaluation → Evaluation Metrics: summarizes results
tools/twin/twin_eval.py - Seeded Evaluation → Sorting Scenes: plans scenes
tools/twin/twin_eval.py - Seeded Evaluation → Policy Server: evaluates policy
tools/twin/twin_eval.py - Semantic Contracts → Motion Contracts: defines shared types
src/vla_robot_sorting/contracts/motion.py