How openbmb/minicpm-robot works
MiniCPM-Robot is a two-part embodied-AI repository: RobotManip serves a streaming vision-language-action policy and evaluates it across manipulation benchmarks; RobotTrack trains/evaluates an on-device visual tracker and deploys it for real-world Go2 tracking. The graph separates model construction, benchmark evaluation, simulation, and robot deployment. Source-index references support the listed internal imports; README capability statements identify product branches. Several server and runtime connections remain unverified because their relevant implementation was not sampled.
Main entry points: training.py Model Training · rearrange_sim_v2.py Rearrangement Simulator · vla_infer.py VLA Inference · runner.py Evaluation Runner
Components 29
RobotManip
- VLA Inference
MiniCPM-RobotManip/vla_infer.py - MemVLA Inference
MiniCPM-RobotManip/vla_infer_memvla.py - MemVLA Recipe
MiniCPM-RobotManip/deployment/model_server/memvla_recipe.py - MemVLA Policy
MiniCPM-RobotManip/deployment/model_server/memvla_policy_wrapper.py - Policy Server
MiniCPM-RobotManip/deployment/model_server/tools/websocket_policy_server.py - WebSocket Client
MiniCPM-RobotManip/deployment/model_server/tools/websocket_policy_client.py - MessagePack Codec
MiniCPM-RobotManip/deployment/model_server/tools/msgpack_numpy.py - CALVIN Evaluation
MiniCPM-RobotManip/evaluation/calvin/model2calvin_interface.py - LIBERO Evaluation
MiniCPM-RobotManip/evaluation/libero/parallel_worker.py - RoboTwin Evaluation
MiniCPM-RobotManip/evaluation/robotwin/launcher.py - RMBench Environment
MiniCPM-RobotManip/evaluation/rmbench/rmbench_env.py
Tracking Model
- Model Training
MiniCPM-RobotTrack/minicpm_robot_track/training.py - Tracking Data
MiniCPM-RobotTrack/minicpm_robot_track/data.py - Vision Features
MiniCPM-RobotTrack/minicpm_robot_track/vision.py - Tracking Model
MiniCPM-RobotTrack/minicpm_robot_track/modeling.py - Model Config
MiniCPM-RobotTrack/minicpm_robot_track/config.py - Checkpoint
MiniCPM-RobotTrack/minicpm_robot_track/checkpoint.py
Tracking Evaluation
- Evaluation Runner
MiniCPM-RobotTrack/minicpm_robot_track/evaluation/runner.py - Evaluation Policy
MiniCPM-RobotTrack/minicpm_robot_track/evaluation/policy.py - Evaluation Splits
MiniCPM-RobotTrack/minicpm_robot_track/evaluation/splits.py
EVT Simulation
- Rearrangement Simulator
MiniCPM-RobotTrack/evt_bench/rearrange_sim_v2.py - Navigation Actions
MiniCPM-RobotTrack/evt_bench/additional_action.py - Tracking Metrics
MiniCPM-RobotTrack/evt_bench/additional_metric.py - Additional Sensors
MiniCPM-RobotTrack/evt_bench/additional_sensor.py
Robot Deployment
- Go2 Tracking Client
MiniCPM-RobotTrack/realworld/client/http_minicpm_robot_track_client.py - Waypoint Selection
MiniCPM-RobotTrack/realworld/client/waypoint_selection.py - Go2 Runtime
MiniCPM-RobotTrack/go2_runtime.py
Other components
- User / Operator
- Tracking Developer
Connections 24
- User / Operator → VLA Inference: requests action
README.md - MemVLA Inference → MemVLA Recipe: uses settings
MiniCPM-RobotManip/vla_infer_memvla.py - MemVLA Policy → MemVLA Recipe: uses settings
MiniCPM-RobotManip/deployment/model_server/memvla_policy_wrapper.py - WebSocket Client → MessagePack Codec: packs messages
MiniCPM-RobotManip/deployment/model_server/tools/websocket_policy_client.py - CALVIN Evaluation → WebSocket Client: uses client
MiniCPM-RobotManip/evaluation/calvin/model2calvin_interface.py - LIBERO Evaluation → VLA Inference: evaluates policy
README.md - RoboTwin Evaluation → VLA Inference: evaluates policy
README.md - RMBench Environment → VLA Inference: evaluates policy
README.md - Tracking Developer → Model Training: trains tracker
- Model Training → Tracking Data: loads samples
MiniCPM-RobotTrack/minicpm_robot_track/training.py - Model Training → Model Config: loads settings
MiniCPM-RobotTrack/minicpm_robot_track/training.py - Model Training → Tracking Model: optimizes model
MiniCPM-RobotTrack/minicpm_robot_track/training.py - Model Training → Checkpoint: saves weights
MiniCPM-RobotTrack/minicpm_robot_track/training.py - Tracking Model → Model Config: validates config
MiniCPM-RobotTrack/minicpm_robot_track/modeling.py - Evaluation Runner → Evaluation Policy: runs policy
MiniCPM-RobotTrack/minicpm_robot_track/evaluation/runner.py - Evaluation Runner → Evaluation Splits: loads splits
MiniCPM-RobotTrack/minicpm_robot_track/evaluation/runner.py - Evaluation Policy → Checkpoint: loads weights
MiniCPM-RobotTrack/minicpm_robot_track/evaluation/policy.py - Evaluation Policy → Vision Features: uses vision
MiniCPM-RobotTrack/minicpm_robot_track/evaluation/policy.py - Evaluation Runner → Rearrangement Simulator: runs episodes
MiniCPM-RobotTrack/minicpm_robot_track/evaluation/runner.py - Rearrangement Simulator → Navigation Actions: registers actions
MiniCPM-RobotTrack/evt_bench/__init__.py - Rearrangement Simulator → Additional Sensors: registers sensors
MiniCPM-RobotTrack/evt_bench/__init__.py - Rearrangement Simulator → Tracking Metrics: registers metrics
MiniCPM-RobotTrack/evt_bench/__init__.py - Go2 Tracking Client → Waypoint Selection: uses selection
MiniCPM-RobotTrack/realworld/client/http_minicpm_robot_track_client.py - Go2 Tracking Client → Go2 Runtime: dispatches commands