# openbmb/minicpm-robot architecture

> Architecture diagram of the GitHub repository openbmb/minicpm-robot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openbmb/minicpm-robot
- Repository: https://github.com/openbmb/minicpm-robot

## Overview

MiniCPM-Robot is a two-part embodied-AI repository: RobotManip serves a streaming vision-language-action policy and evaluates it across manipulation benchmarks; RobotTrack trains/evaluates an on-device visual tracker and deploys it for real-world Go2 tracking. The graph separates model construction, benchmark evaluation, simulation, and robot deployment. Source-index references support the listed internal imports; README capability statements identify product branches. Several server and runtime connections remain unverified because their relevant implementation was not sampled.

Main entry points: [`MiniCPM-RobotTrack/minicpm_robot_track/training.py`](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/training.py) (Model Training), [`MiniCPM-RobotTrack/evt_bench/rearrange_sim_v2.py`](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/evt_bench/rearrange_sim_v2.py) (Rearrangement Simulator), [`MiniCPM-RobotManip/vla_infer.py`](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/vla_infer.py) (VLA Inference), [`MiniCPM-RobotTrack/minicpm_robot_track/evaluation/runner.py`](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/evaluation/runner.py) (Evaluation Runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_manip["RobotManip"]
  node_manip_infer["VLA Inference<br/>[vla_infer.py]"]
  node_mem_infer["MemVLA Inference"]
  node_recipe["MemVLA Recipe<br/>[memvla_recipe.py]"]
  node_mem_wrapper["MemVLA Policy"]
  node_policy_server["Policy Server"]
  node_ws_client["WebSocket Client"]
  node_msgpack["MessagePack Codec<br/>[msgpack_numpy.py]"]
  node_calvin["CALVIN Evaluation"]
  node_libero["LIBERO Evaluation<br/>[parallel_worker.py]"]
  node_robotwin["RoboTwin Evaluation<br/>[launcher.py]"]
  node_rmbench["RMBench Environment<br/>[rmbench_env.py]"]
end

subgraph group_trackmodel["Tracking Model"]
  node_training["Model Training<br/>[training.py]"]
  node_data["Tracking Data<br/>[data.py]"]
  node_features["Vision Features<br/>[vision.py]"]
  node_tracker["Tracking Model<br/>[modeling.py]"]
  node_config["Model Config<br/>[config.py]"]
  node_checkpoint["Checkpoint<br/>[checkpoint.py]"]
end

subgraph group_trackeval["Tracking Evaluation"]
  node_eval_runner["Evaluation Runner<br/>[runner.py]"]
  node_eval_policy["Evaluation Policy<br/>[policy.py]"]
  node_eval_splits["Evaluation Splits<br/>[splits.py]"]
end

subgraph group_sim["EVT Simulation"]
  node_evt_sim["Rearrangement Simulator"]
  node_evt_actions["Navigation Actions"]
  node_evt_metrics["Tracking Metrics"]
  node_evt_sensors["Additional Sensors"]
end

subgraph group_robot["Robot Deployment"]
  node_go2_client["Go2 Tracking Client"]
  node_waypoint["Waypoint Selection"]
  node_go2_runtime["Go2 Runtime<br/>[go2_runtime.py]"]
end

node_operator(("User / Operator"))
node_track_operator(("Tracking Developer"))

node_operator -.->|"requests action"| node_manip_infer
node_mem_infer -->|"uses settings"| node_recipe
node_mem_wrapper -->|"uses settings"| node_recipe
node_ws_client -->|"packs messages"| node_msgpack
node_calvin -->|"uses client"| node_ws_client
node_libero -.->|"evaluates policy"| node_manip_infer
node_robotwin -.->|"evaluates policy"| node_manip_infer
node_rmbench -.->|"evaluates policy"| node_manip_infer
node_track_operator -.->|"trains tracker"| node_training
node_training -->|"loads samples"| node_data
node_training -->|"loads settings"| node_config
node_training -->|"optimizes model"| node_tracker
node_training -->|"saves weights"| node_checkpoint
node_tracker -->|"validates config"| node_config
node_eval_runner -->|"runs policy"| node_eval_policy
node_eval_runner -->|"loads splits"| node_eval_splits
node_eval_policy -->|"loads weights"| node_checkpoint
node_eval_policy -->|"uses vision"| node_features
node_eval_runner -->|"runs episodes"| node_evt_sim
node_evt_sim -->|"registers actions"| node_evt_actions
node_evt_sim -->|"registers sensors"| node_evt_sensors
node_evt_sim -->|"registers metrics"| node_evt_metrics
node_go2_client -->|"uses selection"| node_waypoint
node_go2_client -.->|"dispatches commands"| node_go2_runtime

click node_manip_infer "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/vla_infer.py"
click node_mem_infer "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/vla_infer_memvla.py"
click node_recipe "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/deployment/model_server/memvla_recipe.py"
click node_mem_wrapper "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/deployment/model_server/memvla_policy_wrapper.py"
click node_policy_server "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/deployment/model_server/tools/websocket_policy_server.py"
click node_ws_client "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/deployment/model_server/tools/websocket_policy_client.py"
click node_msgpack "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/deployment/model_server/tools/msgpack_numpy.py"
click node_calvin "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/evaluation/calvin/model2calvin_interface.py"
click node_libero "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/evaluation/libero/parallel_worker.py"
click node_robotwin "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/evaluation/robotwin/launcher.py"
click node_rmbench "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/evaluation/rmbench/rmbench_env.py"
click node_training "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/training.py"
click node_data "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/data.py"
click node_features "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/vision.py"
click node_tracker "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/modeling.py"
click node_config "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/config.py"
click node_checkpoint "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/checkpoint.py"
click node_eval_runner "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/evaluation/runner.py"
click node_eval_policy "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/evaluation/policy.py"
click node_eval_splits "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/evaluation/splits.py"
click node_evt_sim "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/evt_bench/rearrange_sim_v2.py"
click node_evt_actions "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/evt_bench/additional_action.py"
click node_evt_metrics "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/evt_bench/additional_metric.py"
click node_evt_sensors "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/evt_bench/additional_sensor.py"
click node_go2_client "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/realworld/client/http_minicpm_robot_track_client.py"
click node_waypoint "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/realworld/client/waypoint_selection.py"
click node_go2_runtime "https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/go2_runtime.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_manip_infer,node_mem_infer,node_recipe,node_mem_wrapper,node_policy_server,node_ws_client,node_msgpack,node_calvin,node_libero,node_robotwin,node_rmbench,node_operator toneBlue
class node_training,node_data,node_features,node_tracker,node_config,node_checkpoint toneAmber
class node_eval_runner,node_eval_policy,node_eval_splits toneMint
class node_evt_sim,node_evt_actions,node_evt_metrics,node_evt_sensors toneRose
class node_go2_client,node_waypoint,node_go2_runtime,node_track_operator toneIndigo
```

## Components

### RobotManip

- [VLA Inference](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/vla_infer.py): `MiniCPM-RobotManip/vla_infer.py`
- [MemVLA Inference](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/vla_infer_memvla.py): `MiniCPM-RobotManip/vla_infer_memvla.py`
- [MemVLA Recipe](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/deployment/model_server/memvla_recipe.py): `MiniCPM-RobotManip/deployment/model_server/memvla_recipe.py`
- [MemVLA Policy](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/deployment/model_server/memvla_policy_wrapper.py): `MiniCPM-RobotManip/deployment/model_server/memvla_policy_wrapper.py`
- [Policy Server](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/deployment/model_server/tools/websocket_policy_server.py): `MiniCPM-RobotManip/deployment/model_server/tools/websocket_policy_server.py`
- [WebSocket Client](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/deployment/model_server/tools/websocket_policy_client.py): `MiniCPM-RobotManip/deployment/model_server/tools/websocket_policy_client.py`
- [MessagePack Codec](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/deployment/model_server/tools/msgpack_numpy.py): `MiniCPM-RobotManip/deployment/model_server/tools/msgpack_numpy.py`
- [CALVIN Evaluation](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/evaluation/calvin/model2calvin_interface.py): `MiniCPM-RobotManip/evaluation/calvin/model2calvin_interface.py`
- [LIBERO Evaluation](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/evaluation/libero/parallel_worker.py): `MiniCPM-RobotManip/evaluation/libero/parallel_worker.py`
- [RoboTwin Evaluation](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/evaluation/robotwin/launcher.py): `MiniCPM-RobotManip/evaluation/robotwin/launcher.py`
- [RMBench Environment](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotManip/evaluation/rmbench/rmbench_env.py): `MiniCPM-RobotManip/evaluation/rmbench/rmbench_env.py`

### Tracking Model

- [Model Training](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/training.py): `MiniCPM-RobotTrack/minicpm_robot_track/training.py`
- [Tracking Data](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/data.py): `MiniCPM-RobotTrack/minicpm_robot_track/data.py`
- [Vision Features](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/vision.py): `MiniCPM-RobotTrack/minicpm_robot_track/vision.py`
- [Tracking Model](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/modeling.py): `MiniCPM-RobotTrack/minicpm_robot_track/modeling.py`
- [Model Config](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/config.py): `MiniCPM-RobotTrack/minicpm_robot_track/config.py`
- [Checkpoint](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/checkpoint.py): `MiniCPM-RobotTrack/minicpm_robot_track/checkpoint.py`

### Tracking Evaluation

- [Evaluation Runner](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/evaluation/runner.py): `MiniCPM-RobotTrack/minicpm_robot_track/evaluation/runner.py`
- [Evaluation Policy](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/evaluation/policy.py): `MiniCPM-RobotTrack/minicpm_robot_track/evaluation/policy.py`
- [Evaluation Splits](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/minicpm_robot_track/evaluation/splits.py): `MiniCPM-RobotTrack/minicpm_robot_track/evaluation/splits.py`

### EVT Simulation

- [Rearrangement Simulator](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/evt_bench/rearrange_sim_v2.py): `MiniCPM-RobotTrack/evt_bench/rearrange_sim_v2.py`
- [Navigation Actions](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/evt_bench/additional_action.py): `MiniCPM-RobotTrack/evt_bench/additional_action.py`
- [Tracking Metrics](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/evt_bench/additional_metric.py): `MiniCPM-RobotTrack/evt_bench/additional_metric.py`
- [Additional Sensors](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/evt_bench/additional_sensor.py): `MiniCPM-RobotTrack/evt_bench/additional_sensor.py`

### Robot Deployment

- [Go2 Tracking Client](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/realworld/client/http_minicpm_robot_track_client.py): `MiniCPM-RobotTrack/realworld/client/http_minicpm_robot_track_client.py`
- [Waypoint Selection](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/realworld/client/waypoint_selection.py): `MiniCPM-RobotTrack/realworld/client/waypoint_selection.py`
- [Go2 Runtime](https://github.com/openbmb/minicpm-robot/blob/main/MiniCPM-RobotTrack/go2_runtime.py): `MiniCPM-RobotTrack/go2_runtime.py`

### Other components

- **User / Operator**
- **Tracking Developer**

## Connections

- User / Operator → VLA Inference: requests action (evidence: `README.md`)
- MemVLA Inference → MemVLA Recipe: uses settings (evidence: `MiniCPM-RobotManip/vla_infer_memvla.py`)
- MemVLA Policy → MemVLA Recipe: uses settings (evidence: `MiniCPM-RobotManip/deployment/model_server/memvla_policy_wrapper.py`)
- WebSocket Client → MessagePack Codec: packs messages (evidence: `MiniCPM-RobotManip/deployment/model_server/tools/websocket_policy_client.py`)
- CALVIN Evaluation → WebSocket Client: uses client (evidence: `MiniCPM-RobotManip/evaluation/calvin/model2calvin_interface.py`)
- LIBERO Evaluation → VLA Inference: evaluates policy (evidence: `README.md`)
- RoboTwin Evaluation → VLA Inference: evaluates policy (evidence: `README.md`)
- RMBench Environment → VLA Inference: evaluates policy (evidence: `README.md`)
- Tracking Developer → Model Training: trains tracker
- Model Training → Tracking Data: loads samples (evidence: `MiniCPM-RobotTrack/minicpm_robot_track/training.py`)
- Model Training → Model Config: loads settings (evidence: `MiniCPM-RobotTrack/minicpm_robot_track/training.py`)
- Model Training → Tracking Model: optimizes model (evidence: `MiniCPM-RobotTrack/minicpm_robot_track/training.py`)
- Model Training → Checkpoint: saves weights (evidence: `MiniCPM-RobotTrack/minicpm_robot_track/training.py`)
- Tracking Model → Model Config: validates config (evidence: `MiniCPM-RobotTrack/minicpm_robot_track/modeling.py`)
- Evaluation Runner → Evaluation Policy: runs policy (evidence: `MiniCPM-RobotTrack/minicpm_robot_track/evaluation/runner.py`)
- Evaluation Runner → Evaluation Splits: loads splits (evidence: `MiniCPM-RobotTrack/minicpm_robot_track/evaluation/runner.py`)
- Evaluation Policy → Checkpoint: loads weights (evidence: `MiniCPM-RobotTrack/minicpm_robot_track/evaluation/policy.py`)
- Evaluation Policy → Vision Features: uses vision (evidence: `MiniCPM-RobotTrack/minicpm_robot_track/evaluation/policy.py`)
- Evaluation Runner → Rearrangement Simulator: runs episodes (evidence: `MiniCPM-RobotTrack/minicpm_robot_track/evaluation/runner.py`)
- Rearrangement Simulator → Navigation Actions: registers actions (evidence: `MiniCPM-RobotTrack/evt_bench/__init__.py`)
- Rearrangement Simulator → Additional Sensors: registers sensors (evidence: `MiniCPM-RobotTrack/evt_bench/__init__.py`)
- Rearrangement Simulator → Tracking Metrics: registers metrics (evidence: `MiniCPM-RobotTrack/evt_bench/__init__.py`)
- Go2 Tracking Client → Waypoint Selection: uses selection (evidence: `MiniCPM-RobotTrack/realworld/client/http_minicpm_robot_track_client.py`)
- Go2 Tracking Client → Go2 Runtime: dispatches commands
