How huixi-ai/rhinovla works
RhinoVLA is a cross-embodiment vision-language-action system with a fine-tuning stack and an RPU inference backend. The main workflow maps LeRobot demonstrations into the 72D contract, trains a Qwen3-VL-plus-action-expert policy, then uses prepared artifacts and normalization data for robot inference. The graph also includes documented evaluation and visualization capabilities. Sampled excerpts support the principal data and model relationships; inference policy internals and several training subsystems are represented only at their documented enclosing paths.
Main entry points: train.py Training orchestration · executor.py RPU executor · lerobot_native72.py LeRobot adapter · compute_norm_json.py Norm computation
Components 23
Data preparation
- 72D mapping
configs/data_mappings/demo_lerobot_native72.yaml - Norm computation
datasets/compute_norm_json.py - LeRobot adapter
rhinovla/dataloader/lerobot_native72.py
Policy model
- RhinoVLA framework
rhinovla/model/framework/RhinoVLA.py - Qwen3-VL encoder
rhinovla/model/modules/qwen.py - Qwen processor assets
rhinovla/assets/qwen3_vl_processor - Action expert
rhinovla/model/modules/action_expert.py
Training and evaluation
- Fine-tune config
configs/training/demo_full_finetune.yaml - Training orchestration
rhinovla/training/train.py - Checkpoint and optimizer tools
rhinovla/training/trainer_utils/trainer_tools.py - Evaluation and metrics
rhinovla/training/eval.py - Training visualizations
rhinovla/training/visualize.py
RPU inference
- Inference CLI
rhinovla/inference/rpu_backend/cli.py - RPU executor
rhinovla/inference/rpu_backend/executor.py - Inference normalization
rhinovla/inference/rpu_backend/norm.py
Other components
- Model engineer
- LeRobot demonstrations
datasets/example_lerobot_v3 - Normalization stats
datasets/example_lerobot_v3/meta/norm.json - Trained checkpoint
- Robot operator
- Prepared RPU policy
- Robot cameras and state
- Robot action interface
Connections 26
- Model engineer → Norm computation: computes stats
- LeRobot demonstrations → Norm computation: reads demonstrations
- 72D mapping → Norm computation: defines slots
- Norm computation → Normalization stats: writes stats
- Model engineer → Training orchestration: starts fine-tuning
- Fine-tune config → Training orchestration: configures run
- Training orchestration → LeRobot adapter: builds batches
- LeRobot adapter → LeRobot demonstrations: reads samples
- LeRobot adapter → 72D mapping: maps state and actions
- LeRobot adapter → Normalization stats: checks statistics
- Training orchestration → RhinoVLA framework: trains policy
- RhinoVLA framework → Qwen3-VL encoder: encodes vision and language
- Qwen3-VL encoder → Qwen processor assets: loads processor
- RhinoVLA framework → Action expert: predicts actions
- Training orchestration → Checkpoint and optimizer tools: manages optimization
- Training orchestration → Evaluation and metrics: evaluates policy
- Training orchestration → Training visualizations: logs visualizations
- Checkpoint and optimizer tools → Trained checkpoint: saves checkpoint
- Robot operator → Inference CLI: requests inference
- Inference CLI → RPU executor: configures executor
- RPU executor → Prepared RPU policy: loads prepared policy
- RPU executor → Inference normalization: loads normalization
- Inference normalization → 72D mapping: expands native slots
- RPU executor → Robot cameras and state: submits observations
- RPU executor → Robot action interface: returns action chunks
- Trained checkpoint → RPU executor: optional checkpoint