How internrobotics/internvla-m1 works
Project Type and Purpose
• InternVLA-M1 is an open-source spatially grounded vision–language–action (VLA) foundation model for robotic manipulation.
• It provides modular components for data loading, model definition (language and action heads), training, and deployment (real-time server).
Main Architectural Layers and Components
A. Data Layer
• InternVLA/dataloader
– gr00t_lerobot: standardized robot datasets & transforms
– qwenvl_llavajson & vlm_datasets: multimodal datasets
– lerobot_datasets.py: utility loader
• playground/demo_data
– sample episodes (video + metadata) for local testing
• assets/MODEL_ZOO.md & assets PDFs
B. Configuration Layer
• InternVLA/config
– deepseeds/*: DeepSpeed Z2/Z3 configs
– training/*.yaml: Hydra-style training configs
• pyproject.toml, requirements.txt, Makefile
C. Model Layer
• InternVLA/model/framework
– base_framework.py, M1.py: core model class with .from_pretrained(), .chat_with_M1(), .predict_action()
• InternVLA/model/modules
– action_model/DiT_modules & header: diffusion-based action head
– dino_model: visual encoder
– projector/QFormer: vision→language cross-attention
– vlm/QWen2_5: pretrained VLM backbone
D. Training Layer
• InternVLA/training
– train_internvla.py, train_internvla_cotrain.py
– trainer_utils: metrics logging & Overwatch (checkpointing)
• scripts/run_scripts/*.sh: wrapper scripts to launch experiments
E. Deployment Layer
• deployment/model_server
– server_policy_M1.py / debug_server_policy.py: WebSocket-based inference service
– tools/image_tools.py, msgpack_numpy.py: data serialization
• deployment/upload/push_model_to_hf.py: pushes checkpoints to Hugging Face
F. Examples & Clients
• examples/SimplerEnv & examples/LIBERO & examples/real_robot
– eval_*.py, model2*_interface.py, start_server.sh: user-facing launches and evaluation pipelines
• playground: ad-hoc testing
Relationships and Data Flows
• Data Layer → Model Layer: raw data (images, states, text) is loaded & transformed.
• Config Layer → Training Layer: training scripts ingest Hydra/DeepSpeed configs.
• Model Layer + Training Layer → Checkpoints: produce .pt model artifacts.
• Checkpoints + Model Layer + Deployment Layer → Model Server: load weights into InternVLA_M1 class, expose inference API.
• Clients (Examples, Real Robots, SimplerEnv, LIBERO) → Model Server: send image(s) + instruction over WebSocket/HTTP → receive text answers or action sequences.
• Optional: Model Zoo (Hugging Face) is both a sink (upload) and source (download pretrained).
Architectural Patterns & Design Principles
• Modular & Extensible: each directory is a plug-in point (swap datasets, diffusion modules, VLM backbones).
• Dual-System Heads: two parallel pathways (language head & diffusion-based action head) unified in M1 framework.
• Config-driven / Hydra + DeepSpeed: separation of code vs. hyperparameters.
• Client-Server Inference: WebSocket or msgpack RPC for real-time robot control.