# internrobotics/internvla-m1 architecture

> Architecture diagram of the GitHub repository internrobotics/internvla-m1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/internrobotics/internvla-m1
- Repository: https://github.com/internrobotics/internvla-m1

## Overview

##### Project Type and Purpose
- InternVLA-M1 is an open-source spatially grounded vision–language–action (VLA) foundation model for robotic manipulation.
- It provides modular components for data loading, model definition (language and action heads), training, and deployment (real-time server).

##### Main Architectural Layers and Components

A. Data Layer
- InternVLA/dataloader
– gr00t_lerobot: standardized robot datasets & transforms
– qwenvl_llavajson & vlm_datasets: multimodal datasets
– lerobot_datasets.py: utility loader
- playground/demo_data
– sample episodes (video + metadata) for local testing
- assets/MODEL_ZOO.md & assets PDFs

B. Configuration Layer
- InternVLA/config
– deepseeds/*: DeepSpeed Z2/Z3 configs
– training/*.yaml: Hydra-style training configs
- pyproject.toml, requirements.txt, Makefile

C. Model Layer
- InternVLA/model/framework
– base_framework.py, M1.py: core model class with .from_pretrained(), .chat_with_M1(), .predict_action()
- InternVLA/model/modules
– action_model/DiT_modules & header: diffusion-based action head
– dino_model: visual encoder
– projector/QFormer: vision→language cross-attention
– vlm/QWen2_5: pretrained VLM backbone

D. Training Layer
- InternVLA/training
– train_internvla.py, train_internvla_cotrain.py
– trainer_utils: metrics logging & Overwatch (checkpointing)
- scripts/run_scripts/*.sh: wrapper scripts to launch experiments

E. Deployment Layer
- deployment/model_server
– server_policy_M1.py / debug_server_policy.py: WebSocket-based inference service
– tools/image_tools.py, msgpack_numpy.py: data serialization
- deployment/upload/push_model_to_hf.py: pushes checkpoints to Hugging Face

F. Examples & Clients
- examples/SimplerEnv & examples/LIBERO & examples/real_robot
– eval_*.py, model2*_interface.py, start_server.sh: user-facing launches and evaluation pipelines
- playground: ad-hoc testing

##### Relationships and Data Flows
- Data Layer → Model Layer: raw data (images, states, text) is loaded & transformed.
- Config Layer → Training Layer: training scripts ingest Hydra/DeepSpeed configs.
- Model Layer + Training Layer → Checkpoints: produce .pt model artifacts.
- Checkpoints + Model Layer + Deployment Layer → Model Server: load weights into InternVLA_M1 class, expose inference API.
- Clients (Examples, Real Robots, SimplerEnv, LIBERO) → Model Server: send image(s) + instruction over WebSocket/HTTP → receive text answers or action sequences.
- Optional: Model Zoo (Hugging Face) is both a sink (upload) and source (download pretrained).

##### Architectural Patterns & Design Principles
- Modular & Extensible: each directory is a plug-in point (swap datasets, diffusion modules, VLM backbones).
- Dual-System Heads: two parallel pathways (language head & diffusion-based action head) unified in M1 framework.
- Config-driven / Hydra + DeepSpeed: separation of code vs. hyperparameters.
- Client-Server Inference: WebSocket or msgpack RPC for real-time robot control.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DL1["InternVLA/dataloader"]:::code
        DL2["playground/demo_data"]:::data
        DL3["assets/MODEL_ZOO.md"]:::data
        DL4["assets/InternVLA_M1.pdf"]:::data
    end

    %% Config Layer
    subgraph "Configuration Layer"
        direction TB
        CL1["InternVLA/config/deepseeds"]:::config
        CL2["InternVLA/config/training"]:::config
        CL3["pyproject.toml"]:::config
        CL4["requirements.txt"]:::config
        CL5["Makefile"]:::config
    end

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        ML1["base_framework.py"]:::code
        ML2["M1.py"]:::code
        ML3["DiTActionHeader.py"]:::code
        ML4["DiT_modules"]:::code
        ML5["dino_model"]:::code
        ML6["QFormer.py"]:::code
        ML7["QWen2_5.py"]:::code
        ML8["tools.py"]:::code
    end

    %% Training Layer
    subgraph "Training Layer"
        direction TB
        TL1["train_internvla.py"]:::code
        TL2["train_internvla_cotrain.py"]:::code
        TL3["metrics.py"]:::code
        TL4["overwatch.py"]:::code
        TL5["scripts/run_scripts/*.sh"]:::runtime
    end

    %% Deployment Layer
    subgraph "Deployment Layer"
        direction TB
        DLv1["server_policy_M1.py"]:::runtime
        DLv2["debug_server_policy.py"]:::runtime
        DLv3["image_tools.py"]:::runtime
        DLv4["msgpack_numpy.py"]:::runtime
        DLv5["websocket_policy_client.py"]:::runtime
        DLv6["websocket_policy_server.py"]:::runtime
        DLv7["push_model_to_hf.py"]:::runtime
    end

    %% Clients Layer
    subgraph "Examples & Clients"
        direction TB
        CLt1["examples/SimplerEnv"]:::code
        CLt2["examples/LIBERO"]:::code
        CLt3["examples/real_robot"]:::code
        CLt4["playground"]:::data
    end

    %% Relationships
    DL1 -->|"raw data"| ML1
    DL2 -->|"demo episodes"| ML1
    DL3 -->|"model zoo docs"| DLv7
    CL1 --> TL1
    CL2 --> TL1
    CL3 --> TL1
    CL4 --> TL1
    CL5 --> TL1
    ML1 --> TL1
    ML2 --> TL1
    ML3 --> TL1
    ML4 --> TL1
    ML5 --> TL1
    ML6 --> TL1
    ML7 --> TL1
    ML8 --> TL1
    TL1 -->|"checkpoints"| DLv1
    TL2 -->|"checkpoints"| DLv1
    DLv1 -->|"inference API"| CLt1
    DLv1 -->|"inference API"| CLt2
    DLv1 -->|"inference API"| CLt3
    CLt4 --> ML1
    DLv7 -->|"upload/download"| DL3

    %% Click Events
    click DL1 "https://github.com/internrobotics/internvla-m1/tree/InternVLA-M1/InternVLA/dataloader"
    click DL2 "https://github.com/internrobotics/internvla-m1/tree/InternVLA-M1/playground/demo_data"
    click DL3 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/assets/MODEL_ZOO.md"
    click DL4 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/assets/InternVLA_M1.pdf"
    click CL1 "https://github.com/internrobotics/internvla-m1/tree/InternVLA-M1/InternVLA/config/deepseeds"
    click CL2 "https://github.com/internrobotics/internvla-m1/tree/InternVLA-M1/InternVLA/config/training"
    click ML1 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/InternVLA/model/framework/base_framework.py"
    click ML2 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/InternVLA/model/framework/M1.py"
    click ML3 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/InternVLA/model/modules/action_model/DiTActionHeader.py"
    click ML4 "https://github.com/internrobotics/internvla-m1/tree/InternVLA-M1/InternVLA/model/modules/action_model/DiT_modules"
    click ML5 "https://github.com/internrobotics/internvla-m1/tree/InternVLA-M1/InternVLA/model/modules/dino_model"
    click ML6 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/InternVLA/model/modules/projector/QFormer.py"
    click ML7 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/InternVLA/model/modules/vlm/QWen2_5.py"
    click ML8 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/InternVLA/model/tools.py"
    click TL1 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/InternVLA/training/train_internvla.py"
    click TL2 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/InternVLA/training/train_internvla_cotrain.py"
    click TL3 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/InternVLA/training/trainer_utils/metrics.py"
    click TL4 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/InternVLA/training/trainer_utils/overwatch.py"
    click TL5 "https://github.com/internrobotics/internvla-m1/tree/InternVLA-M1/scripts/run_scripts"
    click DLv1 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/deployment/model_server/server_policy_M1.py"
    click DLv2 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/deployment/model_server/debug_server_policy.py"
    click DLv3 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/deployment/model_server/tools/image_tools.py"
    click DLv4 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/deployment/model_server/tools/msgpack_numpy.py"
    click DLv5 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/deployment/model_server/tools/websocket_policy_client.py"
    click DLv6 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/deployment/model_server/tools/websocket_policy_server.py"
    click DLv7 "https://github.com/internrobotics/internvla-m1/blob/InternVLA-M1/deployment/upload/push_model_to_hf.py"
    click CLt1 "https://github.com/internrobotics/internvla-m1/tree/InternVLA-M1/examples/SimplerEnv"
    click CLt2 "https://github.com/internrobotics/internvla-m1/tree/InternVLA-M1/examples/LIBERO"
    click CLt3 "https://github.com/internrobotics/internvla-m1/tree/InternVLA-M1/examples/real_robot"
    click CLt4 "https://github.com/internrobotics/internvla-m1/tree/InternVLA-M1/playground"

    %% Styles
    classDef code fill:#ADD8E6,stroke:#000
    classDef data fill:#90EE90,stroke:#000
    classDef config fill:#FFA500,stroke:#000
    classDef runtime fill:#DDA0DD,stroke:#000
```
