# keivalya/mini-vla architecture

> Architecture diagram of the GitHub repository keivalya/mini-vla, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/keivalya/mini-vla
- Repository: https://github.com/keivalya/mini-vla

## Overview

##### What this project is (type + purpose)

**mini-VLA** is a **research/educational ML codebase** that trains and evaluates a **Vision–Language–Action diffusion policy** for continuous robot control in simulation (Meta-World). It is **not** a deployed service; it’s a **training + inference pipeline** around:
- a **simulation environment** (Meta-World MT1 tasks),
- a **data collection script** (expert demonstrations → `.npz`),
- a **PyTorch model** (image/text/state encoders → fusion → diffusion action head),
- **train/test scripts** (offline imitation learning + rollout evaluation with video recording).

Top-level structure and what each folder “means” architecturally:

##### `envs/`
- `envs/metaworld_env.py`, `envs/metaworld_mt1.py`
- **Role in system**: environment wrappers/adapters around Meta-World tasks.

##### `models/` (the core VLA)
- `encoders.py`: image encoder + text encoder + robot-state encoder
- `fusion.py`: combines embeddings (MLP)
- `diffusion_head.py`: diffusion policy head that denoises to actions
- `vla_diffusion_policy.py`: composes encoders + fusion + diffusion head into one policy module

##### `utils/tokenizer.py`
- **Role**: text preprocessing/tokenization used by the text encoder and scripts.

##### `scripts/`
- `collect_data.py`: runs expert policy in env → saves `.npz`
- `train.py`: loads `.npz` → trains model → saves checkpoint
- `test.py`: loads checkpoint → runs rollouts in env using text instruction → saves videos
- `inference.py`: planned/partial; treat as “future inference entrypoint”

##### `assets/`
- demo videos/gifs; not architectural, but shows “video output artifact”.

##### Meta-World Simulator (MT1 env)
- camera view (RGB frames)
- robot state (proprioception)
- accepts continuous actions
##### Expert Policy (for demonstrations)
- used only by `collect_data.py`
##### Dataset Artifact: `.npz` trajectories
- images / states / actions / (task instruction)
##### Training Loop (`scripts/train.py`)
- dataloader
- loss/optimizer
##### Evaluation / Rollout (`scripts/test.py`)
- loads checkpoint
- runs closed-loop policy in env
- optionally writes videos

- Meta-World + Expert Policy → (trajectories) → `.npz`
- `.npz` → Train Script → (gradients update) → VLA Model → Checkpoint `.pt`
- Checkpoint `.pt` + Instruction Text + Env Observations → Test Script → Actions → Env
- Test Script → Videos folder (optional artifact)

This is the component-level view: how tensors move inside the model.

##### Inputs
- Image observation `I_t` (from env render / camera)
- Robot state `s_t`
- Text instruction `T` (string)
##### Tokenizer (`utils/tokenizer.py`)
- `T` → tokens
##### Encoders (`models/encoders.py`)
- Vision encoder: `I_t → e_img`
- Text encoder: `tokens → e_txt`
- State encoder: `s_t → e_state`
##### Fusion MLP (`models/fusion.py`)
- `(e_img, e_txt, e_state) → e_fused` (single conditioning vector)
##### Diffusion Head (`models/diffusion_head.py`)
- Takes `e_fused` as conditioning
- Iterative denoising over action (or action sequence)
- Outputs continuous action `a_t` (or horizon of actions)
##### Policy wrapper (`models/vla_diffusion_policy.py`)
- Exposes `forward()` / `act()` used by train/test scripts

- Env image/state → encoders → fusion → diffusion head → action → env step
- Instruction text → tokenizer → text encoder → fusion

##### Closed-loop control path (during `test.py`)
- `obs_t` (image+state) + instruction → model → `action_t` → env → `obs_{t+1}`
- This is the runtime system behavior.

##### Offline supervised learning path (during `train.py`)
- dataset provides `(image, state, instruction, expert_action)`
- model predicts action (via diffusion)
- loss compares predicted vs expert (or diffusion objective)
- optimizer updates model weights
- checkpoint saved

##### Data collection path (during `collect_data.py`)
- env + expert policy generate trajectories
- dataset written once, reused by training

##### Architectural patterns/design principles present

- **Separation of concerns**
- `scripts/` = orchestration/entrypoints
- `envs/` = simulation adapters
- `models/` = pure model definition
- `utils/` = reusable preprocessing (tokenizer)
- **Composable model architecture**
- encoders + fusion + head are independent swappable modules (easy to prototype)
- **Artifact-driven ML workflow**
- `.npz` dataset and `.pt` checkpoint are explicit boundaries between stages

##### Technology labels to include (only the ones that matter)

Labels on components (not as a giant tech list):
- **Meta-World** (simulator / benchmark)
- **PyTorch** (model + training)
- **Diffusion policy** (action generation head)
- **NPZ dataset** (demonstrations)
- **Video outputs** (evaluation artifact)

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =========================
  %% Diagram A: Offline IL + Online Rollout Pipeline
  %% =========================
  subgraph "Diagram A — Offline IL pipeline (collect → train → evaluate)"
    direction TB

    subgraph "External Simulation + Benchmarks"
      direction TB
      MW["Meta-World Simulator (MT1)"]:::external
      ENVW["Simulation Environment Wrappers/Adapters\n(reset/step/render)"]:::env
      TASKS["MT1 Task Setup/Definitions"]:::env
      EXP["Expert Policy\n(demonstrations only)"]:::external
    end

    subgraph "Artifacts (explicit ML boundaries)"
      direction TB
      NPZ[("NPZ Dataset\n(.npz trajectories:\nRGB, state, action, instruction)")]:::artifact
      CKPT[("Model Checkpoint\n(.pt weights)")]:::artifact
      VIDS[("Videos / GIFs\n(evaluation artifact)")]:::artifact
    end

    subgraph "Entrypoints / Orchestration (scripts/)"
      direction TB
      COLLECT["collect_data.py\n(run expert → save .npz)"]:::script
      TRAIN["train.py\n(offline imitation learning)"]:::script
      TEST["test.py\n(rollouts + record videos)"]:::script
      INFER["inference.py\n(planned/partial entrypoint)"]:::script
    end

    subgraph "Model (PyTorch) — VLA Diffusion Policy"
      direction TB
      POLICY["VLA Diffusion Policy\n(models/vla_diffusion_policy.py)"]:::model

      subgraph "Model Internals (models/)"
        direction TB
        ENC["Encoders (vision/text/state)\n(models/encoders.py)"]:::model
        FUS["Fusion MLP\n(models/fusion.py)"]:::model
        DIFF["Diffusion Policy Head\n(denoise → continuous actions)\n(models/diffusion_head.py)"]:::model
      end

      TOK["Tokenizer\n(utils/tokenizer.py)"]:::util
    end

    %% Offline lane
    MW -->|"wrapped_by"| ENVW
    TASKS -->|"configures"| ENVW
    COLLECT -->|"creates_env"| ENVW
    COLLECT -->|"uses"| EXP
    ENVW -->|"obs:RGB+state"| COLLECT
    EXP -->|"action (expert)"| COLLECT
    COLLECT -->|"writes_trajectories"| NPZ

    TRAIN -->|"loads"| NPZ
    TRAIN -->|"builds_model"| POLICY
    POLICY -->|"uses"| ENC
    POLICY -->|"uses"| FUS
    POLICY -->|"uses"| DIFF
    TRAIN -->|"tokenizes_instruction"| TOK
    TOK -->|"tokens"| ENC
    TRAIN -->|"optimizer_updates (PyTorch)"| POLICY
    TRAIN -->|"saves_weights"| CKPT

    %% Online lane (closed-loop control)
    TEST -->|"loads_weights"| CKPT
    TEST -->|"creates_env"| ENVW
    TEST -->|"runs_policy_loop"| POLICY
    ENVW -->|"RGB_frame (camera:corner)"| POLICY
    ENVW -->|"proprio_state"| POLICY
    TEST -->|"instruction_text"| TOK
    TOK -->|"tokens"| POLICY
    POLICY -->|"continuous_action"| ENVW
    TEST -->|"records"| VIDS

    %% Future inference entrypoint
    INFER -->|"loads_weights"| CKPT
    INFER -->|"calls_act()"| POLICY
  end

  %% =========================
  %% Diagram B: Model Internal Tensor/Data Flow
  %% =========================
  subgraph "Diagram B — Model internal architecture (VLA Diffusion Policy)"
    direction TB

    subgraph "Inputs"
      direction TB
      INSTR["Instruction text T\n(string)"]:::io
      IMG["Image observation I_t\n(RGB frame)"]:::io
      STATE["Robot state s_t\n(proprioception)"]:::io
    end

    subgraph "Text preprocessing"
      direction TB
      TOK2["Tokenizer\n(utils/tokenizer.py)"]:::util
      TOKENS["Tokens"]:::io
    end

    subgraph "Encoders (models/encoders.py)"
      direction TB
      VENC["Vision encoder\nI_t → e_img"]:::model
      TENC["Text encoder\ntokens → e_txt"]:::model
      SENC["State encoder\ns_t → e_state"]:::model
    end

    subgraph "Fusion + Diffusion Action Head"
      direction TB
      FUS2["Fusion MLP (models/fusion.py)\n(e_img,e_txt,e_state) → e_fused"]:::model
      DIFF2["Diffusion Head (models/diffusion_head.py)\niterative denoising conditioned on e_fused"]:::model
      ACT["Action a_t\n(continuous)"]:::io
    end

    POL2["Policy wrapper (models/vla_diffusion_policy.py)\nforward()/act()"]:::model

    %% Flow
    INSTR -->|"instruction_string→tokens"| TOK2 --> TOKENS -->|"encode"| TENC
    IMG -->|"encode"| VENC
    STATE -->|"encode"| SENC

    VENC -->|"e_img"| FUS2
    TENC -->|"e_txt"| FUS2
    SENC -->|"e_state"| FUS2

    FUS2 -->|"conditioning_embedding e_fused"| DIFF2
    DIFF2 -->|"continuous_action"| ACT
    POL2 -->|"exposes"| ACT

    %% Wrapper composition (conceptual)
    POL2 -.->|"contains"| VENC
    POL2 -.->|"contains"| TENC
    POL2 -.->|"contains"| SENC
    POL2 -.->|"contains"| FUS2
    POL2 -.->|"contains"| DIFF2
    POL2 -.->|"uses"| TOK2
  end

  %% =========================
  %% Notes / Legend
  %% =========================
  subgraph "Notes / Design principles"
    direction TB
    NOTE1["Separation of concerns:\n- scripts/ orchestrate\n- envs/ adapt simulator\n- models/ define PyTorch modules\n- utils/ preprocessing"]:::note
    NOTE2["Artifact-driven workflow:\nNPZ dataset ↔ checkpoint are explicit stage boundaries"]:::note
    NOTE3["Composable model:\nEncoders + Fusion + Diffusion head are swappable modules"]:::note
    NOTE4["Dependencies (see requirements.txt):\nPyTorch, Meta-World, etc."]:::note
  end

  %% =========================
  %% Click Events (from component_mapping)
  %% =========================
  click ENVW "https://github.com/keivalya/mini-vla/blob/main/envs/metaworld_env.py"
  click TASKS "https://github.com/keivalya/mini-vla/blob/main/envs/metaworld_mt1.py"
  click ENC "https://github.com/keivalya/mini-vla/blob/main/models/encoders.py"
  click FUS "https://github.com/keivalya/mini-vla/blob/main/models/fusion.py"
  click DIFF "https://github.com/keivalya/mini-vla/blob/main/models/diffusion_head.py"
  click POLICY "https://github.com/keivalya/mini-vla/blob/main/models/vla_diffusion_policy.py"
  click TOK "https://github.com/keivalya/mini-vla/blob/main/utils/tokenizer.py"
  click COLLECT "https://github.com/keivalya/mini-vla/blob/main/scripts/collect_data.py"
  click TRAIN "https://github.com/keivalya/mini-vla/blob/main/scripts/train.py"
  click TEST "https://github.com/keivalya/mini-vla/blob/main/scripts/test.py"
  click INFER "https://github.com/keivalya/mini-vla/blob/main/scripts/inference.py"
  click VIDS "https://github.com/keivalya/mini-vla/blob/main/assets/corner1_push.mp4"
  click VIDS "https://github.com/keivalya/mini-vla/blob/main/assets/corner2_push.mp4"
  click NOTE4 "https://github.com/keivalya/mini-vla/blob/main/requirements.txt"

  %% Also link Diagram B core nodes where applicable
  click TOK2 "https://github.com/keivalya/mini-vla/blob/main/utils/tokenizer.py"
  click VENC "https://github.com/keivalya/mini-vla/blob/main/models/encoders.py"
  click TENC "https://github.com/keivalya/mini-vla/blob/main/models/encoders.py"
  click SENC "https://github.com/keivalya/mini-vla/blob/main/models/encoders.py"
  click FUS2 "https://github.com/keivalya/mini-vla/blob/main/models/fusion.py"
  click DIFF2 "https://github.com/keivalya/mini-vla/blob/main/models/diffusion_head.py"
  click POL2 "https://github.com/keivalya/mini-vla/blob/main/models/vla_diffusion_policy.py"

  %% =========================
  %% Styles
  %% =========================
  classDef external fill:#F7D7C4,stroke:#B85C38,stroke-width:2px,stroke-dasharray: 6 4,color:#1F1F1F;
  classDef env fill:#FFD9A6,stroke:#B26A00,stroke-width:2px,color:#1F1F1F;
  classDef script fill:#CFEED0,stroke:#2E7D32,stroke-width:2px,color:#1F1F1F;
  classDef model fill:#CFE4FF,stroke:#1E5AA8,stroke-width:2px,color:#0F1B2D;
  classDef util fill:#D7F2FF,stroke:#0077A3,stroke-width:2px,color:#0F1B2D;
  classDef artifact fill:#E6E6E6,stroke:#666666,stroke-width:2px,color:#1F1F1F;
  classDef io fill:#FFFFFF,stroke:#666666,stroke-width:1.5px,color:#1F1F1F;
  classDef note fill:#FFF6B8,stroke:#B59B00,stroke-width:1.5px,color:#1F1F1F;
```
