# vimalabs/vima architecture

> Architecture diagram of the GitHub repository vimalabs/vima, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vimalabs/vima
- Repository: https://github.com/vimalabs/vima

## Overview

VIMA is a multimodal-prompt robot-manipulation policy: it combines textual and visual prompt tokens with object-centric observations, then predicts actions for environment interaction. The main path runs through checkpoint loading, prompt/observation encoding, cross-attention sequence modeling, and action decoding; action embeddings support the policy’s action representation. README also documents Gato, Flamingo, and GPT baseline policies. The excerpts show core policy and encoder wiring, but not the demo’s runtime calls or all baseline internals, so those links are omitted where uncertain.

Main entry points: [`vima/policy/vima_policy.py`](https://github.com/vimalabs/vima/blob/main/vima/policy/vima_policy.py) (VIMA policy), [`vima/nn/obj_encoder/obj_encoder.py`](https://github.com/vimalabs/vima/blob/main/vima/nn/obj_encoder/obj_encoder.py) (Object encoder), [`vima/__init__.py`](https://github.com/vimalabs/vima/blob/main/vima/__init__.py) (Checkpoint loader), [`vima/nn/seq_modeling/xattn_gpt/xattn_gpt.py`](https://github.com/vimalabs/vima/blob/main/vima/nn/seq_modeling/xattn_gpt/xattn_gpt.py) (Cross-attention GPT)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Policy entry"]
  node_checkpoint["Checkpoint loader<br/>[__init__.py]"]
  node_policy["VIMA policy<br/>[vima_policy.py]"]
  node_gato["Gato baseline"]
  node_flamingo["Flamingo baseline"]
  node_gpt["GPT baseline<br/>[vima_gpt_policy.py]"]
end

subgraph group_encoding["Multimodal encoding"]
  node_prompt["Prompt encoder<br/>[prompt_encoder.py]"]
  node_words["Word embedding<br/>[word_embd.py]"]
  node_objects["Object encoder<br/>[obj_encoder.py]"]
  node_vision["Vision transformer<br/>[vit.py]"]
  node_perceiver["Objects Perceiver<br/>[perceiver.py]"]
end

subgraph group_model["Sequence models"]
  node_xattn["Cross-attention GPT<br/>[xattn_gpt.py]"]
  node_xcomponents["Attention components<br/>[components.py]"]
  node_gptmodel["GPT sequence model<br/>[gpt.py]"]
end

subgraph group_actions["Action modeling"]
  node_action_embedding["Action embedding<br/>[action_embd.py]"]
  node_decoder["Action decoder<br/>[action_decoder.py]"]
  node_action_dist["Action distributions<br/>[dists.py]"]
end

node_user(("Robot task"))
node_environment(("VIMA-Bench environment"))

node_user -->|"selects checkpoint"| node_checkpoint
node_checkpoint -->|"loads weights"| node_policy
node_user -->|"provides task"| node_policy
node_environment -->|"provides observations"| node_policy
node_policy -->|"encodes text"| node_prompt
node_policy -->|"embeds words"| node_words
node_policy -->|"encodes images"| node_objects
node_objects -->|"encodes crops"| node_vision
node_objects -->|"aggregates features"| node_perceiver
node_policy -->|"embeds actions"| node_action_embedding
node_policy -->|"models token sequence"| node_xattn
node_xattn -->|"uses attention blocks"| node_xcomponents
node_policy -->|"decodes actions"| node_decoder
node_decoder -->|"builds distributions"| node_action_dist
node_policy -->|"returns actions"| node_environment

click node_checkpoint "https://github.com/vimalabs/vima/blob/main/vima/__init__.py"
click node_policy "https://github.com/vimalabs/vima/blob/main/vima/policy/vima_policy.py"
click node_gato "https://github.com/vimalabs/vima/blob/main/vima/policy/vima_gato_policy.py"
click node_flamingo "https://github.com/vimalabs/vima/blob/main/vima/policy/vima_flamingo_policy.py"
click node_gpt "https://github.com/vimalabs/vima/blob/main/vima/policy/vima_gpt_policy.py"
click node_prompt "https://github.com/vimalabs/vima/blob/main/vima/nn/prompt_encoder/prompt_encoder.py"
click node_words "https://github.com/vimalabs/vima/blob/main/vima/nn/prompt_encoder/word_embd.py"
click node_objects "https://github.com/vimalabs/vima/blob/main/vima/nn/obj_encoder/obj_encoder.py"
click node_vision "https://github.com/vimalabs/vima/blob/main/vima/nn/obj_encoder/vit/vit.py"
click node_perceiver "https://github.com/vimalabs/vima/blob/main/vima/nn/obj_encoder/perceiver/perceiver.py"
click node_xattn "https://github.com/vimalabs/vima/blob/main/vima/nn/seq_modeling/xattn_gpt/xattn_gpt.py"
click node_xcomponents "https://github.com/vimalabs/vima/blob/main/vima/nn/seq_modeling/xattn_gpt/components.py"
click node_gptmodel "https://github.com/vimalabs/vima/blob/main/vima/nn/seq_modeling/gpt/gpt.py"
click node_action_embedding "https://github.com/vimalabs/vima/blob/main/vima/nn/action_embd/action_embd.py"
click node_decoder "https://github.com/vimalabs/vima/blob/main/vima/nn/action_decoder/action_decoder.py"
click node_action_dist "https://github.com/vimalabs/vima/blob/main/vima/nn/action_decoder/dists.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_checkpoint,node_policy,node_gato,node_flamingo,node_gpt toneBlue
class node_prompt,node_words,node_objects,node_vision,node_perceiver toneAmber
class node_xattn,node_xcomponents,node_gptmodel toneMint
class node_action_embedding,node_decoder,node_action_dist,node_user toneRose
class node_environment toneIndigo
```

## Components

### Policy entry

- [Checkpoint loader](https://github.com/vimalabs/vima/blob/main/vima/__init__.py): `vima/__init__.py`
- [VIMA policy](https://github.com/vimalabs/vima/blob/main/vima/policy/vima_policy.py): `vima/policy/vima_policy.py`
- [Gato baseline](https://github.com/vimalabs/vima/blob/main/vima/policy/vima_gato_policy.py): `vima/policy/vima_gato_policy.py`
- [Flamingo baseline](https://github.com/vimalabs/vima/blob/main/vima/policy/vima_flamingo_policy.py): `vima/policy/vima_flamingo_policy.py`
- [GPT baseline](https://github.com/vimalabs/vima/blob/main/vima/policy/vima_gpt_policy.py): `vima/policy/vima_gpt_policy.py`

### Multimodal encoding

- [Prompt encoder](https://github.com/vimalabs/vima/blob/main/vima/nn/prompt_encoder/prompt_encoder.py): `vima/nn/prompt_encoder/prompt_encoder.py`
- [Word embedding](https://github.com/vimalabs/vima/blob/main/vima/nn/prompt_encoder/word_embd.py): `vima/nn/prompt_encoder/word_embd.py`
- [Object encoder](https://github.com/vimalabs/vima/blob/main/vima/nn/obj_encoder/obj_encoder.py): `vima/nn/obj_encoder/obj_encoder.py`
- [Vision transformer](https://github.com/vimalabs/vima/blob/main/vima/nn/obj_encoder/vit/vit.py): `vima/nn/obj_encoder/vit/vit.py`
- [Objects Perceiver](https://github.com/vimalabs/vima/blob/main/vima/nn/obj_encoder/perceiver/perceiver.py): `vima/nn/obj_encoder/perceiver/perceiver.py`

### Sequence models

- [Cross-attention GPT](https://github.com/vimalabs/vima/blob/main/vima/nn/seq_modeling/xattn_gpt/xattn_gpt.py): `vima/nn/seq_modeling/xattn_gpt/xattn_gpt.py`
- [Attention components](https://github.com/vimalabs/vima/blob/main/vima/nn/seq_modeling/xattn_gpt/components.py): `vima/nn/seq_modeling/xattn_gpt/components.py`
- [GPT sequence model](https://github.com/vimalabs/vima/blob/main/vima/nn/seq_modeling/gpt/gpt.py): `vima/nn/seq_modeling/gpt/gpt.py`

### Action modeling

- [Action embedding](https://github.com/vimalabs/vima/blob/main/vima/nn/action_embd/action_embd.py): `vima/nn/action_embd/action_embd.py`
- [Action decoder](https://github.com/vimalabs/vima/blob/main/vima/nn/action_decoder/action_decoder.py): `vima/nn/action_decoder/action_decoder.py`
- [Action distributions](https://github.com/vimalabs/vima/blob/main/vima/nn/action_decoder/dists.py): `vima/nn/action_decoder/dists.py`

### Other components

- **Robot task**
- **VIMA-Bench environment**

## Connections

- Robot task → Checkpoint loader: selects checkpoint
- Checkpoint loader → VIMA policy: loads weights
- Robot task → VIMA policy: provides task
- VIMA-Bench environment → VIMA policy: provides observations
- VIMA policy → Prompt encoder: encodes text
- VIMA policy → Word embedding: embeds words
- VIMA policy → Object encoder: encodes images
- Object encoder → Vision transformer: encodes crops
- Object encoder → Objects Perceiver: aggregates features
- VIMA policy → Action embedding: embeds actions
- VIMA policy → Cross-attention GPT: models token sequence
- Cross-attention GPT → Attention components: uses attention blocks
- VIMA policy → Action decoder: decodes actions
- Action decoder → Action distributions: builds distributions
- VIMA policy → VIMA-Bench environment: returns actions
