How vimalabs/vima works
VIMA is a multimodal-prompt robot-manipulation policy: it combines textual and visual prompt tokens with object-centric observations, then predicts actions for environment interaction. The main path runs through checkpoint loading, prompt/observation encoding, cross-attention sequence modeling, and action decoding; action embeddings support the policy’s action representation. README also documents Gato, Flamingo, and GPT baseline policies. The excerpts show core policy and encoder wiring, but not the demo’s runtime calls or all baseline internals, so those links are omitted where uncertain.
Main entry points: vima_policy.py VIMA policy · obj_encoder.py Object encoder · __init__.py Checkpoint loader · xattn_gpt.py Cross-attention GPT
Components 18
Policy entry
- Checkpoint loader
vima/__init__.py - VIMA policy
vima/policy/vima_policy.py - Gato baseline
vima/policy/vima_gato_policy.py - Flamingo baseline
vima/policy/vima_flamingo_policy.py - GPT baseline
vima/policy/vima_gpt_policy.py
Multimodal encoding
- Prompt encoder
vima/nn/prompt_encoder/prompt_encoder.py - Word embedding
vima/nn/prompt_encoder/word_embd.py - Object encoder
vima/nn/obj_encoder/obj_encoder.py - Vision transformer
vima/nn/obj_encoder/vit/vit.py - Objects Perceiver
vima/nn/obj_encoder/perceiver/perceiver.py
Sequence models
- Cross-attention GPT
vima/nn/seq_modeling/xattn_gpt/xattn_gpt.py - Attention components
vima/nn/seq_modeling/xattn_gpt/components.py - GPT sequence model
vima/nn/seq_modeling/gpt/gpt.py
Action modeling
- Action embedding
vima/nn/action_embd/action_embd.py - Action decoder
vima/nn/action_decoder/action_decoder.py - Action distributions
vima/nn/action_decoder/dists.py
Other components
- Robot task
- VIMA-Bench environment
Connections 15
- Robot task → Checkpoint loader: selects checkpoint
- Checkpoint loader → VIMA policy: loads weights
- Robot task → VIMA policy: provides task
- VIMA-Bench environment → VIMA policy: provides observations
- VIMA policy → Prompt encoder: encodes text
- VIMA policy → Word embedding: embeds words
- VIMA policy → Object encoder: encodes images
- Object encoder → Vision transformer: encodes crops
- Object encoder → Objects Perceiver: aggregates features
- VIMA policy → Action embedding: embeds actions
- VIMA policy → Cross-attention GPT: models token sequence
- Cross-attention GPT → Attention components: uses attention blocks
- VIMA policy → Action decoder: decodes actions
- Action decoder → Action distributions: builds distributions
- VIMA policy → VIMA-Bench environment: returns actions