shihao1895/memoryvla

How shihao1895/memoryvla works

MemoryVLA is a vision-language-action system for robotic manipulation. Its main workflow turns demonstration datasets into normalized training batches, trains a multimodal policy with temporal memory and diffusion-based action prediction, then loads the checkpoint for simulated evaluation or real-world deployment. The graph preserves the separately documented Libero and SimplerEnv evaluation paths. Several subsystem links are supported by the README or SOURCE INDEX references; unsampled implementation details are not inferred.

Main entry points: train.py Training entry · load.py Policy loader · base_strategy.py Training strategy · prismatic.py Multimodal VLM

Components 21

Data preparation

Dataset interface vla/datasets/datasets.py
RLDS processing vla/datasets/rlds/dataset.py
Normalization utilities vla/datasets/rlds/utils/data_utils.py
OXE transforms vla/datasets/rlds/oxe/transforms.py

Policy and action

Policy loader vla/load.py
MemoryVLA policy vla/memory_vla.py
Multimodal VLM prismatic/models/vlms/prismatic.py
Vision backbones prismatic/models/backbones/vision
Language backbones prismatic/models/backbones/llm
Action model action_model/action_model.py
Diffusion sampler action_model/gaussian_diffusion.py

Training

Training entry train.py
VLA configuration conf/vla.py
Training strategy training/strategies/base_strategy.py
Metrics tracking training/metrics.py

Evaluation and deployment

Libero evaluation evaluation/libero/eval_libero.py
SimplerEnv evaluation evaluation/simpler_env/simpler_env_inference.py
ManiSkill evaluator evaluation/simpler_env/maniskill2_evaluator.py
Robot deployment deploy.py

Other components

Operator
Robot or simulator

Connections 25

Operator → Training entry: starts training README.md
Training entry → VLA configuration: uses config train.py
Training entry → Policy loader: loads policy train.py
Training entry → Normalization utilities: uses utilities train.py
Training entry → Training strategy: runs strategy
Training strategy → Metrics tracking: records metrics training/strategies/base_strategy.py
Training strategy → Multimodal VLM: trains model training/strategies/base_strategy.py
Training strategy → Normalization utilities: uses utilities training/strategies/base_strategy.py
Dataset interface → RLDS processing: uses RLDS vla/datasets/datasets.py
RLDS processing → Normalization utilities: uses utilities vla/datasets/rlds/dataset.py
OXE transforms → Normalization utilities: uses utilities vla/datasets/rlds/oxe/transforms.py
Policy loader → VLA configuration: reads config vla/load.py
Policy loader → MemoryVLA policy: materializes policy vla/load.py
MemoryVLA policy → Multimodal VLM: uses backbone vla/memory_vla.py
MemoryVLA policy → Action model: uses action model vla/memory_vla.py
Action model → Diffusion sampler: uses diffusion action_model/action_model.py
Multimodal VLM → Vision backbones: uses vision prismatic/models/vlms/prismatic.py
Multimodal VLM → Language backbones: uses language prismatic/models/vlms/prismatic.py
Operator → Libero evaluation: runs evaluation README.md
Libero evaluation → Policy loader: loads policy evaluation/libero/eval_libero.py
Operator → SimplerEnv evaluation: runs evaluation README.md
SimplerEnv evaluation → ManiSkill evaluator: invokes evaluator evaluation/simpler_env/simpler_env_inference.py
SimplerEnv evaluation → Policy loader: builds inference
Operator → Robot deployment: starts deployment README.md
Robot deployment → Robot or simulator: controls robot

More Python diagrams: defog-ai/introspect · openai/vime · prismorsec/prismor · svsdval/video2midi · akatz-ai/comfyui-depthflow-nodes

Drawn by GitDiagram on September 30, 2026 from the repository's file tree, README and source files. Read as Markdown ·