# shihao1895/memoryvla architecture

> Architecture diagram of the GitHub repository shihao1895/memoryvla, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shihao1895/memoryvla
- Repository: https://github.com/shihao1895/memoryvla

## Overview

MemoryVLA is a vision-language-action system for robotic manipulation. Its main workflow turns demonstration datasets into normalized training batches, trains a multimodal policy with temporal memory and diffusion-based action prediction, then loads the checkpoint for simulated evaluation or real-world deployment. The graph preserves the separately documented Libero and SimplerEnv evaluation paths. Several subsystem links are supported by the README or SOURCE INDEX references; unsampled implementation details are not inferred.

Main entry points: [`train.py`](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/train.py) (Training entry), [`vla/load.py`](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/vla/load.py) (Policy loader), [`training/strategies/base_strategy.py`](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/training/strategies/base_strategy.py) (Training strategy), [`prismatic/models/vlms/prismatic.py`](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/prismatic/models/vlms/prismatic.py) (Multimodal VLM)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_dataset_api["Dataset interface<br/>[datasets.py]"]
  node_rlds["RLDS processing<br/>[dataset.py]"]
  node_data_utils["Normalization utilities<br/>[data_utils.py]"]
  node_oxe_transforms["OXE transforms<br/>[transforms.py]"]
end

subgraph group_model["Policy and action"]
  node_loader["Policy loader<br/>[load.py]"]
  node_memory_policy["MemoryVLA policy<br/>[memory_vla.py]"]
  node_vlm["Multimodal VLM<br/>[prismatic.py]"]
  node_vision["Vision backbones"]
  node_language["Language backbones"]
  node_action_model["Action model<br/>[action_model.py]"]
  node_diffusion["Diffusion sampler"]
end

subgraph group_train["Training"]
  node_train_entry["Training entry<br/>[train.py]"]
  node_config["VLA configuration<br/>[vla.py]"]
  node_strategy["Training strategy<br/>[base_strategy.py]"]
  node_metrics["Metrics tracking<br/>[metrics.py]"]
end

subgraph group_eval["Evaluation and deployment"]
  node_libero_eval["Libero evaluation<br/>[eval_libero.py]"]
  node_simpler_entry["SimplerEnv evaluation"]
  node_maniskill["ManiSkill evaluator"]
  node_deployment["Robot deployment<br/>[deploy.py]"]
end

node_engineer(("Operator"))
node_robot(("Robot or simulator"))

node_engineer -->|"starts training"| node_train_entry
node_train_entry -->|"uses config"| node_config
node_train_entry -->|"loads policy"| node_loader
node_train_entry -->|"uses utilities"| node_data_utils
node_train_entry -.->|"runs strategy"| node_strategy
node_strategy -->|"records metrics"| node_metrics
node_strategy -->|"trains model"| node_vlm
node_strategy -->|"uses utilities"| node_data_utils
node_dataset_api -->|"uses RLDS"| node_rlds
node_rlds -->|"uses utilities"| node_data_utils
node_oxe_transforms -->|"uses utilities"| node_data_utils
node_loader -->|"reads config"| node_config
node_loader -->|"materializes policy"| node_memory_policy
node_memory_policy -->|"uses backbone"| node_vlm
node_memory_policy -->|"uses action model"| node_action_model
node_action_model -->|"uses diffusion"| node_diffusion
node_vlm -->|"uses vision"| node_vision
node_vlm -->|"uses language"| node_language
node_engineer -->|"runs evaluation"| node_libero_eval
node_libero_eval -->|"loads policy"| node_loader
node_engineer -->|"runs evaluation"| node_simpler_entry
node_simpler_entry -->|"invokes evaluator"| node_maniskill
node_simpler_entry -.->|"builds inference"| node_loader
node_engineer -->|"starts deployment"| node_deployment
node_deployment -.->|"controls robot"| node_robot

click node_train_entry "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/train.py"
click node_config "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/conf/vla.py"
click node_strategy "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/training/strategies/base_strategy.py"
click node_metrics "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/training/metrics.py"
click node_dataset_api "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/vla/datasets/datasets.py"
click node_rlds "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/vla/datasets/rlds/dataset.py"
click node_data_utils "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/vla/datasets/rlds/utils/data_utils.py"
click node_oxe_transforms "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/vla/datasets/rlds/oxe/transforms.py"
click node_loader "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/vla/load.py"
click node_memory_policy "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/vla/memory_vla.py"
click node_vlm "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/prismatic/models/vlms/prismatic.py"
click node_vision "https://github.com/shihao1895/memoryvla/tree/openvla-codebase/prismatic/models/backbones/vision"
click node_language "https://github.com/shihao1895/memoryvla/tree/openvla-codebase/prismatic/models/backbones/llm"
click node_action_model "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/action_model/action_model.py"
click node_diffusion "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/action_model/gaussian_diffusion.py"
click node_libero_eval "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/evaluation/libero/eval_libero.py"
click node_simpler_entry "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/evaluation/simpler_env/simpler_env_inference.py"
click node_maniskill "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/evaluation/simpler_env/maniskill2_evaluator.py"
click node_deployment "https://github.com/shihao1895/memoryvla/blob/openvla-codebase/deploy.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dataset_api,node_rlds,node_data_utils,node_oxe_transforms toneBlue
class node_loader,node_memory_policy,node_vlm,node_vision,node_language,node_action_model,node_diffusion toneAmber
class node_train_entry,node_config,node_strategy,node_metrics toneMint
class node_libero_eval,node_simpler_entry,node_maniskill,node_deployment toneRose
class node_engineer,node_robot toneIndigo
```

## Components

### Data preparation

- [Dataset interface](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/vla/datasets/datasets.py): `vla/datasets/datasets.py`
- [RLDS processing](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/vla/datasets/rlds/dataset.py): `vla/datasets/rlds/dataset.py`
- [Normalization utilities](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/vla/datasets/rlds/utils/data_utils.py): `vla/datasets/rlds/utils/data_utils.py`
- [OXE transforms](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/vla/datasets/rlds/oxe/transforms.py): `vla/datasets/rlds/oxe/transforms.py`

### Policy and action

- [Policy loader](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/vla/load.py): `vla/load.py`
- [MemoryVLA policy](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/vla/memory_vla.py): `vla/memory_vla.py`
- [Multimodal VLM](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/prismatic/models/vlms/prismatic.py): `prismatic/models/vlms/prismatic.py`
- [Vision backbones](https://github.com/shihao1895/memoryvla/tree/openvla-codebase/prismatic/models/backbones/vision): `prismatic/models/backbones/vision`
- [Language backbones](https://github.com/shihao1895/memoryvla/tree/openvla-codebase/prismatic/models/backbones/llm): `prismatic/models/backbones/llm`
- [Action model](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/action_model/action_model.py): `action_model/action_model.py`
- [Diffusion sampler](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/action_model/gaussian_diffusion.py): `action_model/gaussian_diffusion.py`

### Training

- [Training entry](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/train.py): `train.py`
- [VLA configuration](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/conf/vla.py): `conf/vla.py`
- [Training strategy](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/training/strategies/base_strategy.py): `training/strategies/base_strategy.py`
- [Metrics tracking](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/training/metrics.py): `training/metrics.py`

### Evaluation and deployment

- [Libero evaluation](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/evaluation/libero/eval_libero.py): `evaluation/libero/eval_libero.py`
- [SimplerEnv evaluation](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/evaluation/simpler_env/simpler_env_inference.py): `evaluation/simpler_env/simpler_env_inference.py`
- [ManiSkill evaluator](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/evaluation/simpler_env/maniskill2_evaluator.py): `evaluation/simpler_env/maniskill2_evaluator.py`
- [Robot deployment](https://github.com/shihao1895/memoryvla/blob/openvla-codebase/deploy.py): `deploy.py`

### Other components

- **Operator**
- **Robot or simulator**

## Connections

- Operator → Training entry: starts training (evidence: `README.md`)
- Training entry → VLA configuration: uses config (evidence: `train.py`)
- Training entry → Policy loader: loads policy (evidence: `train.py`)
- Training entry → Normalization utilities: uses utilities (evidence: `train.py`)
- Training entry → Training strategy: runs strategy
- Training strategy → Metrics tracking: records metrics (evidence: `training/strategies/base_strategy.py`)
- Training strategy → Multimodal VLM: trains model (evidence: `training/strategies/base_strategy.py`)
- Training strategy → Normalization utilities: uses utilities (evidence: `training/strategies/base_strategy.py`)
- Dataset interface → RLDS processing: uses RLDS (evidence: `vla/datasets/datasets.py`)
- RLDS processing → Normalization utilities: uses utilities (evidence: `vla/datasets/rlds/dataset.py`)
- OXE transforms → Normalization utilities: uses utilities (evidence: `vla/datasets/rlds/oxe/transforms.py`)
- Policy loader → VLA configuration: reads config (evidence: `vla/load.py`)
- Policy loader → MemoryVLA policy: materializes policy (evidence: `vla/load.py`)
- MemoryVLA policy → Multimodal VLM: uses backbone (evidence: `vla/memory_vla.py`)
- MemoryVLA policy → Action model: uses action model (evidence: `vla/memory_vla.py`)
- Action model → Diffusion sampler: uses diffusion (evidence: `action_model/action_model.py`)
- Multimodal VLM → Vision backbones: uses vision (evidence: `prismatic/models/vlms/prismatic.py`)
- Multimodal VLM → Language backbones: uses language (evidence: `prismatic/models/vlms/prismatic.py`)
- Operator → Libero evaluation: runs evaluation (evidence: `README.md`)
- Libero evaluation → Policy loader: loads policy (evidence: `evaluation/libero/eval_libero.py`)
- Operator → SimplerEnv evaluation: runs evaluation (evidence: `README.md`)
- SimplerEnv evaluation → ManiSkill evaluator: invokes evaluator (evidence: `evaluation/simpler_env/simpler_env_inference.py`)
- SimplerEnv evaluation → Policy loader: builds inference
- Operator → Robot deployment: starts deployment (evidence: `README.md`)
- Robot deployment → Robot or simulator: controls robot