# ylee-sling/vision-predictor-action architecture

> Architecture diagram of the GitHub repository ylee-sling/vision-predictor-action, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ylee-sling/vision-predictor-action
- Repository: https://github.com/ylee-sling/vision-predictor-action

## Overview

VPA is a research implementation for goal-conditioned robot control. Its principal runtime workflow encodes camera and milestone images, selects a symbolic skill, predicts the next latent and its uncertainty, then generates an action chunk; uncertainty determines how many actions the robot executes before replanning. Separate branches prepare demonstration data, train the policy, and evaluate it in LIBERO. The README documents the runtime sequence; source excerpts and the SOURCE INDEX support the training and evaluation connections. Some unsampled implementation details and evaluation wiring are therefore not expanded.

Main entry points: [`pipeline.py`](https://github.com/ylee-sling/vision-predictor-action/blob/main/pipeline.py) (Inference pipeline), [`train.py`](https://github.com/ylee-sling/vision-predictor-action/blob/main/train.py) (Staged training), [`perception.py`](https://github.com/ylee-sling/vision-predictor-action/blob/main/perception.py) (Vision encoder), [`eval.py`](https://github.com/ylee-sling/vision-predictor-action/blob/main/eval.py) (Closed-loop evaluation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data_training["Data & training"]
  node_hdf5[("Demonstration files")]
  node_dataset["LIBERO dataset<br/>[dataset.py]"]
  node_trainer["Staged training<br/>[train.py]"]
end

subgraph group_representations["Encoders &amp; models"]
  node_vision["Vision encoder<br/>[perception.py]"]
  node_text["Text encoder<br/>[perception.py]"]
  node_momentum["Momentum encoder<br/>[perception.py]"]
  node_dino["DINOv2 initialization"]
end

subgraph group_runtime["Closed-loop control"]
  node_selector["Skill selector<br/>[selector.py]"]
  node_tracker["Milestone tracker<br/>[selector.py]"]
  node_predictor["JEPA predictor<br/>[predictor.py]"]
  node_solver["Flow-matching solver<br/>[solver.py]"]
  node_horizon["Uncertainty horizon<br/>[solver.py]"]
  node_pipeline["Inference pipeline<br/>[pipeline.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_libero["LIBERO simulator"]
  node_evaluator["Closed-loop evaluation<br/>[eval.py]"]
  node_latency["Latency measurement<br/>[bench_latency.py]"]
end

node_operator(("Instruction &amp; goals"))
node_camera(("Camera observations"))
node_robot(("Robot"))

node_operator -->|"provides goals"| node_pipeline
node_camera -->|"provides frames"| node_pipeline
node_pipeline -->|"encodes images"| node_vision
node_pipeline -->|"embeds instruction"| node_text
node_pipeline -->|"selects skill"| node_selector
node_pipeline -->|"tracks milestones"| node_tracker
node_pipeline -->|"predicts latent"| node_predictor
node_pipeline -->|"generates chunk"| node_solver
node_pipeline -->|"filters horizon"| node_horizon
node_pipeline -->|"returns actions"| node_robot
node_robot -.->|"supplies next view"| node_camera
node_hdf5 -->|"reads demonstrations"| node_dataset
node_dataset -->|"supplies samples"| node_trainer
node_trainer -->|"trains encoder"| node_vision
node_trainer -->|"updates target"| node_momentum
node_trainer -->|"trains predictor"| node_predictor
node_trainer -->|"trains selector"| node_selector
node_trainer -->|"builds policy"| node_pipeline
node_dino -->|"initializes weights"| node_vision
node_evaluator -->|"runs policy"| node_pipeline
node_evaluator -->|"evaluates episodes"| node_libero
node_latency -->|"times stages"| node_pipeline

click node_dataset "https://github.com/ylee-sling/vision-predictor-action/blob/main/dataset.py"
click node_trainer "https://github.com/ylee-sling/vision-predictor-action/blob/main/train.py"
click node_vision "https://github.com/ylee-sling/vision-predictor-action/blob/main/perception.py"
click node_text "https://github.com/ylee-sling/vision-predictor-action/blob/main/perception.py"
click node_momentum "https://github.com/ylee-sling/vision-predictor-action/blob/main/perception.py"
click node_dino "https://github.com/ylee-sling/vision-predictor-action/blob/main/pretrained_encoder.py"
click node_selector "https://github.com/ylee-sling/vision-predictor-action/blob/main/selector.py"
click node_tracker "https://github.com/ylee-sling/vision-predictor-action/blob/main/selector.py"
click node_predictor "https://github.com/ylee-sling/vision-predictor-action/blob/main/predictor.py"
click node_solver "https://github.com/ylee-sling/vision-predictor-action/blob/main/solver.py"
click node_horizon "https://github.com/ylee-sling/vision-predictor-action/blob/main/solver.py"
click node_pipeline "https://github.com/ylee-sling/vision-predictor-action/blob/main/pipeline.py"
click node_evaluator "https://github.com/ylee-sling/vision-predictor-action/blob/main/eval.py"
click node_latency "https://github.com/ylee-sling/vision-predictor-action/blob/main/bench_latency.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_hdf5,node_dataset,node_trainer toneBlue
class node_vision,node_text,node_momentum,node_dino toneAmber
class node_selector,node_tracker,node_predictor,node_solver,node_horizon,node_pipeline toneMint
class node_libero,node_evaluator,node_latency toneRose
class node_operator,node_camera,node_robot toneIndigo
```

## Components

### Data & training

- **Demonstration files**
- [LIBERO dataset](https://github.com/ylee-sling/vision-predictor-action/blob/main/dataset.py): `dataset.py`
- [Staged training](https://github.com/ylee-sling/vision-predictor-action/blob/main/train.py): `train.py`

### Encoders & models

- [Vision encoder](https://github.com/ylee-sling/vision-predictor-action/blob/main/perception.py): `perception.py`
- [Text encoder](https://github.com/ylee-sling/vision-predictor-action/blob/main/perception.py): `perception.py`
- [Momentum encoder](https://github.com/ylee-sling/vision-predictor-action/blob/main/perception.py): `perception.py`
- [DINOv2 initialization](https://github.com/ylee-sling/vision-predictor-action/blob/main/pretrained_encoder.py): `pretrained_encoder.py`

### Closed-loop control

- [Skill selector](https://github.com/ylee-sling/vision-predictor-action/blob/main/selector.py): `selector.py`
- [Milestone tracker](https://github.com/ylee-sling/vision-predictor-action/blob/main/selector.py): `selector.py`
- [JEPA predictor](https://github.com/ylee-sling/vision-predictor-action/blob/main/predictor.py): `predictor.py`
- [Flow-matching solver](https://github.com/ylee-sling/vision-predictor-action/blob/main/solver.py): `solver.py`
- [Uncertainty horizon](https://github.com/ylee-sling/vision-predictor-action/blob/main/solver.py): `solver.py`
- [Inference pipeline](https://github.com/ylee-sling/vision-predictor-action/blob/main/pipeline.py): `pipeline.py`

### Evaluation

- **LIBERO simulator**
- [Closed-loop evaluation](https://github.com/ylee-sling/vision-predictor-action/blob/main/eval.py): `eval.py`
- [Latency measurement](https://github.com/ylee-sling/vision-predictor-action/blob/main/bench_latency.py): `bench_latency.py`

### Other components

- **Instruction & goals**
- **Camera observations**
- **Robot**

## Connections

- Instruction & goals → Inference pipeline: provides goals (evidence: `README.md`)
- Camera observations → Inference pipeline: provides frames (evidence: `README.md`)
- Inference pipeline → Vision encoder: encodes images (evidence: `README.md`)
- Inference pipeline → Text encoder: embeds instruction (evidence: `README.md`)
- Inference pipeline → Skill selector: selects skill (evidence: `README.md`)
- Inference pipeline → Milestone tracker: tracks milestones (evidence: `README.md`)
- Inference pipeline → JEPA predictor: predicts latent (evidence: `README.md`)
- Inference pipeline → Flow-matching solver: generates chunk (evidence: `README.md`)
- Inference pipeline → Uncertainty horizon: filters horizon (evidence: `README.md`)
- Inference pipeline → Robot: returns actions (evidence: `README.md`)
- Robot → Camera observations: supplies next view
- Demonstration files → LIBERO dataset: reads demonstrations (evidence: `dataset.py`)
- LIBERO dataset → Staged training: supplies samples (evidence: `train.py`)
- Staged training → Vision encoder: trains encoder (evidence: `train.py`)
- Staged training → Momentum encoder: updates target (evidence: `train.py`)
- Staged training → JEPA predictor: trains predictor (evidence: `train.py`)
- Staged training → Skill selector: trains selector (evidence: `train.py`)
- Staged training → Inference pipeline: builds policy (evidence: `train.py`)
- DINOv2 initialization → Vision encoder: initializes weights (evidence: `pretrained_encoder.py`)
- Closed-loop evaluation → Inference pipeline: runs policy (evidence: `README.md`)
- Closed-loop evaluation → LIBERO simulator: evaluates episodes (evidence: `README.md`)
- Latency measurement → Inference pipeline: times stages (evidence: `bench_latency.py`)