ylee-sling/vision-predictor-action

How ylee-sling/vision-predictor-action works

VPA is a research implementation for goal-conditioned robot control. Its principal runtime workflow encodes camera and milestone images, selects a symbolic skill, predicts the next latent and its uncertainty, then generates an action chunk; uncertainty determines how many actions the robot executes before replanning. Separate branches prepare demonstration data, train the policy, and evaluate it in LIBERO. The README documents the runtime sequence; source excerpts and the SOURCE INDEX support the training and evaluation connections. Some unsampled implementation details and evaluation wiring are therefore not expanded.

Main entry points: pipeline.py Inference pipeline · train.py Staged training · perception.py Vision encoder · eval.py Closed-loop evaluation

Components 19

Data & training

Demonstration files
LIBERO dataset dataset.py
Staged training train.py

Encoders & models

Vision encoder perception.py
Text encoder perception.py
Momentum encoder perception.py
DINOv2 initialization pretrained_encoder.py

Closed-loop control

Skill selector selector.py
Milestone tracker selector.py
JEPA predictor predictor.py
Flow-matching solver solver.py
Uncertainty horizon solver.py
Inference pipeline pipeline.py

Evaluation

LIBERO simulator
Closed-loop evaluation eval.py
Latency measurement bench_latency.py

Other components

Instruction & goals
Camera observations
Robot

Connections 22

Instruction & goals → Inference pipeline: provides goals README.md
Camera observations → Inference pipeline: provides frames README.md
Inference pipeline → Vision encoder: encodes images README.md
Inference pipeline → Text encoder: embeds instruction README.md
Inference pipeline → Skill selector: selects skill README.md
Inference pipeline → Milestone tracker: tracks milestones README.md
Inference pipeline → JEPA predictor: predicts latent README.md
Inference pipeline → Flow-matching solver: generates chunk README.md
Inference pipeline → Uncertainty horizon: filters horizon README.md
Inference pipeline → Robot: returns actions README.md
Robot → Camera observations: supplies next view
Demonstration files → LIBERO dataset: reads demonstrations dataset.py
LIBERO dataset → Staged training: supplies samples train.py
Staged training → Vision encoder: trains encoder train.py
Staged training → Momentum encoder: updates target train.py
Staged training → JEPA predictor: trains predictor train.py
Staged training → Skill selector: trains selector train.py
Staged training → Inference pipeline: builds policy train.py
DINOv2 initialization → Vision encoder: initializes weights pretrained_encoder.py
Closed-loop evaluation → Inference pipeline: runs policy README.md
Closed-loop evaluation → LIBERO simulator: evaluates episodes README.md
Latency measurement → Inference pipeline: times stages bench_latency.py

Drawn by GitDiagram on November 11, 2026 from the repository's file tree, README and source files. Read as Markdown ·