How keivalya/mini-vla works
What this project is (type + purpose)
mini-VLA is a research/educational ML codebase that trains and evaluates a Vision–Language–Action diffusion policy for continuous robot control in simulation (Meta-World). It is not a deployed service; it’s a training + inference pipeline around:
• a simulation environment (Meta-World MT1 tasks),
• a data collection script (expert demonstrations → .npz),
• a PyTorch model (image/text/state encoders → fusion → diffusion action head),
• train/test scripts (offline imitation learning + rollout evaluation with video recording).
Top-level structure and what each folder “means” architecturally:
envs/
• envs/metaworld_env.py, envs/metaworld_mt1.py
• Role in system: environment wrappers/adapters around Meta-World tasks.
models/ (the core VLA)
• encoders.py: image encoder + text encoder + robot-state encoder
• fusion.py: combines embeddings (MLP)
• diffusion_head.py: diffusion policy head that denoises to actions
• vla_diffusion_policy.py: composes encoders + fusion + diffusion head into one policy module
utils/tokenizer.py
• Role: text preprocessing/tokenization used by the text encoder and scripts.
scripts/
• collect_data.py: runs expert policy in env → saves .npz
• train.py: loads .npz → trains model → saves checkpoint
• test.py: loads checkpoint → runs rollouts in env using text instruction → saves videos
• inference.py: planned/partial; treat as “future inference entrypoint”
assets/
• demo videos/gifs; not architectural, but shows “video output artifact”.
Meta-World Simulator (MT1 env)
• camera view (RGB frames)
• robot state (proprioception)
• accepts continuous actions
Expert Policy (for demonstrations)
• used only by collect_data.py
Dataset Artifact: .npz trajectories
• images / states / actions / (task instruction)
Training Loop (scripts/train.py)
• dataloader
• loss/optimizer
Evaluation / Rollout (scripts/test.py)
• loads checkpoint
• runs closed-loop policy in env
• optionally writes videos
• Meta-World + Expert Policy → (trajectories) → .npz
• .npz → Train Script → (gradients update) → VLA Model → Checkpoint .pt
• Checkpoint .pt + Instruction Text + Env Observations → Test Script → Actions → Env
• Test Script → Videos folder (optional artifact)
This is the component-level view: how tensors move inside the model.
Inputs
• Image observation I_t (from env render / camera)
• Robot state s_t
• Text instruction T (string)
Tokenizer (utils/tokenizer.py)
• T → tokens
Encoders (models/encoders.py)
• Vision encoder: I_t → e_img
• Text encoder: tokens → e_txt
• State encoder: s_t → e_state
Fusion MLP (models/fusion.py)
• (e_img, e_txt, e_state) → e_fused (single conditioning vector)
Diffusion Head (models/diffusion_head.py)
• Takes e_fused as conditioning
• Iterative denoising over action (or action sequence)
• Outputs continuous action a_t (or horizon of actions)
Policy wrapper (models/vla_diffusion_policy.py)
• Exposes forward() / act() used by train/test scripts
• Env image/state → encoders → fusion → diffusion head → action → env step
• Instruction text → tokenizer → text encoder → fusion
Closed-loop control path (during test.py)
• obs_t (image+state) + instruction → model → action_t → env → obs_{t+1}
• This is the runtime system behavior.
Offline supervised learning path (during train.py)
• dataset provides (image, state, instruction, expert_action)
• model predicts action (via diffusion)
• loss compares predicted vs expert (or diffusion objective)
• optimizer updates model weights
• checkpoint saved
Data collection path (during collect_data.py)
• env + expert policy generate trajectories
• dataset written once, reused by training
Architectural patterns/design principles present
• Separation of concerns
• scripts/ = orchestration/entrypoints
• envs/ = simulation adapters
• models/ = pure model definition
• utils/ = reusable preprocessing (tokenizer)
• Composable model architecture
• encoders + fusion + head are independent swappable modules (easy to prototype)
• Artifact-driven ML workflow
• .npz dataset and .pt checkpoint are explicit boundaries between stages
Technology labels to include (only the ones that matter)
Labels on components (not as a giant tech list):
• Meta-World (simulator / benchmark)
• PyTorch (model + training)
• Diffusion policy (action generation head)
• NPZ dataset (demonstrations)
• Video outputs (evaluation artifact)