How scyonggg/vjepa2_merl_action_recognition works
Project Type and Purpose
• This is an open-source machine-learning framework for self-supervised video representation learning, pre-training, probe-based evaluation, and action-conditioned robotics post-training.
• It provides CLI entrypoints (app/, evals/), a configuration-driven experiment setup (configs/), model definitions and data pipelines (src/), Jupyter demos (notebooks/), integration with PyTorch Hub and HuggingFace, plus CI and testing.
File-Structure Analysis
• Top-level modules:
• app/ → training loops for V-JEPA pre-training and action-conditioned post-training
• evals/ → evaluation loops (frozen backbone + attentive probes)
• src/ → core library: datasets, models, masks, utilities
• configs/ → YAML experiment definitions for train, eval, inference
• notebooks/ → interactive demos and examples
• scripts/ → small helper scripts (run_demo.sh, run_fine_tune.sh)
• hubconf.py → PyTorch Hub integration
• tests/ → unit tests for src modules
• Config-driven: every experiment (pretrain, cooldown, post-train, probe eval, inference) is parameterized via a YAML under configs/.
• Distribution support: main_distributed.py in both app/ and evals/, plus src/utils/distributed.py and GitHub workflows for CI.
README Insights
• High-level flowchart (assets/flowchart.png) shows:
• Video ➜ Encoder ➜ Masked-latent prediction ➜ downstream tasks or action-conditioned world model
• Two phases:
(a) V-JEPA 2 self-supervised pre-training on internet videos
(b) V-JEPA 2-AC post-training on robot trajectories
• Evaluation: attentive probe training (frozen encoder) for video/image/action tasks
• Inference: demos via PyTorch Hub, HuggingFace, notebooks.