# scyonggg/vjepa2_merl_action_recognition architecture

> Architecture diagram of the GitHub repository scyonggg/vjepa2_merl_action_recognition, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/scyonggg/vjepa2_merl_action_recognition
- Repository: https://github.com/scyonggg/vjepa2_merl_action_recognition

## Overview

##### Project Type and Purpose
- This is an open-source machine-learning framework for self-supervised video representation learning, pre-training, probe-based evaluation, and action-conditioned robotics post-training.
- It provides CLI entrypoints (app/, evals/), a configuration-driven experiment setup (configs/), model definitions and data pipelines (src/), Jupyter demos (notebooks/), integration with PyTorch Hub and HuggingFace, plus CI and testing.

##### File-Structure Analysis
- Top-level modules:
- app/           → training loops for V-JEPA pre-training and action-conditioned post-training
- evals/         → evaluation loops (frozen backbone + attentive probes)
- src/           → core library: datasets, models, masks, utilities
- configs/       → YAML experiment definitions for train, eval, inference
- notebooks/     → interactive demos and examples
- scripts/       → small helper scripts (run_demo.sh, run_fine_tune.sh)
- hubconf.py     → PyTorch Hub integration
- tests/         → unit tests for src modules
- Config-driven: every experiment (pretrain, cooldown, post-train, probe eval, inference) is parameterized via a YAML under configs/.
- Distribution support: main_distributed.py in both app/ and evals/, plus src/utils/distributed.py and GitHub workflows for CI.

##### README Insights
- High-level flowchart (assets/flowchart.png) shows:
- Video ➜ Encoder ➜ Masked-latent prediction ➜ downstream tasks or action-conditioned world model
- Two phases:
(a) V-JEPA 2 self-supervised pre-training on internet videos
(b) V-JEPA 2-AC post-training on robot trajectories
- Evaluation: attentive probe training (frozen encoder) for video/image/action tasks
- Inference: demos via PyTorch Hub, HuggingFace, notebooks.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Configurations"
        CFG_TRAIN["configs/train/ (YAMLs)"]:::config
        CFG_EVAL["configs/eval/ (YAMLs)"]:::config
        CFG_INF["configs/inference/ (YAMLs)"]:::config
    end

    subgraph "CLI Entrypoints"
        APP_MAIN["app/main.py (Pre-train CLI)"]:::code
        APP_DIST["app/main_distributed.py (Distributed CLI)"]:::code
        EVAL_MAIN["evals/main.py (Probe Eval CLI)"]:::code
        EVAL_DIST["evals/main_distributed.py (Distributed Eval CLI)"]:::code
    end

    subgraph "Config Manager"
        CONF_MGR["YAML Parser"]:::code
    end

    subgraph "Data Pipeline"
        DATA_FILES["Video & Traj Data"]:::data
        SRCDS["src/datasets/"]:::code
        APP_TRANS["app/vjepa/transforms.py"]:::code
        APPDROID_TRANS["app/vjepa_droid/transforms.py"]:::code
    end

    subgraph "Model Definitions"
        MODELS["src/models/"]:::code
        MASKS["src/masks/"]:::code
    end

    subgraph "Training Engine"
        SCAFFOLD_APP["app/scaffold.py"]:::code
        VJEPA_TRAIN["app/vjepa/train.py"]:::code
        DROID_TRAIN["app/vjepa_droid/train.py"]:::code
    end

    subgraph "Evaluation Engine"
        SCAFFOLD_EVAL["evals/scaffold.py"]:::code
        EVAL_AA["evals/action_anticipation_frozen/"]:::code
        EVAL_IMG["evals/image_classification_frozen/"]:::code
        EVAL_VID["evals/video_classification_frozen/"]:::code
    end

    subgraph "Distributed Orchestrator"
        DIST_UTIL["src/utils/distributed.py"]:::code
        SLURM["SLURM Cluster"]:::external
    end

    subgraph "Checkpoint & Logging"
        CKPT_LOADER["src/utils/checkpoint_loader.py"]:::code
        LOGGING["src/utils/logging.py"]:::code
        MONITOR["src/utils/monitoring.py"]:::code
        CHECKPTS["Checkpoints & Logs"]:::checkpoint
    end

    subgraph "Inference & Demos"
        NB_DEMO["notebooks/vjepa2_demo.ipynb"]:::code
        GRADIO["vjepa2_gradio_demo.py"]:::code
        HUBCONF["hubconf.py"]:::code
        HF["HuggingFace API"]:::external
    end

    subgraph "Outputs & Metrics"
        METRICS["Evaluation Metrics"]:::checkpoint
        DEMO_OUT["Demo Results"]:::checkpoint
    end

    CFG_TRAIN --> APP_MAIN
    CFG_TRAIN --> APP_DIST
    CFG_EVAL --> EVAL_MAIN
    CFG_EVAL --> EVAL_DIST
    CFG_INF --> NB_DEMO
    CFG_INF --> GRADIO

    APP_MAIN --> CONF_MGR
    APP_DIST --> CONF_MGR
    EVAL_MAIN --> CONF_MGR
    EVAL_DIST --> CONF_MGR

    CONF_MGR --> SRCDS
    CONF_MGR --> DATA_FILES

    DATA_FILES --> SRCDS
    SRCDS --> APP_TRANS
    SRCDS --> APPDROID_TRANS

    APP_TRANS --> VJEPA_TRAIN
    APPDROID_TRANS --> DROID_TRAIN

    CONF_MGR --> MODELS
    MODELS --> MASKS

    MODELS --> VJEPA_TRAIN
    MASKS --> VJEPA_TRAIN

    VJEPA_TRAIN --> SCAFFOLD_APP
    DROID_TRAIN --> SCAFFOLD_APP
    SCAFFOLD_APP --> DIST_UTIL
    APP_DIST --> DIST_UTIL
    EVAL_DIST --> DIST_UTIL
    DIST_UTIL --> SLURM

    VJEPA_TRAIN --> CKPT_LOADER
    DROID_TRAIN --> CKPT_LOADER
    CKPT_LOADER --> CHECKPTS
    LOGGING --> CHECKPTS
    MONITOR --> CHECKPTS

    EVAL_MAIN --> SCAFFOLD_EVAL
    SCAFFOLD_EVAL --> EVAL_AA
    SCAFFOLD_EVAL --> EVAL_IMG
    SCAFFOLD_EVAL --> EVAL_VID
    EVAL_AA --> CKPT_LOADER
    EVAL_IMG --> CKPT_LOADER
    EVAL_VID --> CKPT_LOADER
    CKPT_LOADER --> METRICS

    NB_DEMO --> CHECKPTS
    GRADIO --> CHECKPTS
    HUBCONF --> CHECKPTS
    HUBCONF --> HF
    HF --> DEMO_OUT

    click APP_MAIN "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/app/main.py"
    click APP_DIST "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/app/main_distributed.py"
    click EVAL_MAIN "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/evals/main.py"
    click EVAL_DIST "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/evals/main_distributed.py"
    click NB_DEMO "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/notebooks/vjepa2_demo.ipynb"
    click GRADIO "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/vjepa2_gradio_demo.py"
    click CFG_TRAIN "https://github.com/scyonggg/vjepa2_merl_action_recognition/tree/main/configs/train/"
    click CFG_EVAL "https://github.com/scyonggg/vjepa2_merl_action_recognition/tree/main/configs/eval/"
    click CFG_INF "https://github.com/scyonggg/vjepa2_merl_action_recognition/tree/main/configs/inference/"
    click SRCDS "https://github.com/scyonggg/vjepa2_merl_action_recognition/tree/main/src/datasets/"
    click APP_TRANS "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/app/vjepa/transforms.py"
    click APPDROID_TRANS "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/app/vjepa_droid/transforms.py"
    click MODELS "https://github.com/scyonggg/vjepa2_merl_action_recognition/tree/main/src/models/"
    click MASKS "https://github.com/scyonggg/vjepa2_merl_action_recognition/tree/main/src/masks/"
    click SCAFFOLD_APP "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/app/scaffold.py"
    click VJEPA_TRAIN "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/app/vjepa/train.py"
    click DROID_TRAIN "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/app/vjepa_droid/train.py"
    click SCAFFOLD_EVAL "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/evals/scaffold.py"
    click EVAL_AA "https://github.com/scyonggg/vjepa2_merl_action_recognition/tree/main/evals/action_anticipation_frozen/"
    click EVAL_IMG "https://github.com/scyonggg/vjepa2_merl_action_recognition/tree/main/evals/image_classification_frozen/"
    click EVAL_VID "https://github.com/scyonggg/vjepa2_merl_action_recognition/tree/main/evals/video_classification_frozen/"
    click DIST_UTIL "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/src/utils/distributed.py"
    click CKPT_LOADER "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/src/utils/checkpoint_loader.py"
    click LOGGING "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/src/utils/logging.py"
    click MONITOR "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/src/utils/monitoring.py"
    click HUBCONF "https://github.com/scyonggg/vjepa2_merl_action_recognition/blob/main/hubconf.py"

    classDef code fill:#bbdefb,stroke:#1e88e5,color:#000;
    classDef config fill:#c8e6c9,stroke:#388e3c,color:#000;
    classDef data fill:#fff59d,stroke:#fbc02d,color:#000;
    classDef checkpoint fill:#ffcc80,stroke:#fb8c00,color:#000;
    classDef external fill:#e0e0e0,stroke:#9e9e9e,color:#000;
```
