# facebookresearch/jepa architecture

> Architecture diagram of the GitHub repository facebookresearch/jepa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/jepa
- Repository: https://github.com/facebookresearch/jepa

## Overview

##### Main Components

a) Core Architecture Components:
- Video Input Processing Pipeline
- Encoder (ViT-based architecture)
- Predictor Network
- Masking System
- Feature Prediction System
- Evaluation Probes

b) Data Flow Components:
- Data Management System
- Training Pipeline
- Evaluation Pipeline
- Distributed Training Infrastructure

##### Detailed Component Breakdown

A. Input Processing Layer:
- Video Input Handler (/src/datasets/video_dataset.py)
- Image Input Handler (/src/datasets/image_dataset.py)
- Data Manager (/src/datasets/data_manager.py)
- Transform Pipeline (/src/datasets/utils/video/transforms.py)

B. Model Architecture Layer:
- Vision Transformer (/src/models/vision_transformer.py)
- Attentive Pooler (/src/models/attentive_pooler.py)
- Predictor Network (/src/models/predictor.py)

C. Masking System:
- Mask Generators (/src/masks/)
- Default Masking
- MultiBlock3D
- Random Tube

D. Training Infrastructure:
- Distributed Training System (/src/utils/distributed.py)
- Monitoring System (/src/utils/monitoring.py)
- Scheduling System (/src/utils/schedulers.py)

##### Key Relationships to Highlight

a) Primary Data Flow:
**Video Input → Data Processing → Encoder → Masking → Predictor → Feature Prediction**

b) Training Flow:
**Training Data → Distributed Training System → Model Components → Loss Computation → Parameter Updates**

c) Evaluation Flow:
**Test Data → Frozen Backbone → Attentive Probe → Task-Specific Output**

##### Implementation Details

**Layer 1: Data Input/Processing**
**Layer 2: Core Model Architecture**
**Layer 3: Training/Evaluation Infrastructure**
**Layer 4: Output/Results**

##### Additional Elements

- Version information and model configurations
- Relationship with external services (if any)
- Monitoring and logging components

##### Technical Notes

- Use clear labeling for all components
- Input/output specifications for each major component
- Configuration system that manages different model variants
- Distributed training capabilities

This system design should be implemented using a professional diagramming tool that supports UML or similar notation, ensuring that the complex relationships between components are clearly visible while maintaining readability.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Data Input Layer"
        VI["Video Input Processing"]
        II["Image Input Processing"]
        DM["Data Management System"]
        TP["Transform Pipeline"]
    end

    subgraph "Core Model Architecture"
        VT["Vision Transformer Encoder"]
        AP["Attentive Pooler"]
        PN["Predictor Network"]
    end

    subgraph "Masking System"
        MS["Masking Controller"]
        DM_M["Default Masking"]
        MB["MultiBlock3D"]
        RT["Random Tube"]
    end

    subgraph "Training & Evaluation"
        TP_T["Training Pipeline"]
        EP["Evaluation Pipeline"]
        DTI["Distributed Training"]
        MON["Monitoring System"]
        SCH["Scheduling System"]
    end

    subgraph "Configuration"
        CS["Configuration System"]
        PC["Pretraining Configs"]
        EC["Evaluation Configs"]
    end

    %% Data flow connections
    VI --> TP
    II --> TP
    TP --> DM
    DM --> VT
    VT --> AP
    VT --> PN
    MS --> VT
    DM_M --> MS
    MB --> MS
    RT --> MS
    
    %% Training and evaluation connections
    TP_T --> DTI
    DTI --> VT
    MON --> DTI
    SCH --> DTI
    EP --> VT
    
    %% Configuration connections
    CS --> PC
    CS --> EC
    PC --> TP_T
    EC --> EP

    %% Click events for component mapping
    click VI "https://github.com/facebookresearch/jepa/blob/main/src/datasets/video_dataset.py"
    click II "https://github.com/facebookresearch/jepa/blob/main/src/datasets/image_dataset.py"
    click DM "https://github.com/facebookresearch/jepa/blob/main/src/datasets/data_manager.py"
    click TP "https://github.com/facebookresearch/jepa/blob/main/src/datasets/utils/video/transforms.py"
    click VT "https://github.com/facebookresearch/jepa/blob/main/src/models/vision_transformer.py"
    click PN "https://github.com/facebookresearch/jepa/blob/main/src/models/predictor.py"
    click AP "https://github.com/facebookresearch/jepa/blob/main/src/models/attentive_pooler.py"
    click MS "https://github.com/facebookresearch/jepa/tree/main/src/masks/"
    click DM_M "https://github.com/facebookresearch/jepa/blob/main/src/masks/default.py"
    click MB "https://github.com/facebookresearch/jepa/blob/main/src/masks/multiblock3d.py"
    click RT "https://github.com/facebookresearch/jepa/blob/main/src/masks/random_tube.py"
    click TP_T "https://github.com/facebookresearch/jepa/blob/main/app/vjepa/train.py"
    click EP "https://github.com/facebookresearch/jepa/blob/main/evals/main.py"
    click DTI "https://github.com/facebookresearch/jepa/blob/main/src/utils/distributed.py"
    click MON "https://github.com/facebookresearch/jepa/blob/main/src/utils/monitoring.py"
    click SCH "https://github.com/facebookresearch/jepa/blob/main/src/utils/schedulers.py"
    click CS "https://github.com/facebookresearch/jepa/tree/main/configs/"
    click PC "https://github.com/facebookresearch/jepa/tree/main/configs/pretrain/"
    click EC "https://github.com/facebookresearch/jepa/tree/main/configs/evals/"

    %% Styling
    classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px
    classDef input fill:#d4e6f1,stroke:#2874a6
    classDef model fill:#d5f5e3,stroke:#196f3d
    classDef mask fill:#fdebd0,stroke:#d35400
    classDef infra fill:#ebdef0,stroke:#8e44ad
    classDef config fill:#f2f4f4,stroke:#2c3e50

    class VI,II,DM,TP input
    class VT,AP,PN model
    class MS,DM_M,MB,RT mask
    class TP_T,EP,DTI,MON,SCH infra
    class CS,PC,EC config
```
