# saketh-04/lv-mae-event-span architecture

> Architecture diagram of the GitHub repository saketh-04/lv-mae-event-span, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saketh-04/lv-mae-event-span
- Repository: https://github.com/saketh-04/lv-mae-event-span

## Overview

This repository implements event-span-masked LV-MAE pretraining for long-video representations, with XD-Violence attentive-probe classification documented as the downstream use. The main runtime path loads segment embeddings, applies event-aware masking, trains the MAE, and records training progress. The README also describes embedding extraction and CSV preparation. LanguageBind includes several modality model families, though the documented experiment uses video only. Some links rely on the README or SOURCE INDEX; downstream probe implementation is not present in the supplied tree.

Main entry points: [`main_pretrain.py`](https://github.com/saketh-04/lv-mae-event-span/blob/main/main_pretrain.py) (Pretraining entry), [`engine_pretrain.py`](https://github.com/saketh-04/lv-mae-event-span/blob/main/engine_pretrain.py) (Train and validate), [`data/CombinedData.py`](https://github.com/saketh-04/lv-mae-event-span/blob/main/data/CombinedData.py) (Embedding dataset), [`data_preprocess/extract_xd.py`](https://github.com/saketh-04/lv-mae-event-span/blob/main/data_preprocess/extract_xd.py) (Video embeddings)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_extract["Video embeddings<br/>[extract_xd.py]"]
  node_csv["XD-Violence CSV<br/>[build_xd_csv.py]"]
  node_dataset["Embedding dataset<br/>[CombinedData.py]"]
  node_mask["Event-span masking"]
end

subgraph group_training["Masked pretraining"]
  node_main["Pretraining entry<br/>[main_pretrain.py]"]
  node_engine["Train and validate<br/>[engine_pretrain.py]"]
  node_mae["Masked autoencoder<br/>[models_mae.py]"]
end

subgraph group_encoders["LanguageBind models"]
  node_video_model["Video model<br/>[modeling_video.py]"]
  node_video_config["Video configuration"]
  node_video_processor["Video processor"]
  node_image_model["Image model<br/>[modeling_image.py]"]
  node_image_config["Image configuration"]
  node_audio_model["Audio model<br/>[modeling_audio.py]"]
  node_audio_config["Audio configuration"]
  node_depth_model["Depth model<br/>[modeling_depth.py]"]
  node_depth_config["Depth configuration"]
  node_thermal_model["Thermal model"]
  node_thermal_config["Thermal configuration"]
end

subgraph group_support["Training support"]
  node_misc["Metrics and checkpoints<br/>[misc.py]"]
  node_schedule["Learning-rate schedule<br/>[lr_sched.py]"]
  node_logger["Logger adapters<br/>[base_logger.py]"]
end

node_user(("Experiment operator"))
node_probe["Attentive probe"]

node_user -->|"runs extraction"| node_extract
node_extract -.->|"encodes video"| node_video_model
node_user -->|"builds splits"| node_csv
node_csv -.->|"provides CSV"| node_dataset
node_main -->|"loads data"| node_dataset
node_dataset -->|"applies masking"| node_mask
node_main -->|"sets temperature"| node_mask
node_main -->|"runs epochs"| node_engine
node_engine -->|"computes loss"| node_mae
node_main -->|"constructs model"| node_mae
node_engine -->|"logs metrics"| node_misc
node_main -->|"saves checkpoints"| node_misc
node_engine -->|"updates learning rate"| node_schedule
node_main -->|"configures logging"| node_logger
node_video_model -->|"uses configuration"| node_video_config
node_image_model -->|"uses configuration"| node_image_config
node_audio_model -->|"uses configuration"| node_audio_config
node_depth_model -->|"uses configuration"| node_depth_config
node_thermal_model -->|"uses configuration"| node_thermal_config
node_user -.->|"evaluates encoder"| node_probe

click node_extract "https://github.com/saketh-04/lv-mae-event-span/blob/main/data_preprocess/extract_xd.py"
click node_csv "https://github.com/saketh-04/lv-mae-event-span/blob/main/data_preprocess/build_xd_csv.py"
click node_dataset "https://github.com/saketh-04/lv-mae-event-span/blob/main/data/CombinedData.py"
click node_mask "https://github.com/saketh-04/lv-mae-event-span/blob/main/data/event_span_masking.py"
click node_main "https://github.com/saketh-04/lv-mae-event-span/blob/main/main_pretrain.py"
click node_engine "https://github.com/saketh-04/lv-mae-event-span/blob/main/engine_pretrain.py"
click node_mae "https://github.com/saketh-04/lv-mae-event-span/blob/main/models_mae.py"
click node_video_model "https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/video/modeling_video.py"
click node_video_config "https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/video/configuration_video.py"
click node_video_processor "https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/video/processing_video.py"
click node_image_model "https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/image/modeling_image.py"
click node_image_config "https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/image/configuration_image.py"
click node_audio_model "https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/audio/modeling_audio.py"
click node_audio_config "https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/audio/configuration_audio.py"
click node_depth_model "https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/depth/modeling_depth.py"
click node_depth_config "https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/depth/configuration_depth.py"
click node_thermal_model "https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/thermal/modeling_thermal.py"
click node_thermal_config "https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/thermal/configuration_thermal.py"
click node_misc "https://github.com/saketh-04/lv-mae-event-span/blob/main/util/misc.py"
click node_schedule "https://github.com/saketh-04/lv-mae-event-span/blob/main/util/lr_sched.py"
click node_logger "https://github.com/saketh-04/lv-mae-event-span/blob/main/loggers/base_logger.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_extract,node_csv,node_dataset,node_mask toneBlue
class node_main,node_engine,node_mae toneAmber
class node_video_model,node_video_config,node_video_processor,node_image_model,node_image_config,node_audio_model,node_audio_config,node_depth_model,node_depth_config,node_thermal_model,node_thermal_config toneMint
class node_misc,node_schedule,node_logger toneRose
class node_user,node_probe toneIndigo
```

## Components

### Data preparation

- [Video embeddings](https://github.com/saketh-04/lv-mae-event-span/blob/main/data_preprocess/extract_xd.py): `data_preprocess/extract_xd.py`
- [XD-Violence CSV](https://github.com/saketh-04/lv-mae-event-span/blob/main/data_preprocess/build_xd_csv.py): `data_preprocess/build_xd_csv.py`
- [Embedding dataset](https://github.com/saketh-04/lv-mae-event-span/blob/main/data/CombinedData.py): `data/CombinedData.py`
- [Event-span masking](https://github.com/saketh-04/lv-mae-event-span/blob/main/data/event_span_masking.py): `data/event_span_masking.py`

### Masked pretraining

- [Pretraining entry](https://github.com/saketh-04/lv-mae-event-span/blob/main/main_pretrain.py): `main_pretrain.py`
- [Train and validate](https://github.com/saketh-04/lv-mae-event-span/blob/main/engine_pretrain.py): `engine_pretrain.py`
- [Masked autoencoder](https://github.com/saketh-04/lv-mae-event-span/blob/main/models_mae.py): `models_mae.py`

### LanguageBind models

- [Video model](https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/video/modeling_video.py): `languagebind/video/modeling_video.py`
- [Video configuration](https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/video/configuration_video.py): `languagebind/video/configuration_video.py`
- [Video processor](https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/video/processing_video.py): `languagebind/video/processing_video.py`
- [Image model](https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/image/modeling_image.py): `languagebind/image/modeling_image.py`
- [Image configuration](https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/image/configuration_image.py): `languagebind/image/configuration_image.py`
- [Audio model](https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/audio/modeling_audio.py): `languagebind/audio/modeling_audio.py`
- [Audio configuration](https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/audio/configuration_audio.py): `languagebind/audio/configuration_audio.py`
- [Depth model](https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/depth/modeling_depth.py): `languagebind/depth/modeling_depth.py`
- [Depth configuration](https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/depth/configuration_depth.py): `languagebind/depth/configuration_depth.py`
- [Thermal model](https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/thermal/modeling_thermal.py): `languagebind/thermal/modeling_thermal.py`
- [Thermal configuration](https://github.com/saketh-04/lv-mae-event-span/blob/main/languagebind/thermal/configuration_thermal.py): `languagebind/thermal/configuration_thermal.py`

### Training support

- [Metrics and checkpoints](https://github.com/saketh-04/lv-mae-event-span/blob/main/util/misc.py): `util/misc.py`
- [Learning-rate schedule](https://github.com/saketh-04/lv-mae-event-span/blob/main/util/lr_sched.py): `util/lr_sched.py`
- [Logger adapters](https://github.com/saketh-04/lv-mae-event-span/blob/main/loggers/base_logger.py): `loggers/base_logger.py`

### Other components

- **Experiment operator**
- **Attentive probe**

## Connections

- Experiment operator → Video embeddings: runs extraction
- Video embeddings → Video model: encodes video
- Experiment operator → XD-Violence CSV: builds splits
- XD-Violence CSV → Embedding dataset: provides CSV
- Pretraining entry → Embedding dataset: loads data (evidence: `main_pretrain.py`)
- Embedding dataset → Event-span masking: applies masking (evidence: `data/CombinedData.py`)
- Pretraining entry → Event-span masking: sets temperature (evidence: `main_pretrain.py`)
- Pretraining entry → Train and validate: runs epochs (evidence: `main_pretrain.py`)
- Train and validate → Masked autoencoder: computes loss (evidence: `engine_pretrain.py`)
- Pretraining entry → Masked autoencoder: constructs model (evidence: `main_pretrain.py`)
- Train and validate → Metrics and checkpoints: logs metrics (evidence: `engine_pretrain.py`)
- Pretraining entry → Metrics and checkpoints: saves checkpoints (evidence: `main_pretrain.py`)
- Train and validate → Learning-rate schedule: updates learning rate (evidence: `engine_pretrain.py`)
- Pretraining entry → Logger adapters: configures logging (evidence: `main_pretrain.py`)
- Video model → Video configuration: uses configuration (evidence: `languagebind/video/modeling_video.py`)
- Image model → Image configuration: uses configuration (evidence: `languagebind/image/modeling_image.py`)
- Audio model → Audio configuration: uses configuration (evidence: `languagebind/audio/modeling_audio.py`)
- Depth model → Depth configuration: uses configuration (evidence: `languagebind/depth/modeling_depth.py`)
- Thermal model → Thermal configuration: uses configuration (evidence: `languagebind/thermal/modeling_thermal.py`)
- Experiment operator → Attentive probe: evaluates encoder