How saketh-04/lv-mae-event-span works
This repository implements event-span-masked LV-MAE pretraining for long-video representations, with XD-Violence attentive-probe classification documented as the downstream use. The main runtime path loads segment embeddings, applies event-aware masking, trains the MAE, and records training progress. The README also describes embedding extraction and CSV preparation. LanguageBind includes several modality model families, though the documented experiment uses video only. Some links rely on the README or SOURCE INDEX; downstream probe implementation is not present in the supplied tree.
Main entry points: main_pretrain.py Pretraining entry · engine_pretrain.py Train and validate · CombinedData.py Embedding dataset · extract_xd.py Video embeddings
Components 23
Data preparation
- Video embeddings
data_preprocess/extract_xd.py - XD-Violence CSV
data_preprocess/build_xd_csv.py - Embedding dataset
data/CombinedData.py - Event-span masking
data/event_span_masking.py
Masked pretraining
- Pretraining entry
main_pretrain.py - Train and validate
engine_pretrain.py - Masked autoencoder
models_mae.py
LanguageBind models
- Video model
languagebind/video/modeling_video.py - Video configuration
languagebind/video/configuration_video.py - Video processor
languagebind/video/processing_video.py - Image model
languagebind/image/modeling_image.py - Image configuration
languagebind/image/configuration_image.py - Audio model
languagebind/audio/modeling_audio.py - Audio configuration
languagebind/audio/configuration_audio.py - Depth model
languagebind/depth/modeling_depth.py - Depth configuration
languagebind/depth/configuration_depth.py - Thermal model
languagebind/thermal/modeling_thermal.py - Thermal configuration
languagebind/thermal/configuration_thermal.py
Training support
- Metrics and checkpoints
util/misc.py - Learning-rate schedule
util/lr_sched.py - Logger adapters
loggers/base_logger.py
Other components
- Experiment operator
- Attentive probe
Connections 20
- Experiment operator → Video embeddings: runs extraction
- Video embeddings → Video model: encodes video
- Experiment operator → XD-Violence CSV: builds splits
- XD-Violence CSV → Embedding dataset: provides CSV
- Pretraining entry → Embedding dataset: loads data
main_pretrain.py - Embedding dataset → Event-span masking: applies masking
data/CombinedData.py - Pretraining entry → Event-span masking: sets temperature
main_pretrain.py - Pretraining entry → Train and validate: runs epochs
main_pretrain.py - Train and validate → Masked autoencoder: computes loss
engine_pretrain.py - Pretraining entry → Masked autoencoder: constructs model
main_pretrain.py - Train and validate → Metrics and checkpoints: logs metrics
engine_pretrain.py - Pretraining entry → Metrics and checkpoints: saves checkpoints
main_pretrain.py - Train and validate → Learning-rate schedule: updates learning rate
engine_pretrain.py - Pretraining entry → Logger adapters: configures logging
main_pretrain.py - Video model → Video configuration: uses configuration
languagebind/video/modeling_video.py - Image model → Image configuration: uses configuration
languagebind/image/modeling_image.py - Audio model → Audio configuration: uses configuration
languagebind/audio/modeling_audio.py - Depth model → Depth configuration: uses configuration
languagebind/depth/modeling_depth.py - Thermal model → Thermal configuration: uses configuration
languagebind/thermal/modeling_thermal.py - Experiment operator → Attentive probe: evaluates encoder