How zqsiat/ov-oad works
Main Components
a. Data Ingestion & Feature Extraction
• extract_features/ scripts & utilities
• Video frame extractor (save_jpg.py)
• Feature extractor (internvid_extract.py calls a pretrained CLIP or Vision Transformer)
• Output: saved image frames and feature tensors
b. Dataset & Configuration Management
• datasets/ and oad/datasets/ folders
• Builder classes (datasets/builder.py, oad/evaluation/builder.py)
• Tokenizers, samplers, formatters
• YAML config files (configs/*.yml, oad/configs)
c. Model Definition & Builders
• clip/ module (clip.py, model.py, tokenizer) – vision‐language model
• models/ folder (clipmodel, transformer, group_vit, adapter, losses, multi_label_contrastive)
• models/builder.py stitches backbone + decoder + aggregator
d. Training & Pretraining Engine
• main_pretrain.py (video‐text pretraining)
• Training loop: load data via datasets.builder, build model via models.builder, apply losses, optimizer, lr_scheduler
• Checkpointing & logging via utils/ (checkpoint.py, logger.py)
e. Online Action Detection & Evaluation
• main_oad.py (loads pretrained weights, runs frame‐by‐frame inference)
• oad/evaluation/ scripts (group_vit_oad.py, builder) calculate mAP, CAP
• Output: real‐time action labels per frame
Relationships & Data Flow
• Raw Video → Data Ingestion & Frame Extraction → Saved Frames
→ Feature Extraction → Feature Store (disk)
→ Dataset Builder (samples features + text queries)
→ Model Training / Pretraining → Checkpoints
→ Pretrained Model → Online Action Detection
→ Inference Outputs → Evaluation Module → Metrics
Architectural Patterns & Principles
• Pipeline pattern: clearly separated stages (extract→dataset→train→infer)
• Modular design: interchangeable backbones (GroupViT, Uniformer), tokenizers, and adapters
Technologies & Frameworks
• Python & PyTorch
• Pretrained vision-language model (clip/)
• Transformer architectures (models/transformer.py, vision_transformer.py)
• GroupViT codebase adaptation
• YAML configs & EasyDict
• Bash scripts for orchestration (extract_features/*.sh)