# zqsiat/ov-oad architecture

> Architecture diagram of the GitHub repository zqsiat/ov-oad, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zqsiat/ov-oad
- Repository: https://github.com/zqsiat/ov-oad

## Overview

##### Main Components
a. Data Ingestion & Feature Extraction
- extract_features/ scripts & utilities
- Video frame extractor (save_jpg.py)
- Feature extractor (internvid_extract.py calls a pretrained CLIP or Vision Transformer)
- Output: saved image frames and feature tensors
b. Dataset & Configuration Management
- datasets/ and oad/datasets/ folders
- Builder classes (datasets/builder.py, oad/evaluation/builder.py)
- Tokenizers, samplers, formatters
- YAML config files (configs/*.yml, oad/configs)
c. Model Definition & Builders
- clip/ module (clip.py, model.py, tokenizer) – vision‐language model
- models/ folder (clipmodel, transformer, group_vit, adapter, losses, multi_label_contrastive)
- models/builder.py stitches backbone + decoder + aggregator
d. Training & Pretraining Engine
- main_pretrain.py (video‐text pretraining)
- Training loop: load data via datasets.builder, build model via models.builder, apply losses, optimizer, lr_scheduler
- Checkpointing & logging via utils/ (checkpoint.py, logger.py)
e. Online Action Detection & Evaluation
- main_oad.py (loads pretrained weights, runs frame‐by‐frame inference)
- oad/evaluation/ scripts (group_vit_oad.py, builder) calculate mAP, CAP
- Output: real‐time action labels per frame

##### Relationships & Data Flow
- Raw Video → Data Ingestion & Frame Extraction → Saved Frames
→ Feature Extraction → Feature Store (disk)
→ Dataset Builder (samples features + text queries)
→ Model Training / Pretraining → Checkpoints
→ Pretrained Model → Online Action Detection
→ Inference Outputs → Evaluation Module → Metrics

##### Architectural Patterns & Principles
- Pipeline pattern: clearly separated stages (extract→dataset→train→infer)
- Modular design: interchangeable backbones (GroupViT, Uniformer), tokenizers, and adapters

##### Technologies & Frameworks
- Python & PyTorch
- Pretrained vision-language model (clip/)
- Transformer architectures (models/transformer.py, vision_transformer.py)
- GroupViT codebase adaptation
- YAML configs & EasyDict
- Bash scripts for orchestration (extract_features/*.sh)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer (blue)
    RawVideo["Raw Video Files"]:::data
    FrameExtractor["Frame Extractor\n(save_jpg.py)"]:::data
    click FrameExtractor "https://github.com/zqsiat/ov-oad/blob/main/extract_features/save_jpg.py"
    ImageFrames["Image Frames\n(Folder)"]:::data
    FeatureExtractor["Feature Extractor\n(internvid_extract.py)"]:::data
    click FeatureExtractor "https://github.com/zqsiat/ov-oad/blob/main/extract_features/internvid_extract.py"
    RunSaveImgs["run_save_imgs.sh"]:::data
    click RunSaveImgs "https://github.com/zqsiat/ov-oad/blob/main/extract_features/run_save_imgs.sh"
    RunExtFeat["run_extc_feat.sh"]:::data
    click RunExtFeat "https://github.com/zqsiat/ov-oad/blob/main/extract_features/run_extc_feat.sh"
    DataInfo["new_data_info.json"]:::data
    click DataInfo "https://github.com/zqsiat/ov-oad/blob/main/extract_features/data_list/new_data_info.json"
    FeatureStore["Feature Store\n(Tensors)"]:::storage

    RawVideo --> FrameExtractor --> ImageFrames
    ImageFrames --> RunSaveImgs --> FeatureExtractor --> RunExtFeat --> DataInfo --> FeatureStore

    %% Dataset & Configuration Management (gray)
    subgraph "Dataset & Config" 
        direction TB
        DatasetBuilder["Dataset Builder\n(datasets/builder.py)"]:::config
        click DatasetBuilder "https://github.com/zqsiat/ov-oad/blob/main/datasets/builder.py"
        BaseDataset["Base Dataset\n(datasets/base_dataset.py)"]:::config
        click BaseDataset "https://github.com/zqsiat/ov-oad/blob/main/datasets/base_dataset.py"
        ClipDataset["CLIP Dataset\n(datasets/clip_dataset.py)"]:::config
        click ClipDataset "https://github.com/zqsiat/ov-oad/blob/main/datasets/clip_dataset.py"
        VideoClipDataset["Video-CLIP Dataset\n(datasets/video_clip_dataset.py)"]:::config
        click VideoClipDataset "https://github.com/zqsiat/ov-oad/blob/main/datasets/video_clip_dataset.py"
        InternvidLoader["InternVid Loader\n(datasets/internvid_loader.py)"]:::config
        click InternvidLoader "https://github.com/zqsiat/ov-oad/blob/main/datasets/internvid_loader.py"
        Formatting["Formatting\n(datasets/formatting.py)"]:::config
        click Formatting "https://github.com/zqsiat/ov-oad/blob/main/datasets/formatting.py"
        Sampler["Sampler\n(datasets/sampler.py)"]:::config
        click Sampler "https://github.com/zqsiat/ov-oad/blob/main/datasets/sampler.py"
        Tokenizer["Tokenizer\n(datasets/tokenizer.py)"]:::config
        click Tokenizer "https://github.com/zqsiat/ov-oad/blob/main/datasets/tokenizer.py"
        ConfigDefault["default.yml"]:::config
        click ConfigDefault "https://github.com/zqsiat/ov-oad/blob/main/configs/default.yml"
        ConfigAnet["enc32_anet_3layers_lsxattn.yml"]:::config
        click ConfigAnet "https://github.com/zqsiat/ov-oad/blob/main/configs/enc32_anet_3layers_lsxattn.yml"
        OADUtils["OAD Dataset Utils\n(oad/datasets/datasets_utils.py)"]:::config
        click OADUtils "https://github.com/zqsiat/ov-oad/blob/main/oad/datasets/datasets_utils.py"
        OADDatasetsBase["Thumos14 Config\n(oad/configs/_base_/datasets/thomus14.py)"]:::config
        click OADDatasetsBase "https://github.com/zqsiat/ov-oad/blob/main/oad/configs/_base_/datasets/thomus14.py"
    end
    FeatureStore --> DatasetBuilder

    %% Model Definition & Builders (orange)
    subgraph "Model Definition & Builders"
        direction TB
        ModelsBuilder["Models Builder\n(models/builder.py)"]:::model
        click ModelsBuilder "https://github.com/zqsiat/ov-oad/blob/main/models/builder.py"
        CLIPModel["CLIP Model\n(clip/clip.py & clip/model.py)"]:::model
        click CLIPModel "https://github.com/zqsiat/ov-oad/blob/main/clip/clip.py"
        click CLIPModel "https://github.com/zqsiat/ov-oad/blob/main/clip/model.py"
        SimpleTokenizer["Simple Tokenizer\n(clip/simple_tokenizer.py)"]:::model
        click SimpleTokenizer "https://github.com/zqsiat/ov-oad/blob/main/clip/simple_tokenizer.py"
        OADTR["OADTR\n(clip/oadtr.py)"]:::model
        click OADTR "https://github.com/zqsiat/ov-oad/blob/main/clip/oadtr.py"
        ZSOADVTC["ZSOAD VTC\n(clip/zsoad_vtc.py)"]:::model
        click ZSOADVTC "https://github.com/zqsiat/ov-oad/blob/main/clip/zsoad_vtc.py"
        VisionTransformer["Vision Transformer\n(models/vision_transformer.py)"]:::model
        click VisionTransformer "https://github.com/zqsiat/ov-oad/blob/main/models/vision_transformer.py"
        Transformer["Transformer\n(models/transformer.py)"]:::model
        click Transformer "https://github.com/zqsiat/ov-oad/blob/main/models/transformer.py"
        GroupViT["GroupViT\n(models/group_vit.py)"]:::model
        click GroupViT "https://github.com/zqsiat/ov-oad/blob/main/models/group_vit.py"
        Adapter["Adapter\n(models/adapter.py)"]:::model
        click Adapter "https://github.com/zqsiat/ov-oad/blob/main/models/adapter.py"
        Contrastive["Multi-Label Contrastive\n(models/multi_label_contrastive.py)"]:::model
        click Contrastive "https://github.com/zqsiat/ov-oad/blob/main/models/multi_label_contrastive.py"
        Losses["Losses\n(models/losses.py)"]:::model
        click Losses "https://github.com/zqsiat/ov-oad/blob/main/models/losses.py"
    end
    DatasetBuilder --> ModelsBuilder
    CLIPModel --> ModelsBuilder
    VisionTransformer --> ModelsBuilder

    %% Training & Pretraining Engine (green)
    subgraph "Training & Pretraining"
        direction TB
        PretrainMain["main_pretrain.py"]:::train
        click PretrainMain "https://github.com/zqsiat/ov-oad/blob/main/main_pretrain.py"
        Checkpoint["Checkpoint Manager\n(utils/checkpoint.py)"]:::train
        click Checkpoint "https://github.com/zqsiat/ov-oad/blob/main/utils/checkpoint.py"
        Logger["Logger\n(utils/logger.py)"]:::train
        click Logger "https://github.com/zqsiat/ov-oad/blob/main/utils/logger.py"
        Optimizer["Optimizer\n(utils/optimizer.py)"]:::train
        click Optimizer "https://github.com/zqsiat/ov-oad/blob/main/utils/optimizer.py"
        LRScheduler["LR Scheduler\n(utils/lr_scheduler.py)"]:::train
        click LRScheduler "https://github.com/zqsiat/ov-oad/blob/main/utils/lr_scheduler.py"
        ConfigUtil["Config Utils\n(utils/config.py)"]:::train
        click ConfigUtil "https://github.com/zqsiat/ov-oad/blob/main/utils/config.py"
        EasyDict["EasyDict\n(utils/easydict.py)"]:::train
        click EasyDict "https://github.com/zqsiat/ov-oad/blob/main/utils/easydict.py"
    end
    ModelsBuilder --> PretrainMain
    DatasetBuilder --> PretrainMain
    PretrainMain --> Checkpoint
    PretrainMain --> Logger

    %% Online Action Detection & Evaluation (purple)
    subgraph "Online Inference & Evaluation"
        direction TB
        OADMain["main_oad.py"]:::eval
        click OADMain "https://github.com/zqsiat/ov-oad/blob/main/main_oad.py"
        EvalBuilder["Evaluation Builder\n(oad/evaluation/builder.py)"]:::eval
        click EvalBuilder "https://github.com/zqsiat/ov-oad/blob/main/oad/evaluation/builder.py"
        GroupViTOAD["GroupViT OAD\n(oad/evaluation/group_vit_oad.py)"]:::eval
        click GroupViTOAD "https://github.com/zqsiat/ov-oad/blob/main/oad/evaluation/group_vit_oad.py"
        Palette["Group Palette\n(oad/evaluation/group_palette.txt)"]:::eval
        click Palette "https://github.com/zqsiat/ov-oad/blob/main/oad/evaluation/group_palette.txt"
        OADConfigs["OAD Configs\n(oad/configs)"]:::eval
        click OADConfigs "https://github.com/zqsiat/ov-oad/tree/main/oad/configs"
        OADDatasets["OAD Datasets\n(oad/datasets)"]:::eval
        click OADDatasets "https://github.com/zqsiat/ov-oad/tree/main/oad/datasets"
        Metrics["Metrics\n(mAP, CAP)"]:::eval
    end
    Checkpoint --> OADMain
    OADMain --> EvalBuilder --> GroupViTOAD --> Metrics

    %% Styles
    classDef data fill:#bbdefb,stroke:#1e88e5,color:#0d47a1;
    classDef storage fill:#e3f2fd,stroke:#64b5f6,color:#1565c0;
    classDef config fill:#cfd8dc,stroke:#90a4ae,color:#37474f;
    classDef model fill:#ffe0b2,stroke:#ffb74d,color:#e65100;
    classDef train fill:#c8e6c9,stroke:#81c784,color:#2e7d32;
    classDef eval fill:#e1bee7,stroke:#ce93d8,color:#6a1b9a;
```
