# sail-sg/dualformer architecture

> Architecture diagram of the GitHub repository sail-sg/dualformer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sail-sg/dualformer
- Repository: https://github.com/sail-sg/dualformer

## Overview

##### Project Type & Purpose
- This is a research‐oriented machine‐learning framework (built atop mmaction2) for efficient video recognition.
- It provides:
- Config files for model variants (DualFormer‐Tiny/Small/Base, schedules, datasets)
- A core library (`mmaction/…`) implementing data pipelines, model definitions, training/inference APIs
- Demo and tooling scripts for visualization, analysis, training, testing, and deployment

##### Main Components

A. Configuration Layer (configs/)
- Default and experiment‐specific config files (model architecture, optimizer, schedule, dataset settings)
- Hierarchy: `_base_` modules → recognition/dualformer/*.py

B. Core Library (`mmaction/`)
- apis/: CLI‐style training/test/inference entry points
- core/: runners (distributed, omnisource), schedulers, hooks, optimizers, evaluation
- datasets/: data‐loading pipelines, augmentations, samplers for video, raw frames, audio
- models/:
- backbones/dualformer.py (local‐global transformer)
- heads/ (I3D head, classification, bbox, etc.)
- recognizers (with/without token‐labelling)
- localizers (proposal, detection utilities)
- losses, necks, roi_extractors
- utils/: logging, grad‐cam, optimizer wrappers, module hooks

C. Custom MMCV Extensions (`mmcv_custom/`)
- Modified runner and checkpoint logic
- Integration layer injecting mmcv APIs into mmaction2 core

D. Tools & Scripts (tools/)
- train.py, test.py, dist_train.sh, dist_test.sh
- analysis/ (log parsing, benchmark, flops, accuracy reports)
- deployment (pytorch2onnx, publish_model)
- misc (feature extraction, proposal generation)

E. Demo & Visualization (demo/)
- Jupyter notebooks and Python scripts for:
- Running inference on videos or webcam
- Grad‐CAM heatmap visualization
- Spatio‐temporal detection
- Video I/O and sample outputs

F. External Dependencies & Infrastructure
- PyTorch & CUDA on GPU cluster (e.g., 8×A100)
- MMCV, timm, NVIDIA Apex (optional)
- Data storage (local disk or cloud buckets for Kinetics, UCF-101, etc.)
- Logging systems and checkpoints

##### Relationships & Data Flow
- Configs → apis/train.py or dist_train.sh → Core Library (runner)
- Runner calls Datasets → DataLoader → Model (Backbone + Head)
- Loss & Optimizer defined by config → training loop → checkpoint via mmcv_custom
- Runner → Evaluation hooks → metrics (accuracy, detection mAP)
- For inference: apis/inference.py or demo scripts consume a checkpoint and config → Model → outputs (predictions, heatmaps)
- Tools/dist_test.sh and tools/deployment scripts plug into the same inference APIs

##### Architectural Patterns & Principles
- Modular configuration‐driven design: decoupling of configs from code
- Plugin architecture: every component (dataset, model, optimizer, scheduler) is registered and can be swapped
- Separation of concerns:
- “Core” for reusable library code
- “Tools” for CLI and helper scripts
- “Demo” for end‐user examples
- Extension of upstream mmaction2 without modifying its core: via mmcv_custom

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Configuration Layer"
        config_base["_base_ modules"]:::config
        config_model["dualformer_*.py"]:::config
        config_schedule["schedules"]:::config
        config_recog["configs/recognition/dualformer"]:::config
    end

    subgraph "Core Library"
        subgraph "APIs"
            train_api["train.py"]:::core
            test_api["test.py"]:::core
            infer_api["inference.py"]:::core
        end
        subgraph "Runner & Extensions"
            runner["omnisource_runner.py"]:::core
            scheduler["lr_updater.py"]:::core
            hook["output.py"]:::core
            optimizer_constructor["tsm_optimizer_constructor.py"]:::core
            eval_acc["accuracy.py"]:::core
            eval_det["eval_detection.py"]:::core
        end
        subgraph "Data"
            dataset_files["rawvideo/video/audio datasets"]:::core
            pipelines["pipelines"]:::core
            sampler["distributed_sampler.py"]:::core
        end
        subgraph "Models"
            backbone["dualformer.py"]:::core
            heads["heads"]:::core
            recognizers["recognizers"]:::core
            localizers["localizers"]:::core
            losses["losses"]:::core
            necks["tpn.py"]:::core
            roi["single_straight3d.py"]:::core
        end
        subgraph "Utilities"
            logger["logger.py"]:::core
            gradcam["gradcam_utils.py"]:::core
            modhooks["module_hooks.py"]:::core
            optutils["optimizer.py"]:::core
        end
    end

    subgraph "Custom MMCV Extensions"
        ext_runner["epoch_based_runner.py"]:::ext
        ext_checkpoint["checkpoint.py"]:::ext
    end

    subgraph "Tools & Scripts"
        tool_train["train.py"]:::tool
        tool_test["test.py"]:::tool
        dist_train["dist_train.sh"]:::tool
        dist_test["dist_test.sh"]:::tool
        analysis["analyze_logs.py"]:::tool
        deployment["pytorch2onnx.py & publish_model.py"]:::tool
        misc["misc scripts"]:::tool
    end

    subgraph "Demo & Visualization"
        demo_py["demo scripts"]:::demo
        notebooks["Jupyter notebooks"]:::demo
        supporting["supporting files"]:::demo
    end

    subgraph "External Dependencies & Infra"
        gpu["GPU Cluster (8×A100)"]:::extsys
        storage["Data Storage"]:::extsys
        deps["PyTorch, MMCV, timm, Apex"]:::extsys
        logsys["Logging & Checkpoints"]:::extsys
    end

    %% Relationships - Training
    config_base --> train_api
    config_model --> train_api
    config_schedule --> train_api
    config_recog --> train_api
    tool_train --> train_api
    dist_train --> train_api

    train_api --> runner
    runner --> dataset_files
    dataset_files --> pipelines
    pipelines --> sampler
    sampler --> backbone
    backbone --> heads
    heads --> recognizers
    recognizers --> localizers
    localizers --> losses
    losses --> necks
    necks --> roi
    train_api --> optimizer_constructor
    optimizer_constructor --> runner
    runner --> scheduler
    runner --> hook
    hook --> eval_acc
    hook --> eval_det
    runner --> ext_runner
    ext_runner --> ext_checkpoint
    ext_checkpoint --> logsys
    dataset_files --> storage
    runner --> gpu
    deps --> train_api
    deps --> runner

    %% Relationships - Inference
    config_base --> infer_api
    config_model --> infer_api
    config_schedule --> infer_api
    config_recog --> infer_api
    tool_test --> test_api
    dist_test --> test_api
    tool_test --> infer_api
    dist_test --> infer_api
    deployment --> infer_api

    infer_api --> backbone
    infer_api --> heads
    infer_api --> recognizers
    infer_api --> demo_py
    demo_py --> notebooks
    demo_py --> supporting
    infer_api --> logsys

    %% Tools analysis
    analysis --> logsys

    %% Click Events
    click config_base "https://github.com/sail-sg/dualformer/blob/main/configs/_base_/default_runtime.py"
    click config_model "https://github.com/sail-sg/dualformer/blob/main/configs/_base_/models/dualformer/dualformer_*.py"
    click config_schedule "https://github.com/sail-sg/dualformer/tree/main/configs/_base_/schedules/"
    click config_recog "https://github.com/sail-sg/dualformer/tree/main/configs/recognition/dualformer/"
    click train_api "https://github.com/sail-sg/dualformer/blob/main/mmaction/apis/train.py"
    click test_api "https://github.com/sail-sg/dualformer/blob/main/mmaction/apis/test.py"
    click infer_api "https://github.com/sail-sg/dualformer/blob/main/mmaction/apis/inference.py"
    click runner "https://github.com/sail-sg/dualformer/blob/main/mmaction/core/runner/omnisource_runner.py"
    click scheduler "https://github.com/sail-sg/dualformer/blob/main/mmaction/core/scheduler/lr_updater.py"
    click hook "https://github.com/sail-sg/dualformer/blob/main/mmaction/core/hooks/output.py"
    click optimizer_constructor "https://github.com/sail-sg/dualformer/blob/main/mmaction/core/optimizer/tsm_optimizer_constructor.py"
    click eval_acc "https://github.com/sail-sg/dualformer/blob/main/mmaction/core/evaluation/accuracy.py"
    click eval_det "https://github.com/sail-sg/dualformer/blob/main/mmaction/core/evaluation/eval_detection.py"
    click dataset_files "https://github.com/sail-sg/dualformer/blob/main/mmaction/datasets/rawvideo_dataset.py"
    click pipelines "https://github.com/sail-sg/dualformer/tree/main/mmaction/datasets/pipelines/"
    click sampler "https://github.com/sail-sg/dualformer/blob/main/mmaction/datasets/samplers/distributed_sampler.py"
    click backbone "https://github.com/sail-sg/dualformer/blob/main/mmaction/models/backbones/dualformer.py"
    click heads "https://github.com/sail-sg/dualformer/tree/main/mmaction/models/heads/"
    click recognizers "https://github.com/sail-sg/dualformer/tree/main/mmaction/models/recognizers/"
    click localizers "https://github.com/sail-sg/dualformer/tree/main/mmaction/models/localizers/"
    click losses "https://github.com/sail-sg/dualformer/tree/main/mmaction/models/losses/"
    click necks "https://github.com/sail-sg/dualformer/blob/main/mmaction/models/necks/tpn.py"
    click roi "https://github.com/sail-sg/dualformer/blob/main/mmaction/models/roi_extractors/single_straight3d.py"
    click logger "https://github.com/sail-sg/dualformer/blob/main/mmaction/utils/logger.py"
    click gradcam "https://github.com/sail-sg/dualformer/blob/main/mmaction/utils/gradcam_utils.py"
    click modhooks "https://github.com/sail-sg/dualformer/blob/main/mmaction/utils/module_hooks.py"
    click optutils "https://github.com/sail-sg/dualformer/blob/main/mmaction/utils/optimizer.py"
    click ext_runner "https://github.com/sail-sg/dualformer/blob/main/mmcv_custom/runner/epoch_based_runner.py"
    click ext_checkpoint "https://github.com/sail-sg/dualformer/blob/main/mmcv_custom/runner/checkpoint.py"
    click tool_train "https://github.com/sail-sg/dualformer/blob/main/tools/train.py"
    click tool_test "https://github.com/sail-sg/dualformer/blob/main/tools/test.py"
    click dist_train "https://github.com/sail-sg/dualformer/blob/main/tools/dist_train.sh"
    click dist_test "https://github.com/sail-sg/dualformer/blob/main/tools/dist_test.sh"
    click analysis "https://github.com/sail-sg/dualformer/blob/main/tools/analysis/analyze_logs.py"
    click deployment "https://github.com/sail-sg/dualformer/blob/main/tools/deployment/pytorch2onnx.py"
    click misc "https://github.com/sail-sg/dualformer/blob/main/tools/misc/flow_extraction.py"
    click demo_py "https://github.com/sail-sg/dualformer/blob/main/demo/demo.py"
    click notebooks "https://github.com/sail-sg/dualformer/blob/main/demo/demo.ipynb"
    click supporting "https://github.com/sail-sg/dualformer/blob/main/demo/fuse/data_list.txt"

    %% Styles
    classDef config fill:#bbdefb,stroke:#333,stroke-width:1px
    classDef core fill:#c8e6c9,stroke:#333,stroke-width:1px
    classDef ext fill:#ffe0b2,stroke:#333,stroke-width:1px
    classDef tool fill:#fff9c4,stroke:#333,stroke-width:1px
    classDef demo fill:#e1bee7,stroke:#333,stroke-width:1px
    classDef extsys fill:#eeeeee,stroke:#333,stroke-width:1px
```
