How sail-sg/dualformer works
Project Type & Purpose
• This is a research‐oriented machine‐learning framework (built atop mmaction2) for efficient video recognition.
• It provides:
• Config files for model variants (DualFormer‐Tiny/Small/Base, schedules, datasets)
• A core library (mmaction/…) implementing data pipelines, model definitions, training/inference APIs
• Demo and tooling scripts for visualization, analysis, training, testing, and deployment
Main Components
A. Configuration Layer (configs/)
• Default and experiment‐specific config files (model architecture, optimizer, schedule, dataset settings)
• Hierarchy: _base_ modules → recognition/dualformer/*.py
B. Core Library (mmaction/)
• apis/: CLI‐style training/test/inference entry points
• core/: runners (distributed, omnisource), schedulers, hooks, optimizers, evaluation
• datasets/: data‐loading pipelines, augmentations, samplers for video, raw frames, audio
• models/:
• backbones/dualformer.py (local‐global transformer)
• heads/ (I3D head, classification, bbox, etc.)
• recognizers (with/without token‐labelling)
• localizers (proposal, detection utilities)
• losses, necks, roi_extractors
• utils/: logging, grad‐cam, optimizer wrappers, module hooks
C. Custom MMCV Extensions (mmcv_custom/)
• Modified runner and checkpoint logic
• Integration layer injecting mmcv APIs into mmaction2 core
D. Tools & Scripts (tools/)
• train.py, test.py, dist_train.sh, dist_test.sh
• analysis/ (log parsing, benchmark, flops, accuracy reports)
• deployment (pytorch2onnx, publish_model)
• misc (feature extraction, proposal generation)
E. Demo & Visualization (demo/)
• Jupyter notebooks and Python scripts for:
• Running inference on videos or webcam
• Grad‐CAM heatmap visualization
• Spatio‐temporal detection
• Video I/O and sample outputs
F. External Dependencies & Infrastructure
• PyTorch & CUDA on GPU cluster (e.g., 8×A100)
• MMCV, timm, NVIDIA Apex (optional)
• Data storage (local disk or cloud buckets for Kinetics, UCF-101, etc.)
• Logging systems and checkpoints
Relationships & Data Flow
• Configs → apis/train.py or dist_train.sh → Core Library (runner)
• Runner calls Datasets → DataLoader → Model (Backbone + Head)
• Loss & Optimizer defined by config → training loop → checkpoint via mmcv_custom
• Runner → Evaluation hooks → metrics (accuracy, detection mAP)
• For inference: apis/inference.py or demo scripts consume a checkpoint and config → Model → outputs (predictions, heatmaps)
• Tools/dist_test.sh and tools/deployment scripts plug into the same inference APIs
Architectural Patterns & Principles
• Modular configuration‐driven design: decoupling of configs from code
• Plugin architecture: every component (dataset, model, optimizer, scheduler) is registered and can be swapped
• Separation of concerns:
• “Core” for reusable library code
• “Tools” for CLI and helper scripts
• “Demo” for end‐user examples
• Extension of upstream mmaction2 without modifying its core: via mmcv_custom