How hustvl/vim works
Project Type and Purpose
• Vision Mamba (“Vim”) is a research-oriented deep-learning framework for visual representation learning using bidirectional state-space models.
• It is packaged as a Python library, with C++/CUDA extensions (causal-conv1d, mamba-1p1p1) and companion projects for detection (det/Detectron2) and segmentation (seg).
Top-Level Components
A. Core Vim Package (folder: vim/)
• CLI entry point: main.py (parses args for train/eval)
• Dataset & DataLoader: datasets.py, samplers.py
• Transforms & Augmentations: augment.py, utils.py, rope.py (positional embeddings)
• Model Definition: models_mamba.py (builds Vision-Mamba backbone with MambaBlocks)
• Loss Functions: losses.py
• Engine & Training Loop: engine.py (train/validate/test loops), run_with_submitit.py
• PyTorch Hub support: hubconf.py
B. External CUDA Extensions
• causal-conv1d/: a Python package with C++/CUDA kernels for 1D causal conv (used for state-space computations)
• mamba-1p1p1/: the underlying bidirectional state-space model implementation with its own C++/CUDA/triton ops
C. Companion Projects
• det/: Detectron2 integration (configs, modeling, data, engine, evaluation modules)
• seg/: segmentation framework (based on mmcv, configs, custom runner)
D. Scripts & Utilities
• vim/scripts/: bash scripts to launch pretraining/finetuning (pt-vim-s.sh, ft-vim-t.sh)
• seg/tools/, det/tools/: analysis, deployment, inference scripts
E. Packaging & Tests
• setup.py, *.cfg for each subproject
• tests/ under each (e.g., causal-conv1d/tests, mamba-1p1p1/tests, det/tests, seg/test.py)
Relationships & Data Flow
• User invokes a bash script (e.g., vim/scripts/pt-vim-t.sh) → calls python main.py
• main.py → Engine (engine.py)
→ Instantiates Dataset (datasets.py)
→ Applies Transforms/Augmentations → DataLoader with Sampler
→ Feeds batches to Model (models_mamba.py)
↳ Internally uses mamba-1p1p1 ops for StateSpaceBlock
↳ Uses causal-conv1d ops for causal convolutions
→ Computes Loss (losses.py) → Backprop via PyTorch
→ Engine updates parameters, logs metrics, checkpoints
• At evaluation/inference: similar flow, but Engine triggers evaluation hooks (engine.py) and Predictor (in det/demo or seg/scripts)
Architectural Patterns & Principles
• Modular layering:
1) Data layer (datasets, transforms, samplers)
2) Model layer (pure Python + PyTorch + C++/CUDA extensions)
3) Training engine (loop, hooks, logging, checkpointing)
4) CLI/Scripting layer (scripts, hubconf)
• Extension pattern: build critical performance kernels in C++/CUDA/Triton, expose Python interfaces
• Configuration-driven: YAML/py config files under det/configs, seg/configs, twin with detectron2 & mmcv patterns