# hustvl/vim architecture

> Architecture diagram of the GitHub repository hustvl/vim, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hustvl/vim
- Repository: https://github.com/hustvl/vim

## Overview

##### Project Type and Purpose
- Vision Mamba (“Vim”) is a research-oriented deep-learning framework for visual representation learning using bidirectional state-space models.
- It is packaged as a Python library, with C++/CUDA extensions (causal-conv1d, mamba-1p1p1) and companion projects for detection (det/Detectron2) and segmentation (seg).

##### Top-Level Components
A. Core Vim Package (folder: vim/)
- CLI entry point: main.py (parses args for train/eval)
- Dataset & DataLoader: datasets.py, samplers.py
- Transforms & Augmentations: augment.py, utils.py, rope.py (positional embeddings)
- Model Definition: models_mamba.py (builds Vision-Mamba backbone with MambaBlocks)
- Loss Functions: losses.py
- Engine & Training Loop: engine.py (train/validate/test loops), run_with_submitit.py
- PyTorch Hub support: hubconf.py

B. External CUDA Extensions
- causal-conv1d/: a Python package with C++/CUDA kernels for 1D causal conv (used for state-space computations)
- mamba-1p1p1/: the underlying bidirectional state-space model implementation with its own C++/CUDA/triton ops

C. Companion Projects
- det/: Detectron2 integration (configs, modeling, data, engine, evaluation modules)
- seg/: segmentation framework (based on mmcv, configs, custom runner)

D. Scripts & Utilities
- vim/scripts/: bash scripts to launch pretraining/finetuning (pt-vim-s.sh, ft-vim-t.sh)
- seg/tools/, det/tools/: analysis, deployment, inference scripts

E. Packaging & Tests
- setup.py, *.cfg for each subproject
- tests/ under each (e.g., causal-conv1d/tests, mamba-1p1p1/tests, det/tests, seg/test.py)

##### Relationships & Data Flow
- User invokes a bash script (e.g., vim/scripts/pt-vim-t.sh) → calls python main.py
- main.py → Engine (engine.py)
→ Instantiates Dataset (datasets.py)
→ Applies Transforms/Augmentations → DataLoader with Sampler
→ Feeds batches to Model (models_mamba.py)
↳ Internally uses mamba-1p1p1 ops for StateSpaceBlock
↳ Uses causal-conv1d ops for causal convolutions
→ Computes Loss (losses.py) → Backprop via PyTorch
→ Engine updates parameters, logs metrics, checkpoints

- At evaluation/inference: similar flow, but Engine triggers evaluation hooks (engine.py) and Predictor (in det/demo or seg/scripts)

##### Architectural Patterns & Principles
- Modular layering:
1) Data layer (datasets, transforms, samplers)
2) Model layer (pure Python + PyTorch + C++/CUDA extensions)
3) Training engine (loop, hooks, logging, checkpointing)
4) CLI/Scripting layer (scripts, hubconf)
- Extension pattern: build critical performance kernels in C++/CUDA/Triton, expose Python interfaces
- Configuration-driven: YAML/py config files under det/configs, seg/configs, twin with detectron2 & mmcv patterns

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Scripts/CLI Layer
    subgraph "Scripts & CLI" 
        direction TB
        PT_S["pt-vim-s.sh"]:::scripts
        PT_T["pt-vim-t.sh"]:::scripts
        FT_S["ft-vim-s.sh"]:::scripts
        FT_T["ft-vim-t.sh"]:::scripts
        CLI["main.py (CLI)"]:::core
    end

    %% Core Vim Package
    subgraph "Core Vim Package" 
        direction TB
        Engine["engine.py (Training/Eval)"]:::core
        RunSubmit["run_with_submitit.py"]:::core
        Dataset["datasets.py"]:::core
        Sampler["samplers.py"]:::core
        Augment["augment.py"]:::core
        Utils["utils.py"]:::core
        Rope["rope.py"]:::core
        Model["models_mamba.py (Vision Backbone)"]:::core
        Loss["losses.py"]:::core
        Hub["hubconf.py"]:::core
        Checkpoint["Model Checkpoints"]:::store
    end

    %% External Extensions
    subgraph "External CUDA Extensions" 
        direction TB
        ExtCausal(["CausalConv1D Package"]):::external
        ExtMamba(["Mamba SSM Package"]):::external
    end

    %% Companion Projects
    subgraph "Companion Projects" 
        direction TB
        style CompanionProjects stroke-dasharray: 5 5
        det["det/ (Detectron2 Integration)"]:::companion
        seg["seg/ (MMCV-based Segmentation)"]:::companion
    end

    %% Packaging & Tests
    subgraph "Packaging & Tests"
        direction TB
        Setup["setup.py & cfg"]:::scripts
        TCausal["causal-conv1d/tests"]:::scripts
        TMamba["mamba-1p1p1/tests"]:::scripts
        TDet["det/tests"]:::scripts
        TSeg["seg/test.py"]:::scripts
    end

    %% Data Flow
    PT_S --> CLI
    PT_T --> CLI
    FT_S --> CLI
    FT_T --> CLI
    CLI --> Engine
    Engine --> Dataset
    Engine --> Sampler
    Engine --> Augment
    Engine --> Utils
    Engine --> Rope
    Dataset -->|"loads data"| Dataset
    Dataset -->|"uses sampler"| Sampler
    Engine -->|"prep transforms"| Augment
    Engine -->|"prep transforms"| Utils
    Engine -->|"prep transforms"| Rope
    Engine -->|"creates batches"| Dataset
    Engine --> Model
    Model --> Loss
    Loss --> Engine
    Engine --> Checkpoint

    %% Dependencies
    Model -- "uses C++/CUDA ops" --> ExtCausal
    Model -- "uses C++/CUDA/Triton ops" --> ExtMamba
    CLI --> Hub

    %% Legends (class definitions)
    classDef core fill:#b7e4c7,stroke:#2d6a4f,color:#1b4332;
    classDef external fill:#ffd97d,stroke:#d9480f,color:#a74400;
    classDef companion fill:#a3cef1,stroke:#1864ab,color:#023e8a;
    classDef scripts fill:#e0e0e0,stroke:#495057,color:#212529;
    classDef store fill:#c9c9ff,stroke:#3a0ca3,color:#1a056d;

    %% Click Events
    click PT_S "https://github.com/hustvl/vim/blob/main/vim/scripts/pt-vim-s.sh"
    click PT_T "https://github.com/hustvl/vim/blob/main/vim/scripts/pt-vim-t.sh"
    click FT_S "https://github.com/hustvl/vim/blob/main/vim/scripts/ft-vim-s.sh"
    click FT_T "https://github.com/hustvl/vim/blob/main/vim/scripts/ft-vim-t.sh"
    click CLI "https://github.com/hustvl/vim/blob/main/vim/main.py"
    click Engine "https://github.com/hustvl/vim/blob/main/vim/engine.py"
    click RunSubmit "https://github.com/hustvl/vim/blob/main/vim/run_with_submitit.py"
    click Dataset "https://github.com/hustvl/vim/blob/main/vim/datasets.py"
    click Sampler "https://github.com/hustvl/vim/blob/main/vim/samplers.py"
    click Augment "https://github.com/hustvl/vim/blob/main/vim/augment.py"
    click Utils "https://github.com/hustvl/vim/blob/main/vim/utils.py"
    click Rope "https://github.com/hustvl/vim/blob/main/vim/rope.py"
    click Model "https://github.com/hustvl/vim/blob/main/vim/models_mamba.py"
    click Loss "https://github.com/hustvl/vim/blob/main/vim/losses.py"
    click Hub "https://github.com/hustvl/vim/blob/main/vim/hubconf.py"
    click ExtCausal "https://github.com/hustvl/vim/tree/main/causal-conv1d/csrc/"
    click ExtCausal "https://github.com/hustvl/vim/tree/main/causal-conv1d/causal_conv1d/"
    click ExtCausal "https://github.com/hustvl/vim/blob/main/causal-conv1d/setup.py"
    click ExtMamba "https://github.com/hustvl/vim/tree/main/mamba-1p1p1/mamba_ssm/"
    click ExtMamba "https://github.com/hustvl/vim/tree/main/mamba-1p1p1/csrc/"
    click ExtMamba "https://github.com/hustvl/vim/blob/main/mamba-1p1p1/setup.py"
    click det "https://github.com/hustvl/vim/tree/main/det/"
    click seg "https://github.com/hustvl/vim/tree/main/seg/"
    click Setup "https://github.com/hustvl/vim/blob/main/setup.py"
    click TCausal "https://github.com/hustvl/vim/blob/main/causal-conv1d/tests/test_causal_conv1d.py"
    click TMamba "https://github.com/hustvl/vim/tree/main/mamba-1p1p1/tests/ops/"
    click TDet "https://github.com/hustvl/vim/tree/main/det/tests/"
    click TSeg "https://github.com/hustvl/vim/blob/main/seg/test.py"
```
