# opengvlab/dcnv4 architecture

> Architecture diagram of the GitHub repository opengvlab/dcnv4, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opengvlab/dcnv4
- Repository: https://github.com/opengvlab/dcnv4

## Overview

1. This repository is a multi-task vision model stack centered on the FlashInternImage backbone and its custom deformable operators. It is split into one low-level operator package plus three downstream application trees: classification, detection, and segmentation.

2. The lowest layer is `DCNv4_op/`, which implements the DCNv4 and flash deformable attention operators as PyTorch native extensions. The Python side lives in `DCNv4_op/DCNv4/functions` and `modules`, while the CUDA/C++ kernels live under `DCNv4_op/src`. `setup.py` and `make.sh` define the build boundary for compiling those ops into importable modules.

3. `DCNv4_op/scripts/` is a performance-tuning and validation layer. It contains search utilities for forward/backward kernel selection and small test programs, so the repo is not just model code but also operator benchmarking and kernel tuning infrastructure.

4. Each task subproject vendors its own compiled deformable-conv extension under `ops_dcnv3/`. These packages mirror the same pattern: thin Python wrappers, a `setup.py` build, and CPU/CUDA sources. Architecturally, this isolates native code from the higher-level task code and lets each application build the operator it needs independently.

5. `classification/` is a self-contained image classification training stack. Data enters through dataset builders, cached image-folder readers, samplers, and transform utilities; flows into model definitions in `classification/models/`; then through optimizers, LR schedulers, EMA/DeepSpeed hooks, and training entry points such as `main.py`, `main_deepspeed.py`, and `main_accelerate.py`. It also includes export and feature-extraction paths.

6. In classification, `flash_intern_image.py` is the key model implementation, with configs in YAML files describing model size, training recipe, and resolution. This subtree is designed around standard PyTorch training but with task-specific glue for distributed training and ImageNet-style datasets.

7. `detection/` is built as a customized MMDetection-style stack. The `mmdet_custom/` tree provides registered backbones, detectors, dense heads, necks, and utility modules; `mmcv_custom/` provides optimizer construction and layer-decay logic. Configs under `detection/configs/` define COCO and CrowdHuman experiments, while `train.py`, `test.py`, distributed shell scripts, and demos form the execution layer.

8. The detection data flow is: dataset/config → custom backbone (`flash_intern_image`) → FPN/neck → task head or detector (`Mask R-CNN`, `Cascade`, `DINO`, etc.) → losses and evaluation. The repo keeps detector-specific logic separate from the backbone so the same FlashInternImage can be reused across multiple detector families.

9. `segmentation/` mirrors the detection structure but for MMSegmentation-like workloads. It has `mmseg_custom/` for custom backbones, decode heads, segmentors, losses, assigners, and dataset/pipeline logic; `mmcv_custom/` for optimizer/layer-decay support; and `configs/` for ADE20K and other segmentation benchmarks.

10. The segmentation runtime path is dataset/pipeline → FlashInternImage backbone → decoder head or Mask2Former/UperNet-style segmentor → segmentation losses and evaluation utilities. This subtree also includes deployment configs for ONNX and TensorRT, which is an important architectural boundary: training-time graph vs export/deployment graph.

11. Across detection and segmentation, the custom backbone is the same architectural bridge: `flash_intern_image.py` adapts the operator-backed feature extractor into each framework’s registry and feature-map conventions. The rest of the task-specific model code mainly handles necks, heads, assigners, losses, and dataset-specific preprocessing.

12. The repo also contains task-specific orchestration scripts for distributed execution: `train.sh`, `dist_train.sh`, `dist_test.sh`, and `slurm_*` variants. These are the operational boundary between the model code and cluster/runtime environments such as multi-GPU training, DeepSpeed, or Slurm-managed jobs.

13. Overall, the main diagram should show three layers: native operator kernels at the bottom, shared FlashInternImage/backbone code in the middle, and three independent application stacks on top. The critical cross-cutting concern is that all three tasks depend on custom CUDA extensions for deformable convolution, which is what gives the repo its performance-oriented character.

Main entry points: [`segmentation/train.py`](https://github.com/opengvlab/dcnv4/blob/main/segmentation/train.py) (Seg Runtime), [`classification/main.py`](https://github.com/opengvlab/dcnv4/blob/main/classification/main.py) (Train Runtime), [`detection/mmdet_custom/models`](https://github.com/opengvlab/dcnv4/tree/main/detection/mmdet_custom/models) (Det Models), [`segmentation/mmseg_custom/models`](https://github.com/opengvlab/dcnv4/tree/main/segmentation/mmseg_custom/models) (Seg Models)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_native_ops["Native Ops"]
  node_dcnv4_op["DCNv4 Ops<br/>native extension package"]
  node_dcnv4_tuning["Kernel Search<br/>benchmark/tuning"]
  node_task_dcnv3_ops["Task DCNv3<br/>vendored extension"]
end

subgraph group_classification["Classification"]
  node_cls_data["Data Stack<br/>dataset/input"]
  node_cls_model["Cls Model<br/>model definitions"]
  node_cls_train["Train Runtime<br/>training entrypoints<br/>[main.py]"]
  node_cls_export["Export/Features<br/>inference utilities<br/>[export.py]"]
end

subgraph group_detection["Detection"]
  node_det_runtime["Det Runtime<br/>execution layer<br/>[train.py]"]
  node_det_models["Det Models<br/>custom mmdet stack"]
  node_det_cfg["Det Configs<br/>experiment configs"]
  node_det_ops["Det Ops<br/>vendored extension"]
end

subgraph group_segmentation["Segmentation"]
  node_seg_runtime["Seg Runtime<br/>execution layer<br/>[train.py]"]
  node_seg_models["Seg Models<br/>custom mmseg stack"]
  node_seg_cfg["Seg Configs<br/>experiment configs"]
  node_seg_deploy["Deploy Graph<br/>onnx/tensorrt configs"]
  node_seg_ops["Seg Ops<br/>vendored extension"]
end

node_flash_backbone{{"FlashInternImage<br/>shared backbone"}}

node_dcnv4_op -->|"kernels for"| node_flash_backbone
node_dcnv4_tuning -->|"validates"| node_dcnv4_op
node_task_dcnv3_ops -->|"accelerates"| node_flash_backbone
node_flash_backbone -->|"used by"| node_cls_model
node_cls_data -->|"feeds"| node_cls_train
node_cls_model -->|"trained by"| node_cls_train
node_cls_train -->|"produces"| node_cls_export
node_flash_backbone -->|"used by"| node_det_models
node_det_cfg -->|"drives"| node_det_runtime
node_det_runtime -->|"runs"| node_det_models
node_det_models -->|"calls"| node_det_ops
node_flash_backbone -->|"used by"| node_seg_models
node_seg_cfg -->|"drives"| node_seg_runtime
node_seg_runtime -->|"runs"| node_seg_models
node_seg_models -->|"calls"| node_seg_ops
node_seg_runtime -.->|"exports to"| node_seg_deploy

click node_dcnv4_op "https://github.com/opengvlab/dcnv4/tree/main/DCNv4_op"
click node_dcnv4_tuning "https://github.com/opengvlab/dcnv4/tree/main/DCNv4_op/scripts"
click node_cls_data "https://github.com/opengvlab/dcnv4/tree/main/classification/dataset"
click node_cls_model "https://github.com/opengvlab/dcnv4/tree/main/classification/models"
click node_cls_train "https://github.com/opengvlab/dcnv4/blob/main/classification/main.py"
click node_cls_export "https://github.com/opengvlab/dcnv4/blob/main/classification/export.py"
click node_det_runtime "https://github.com/opengvlab/dcnv4/blob/main/detection/train.py"
click node_det_models "https://github.com/opengvlab/dcnv4/tree/main/detection/mmdet_custom/models"
click node_det_cfg "https://github.com/opengvlab/dcnv4/tree/main/detection/configs"
click node_det_ops "https://github.com/opengvlab/dcnv4/tree/main/detection/ops_dcnv3"
click node_seg_runtime "https://github.com/opengvlab/dcnv4/blob/main/segmentation/train.py"
click node_seg_models "https://github.com/opengvlab/dcnv4/tree/main/segmentation/mmseg_custom/models"
click node_seg_cfg "https://github.com/opengvlab/dcnv4/tree/main/segmentation/configs"
click node_seg_deploy "https://github.com/opengvlab/dcnv4/tree/main/segmentation/deploy/configs"
click node_seg_ops "https://github.com/opengvlab/dcnv4/tree/main/segmentation/ops_dcnv3"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dcnv4_op,node_dcnv4_tuning,node_task_dcnv3_ops toneBlue
class node_cls_data,node_cls_model,node_cls_train,node_cls_export toneAmber
class node_det_runtime,node_det_models,node_det_cfg,node_det_ops toneMint
class node_seg_runtime,node_seg_models,node_seg_cfg,node_seg_deploy,node_seg_ops toneRose
class node_flash_backbone toneNeutral
```

## Components

### Native Ops

Low-level CUDA/C++ operator packages and their tuning scripts.

- [DCNv4 Ops](https://github.com/opengvlab/dcnv4/tree/main/DCNv4_op): native extension package · PyTorch extension package for DCNv4 and flash deformable attention, including Python wrappers and CUDA/C++ kernels. · `DCNv4_op`
- [Kernel Search](https://github.com/opengvlab/dcnv4/tree/main/DCNv4_op/scripts): benchmark/tuning · Search and validation scripts for forward/backward kernel selection and operator tests. · `DCNv4_op/scripts`
- **Task DCNv3**: vendored extension · Per-task deformable-conv extension packages built independently inside each application tree.

### Classification

Image classification training and export stack.

- [Data Stack](https://github.com/opengvlab/dcnv4/tree/main/classification/dataset): dataset/input · Dataset builders, cached readers, samplers, and transforms for ImageNet-style training. · `classification/dataset`
- [Cls Model](https://github.com/opengvlab/dcnv4/tree/main/classification/models): model definitions · Classification models centered on flash_intern_image and related backbone variants. · `classification/models`
- [Train Runtime](https://github.com/opengvlab/dcnv4/blob/main/classification/main.py): training entrypoints · Main training scripts plus optimizer, scheduler, EMA, and distributed hooks. · `classification/main.py`
- [Export/Features](https://github.com/opengvlab/dcnv4/blob/main/classification/export.py): inference utilities · Model export and feature extraction paths for downstream use. · `classification/export.py`

### Detection

MMDetection-style object detection stack.

- [Det Runtime](https://github.com/opengvlab/dcnv4/blob/main/detection/train.py): execution layer · Training, testing, demos, distributed scripts, and experiment configs for detection. · `detection/train.py`
- [Det Models](https://github.com/opengvlab/dcnv4/tree/main/detection/mmdet_custom/models): custom mmdet stack · Backbones, necks, detectors, dense heads, and detection utilities registered into the detection framework. · `detection/mmdet_custom/models`
- [Det Configs](https://github.com/opengvlab/dcnv4/tree/main/detection/configs): experiment configs · COCO and CrowdHuman experiment definitions and base model/runtime templates. · `detection/configs`
- [Det Ops](https://github.com/opengvlab/dcnv4/tree/main/detection/ops_dcnv3): vendored extension · Detection-specific DCNv3 native extension package built alongside the detector stack. · `detection/ops_dcnv3`

### Segmentation

MMSegmentation-style segmentation and deployment stack.

- [Seg Runtime](https://github.com/opengvlab/dcnv4/blob/main/segmentation/train.py): execution layer · Training, testing, demos, distributed scripts, and deployment configs for segmentation. · `segmentation/train.py`
- [Seg Models](https://github.com/opengvlab/dcnv4/tree/main/segmentation/mmseg_custom/models): custom mmseg stack · Backbones, decode heads, segmentors, losses, assigners, and transformer/pixel-decoder utilities. · `segmentation/mmseg_custom/models`
- [Seg Configs](https://github.com/opengvlab/dcnv4/tree/main/segmentation/configs): experiment configs · ADE20K and other segmentation benchmark definitions plus base dataset/model schedules. · `segmentation/configs`
- [Deploy Graph](https://github.com/opengvlab/dcnv4/tree/main/segmentation/deploy/configs): onnx/tensorrt configs · Deployment boundary for exporting segmentation models to ONNX and TensorRT. · `segmentation/deploy/configs`
- [Seg Ops](https://github.com/opengvlab/dcnv4/tree/main/segmentation/ops_dcnv3): vendored extension · Segmentation-specific DCNv3 native extension package built independently of other tasks. · `segmentation/ops_dcnv3`

### Other components

- **FlashInternImage**: shared backbone · Operator-backed feature extractor reused across classification, detection, and segmentation.

## Connections

- DCNv4 Ops → FlashInternImage: kernels for · Shared deformable operators underpin the FlashInternImage backbone.
- Kernel Search → DCNv4 Ops: validates · Benchmarks and search scripts validate operator behavior and performance.
- Task DCNv3 → FlashInternImage: accelerates · Per-task DCNv3 extensions provide the deformable convolution backend used by application models.
- FlashInternImage → Cls Model: used by · The backbone is wrapped into the classification model implementations.
- Data Stack → Train Runtime: feeds · Datasets and samplers feed the training entrypoints.
- Cls Model → Train Runtime: trained by · Model definitions are optimized through the training runtime.
- Train Runtime → Export/Features: produces · Training stack connects to export and feature extraction flows.
- FlashInternImage → Det Models: used by · The shared backbone is registered inside the detection model stack.
- Det Configs → Det Runtime: drives · Configs define the detector family, dataset, and schedule used by runtime scripts.
- Det Runtime → Det Models: runs · Execution layer instantiates the custom detection model components.
- Det Models → Det Ops: calls · Detection models rely on the vendored DCNv3 extension package.
- FlashInternImage → Seg Models: used by · The shared backbone is registered inside the segmentation model stack.
- Seg Configs → Seg Runtime: drives · Configs define the segmentation benchmark, recipe, and runtime settings.
- Seg Runtime → Seg Models: runs · Execution layer instantiates the custom segmentation model components.
- Seg Models → Seg Ops: calls · Segmentation models rely on the vendored DCNv3 extension package.
- Seg Runtime → Deploy Graph: exports to · Training-time models can be exported through the deployment configuration boundary.
