How opengvlab/dcnv4 works
1. This repository is a multi-task vision model stack centered on the FlashInternImage backbone and its custom deformable operators. It is split into one low-level operator package plus three downstream application trees: classification, detection, and segmentation.
2. The lowest layer is DCNv4_op/, which implements the DCNv4 and flash deformable attention operators as PyTorch native extensions. The Python side lives in DCNv4_op/DCNv4/functions and modules, while the CUDA/C++ kernels live under DCNv4_op/src. setup.py and make.sh define the build boundary for compiling those ops into importable modules.
3. DCNv4_op/scripts/ is a performance-tuning and validation layer. It contains search utilities for forward/backward kernel selection and small test programs, so the repo is not just model code but also operator benchmarking and kernel tuning infrastructure.
4. Each task subproject vendors its own compiled deformable-conv extension under ops_dcnv3/. These packages mirror the same pattern: thin Python wrappers, a setup.py build, and CPU/CUDA sources. Architecturally, this isolates native code from the higher-level task code and lets each application build the operator it needs independently.
5. classification/ is a self-contained image classification training stack. Data enters through dataset builders, cached image-folder readers, samplers, and transform utilities; flows into model definitions in classification/models/; then through optimizers, LR schedulers, EMA/DeepSpeed hooks, and training entry points such as main.py, main_deepspeed.py, and main_accelerate.py. It also includes export and feature-extraction paths.
6. In classification, flash_intern_image.py is the key model implementation, with configs in YAML files describing model size, training recipe, and resolution. This subtree is designed around standard PyTorch training but with task-specific glue for distributed training and ImageNet-style datasets.
7. detection/ is built as a customized MMDetection-style stack. The mmdet_custom/ tree provides registered backbones, detectors, dense heads, necks, and utility modules; mmcv_custom/ provides optimizer construction and layer-decay logic. Configs under detection/configs/ define COCO and CrowdHuman experiments, while train.py, test.py, distributed shell scripts, and demos form the execution layer.
8. The detection data flow is: dataset/config → custom backbone (flash_intern_image) → FPN/neck → task head or detector (Mask R-CNN, Cascade, DINO, etc.) → losses and evaluation. The repo keeps detector-specific logic separate from the backbone so the same FlashInternImage can be reused across multiple detector families.
9. segmentation/ mirrors the detection structure but for MMSegmentation-like workloads. It has mmseg_custom/ for custom backbones, decode heads, segmentors, losses, assigners, and dataset/pipeline logic; mmcv_custom/ for optimizer/layer-decay support; and configs/ for ADE20K and other segmentation benchmarks.
10. The segmentation runtime path is dataset/pipeline → FlashInternImage backbone → decoder head or Mask2Former/UperNet-style segmentor → segmentation losses and evaluation utilities. This subtree also includes deployment configs for ONNX and TensorRT, which is an important architectural boundary: training-time graph vs export/deployment graph.
11. Across detection and segmentation, the custom backbone is the same architectural bridge: flash_intern_image.py adapts the operator-backed feature extractor into each framework’s registry and feature-map conventions. The rest of the task-specific model code mainly handles necks, heads, assigners, losses, and dataset-specific preprocessing.
12. The repo also contains task-specific orchestration scripts for distributed execution: train.sh, dist_train.sh, dist_test.sh, and slurm_* variants. These are the operational boundary between the model code and cluster/runtime environments such as multi-GPU training, DeepSpeed, or Slurm-managed jobs.
13. Overall, the main diagram should show three layers: native operator kernels at the bottom, shared FlashInternImage/backbone code in the middle, and three independent application stacks on top. The critical cross-cutting concern is that all three tasks depend on custom CUDA extensions for deformable convolution, which is what gives the repo its performance-oriented character.
Main entry points: train.py Seg Runtime · main.py Train Runtime · mmdet_custom/models Det Models · mmseg_custom/models Seg Models
Components 17
Native Ops
Low-level CUDA/C++ operator packages and their tuning scripts.
- DCNv4 Ops · PyTorch extension package for DCNv4 and flash deformable attention, including Python wrappers and CUDA/C++ kernels.
DCNv4_op - Kernel Search · Search and validation scripts for forward/backward kernel selection and operator tests.
DCNv4_op/scripts - Task DCNv3 · Per-task deformable-conv extension packages built independently inside each application tree.
Classification
Image classification training and export stack.
- Data Stack · Dataset builders, cached readers, samplers, and transforms for ImageNet-style training.
classification/dataset - Cls Model · Classification models centered on flash_intern_image and related backbone variants.
classification/models - Train Runtime · Main training scripts plus optimizer, scheduler, EMA, and distributed hooks.
classification/main.py - Export/Features · Model export and feature extraction paths for downstream use.
classification/export.py
Detection
MMDetection-style object detection stack.
- Det Runtime · Training, testing, demos, distributed scripts, and experiment configs for detection.
detection/train.py - Det Models · Backbones, necks, detectors, dense heads, and detection utilities registered into the detection framework.
detection/mmdet_custom/models - Det Configs · COCO and CrowdHuman experiment definitions and base model/runtime templates.
detection/configs - Det Ops · Detection-specific DCNv3 native extension package built alongside the detector stack.
detection/ops_dcnv3
Segmentation
MMSegmentation-style segmentation and deployment stack.
- Seg Runtime · Training, testing, demos, distributed scripts, and deployment configs for segmentation.
segmentation/train.py - Seg Models · Backbones, decode heads, segmentors, losses, assigners, and transformer/pixel-decoder utilities.
segmentation/mmseg_custom/models - Seg Configs · ADE20K and other segmentation benchmark definitions plus base dataset/model schedules.
segmentation/configs - Deploy Graph · Deployment boundary for exporting segmentation models to ONNX and TensorRT.
segmentation/deploy/configs - Seg Ops · Segmentation-specific DCNv3 native extension package built independently of other tasks.
segmentation/ops_dcnv3
Other components
- FlashInternImage · Operator-backed feature extractor reused across classification, detection, and segmentation.
Connections 16
- DCNv4 Ops → FlashInternImage: kernels for · Shared deformable operators underpin the FlashInternImage backbone.
- Kernel Search → DCNv4 Ops: validates · Benchmarks and search scripts validate operator behavior and performance.
- Task DCNv3 → FlashInternImage: accelerates · Per-task DCNv3 extensions provide the deformable convolution backend used by application models.
- FlashInternImage → Cls Model: used by · The backbone is wrapped into the classification model implementations.
- Data Stack → Train Runtime: feeds · Datasets and samplers feed the training entrypoints.
- Cls Model → Train Runtime: trained by · Model definitions are optimized through the training runtime.
- Train Runtime → Export/Features: produces · Training stack connects to export and feature extraction flows.
- FlashInternImage → Det Models: used by · The shared backbone is registered inside the detection model stack.
- Det Configs → Det Runtime: drives · Configs define the detector family, dataset, and schedule used by runtime scripts.
- Det Runtime → Det Models: runs · Execution layer instantiates the custom detection model components.
- Det Models → Det Ops: calls · Detection models rely on the vendored DCNv3 extension package.
- FlashInternImage → Seg Models: used by · The shared backbone is registered inside the segmentation model stack.
- Seg Configs → Seg Runtime: drives · Configs define the segmentation benchmark, recipe, and runtime settings.
- Seg Runtime → Seg Models: runs · Execution layer instantiates the custom segmentation model components.
- Seg Models → Seg Ops: calls · Segmentation models rely on the vendored DCNv3 extension package.
- Seg Runtime → Deploy Graph: exports to · Training-time models can be exported through the deployment configuration boundary.