How intellindust-ai-lab/deimv2 works
1. Repository purpose
- This is a PyTorch-based object detection framework centered on DEIMv2, which combines DEIM-style detection with DINOv3-derived backbones and several lightweight backbone variants.
- The repo is organized around training, evaluation, deployment export, and runtime inference for detection models.
2. Top-level configuration system
- The configs/ tree is the main orchestration layer.
- configs/base/ defines reusable building blocks such as dataloader, optimizer, and model base settings.
- configs/dataset/ contains dataset-specific wiring for COCO, VOC, CrowdHuman, Object365, and custom detection data.
- Task-specific experiment folders like configs/deimv2/, configs/deim_dfine/, and configs/deim_rtdetrv2/ compose a full experiment from these base pieces.
3. Core runtime and configuration loading
- engine/core/ handles config parsing and workspace setup.
- yaml_config.py, yaml_utils.py, and _config.py indicate a YAML-driven experiment system rather than hardcoded Python training loops.
- workspace.py is the likely boundary for resolving paths, checkpoints, logging, and run directories.
4. Data ingestion and batching
- engine/data/ contains the full input pipeline: dataloaders, datasets, and transforms.
- Dataset implementations focus on COCO-style detection, with specific support for VOC and COCO evaluation utilities.
- engine/data/transforms/ includes augmentation primitives such as mosaic, resize, and composition containers, showing that image/box preprocessing is part of the data layer rather than the model layer.
5. Dataset boundary and annotation format
- The framework expects detection data in COCO format for most training/evaluation paths.
- There is explicit support for custom datasets via category remapping control and COCO-formatted annotations.
- Evaluation is dataset-aware: COCO and VOC have dedicated evaluators, keeping metric logic separated from training code.
6. Backbone subsystem
- engine/backbone/ is a large, modular family of feature extractors.
- It includes conventional CNN backbones such as ResNet, CSPDarknet, HGNetv2, plus adapter/wrapper support for external models (timm_model.py, torchvision_model.py).
- The most distinctive part is the DINOv3 stack under engine/backbone/dinov3/, which contains transformer blocks, attention, patch embedding, normalization, and head/layer utilities.
- This backbone layer is the main architecture knob across model scales: HGNetv2-based variants for lighter deployments, and DINOv3-based variants for larger, higher-accuracy models.
7. Detection model and heads
- engine/deim/ contains the detection-specific architecture and loss logic.
- Key components include:
- deim.py as the model-level composition,
- hybrid_encoder.py and lite_encoder.py for feature aggregation,
- decoders for multiple families: deim_decoder.py, dfine_decoder.py, rtdetrv2_decoder.py,
- deim_criterion.py, matcher.py, and denoising.py for assignment and training targets.
- This suggests the repo supports a family of closely related DETR-like detection heads rather than a single monolithic detector.
8. Training objective and post-processing boundary
- The detection pipeline is split into:
- model feature extraction and decoding,
- criterion/matching during training,
- post-processing for inference outputs.
- postprocessor.py and box_ops.py handle conversion from raw predictions to usable boxes/scores, separating inference semantics from the core model.
9. Solver and optimization flow
- engine/solver/ defines the training/evaluation engine for both classification and detection, with detection-specific solver/engine files as the main path for this repo.
- engine/optim/ contains AMP, EMA, LR scheduling, warmup, and optimizer composition.
- Optimizer configs in YAML are rich enough to assign different parameter groups by regex, which matters because backbone, encoder, and decoder often train with different rules.
10. Distributed training and execution
- The project is designed to run under torchrun, implying distributed multi-GPU training as a first-class path.
- engine/misc/dist_utils.py and logger/profiler helpers support distributed execution and experiment observability.
- train.py is the main entrypoint for train/test/tune flows, with config and checkpoint arguments driving behavior.
11. Deployment and inference boundary
- tools/deployment/ exports models to ONNX and supports an export path for YOLO-style NMS packaging.
- tools/inference/ provides separate runtime scripts for PyTorch, ONNX Runtime, TensorRT, and OpenVINO, which makes deployment a distinct post-training subsystem.
- The architecture therefore has a clear split between research/training code in engine/ and production-facing inference/export code in tools/.
12. Benchmarking, visualization, and dataset utilities
- tools/benchmark/ measures FLOPs/params and TensorRT latency.
- tools/visualization/ integrates with FiftyOne for qualitative inspection.
- tools/dataset/ includes Object365 preprocessing utilities, showing that some dataset adaptation is treated as offline data engineering rather than model code.
13. Model family structure
- The repo is not a single model implementation; it is a platform for several DEIM variants:
- HGNetv2-based lightweight models,
- DINOv3-based S/M/L/X models,
- DEIM, DFINE, and RT-DETRv2-related configurations and decoders.
- The architectural diagram should show a shared backbone/data/solver substrate feeding multiple model heads and variant-specific configs.
14. External dependencies that matter architecturally
- PyTorch is the core runtime.
- Hugging Face is used for published checkpoints and model loading.
- TensorRT and ONNX Runtime are important inference backends, while OpenVINO appears as an additional deployment target.
- These are not just convenience tools; they define the repo’s training-to-deployment pipeline and supported execution environments.
Main entry points: core Config core · deim DEIM models · data Data pipeline · backbone Backbones
Components 23
Configs
YAML experiment composition for datasets, bases, and model variants.
- runtime.yml · Top-level runtime defaults and execution settings.
configs/runtime.yml - Base configs · Reusable dataloader, optimizer, and model defaults.
configs/base - Dataset configs · COCO, VOC, CrowdHuman, Object365, and custom dataset presets.
configs/dataset - DEIMv2 configs · DEIMv2 experiment families across HGNetv2 and DINOv3 scales.
configs/deimv2 - DFINE configs · DEIM and DFINE experiment family definitions.
configs/deim_dfine - RT-DETRv2 configs · DEIM and RT-DETRv2 experiment family definitions.
configs/deim_rtdetrv2
Engine
Core training, data, models, optimization, and runtime infrastructure.
- train.py · Main training/testing/tuning entrypoint.
train.py - Config core · Parses experiment configs and resolves workspace state.
engine/core - Data pipeline · Dataloaders, dataset adapters, and transform assembly.
engine/data - COCO/VOC data · Detection datasets and evaluation utilities for COCO and VOC.
engine/data/dataset - Transforms · Resize, mosaic, composition, and preprocessing primitives.
engine/data/transforms - Backbones · Shared backbone family for detection models.
engine/backbone - DINOv3 stack · Transformer backbone blocks, attention, patch embedding, and heads.
engine/backbone/dinov3 - DEIM models · Model composition, encoders, decoders, matching, and losses.
engine/deim - Solver · Training and evaluation loops for detection and classification.
engine/solver - Optimization · AMP, EMA, learning-rate scheduling, warmup, and optimizer assembly.
engine/optim - Dist & logs · Distributed utilities, logging, profiling, and visualization helpers.
engine/misc - Postprocess · Box conversion and prediction cleanup for exported or runtime inference.
engine/deim/postprocessor.py
Tools
Export, inference, benchmarking, dataset adaptation, and visualization utilities.
- Export · ONNX and YOLO-style NMS export paths.
tools/deployment - Inference · PyTorch, ONNX Runtime, TensorRT, and OpenVINO inference scripts.
tools/inference - Benchmark · FLOPs, params, and TensorRT latency benchmarks.
tools/benchmark - Data tools · Object365 remapping and resizing utilities.
tools/dataset - Visualization · FiftyOne-based qualitative inspection utilities.
tools/visualization
Connections 24
- train.py → runtime.yml: reads defaults · Entry point loads runtime settings and experiment overrides.
- train.py → Config core: loads config · Training is driven through the YAML config system.
- Config core → Base configs: composes · Core loader resolves reusable base experiment pieces.
- Config core → Dataset configs: binds data · Dataset wiring is injected through configuration composition.
- Config core → DEIMv2 configs: selects variant · Model family presets are chosen through config composition.
- Config core → DFINE configs: selects variant · Alternative detector families are also assembled from configs.
- Config core → RT-DETRv2 configs: selects variant · RT-DETRv2-related experiments are assembled from configs.
- Config core → Solver: starts run · Resolved config drives the training/evaluation engine.
- Config core → Data pipeline: builds input · Config defines dataloaders, datasets, and transforms.
- Data pipeline → COCO/VOC data: loads samples · Dataset layer reads COCO-style annotations and evaluation targets.
- Data pipeline → Transforms: augments · Transforms are applied during preprocessing and batching.
- Config core → Backbones: chooses backbone · Model configs determine which backbone family is instantiated.
- Backbones → DINOv3 stack: specializes · DINOv3 is the main transformer-based backbone branch.
- Backbones → DEIM models: feeds features · Backbone outputs are consumed by detector encoders and heads.
- DEIM models → Postprocess: outputs boxes · Raw predictions are converted for inference consumption.
- DEIM models → Optimization: trains with · Detector training uses optimizer, AMP, EMA, and scheduling utilities.
- Solver → Dist & logs: uses runtime support · Distributed execution, logging, and profiling support the solver.
- Solver → Optimization: updates weights · Solver coordinates optimization and learning-rate control.
- Export → DEIM models: exports · Export utilities serialize detector graphs for deployment.
- Inference → Postprocess: runs postprocess · Inference backends consume postprocessed detection outputs.
- Inference → Export: consumes artifacts · Inference scripts operate on exported ONNX or deployment packages.
- Benchmark → DEIM models: profiles · Benchmarking measures model cost and latency.
- Data tools → COCO/VOC data: prepares · Offline utilities adapt dataset annotations and image sizes.
- Visualization → COCO/VOC data: inspects · Visualization tools inspect dataset samples and predictions.