# intellindust-ai-lab/deimv2 architecture

> Architecture diagram of the GitHub repository intellindust-ai-lab/deimv2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/intellindust-ai-lab/deimv2
- Repository: https://github.com/intellindust-ai-lab/deimv2

## Overview

1. **Repository purpose**
   - This is a PyTorch-based object detection framework centered on **DEIMv2**, which combines DEIM-style detection with **DINOv3-derived backbones** and several lightweight backbone variants.
   - The repo is organized around training, evaluation, deployment export, and runtime inference for detection models.

2. **Top-level configuration system**
   - The `configs/` tree is the main orchestration layer.
   - `configs/base/` defines reusable building blocks such as dataloader, optimizer, and model base settings.
   - `configs/dataset/` contains dataset-specific wiring for COCO, VOC, CrowdHuman, Object365, and custom detection data.
   - Task-specific experiment folders like `configs/deimv2/`, `configs/deim_dfine/`, and `configs/deim_rtdetrv2/` compose a full experiment from these base pieces.

3. **Core runtime and configuration loading**
   - `engine/core/` handles config parsing and workspace setup.
   - `yaml_config.py`, `yaml_utils.py`, and `_config.py` indicate a YAML-driven experiment system rather than hardcoded Python training loops.
   - `workspace.py` is the likely boundary for resolving paths, checkpoints, logging, and run directories.

4. **Data ingestion and batching**
   - `engine/data/` contains the full input pipeline: dataloaders, datasets, and transforms.
   - Dataset implementations focus on **COCO-style detection**, with specific support for VOC and COCO evaluation utilities.
   - `engine/data/transforms/` includes augmentation primitives such as mosaic, resize, and composition containers, showing that image/box preprocessing is part of the data layer rather than the model layer.

5. **Dataset boundary and annotation format**
   - The framework expects detection data in **COCO format** for most training/evaluation paths.
   - There is explicit support for custom datasets via category remapping control and COCO-formatted annotations.
   - Evaluation is dataset-aware: COCO and VOC have dedicated evaluators, keeping metric logic separated from training code.

6. **Backbone subsystem**
   - `engine/backbone/` is a large, modular family of feature extractors.
   - It includes conventional CNN backbones such as **ResNet, CSPDarknet, HGNetv2**, plus adapter/wrapper support for external models (`timm_model.py`, `torchvision_model.py`).
   - The most distinctive part is the **DINOv3 stack** under `engine/backbone/dinov3/`, which contains transformer blocks, attention, patch embedding, normalization, and head/layer utilities.
   - This backbone layer is the main architecture knob across model scales: HGNetv2-based variants for lighter deployments, and DINOv3-based variants for larger, higher-accuracy models.

7. **Detection model and heads**
   - `engine/deim/` contains the detection-specific architecture and loss logic.
   - Key components include:
     - `deim.py` as the model-level composition,
     - `hybrid_encoder.py` and `lite_encoder.py` for feature aggregation,
     - decoders for multiple families: `deim_decoder.py`, `dfine_decoder.py`, `rtdetrv2_decoder.py`,
     - `deim_criterion.py`, `matcher.py`, and `denoising.py` for assignment and training targets.
   - This suggests the repo supports a family of closely related DETR-like detection heads rather than a single monolithic detector.

8. **Training objective and post-processing boundary**
   - The detection pipeline is split into:
     - model feature extraction and decoding,
     - criterion/matching during training,
     - post-processing for inference outputs.
   - `postprocessor.py` and `box_ops.py` handle conversion from raw predictions to usable boxes/scores, separating inference semantics from the core model.

9. **Solver and optimization flow**
   - `engine/solver/` defines the training/evaluation engine for both classification and detection, with detection-specific solver/engine files as the main path for this repo.
   - `engine/optim/` contains AMP, EMA, LR scheduling, warmup, and optimizer composition.
   - Optimizer configs in YAML are rich enough to assign different parameter groups by regex, which matters because backbone, encoder, and decoder often train with different rules.

10. **Distributed training and execution**
   - The project is designed to run under `torchrun`, implying distributed multi-GPU training as a first-class path.
   - `engine/misc/dist_utils.py` and logger/profiler helpers support distributed execution and experiment observability.
   - `train.py` is the main entrypoint for train/test/tune flows, with config and checkpoint arguments driving behavior.

11. **Deployment and inference boundary**
   - `tools/deployment/` exports models to **ONNX** and supports an export path for YOLO-style NMS packaging.
   - `tools/inference/` provides separate runtime scripts for **PyTorch**, **ONNX Runtime**, **TensorRT**, and **OpenVINO**, which makes deployment a distinct post-training subsystem.
   - The architecture therefore has a clear split between research/training code in `engine/` and production-facing inference/export code in `tools/`.

12. **Benchmarking, visualization, and dataset utilities**
   - `tools/benchmark/` measures FLOPs/params and TensorRT latency.
   - `tools/visualization/` integrates with FiftyOne for qualitative inspection.
   - `tools/dataset/` includes Object365 preprocessing utilities, showing that some dataset adaptation is treated as offline data engineering rather than model code.

13. **Model family structure**
   - The repo is not a single model implementation; it is a **platform for several DEIM variants**:
     - HGNetv2-based lightweight models,
     - DINOv3-based S/M/L/X models,
     - DEIM, DFINE, and RT-DETRv2-related configurations and decoders.
   - The architectural diagram should show a shared backbone/data/solver substrate feeding multiple model heads and variant-specific configs.

14. **External dependencies that matter architecturally**
   - PyTorch is the core runtime.
   - Hugging Face is used for published checkpoints and model loading.
   - TensorRT and ONNX Runtime are important inference backends, while OpenVINO appears as an additional deployment target.
   - These are not just convenience tools; they define the repo’s training-to-deployment pipeline and supported execution environments.

Main entry points: [`engine/core`](https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/core) (Config core), [`engine/deim`](https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/deim) (DEIM models), [`engine/data`](https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/data) (Data pipeline), [`engine/backbone`](https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/backbone) (Backbones)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_configs["Configs"]
  node_runtime_yml["runtime.yml<br/>runtime config<br/>[runtime.yml]"]
  node_config_base["Base configs<br/>experiment base"]
  node_config_dataset["Dataset configs<br/>dataset wiring"]
  node_config_deimv2["DEIMv2 configs<br/>model variants"]
  node_config_dfine["DFINE configs<br/>model variants"]
  node_config_rtdetrv2["RT-DETRv2 configs<br/>model variants"]
end

subgraph group_grp_engine["Engine"]
  node_train_py["train.py<br/>entrypoint<br/>[train.py]"]
  node_config_core["Config core<br/>yaml loader"]
  node_data_pipeline["Data pipeline<br/>input pipeline"]
  node_datasets[("COCO/VOC data<br/>dataset layer")]
  node_transforms["Transforms<br/>augmentation"]
  node_backbone_core["Backbones<br/>feature extractors"]
  node_dinov3_stack["DINOv3 stack<br/>vision transformer"]
  node_deim_models["DEIM models<br/>detector family"]
  node_solver["Solver<br/>training engine"]
  node_optim["Optimization<br/>optim stack"]
  node_dist_misc["Dist & logs<br/>runtime support"]
  node_postproc["Postprocess<br/>inference utils<br/>[postprocessor.py]"]
end

subgraph group_grp_tools["Tools"]
  node_tools_export["Export<br/>deployment"]
  node_tools_infer["Inference<br/>runtime backends"]
  node_tools_bench["Benchmark<br/>measurement"]
  node_tools_data["Data tools<br/>dataset prep"]
  node_tools_viz["Visualization<br/>inspection"]
end

node_train_py -->|"reads defaults"| node_runtime_yml
node_train_py -->|"loads config"| node_config_core
node_config_core -->|"composes"| node_config_base
node_config_core -->|"binds data"| node_config_dataset
node_config_core -->|"selects variant"| node_config_deimv2
node_config_core -->|"selects variant"| node_config_dfine
node_config_core -->|"selects variant"| node_config_rtdetrv2
node_config_core -->|"starts run"| node_solver
node_config_core -->|"builds input"| node_data_pipeline
node_data_pipeline -->|"loads samples"| node_datasets
node_data_pipeline -->|"augments"| node_transforms
node_config_core -->|"chooses backbone"| node_backbone_core
node_backbone_core -->|"specializes"| node_dinov3_stack
node_backbone_core -->|"feeds features"| node_deim_models
node_deim_models -->|"outputs boxes"| node_postproc
node_deim_models -->|"trains with"| node_optim
node_solver -->|"uses runtime support"| node_dist_misc
node_solver -->|"updates weights"| node_optim
node_tools_export -->|"exports"| node_deim_models
node_tools_infer -->|"runs postprocess"| node_postproc
node_tools_infer -->|"consumes artifacts"| node_tools_export
node_tools_bench -->|"profiles"| node_deim_models
node_tools_data -->|"prepares"| node_datasets
node_tools_viz -->|"inspects"| node_datasets

click node_train_py "https://github.com/intellindust-ai-lab/deimv2/blob/main/train.py"
click node_runtime_yml "https://github.com/intellindust-ai-lab/deimv2/blob/main/configs/runtime.yml"
click node_config_core "https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/core"
click node_config_base "https://github.com/intellindust-ai-lab/deimv2/tree/main/configs/base"
click node_config_dataset "https://github.com/intellindust-ai-lab/deimv2/tree/main/configs/dataset"
click node_config_deimv2 "https://github.com/intellindust-ai-lab/deimv2/tree/main/configs/deimv2"
click node_config_dfine "https://github.com/intellindust-ai-lab/deimv2/tree/main/configs/deim_dfine"
click node_config_rtdetrv2 "https://github.com/intellindust-ai-lab/deimv2/tree/main/configs/deim_rtdetrv2"
click node_data_pipeline "https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/data"
click node_datasets "https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/data/dataset"
click node_transforms "https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/data/transforms"
click node_backbone_core "https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/backbone"
click node_dinov3_stack "https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/backbone/dinov3"
click node_deim_models "https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/deim"
click node_solver "https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/solver"
click node_optim "https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/optim"
click node_dist_misc "https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/misc"
click node_postproc "https://github.com/intellindust-ai-lab/deimv2/blob/main/engine/deim/postprocessor.py"
click node_tools_export "https://github.com/intellindust-ai-lab/deimv2/tree/main/tools/deployment"
click node_tools_infer "https://github.com/intellindust-ai-lab/deimv2/tree/main/tools/inference"
click node_tools_bench "https://github.com/intellindust-ai-lab/deimv2/tree/main/tools/benchmark"
click node_tools_data "https://github.com/intellindust-ai-lab/deimv2/tree/main/tools/dataset"
click node_tools_viz "https://github.com/intellindust-ai-lab/deimv2/tree/main/tools/visualization"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runtime_yml,node_config_base,node_config_dataset,node_config_deimv2,node_config_dfine,node_config_rtdetrv2 toneBlue
class node_train_py,node_config_core,node_data_pipeline,node_datasets,node_transforms,node_backbone_core,node_dinov3_stack,node_deim_models,node_solver,node_optim,node_dist_misc,node_postproc toneAmber
class node_tools_export,node_tools_infer,node_tools_bench,node_tools_data,node_tools_viz toneMint
```

## Components

### Configs

YAML experiment composition for datasets, bases, and model variants.

- [runtime.yml](https://github.com/intellindust-ai-lab/deimv2/blob/main/configs/runtime.yml): runtime config · Top-level runtime defaults and execution settings. · `configs/runtime.yml`
- [Base configs](https://github.com/intellindust-ai-lab/deimv2/tree/main/configs/base): experiment base · Reusable dataloader, optimizer, and model defaults. · `configs/base`
- [Dataset configs](https://github.com/intellindust-ai-lab/deimv2/tree/main/configs/dataset): dataset wiring · COCO, VOC, CrowdHuman, Object365, and custom dataset presets. · `configs/dataset`
- [DEIMv2 configs](https://github.com/intellindust-ai-lab/deimv2/tree/main/configs/deimv2): model variants · DEIMv2 experiment families across HGNetv2 and DINOv3 scales. · `configs/deimv2`
- [DFINE configs](https://github.com/intellindust-ai-lab/deimv2/tree/main/configs/deim_dfine): model variants · DEIM and DFINE experiment family definitions. · `configs/deim_dfine`
- [RT-DETRv2 configs](https://github.com/intellindust-ai-lab/deimv2/tree/main/configs/deim_rtdetrv2): model variants · DEIM and RT-DETRv2 experiment family definitions. · `configs/deim_rtdetrv2`

### Engine

Core training, data, models, optimization, and runtime infrastructure.

- [train.py](https://github.com/intellindust-ai-lab/deimv2/blob/main/train.py): entrypoint · Main training/testing/tuning entrypoint. · `train.py`
- [Config core](https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/core): yaml loader · Parses experiment configs and resolves workspace state. · `engine/core`
- [Data pipeline](https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/data): input pipeline · Dataloaders, dataset adapters, and transform assembly. · `engine/data`
- [COCO/VOC data](https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/data/dataset): dataset layer · Detection datasets and evaluation utilities for COCO and VOC. · `engine/data/dataset`
- [Transforms](https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/data/transforms): augmentation · Resize, mosaic, composition, and preprocessing primitives. · `engine/data/transforms`
- [Backbones](https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/backbone): feature extractors · Shared backbone family for detection models. · `engine/backbone`
- [DINOv3 stack](https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/backbone/dinov3): vision transformer · Transformer backbone blocks, attention, patch embedding, and heads. · `engine/backbone/dinov3`
- [DEIM models](https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/deim): detector family · Model composition, encoders, decoders, matching, and losses. · `engine/deim`
- [Solver](https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/solver): training engine · Training and evaluation loops for detection and classification. · `engine/solver`
- [Optimization](https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/optim): optim stack · AMP, EMA, learning-rate scheduling, warmup, and optimizer assembly. · `engine/optim`
- [Dist & logs](https://github.com/intellindust-ai-lab/deimv2/tree/main/engine/misc): runtime support · Distributed utilities, logging, profiling, and visualization helpers. · `engine/misc`
- [Postprocess](https://github.com/intellindust-ai-lab/deimv2/blob/main/engine/deim/postprocessor.py): inference utils · Box conversion and prediction cleanup for exported or runtime inference. · `engine/deim/postprocessor.py`

### Tools

Export, inference, benchmarking, dataset adaptation, and visualization utilities.

- [Export](https://github.com/intellindust-ai-lab/deimv2/tree/main/tools/deployment): deployment · ONNX and YOLO-style NMS export paths. · `tools/deployment`
- [Inference](https://github.com/intellindust-ai-lab/deimv2/tree/main/tools/inference): runtime backends · PyTorch, ONNX Runtime, TensorRT, and OpenVINO inference scripts. · `tools/inference`
- [Benchmark](https://github.com/intellindust-ai-lab/deimv2/tree/main/tools/benchmark): measurement · FLOPs, params, and TensorRT latency benchmarks. · `tools/benchmark`
- [Data tools](https://github.com/intellindust-ai-lab/deimv2/tree/main/tools/dataset): dataset prep · Object365 remapping and resizing utilities. · `tools/dataset`
- [Visualization](https://github.com/intellindust-ai-lab/deimv2/tree/main/tools/visualization): inspection · FiftyOne-based qualitative inspection utilities. · `tools/visualization`

## Connections

- train.py → runtime.yml: reads defaults · Entry point loads runtime settings and experiment overrides.
- train.py → Config core: loads config · Training is driven through the YAML config system.
- Config core → Base configs: composes · Core loader resolves reusable base experiment pieces.
- Config core → Dataset configs: binds data · Dataset wiring is injected through configuration composition.
- Config core → DEIMv2 configs: selects variant · Model family presets are chosen through config composition.
- Config core → DFINE configs: selects variant · Alternative detector families are also assembled from configs.
- Config core → RT-DETRv2 configs: selects variant · RT-DETRv2-related experiments are assembled from configs.
- Config core → Solver: starts run · Resolved config drives the training/evaluation engine.
- Config core → Data pipeline: builds input · Config defines dataloaders, datasets, and transforms.
- Data pipeline → COCO/VOC data: loads samples · Dataset layer reads COCO-style annotations and evaluation targets.
- Data pipeline → Transforms: augments · Transforms are applied during preprocessing and batching.
- Config core → Backbones: chooses backbone · Model configs determine which backbone family is instantiated.
- Backbones → DINOv3 stack: specializes · DINOv3 is the main transformer-based backbone branch.
- Backbones → DEIM models: feeds features · Backbone outputs are consumed by detector encoders and heads.
- DEIM models → Postprocess: outputs boxes · Raw predictions are converted for inference consumption.
- DEIM models → Optimization: trains with · Detector training uses optimizer, AMP, EMA, and scheduling utilities.
- Solver → Dist & logs: uses runtime support · Distributed execution, logging, and profiling support the solver.
- Solver → Optimization: updates weights · Solver coordinates optimization and learning-rate control.
- Export → DEIM models: exports · Export utilities serialize detector graphs for deployment.
- Inference → Postprocess: runs postprocess · Inference backends consume postprocessed detection outputs.
- Inference → Export: consumes artifacts · Inference scripts operate on exported ONNX or deployment packages.
- Benchmark → DEIM models: profiles · Benchmarking measures model cost and latency.
- Data tools → COCO/VOC data: prepares · Offline utilities adapt dataset annotations and image sizes.
- Visualization → COCO/VOC data: inspects · Visualization tools inspect dataset samples and predictions.