# xiuqhou/salience-detr architecture

> Architecture diagram of the GitHub repository xiuqhou/salience-detr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xiuqhou/salience-detr
- Repository: https://github.com/xiuqhou/salience-detr

## Overview

##### Project Type & Purpose
- Type: Open-source research code / PyTorch model library for object detection.
- Purpose: Implements “Salience DETR,” a Detection Transformer enhanced with hierarchical salience filtering. Provides training, evaluation, inference, ONNX export, and data‐visualization tooling.

##### High-Level Architecture
- Data Ingestion Layer
– COCO (or custom) dataset files under `data/` loaded via `datasets/coco.py`.
– Preprocessing & augmentation modules under `transforms/` (and `transforms/v2/`).
- Model Definition Layer
– Backbone modules (`models/backbones/*`): ResNet, ConvNeXt, Swin, FocalNet, ViT.
– Transformer “bricks” (`models/bricks/*`): multi‐scale deformable attention, salience heads, position encodings, loss computations.
– Detector wrapper (`models/detectors/salience_detr.py`) composes backbone + transformer + post‐processing.
– Matcher (`models/matcher/hungarian_matcher.py`) implements bipartite matching for loss assignment.
- Configuration & Orchestration
– Model configs (`configs/salience_detr/*.py`) declare architecture hyperparameters (backbone choice, input size, num queries, num classes).
– Training config (`configs/train_config.py`) declares optimizer, scheduler, dataset paths, mixed‐GPU settings using `accelerate`.
– Entry points: `main.py` orchestrates training; `test.py` runs evaluation; `inference.py` handles image‐folder inference; `tools/*` provide benchmarking, ONNX export, dataset visualization.
- Utilities & Support
– `util/engine.py`: training loop, evaluation loop, checkpointing, logging.
– `util/*.py`: distributed setup, metrics (COCO eval), data collate, file I/O, environment reporting, visualization helpers.
– CUDA kernels under `models/bricks/ops/cuda` auto‐built on first run.

##### Relationships & Data Flow
1. COCO JSON + images → `datasets.CocoDetection` → PyTorch DataLoader (with `util.collate_fn`).
2. Each batch → `transforms` pipeline → tensor inputs.
3. Inputs → backbone → multiscale feature maps.
4. Features + learnable queries → transformer encoder/decoder (salience filtering layers) → raw predictions.
6. Losses → backprop → `torch.optim.AdamW` (param groups from `optimizer/param_dict.py`) → update model.
7. Periodic eval (`util.engine.evaluate`) → COCO metrics → logs & checkpoints.

##### Architectural Patterns & Principles
- Modular plugin pattern: Backbones, heads, and matchers implement base classes—easy to add new modules.
- Config‐driven design: Behavior controlled entirely by `.py` config files—no hard‐coded hyperparameters.
- Separation of concerns: Data, model, training loop, utils, and tools reside in distinct directories.
- Lazy CUDA op compilation: abstracts away manual build steps.
- Use of `accelerate` for multi‐GPU scaling with minimal boilerplate.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        D1["COCO Loader\ndatasets/coco.py"]:::data
        click D1 "https://github.com/xiuqhou/salience-detr/blob/main/datasets/coco.py"
        D2["Transforms\ntransforms/"]:::data
        click D2 "https://github.com/xiuqhou/salience-detr/tree/main/transforms/"
        D3["Transforms v2\ntransforms/v2/"]:::data
        click D3 "https://github.com/xiuqhou/salience-detr/tree/main/transforms/v2/"
    end

    %% Model Definition Layer
    subgraph "Model Definition Layer"
        subgraph "Backbones"
            B1["ResNet\nmodels/backbones/resnet.py"]:::model
            click B1 "https://github.com/xiuqhou/salience-detr/blob/main/models/backbones/resnet.py"
            B2["ConvNeXt\nmodels/backbones/convnext.py"]:::model
            click B2 "https://github.com/xiuqhou/salience-detr/blob/main/models/backbones/convnext.py"
            B3["FocalNet\nmodels/backbones/focalnet.py"]:::model
            click B3 "https://github.com/xiuqhou/salience-detr/blob/main/models/backbones/focalnet.py"
            B4["Swin\nmodels/backbones/swin.py"]:::model
            click B4 "https://github.com/xiuqhou/salience-detr/blob/main/models/backbones/swin.py"
            B5["ViT\nmodels/backbones/vit.py"]:::model
            click B5 "https://github.com/xiuqhou/salience-detr/blob/main/models/backbones/vit.py"
        end
        subgraph "Transformer Bricks"
            T0["Base Transformer\nmodels/bricks/base_transformer.py"]:::model
            click T0 "https://github.com/xiuqhou/salience-detr/blob/main/models/bricks/base_transformer.py"
            T1["Basic Modules\nmodels/bricks/basic.py"]:::model
            click T1 "https://github.com/xiuqhou/salience-detr/blob/main/models/bricks/basic.py"
            T2["MS Deformable Attention\nmodels/bricks/ms_deform_attn.py"]:::model
            click T2 "https://github.com/xiuqhou/salience-detr/blob/main/models/bricks/ms_deform_attn.py"
            T3["Salience Filtering\nmodels/bricks/salience_transformer.py"]:::model
            click T3 "https://github.com/xiuqhou/salience-detr/blob/main/models/bricks/salience_transformer.py"
            T4["Position Encodings\nmodels/bricks/position_encoding.py"]:::model
            click T4 "https://github.com/xiuqhou/salience-detr/blob/main/models/bricks/position_encoding.py"
            T5["Loss & Criterion\nmodels/bricks/losses.py\nmodels/bricks/set_criterion.py"]:::model
            click T5 "https://github.com/xiuqhou/salience-detr/blob/main/models/bricks/losses.py"
            click T5 "https://github.com/xiuqhou/salience-detr/blob/main/models/bricks/set_criterion.py"
            T6["Post-Processing\nmodels/bricks/post_process.py"]:::model
            click T6 "https://github.com/xiuqhou/salience-detr/blob/main/models/bricks/post_process.py"
            T7["CUDA Ops\nmodels/bricks/ops/cuda/"]:::model
            click T7 "https://github.com/xiuqhou/salience-detr/tree/main/models/bricks/ops/cuda/"
        end
        DWR["Detector Wrapper\nmodels/detectors/salience_detr.py"]:::model
        click DWR "https://github.com/xiuqhou/salience-detr/blob/main/models/detectors/salience_detr.py"
        MTR["Hungarian Matcher\nmodels/matcher/hungarian_matcher.py"]:::model
        click MTR "https://github.com/xiuqhou/salience-detr/blob/main/models/matcher/hungarian_matcher.py"
    end

    %% Configuration & Orchestration
    subgraph "Configuration & Orchestration"
        C1["Model Configs\nconfigs/salience_detr/*.py"]:::config
        click C1 "https://github.com/xiuqhou/salience-detr/tree/main/configs/salience_detr/"
        C2["Training Config\nconfigs/train_config.py"]:::config
        click C2 "https://github.com/xiuqhou/salience-detr/blob/main/configs/train_config.py"
        E1["main.py"]:::config
        click E1 "https://github.com/xiuqhou/salience-detr/blob/main/main.py"
        E2["test.py"]:::config
        click E2 "https://github.com/xiuqhou/salience-detr/blob/main/test.py"
        E3["inference.py"]:::config
        click E3 "https://github.com/xiuqhou/salience-detr/blob/main/inference.py"
    end

    %% Utilities & Support
    subgraph "Utilities & Support"
        U1["Engine\nutil/engine.py"]:::util
        click U1 "https://github.com/xiuqhou/salience-detr/blob/main/util/engine.py"
        U2["COCO Eval\nutil/coco_eval.py\nutil/coco_utils.py"]:::util
        click U2 "https://github.com/xiuqhou/salience-detr/blob/main/util/coco_eval.py"
        click U2 "https://github.com/xiuqhou/salience-detr/blob/main/util/coco_utils.py"
        U3["Collate Fn\nutil/collate_fn.py"]:::util
        click U3 "https://github.com/xiuqhou/salience-detr/blob/main/util/collate_fn.py"
        U4["Logger\nutil/logger.py"]:::util
        click U4 "https://github.com/xiuqhou/salience-detr/blob/main/util/logger.py"
        U5["File I/O & Env\nutil/file_io.py\nutil/collect_env.py"]:::util
        click U5 "https://github.com/xiuqhou/salience-detr/blob/main/util/file_io.py"
        click U5 "https://github.com/xiuqhou/salience-detr/blob/main/util/collect_env.py"
        U6["Misc Helpers\nutil/misc.py\nutil/utils.py"]:::util
        click U6 "https://github.com/xiuqhou/salience-detr/blob/main/util/misc.py"
        click U6 "https://github.com/xiuqhou/salience-detr/blob/main/util/utils.py"
        U7["Vis Helpers\nutil/visualize.py"]:::util
        click U7 "https://github.com/xiuqhou/salience-detr/blob/main/util/visualize.py"
    end

    %% Tools & Export
    subgraph "Tools & Export"
        Tbm["Benchmark\ntools/benchmark_model.py"]:::tool
        click Tbm "https://github.com/xiuqhou/salience-detr/blob/main/tools/benchmark_model.py"
        Tox["ONNX Export\ntools/pytorch2onnx.py"]:::tool
        click Tox "https://github.com/xiuqhou/salience-detr/blob/main/tools/pytorch2onnx.py"
        Tvis["Dataset Viz\ntools/visualize_datasets.py"]:::tool
        click Tvis "https://github.com/xiuqhou/salience-detr/blob/main/tools/visualize_datasets.py"
    end

    %% Data Flow & Relationships
    D1 -->|"load images"| D2
    D2 -->|"batch"| D3
    D3 -->|"tensors"| B1
    D3 -->|"tensors"| B2
    D3 -->|"tensors"| B3
    D3 -->|"tensors"| B4
    D3 -->|"tensors"| B5
    B1 & B2 & B3 & B4 & B5 -->|"features"| T0
    T0 --> T1
    T1 --> T2
    T2 --> T3
    T3 --> T4
    T4 -->|"encoded features"| DWR
    DWR -->|"raw preds"| MTR
    MTR -->|"matched preds"| T5
    T5 -->|"losses"| U1
    U1 -->|"backprop & update"| DWR
    U1 -->|"evaluate"| U2
    U1 -->|"log & checkpoint"| U4

    %% Orchestration & Config Paths
    C1 --> DWR
    C2 --> U1
    E1 --> U1
    E2 --> U2
    E3 --> DWR

    %% Tools Connections
    DWR --> Tbm
    DWR --> Tox
    DWR --> Tvis

    %% Styles
    classDef data fill:#9f6,stroke:#333,stroke-width:1px
    classDef model fill:#6cf,stroke:#333,stroke-width:1px
    classDef config fill:#fc6,stroke:#333,stroke-width:1px
    classDef util fill:#ccc,stroke:#333,stroke-width:1px
    classDef tool fill:#f96,stroke:#333,stroke-width:1px
```
