How xiuqhou/salience-detr works
Project Type & Purpose
• Type: Open-source research code / PyTorch model library for object detection.
• Purpose: Implements “Salience DETR,” a Detection Transformer enhanced with hierarchical salience filtering. Provides training, evaluation, inference, ONNX export, and data‐visualization tooling.
High-Level Architecture
• Data Ingestion Layer
– COCO (or custom) dataset files under data/ loaded via datasets/coco.py.
– Preprocessing & augmentation modules under transforms/ (and transforms/v2/).
• Model Definition Layer
– Backbone modules (models/backbones/*): ResNet, ConvNeXt, Swin, FocalNet, ViT.
– Transformer “bricks” (models/bricks/*): multi‐scale deformable attention, salience heads, position encodings, loss computations.
– Detector wrapper (models/detectors/salience_detr.py) composes backbone + transformer + post‐processing.
– Matcher (models/matcher/hungarian_matcher.py) implements bipartite matching for loss assignment.
• Configuration & Orchestration
– Model configs (configs/salience_detr/*.py) declare architecture hyperparameters (backbone choice, input size, num queries, num classes).
– Training config (configs/train_config.py) declares optimizer, scheduler, dataset paths, mixed‐GPU settings using accelerate.
– Entry points: main.py orchestrates training; test.py runs evaluation; inference.py handles image‐folder inference; tools/* provide benchmarking, ONNX export, dataset visualization.
• Utilities & Support
– util/engine.py: training loop, evaluation loop, checkpointing, logging.
– util/*.py: distributed setup, metrics (COCO eval), data collate, file I/O, environment reporting, visualization helpers.
– CUDA kernels under models/bricks/ops/cuda auto‐built on first run.
Relationships & Data Flow
1. COCO JSON + images → datasets.CocoDetection → PyTorch DataLoader (with util.collate_fn).
2. Each batch → transforms pipeline → tensor inputs.
3. Inputs → backbone → multiscale feature maps.
4. Features + learnable queries → transformer encoder/decoder (salience filtering layers) → raw predictions.
6. Losses → backprop → torch.optim.AdamW (param groups from optimizer/param_dict.py) → update model.
7. Periodic eval (util.engine.evaluate) → COCO metrics → logs & checkpoints.
Architectural Patterns & Principles
• Modular plugin pattern: Backbones, heads, and matchers implement base classes—easy to add new modules.
• Config‐driven design: Behavior controlled entirely by .py config files—no hard‐coded hyperparameters.
• Separation of concerns: Data, model, training loop, utils, and tools reside in distinct directories.
• Lazy CUDA op compilation: abstracts away manual build steps.
• Use of accelerate for multi‐GPU scaling with minimal boilerplate.