How eternaldolphin/lami-detr works
Project Type and Purpose
• LaMI-DETR is an open-source computer-vision research project for open-vocabulary object detection. It adapts the DETR architecture (via Detectron2 and Detrex) with a frozen CLIP image encoder backbone and leverages LLM-generated “instructions” (GPT-3.5-turbo or similar) to sample hard negatives and refine category representations.
High-Level Architectural Overview
The system assembles four layers of components:
A. Data & Dataset Registration
• Raw data (COCO, LVIS, VisualGenome) stored under dataset/.
• Custom dataset registration scripts in detectron2/detectron2/data/datasets and detrex/data/datasets.
• Precomputed clusters & metadata (dataset/cluster, dataset/metadata).
B. Configuration Management
• Project-level configs in configs/common/ drive schedules, optimizers, data parameters, and model settings.
• Model-specific configs in lami_dino/configs/ for LaMI-DETR fine-tuning (scripts point to CLIP checkpoints, LLM query paths).
• Detectron2 & Detrex base configs (sub-repos under detectron2/configs and detrex/config/configs).
C. Model & Training Engine
1. Core Frameworks:
• Detectron2: data pipelines, training loops, model zoo, registry.
• Detrex: DETR-style meta-architecture, transformer layers, dataset mappers.
2. Custom Extensions:
• lami_dino/modeling/: LaMI-DETR head, criterion modules (two-stage, denoising).
• CLIP backbone loader & integration.
• GPT API adapter (through config’s model.eval_query_path) to fetch negative-category instructions.
3. Training Orchestration:
• Entry point: tools/train_net.py
• Engine components in detectron2/detectron2/engine (hooks, launch, train_loop).
• Loss & optimizer scheduling in configs/common/optim.py, configs/common/train.py.
D. Evaluation, Checkpointing & Outputs
• Pretrained-model loader in pretrained_models/
• Checkpoint saving via Detectron2 hooks.
• LVIS / COCO evaluators in detectron2/detectron2/evaluation.
• Scripts for inference and score ensemble (p2 vs p3) in README examples.
Relationships & Data Flow
1. Inputs: Raw images & annotations → Dataset registration → DataLoader
2. DataLoader → Transform pipeline → Batch of tensors
3. Batch → CLIP backbone → feature maps
4. Feature maps → DETR head (in Detrex meta-arch) + LaMI modules → predictions
5. Predictions + ground truth → Loss functions (criterion modules)
6. Loss → Optimizer & LR scheduler → Model weight updates
7. Periodic checkpointing & evaluation (COCO / LVIS metrics)
8. During evaluation/inference, optional LLM API call supplies refined category embeddings to classification head.
Architectural Patterns & Design Principles
• Plugin & registry pattern (Detectron2 registries for backbones, heads, datasets)
• Separation of concerns: configs drive behavior, code is modular across sub-repos (detectron2, detrex, lami_dino)
• Layered architecture: Data → Model → Engine → Output
• Dependency injection via YAML/Python config files
Technologies & Frameworks
• PyTorch + CUDA
• Detectron2 (Facebookresearch)
• Detrex (IDEA-Research)
• OpenAI GPT API (LLM for instructions)
• Numpy for clusters/metadata