# eternaldolphin/lami-detr architecture

> Architecture diagram of the GitHub repository eternaldolphin/lami-detr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eternaldolphin/lami-detr
- Repository: https://github.com/eternaldolphin/lami-detr

## Overview

##### Project Type and Purpose
- LaMI-DETR is an open-source computer-vision research project for open-vocabulary object detection. It adapts the DETR architecture (via Detectron2 and Detrex) with a frozen CLIP image encoder backbone and leverages LLM-generated “instructions” (GPT-3.5-turbo or similar) to sample hard negatives and refine category representations.

##### High-Level Architectural Overview
The system assembles four layers of components:
A. Data & Dataset Registration
- Raw data (COCO, LVIS, VisualGenome) stored under `dataset/`.
- Custom dataset registration scripts in `detectron2/detectron2/data/datasets` and `detrex/data/datasets`.
- Precomputed clusters & metadata (`dataset/cluster`, `dataset/metadata`).
B. Configuration Management
- Project-level configs in `configs/common/` drive schedules, optimizers, data parameters, and model settings.
- Model-specific configs in `lami_dino/configs/` for LaMI-DETR fine-tuning (scripts point to CLIP checkpoints, LLM query paths).
- Detectron2 & Detrex base configs (sub-repos under `detectron2/configs` and `detrex/config/configs`).
C. Model & Training Engine
1. Core Frameworks:
- Detectron2: data pipelines, training loops, model zoo, registry.
- Detrex: DETR-style meta-architecture, transformer layers, dataset mappers.
2. Custom Extensions:
- `lami_dino/modeling/`: LaMI-DETR head, criterion modules (two-stage, denoising).
- CLIP backbone loader & integration.
- GPT API adapter (through config’s `model.eval_query_path`) to fetch negative-category instructions.
3. Training Orchestration:
- Entry point: `tools/train_net.py`
- Engine components in `detectron2/detectron2/engine` (hooks, launch, train_loop).
- Loss & optimizer scheduling in `configs/common/optim.py`, `configs/common/train.py`.
D. Evaluation, Checkpointing & Outputs
- Pretrained-model loader in `pretrained_models/`
- Checkpoint saving via Detectron2 hooks.
- LVIS / COCO evaluators in `detectron2/detectron2/evaluation`.
- Scripts for inference and score ensemble (p2 vs p3) in README examples.

##### Relationships & Data Flow
1. Inputs: Raw images & annotations → Dataset registration → DataLoader
2. DataLoader → Transform pipeline → Batch of tensors
3. Batch → CLIP backbone → feature maps
4. Feature maps → DETR head (in Detrex meta-arch) + LaMI modules → predictions
5. Predictions + ground truth → Loss functions (criterion modules)
6. Loss → Optimizer & LR scheduler → Model weight updates
7. Periodic checkpointing & evaluation (COCO / LVIS metrics)
8. During evaluation/inference, optional LLM API call supplies refined category embeddings to classification head.

##### Architectural Patterns & Design Principles
- Plugin & registry pattern (Detectron2 registries for backbones, heads, datasets)
- Separation of concerns: configs drive behavior, code is modular across sub-repos (`detectron2`, `detrex`, `lami_dino`)
- Layered architecture: Data → Model → Engine → Output
- Dependency injection via YAML/Python config files

##### Technologies & Frameworks
- PyTorch + CUDA
- Detectron2 (Facebookresearch)
- Detrex (IDEA-Research)
- OpenAI GPT API (LLM for instructions)
- Numpy for clusters/metadata

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Dataset Registration
    subgraph "Data & Dataset Registration"
        DS["Data Storage\n(cluster, metadata)"]:::dataStorage
        DR["Dataset Registration\n(Detectron2 & Detrex)"]:::dataset
    end

    %% Configuration Management
    subgraph "Configuration Management"
        PC["Common Configs"]:::config
        LC["LaMI-DETR Configs"]:::config
        DC["Detectron2 Base Configs"]:::config
        TC["Detrex Base Configs"]:::config
    end

    %% Model & Training Engine
    subgraph "Model & Training Engine"
        Loader["DataLoader & Transforms"]:::transform
        Backbone["CLIP Backbone"]:::model
        DEHead["Detrex DETR Head"]:::model
        LMIMod["LaMI-DETR Modules\n(heads & criteria)"]:::model
        LLM["LLM API (GPT)"]:::api
        Engine["Training Engine & Scheduler"]:::engine
    end

    %% Evaluation & Checkpointing
    subgraph "Evaluation & Checkpointing"
        CKPT["Checkpointing"]:::output
        Eval["COCO / LVIS Evaluation"]:::output
    end

    %% Data Flow
    DS --> DR
    DR --> Loader
    Loader --> Backbone
    Backbone --> DEHead
    DEHead --> LMIMod
    LMIMod --> Engine
    LMIMod -->|"optional embeddings"| LLM
    LLM -.->|"embeddings"| DEHead
    Engine --> CKPT
    Engine --> Eval

    %% Config Influences
    PC --> Engine
    LC --> Engine
    DC --> Engine
    TC --> Engine

    %% Click Events
    click DS "https://github.com/eternaldolphin/lami-detr/tree/main/dataset/cluster"
    click DS "https://github.com/eternaldolphin/lami-detr/tree/main/dataset/metadata"
    click DR "https://github.com/eternaldolphin/lami-detr/tree/main/detectron2/detectron2/data/datasets/"
    click DR "https://github.com/eternaldolphin/lami-detr/tree/main/detrex/data/datasets/"
    click PC "https://github.com/eternaldolphin/lami-detr/tree/main/configs/common/"
    click LC "https://github.com/eternaldolphin/lami-detr/tree/main/lami_dino/configs/"
    click DC "https://github.com/eternaldolphin/lami-detr/tree/main/detectron2/configs/"
    click TC "https://github.com/eternaldolphin/lami-detr/tree/main/detrex/config/configs/"
    click Backbone "https://github.com/eternaldolphin/lami-detr/blob/main/detrex/modeling/backbone/timm_backbone.py"
    click DEHead "https://github.com/eternaldolphin/lami-detr/tree/main/detrex/modeling/neck/"
    click DEHead "https://github.com/eternaldolphin/lami-detr/blob/main/detrex/modeling/classifier/zero_shot_classifier.py"
    click LMIMod "https://github.com/eternaldolphin/lami-detr/blob/main/lami_dino/modeling/dino.py"
    click LMIMod "https://github.com/eternaldolphin/lami-detr/blob/main/lami_dino/modeling/dino_transformer.py"
    click LMIMod "https://github.com/eternaldolphin/lami-detr/blob/main/lami_dino/modeling/two_stage_criterion.py"
    click LMIMod "https://github.com/eternaldolphin/lami-detr/blob/main/lami_dino/modeling/dn_criterion.py"
    click Engine "https://github.com/eternaldolphin/lami-detr/blob/main/tools/train_net.py"
    click Engine "https://github.com/eternaldolphin/lami-detr/blob/main/detectron2/detectron2/engine/hooks.py"
    click Engine "https://github.com/eternaldolphin/lami-detr/blob/main/detectron2/detectron2/engine/launch.py"
    click Engine "https://github.com/eternaldolphin/lami-detr/blob/main/detectron2/detectron2/engine/train_loop.py"
    click CKPT "https://github.com/eternaldolphin/lami-detr/tree/main/detectron2/detectron2/checkpoint/"
    click Eval "https://github.com/eternaldolphin/lami-detr/blob/main/detectron2/detectron2/evaluation/coco_evaluation.py"
    click Eval "https://github.com/eternaldolphin/lami-detr/blob/main/detectron2/detectron2/evaluation/lvis_evaluation.py"

    %% Styles
    classDef dataStorage fill:#dae8fc,stroke:#6c8ebf,stroke-width:2px;
    classDef dataset fill:#d5e8d4,stroke:#82b366,stroke-width:2px;
    classDef transform fill:#e1f5a9,stroke:#8dc63f,stroke-width:2px;
    classDef model fill:#f8cecc,stroke:#b85450,stroke-width:2px;
    classDef api fill:#e1d5e7,stroke:#9673a6,stroke-width:2px,stroke-dasharray: 5 5;
    classDef engine fill:#f5f5f5,stroke:#d9534f,stroke-width:2px,stroke-dasharray: 2 2;
    classDef output fill:#f2f2f2,stroke:#888888,stroke-width:2px;
```
