# facebookresearch/detr architecture

> Architecture diagram of the GitHub repository facebookresearch/detr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/detr
- Repository: https://github.com/facebookresearch/detr

## Overview

##### Project Type and Purpose
- The project is an object detection system built on Transformers (DETR). It is implemented in PyTorch and focuses on both object detection and panoptic segmentation.
- The README and file tree show that the project provides training scripts, evaluation routines, model definitions, datasets handling for COCO, and a wrapper for integration with Detectron2 (located in the “d2” folder).

##### Analyzing the File Structure
- Top-level files like README.md, main.py, Dockerfile, and LICENSE provide project-level context.
- Core directories include:
- models: Contains the core neural network modules (backbone, transformer, segmentation, matcher, etc.) which define the DETR model.
- datasets: Contains dataset utilities and COCO-specific evaluators and transforms. This folder manages data loading, pre- and post-processing.
- d2: Provides a Detectron2 wrapper and additional configuration files for integrating DETR into the Detectron2 framework.
- .github: Contains contribution guidelines, issue templates, and the code of conduct for managing community interactions.
- The presence of training scripts (like train_net.py, engine.py, run_with_submitit.py, and main.py) indicates that the project follows a typical ML training pipeline.

##### Key Components
A. Data Handling Layer:
– Components: datasets (files such as coco.py, coco_eval.py, transforms.py)
– Role: Data loading, augmentation, and transformation from the COCO dataset (and panoptic dataset for segmentation).
– Flow: Raw input images → data augmentation and preprocessing → fed as input to the model.

B. Model Architecture:
– Components: models (backbone.py, transformer.py, detr.py, matcher.py, segmentation.py, position_encoding.py)
– Role: Implements the core DETR model, including:
- The backbone (e.g., ResNet-50 / ResNet-101) for feature extraction.
- The Transformer encoder-decoder architecture that processes features and object queries.
- Specialized modules such as the segmentation head (for panoptic segmentation).
– Important note: Incorporate the design decision of handling object queries in a set-based prediction paradigm.

C. Training & Inference Engine:
– Components: main.py, train_net.py, engine.py, run_with_submitit.py
– Role: Manage the training loop and inference:
- Setup of training (distributed training, data loading, evaluation metrics)
- Handling loss computation and optimization (using AdamW with designated hyperparameters).
– Pipeline: Data from datasets → processed via the model → outputs compared with ground truth → loss computed → weights updated.

D. Utilities and Support:
– Components: util (box_ops.py, plot_utils.py)

E. Integration and Deployment:
– Components: hubconf.py, Dockerfile
– Role: Allowing users to load pre-trained models via torch.hub and deployment support through containerization.

F. Extra Features – Detectron2 Integration:
– Components: d2 folder (configs, converter.py, detr submodule)
– Role: Acts as an adapter to integrate with the Detectron2 framework. This set-up provides a different API perspective and additional configuration options for those using Detectron2.

##### Architectural Patterns and Design Principles
- Layered Architecture: The system is broken down into data, model, training, and utility layers.
- Modular Design: Independent modules allow separation of concerns (e.g., datasets vs. model vs. training engine).
- Reusability: Using configuration files and a wrapper layer (d2) to integrate with other frameworks to extend usage.
- Parallelism: The design inherently supports parallel inference and training (e.g., distributed training using PyTorch's distributed launcher).

##### Final Checklist
- Clear labels and distinct blocks for each module.
– From dataset preparation to preprocessed data.
– From the preprocessed data to the model with different branches for detection and segmentation.
- Connection points with key operations (like “bipartite matching”, “feature extraction”, “loss computation”).
- Configuration sources (e.g., configs in d2/) and deployment setups (Dockerfile).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Layer
    subgraph "Input Layer"
        Raw["Raw Images"]:::data
    end

    %% Data Handling Layer
    subgraph "Data Handling Layer"
        D1["COCO Loader"]:::data
        D2["COCO Eval"]:::data
        D3["Transforms"]:::data
        D4["COCO Panoptic"]:::data
        D5["Panoptic Eval"]:::data
    end

    %% Model Architecture
    subgraph "Model Architecture"
        M1["Backbone"]:::model
        M2["DETR Core"]:::model
        M3["Transformer"]:::model
        M4["Matcher"]:::model
        M5["Positional Encoding"]:::model
        M6["Segmentation Head"]:::model
    end

    %% Training & Inference Engine
    subgraph "Training & Inference Engine"
        T1["Main Training"]:::training
        T2["Engine"]:::training
        T3["Submitit Runner"]:::training
        T4["Detectron2 Training"]:::training
    end

    %% Utilities and Support
    subgraph "Utilities"
        U1["Box Operations"]:::utility
        U2["Plot Utils"]:::utility
        U3["Misc Helpers"]:::utility
    end

    %% Integration and Deployment
    subgraph "Integration & Deployment"
        I1["Torch Hub Config"]:::deployment
        I2["Dockerization"]:::deployment
    end

    %% Detectron2 Integration
    subgraph "Detectron2 Integration"
        D21["D2 README"]:::d2
        D22["D2 Configs"]:::d2
        D23["Converter"]:::d2
        D24["DETR D2 Module __init__"]:::d2
        D25["DETR D2 Config"]:::d2
    end

    %% Data Flow
    Raw -->|"loads"| D1
    Raw -->|"loads"| D2
    Raw -->|"loads"| D3
    Raw -->|"loads"| D4
    Raw -->|"loads"| D5

    D1 -->|"preprocess"| M1
    D2 --> M1
    D3 --> M1
    D4 --> M1
    D5 --> M1

    %% Model Internal Flow
    M1 -->|"featureExtraction"| M2
    M3 --> M2
    M5 --> M2
    M6 --> M2
    M2 -->|"bipartiteMatching"| M4

    %% Training & Inference Flow
    M2 -->|"inferenceOutput"| T1
    T1 --> T2
    T2 --> T3
    T1 -->|"triggers"| T4
    T2 -->|"updatesWeights"| M2

    %% Utilities usage
    T1 -->|"logsVisualizes"| U1
    T1 --> U2
    T1 --> U3

    %% Integration & Deployment usage
    T1 --> I1
    T1 --> I2

    %% Detectron2 Integration usage
    T1 --> D21
    T1 --> D22
    T1 --> D23
    T1 --> D24
    T1 --> D25

    %% Click Events for Data Handling Layer
    click D1 "https://github.com/facebookresearch/detr/blob/main/datasets/coco.py"
    click D2 "https://github.com/facebookresearch/detr/blob/main/datasets/coco_eval.py"
    click D3 "https://github.com/facebookresearch/detr/blob/main/datasets/transforms.py"
    click D4 "https://github.com/facebookresearch/detr/blob/main/datasets/coco_panoptic.py"
    click D5 "https://github.com/facebookresearch/detr/blob/main/datasets/panoptic_eval.py"

    %% Click Events for Model Architecture
    click M1 "https://github.com/facebookresearch/detr/blob/main/models/backbone.py"
    click M2 "https://github.com/facebookresearch/detr/blob/main/models/detr.py"
    click M3 "https://github.com/facebookresearch/detr/blob/main/models/transformer.py"
    click M4 "https://github.com/facebookresearch/detr/blob/main/models/matcher.py"
    click M5 "https://github.com/facebookresearch/detr/blob/main/models/position_encoding.py"
    click M6 "https://github.com/facebookresearch/detr/blob/main/models/segmentation.py"

    %% Click Events for Training & Inference Engine
    click T1 "https://github.com/facebookresearch/detr/blob/main/main.py"
    click T2 "https://github.com/facebookresearch/detr/blob/main/engine.py"
    click T3 "https://github.com/facebookresearch/detr/blob/main/run_with_submitit.py"
    click T4 "https://github.com/facebookresearch/detr/blob/main/d2/train_net.py"

    %% Click Events for Utilities
    click U1 "https://github.com/facebookresearch/detr/blob/main/util/box_ops.py"
    click U2 "https://github.com/facebookresearch/detr/blob/main/util/plot_utils.py"
    click U3 "https://github.com/facebookresearch/detr/blob/main/util/misc.py"

    %% Click Events for Integration & Deployment
    click I1 "https://github.com/facebookresearch/detr/blob/main/hubconf.py"
    click I2 "https://github.com/facebookresearch/detr/tree/main/Dockerfile"

    %% Click Events for Detectron2 Integration
    click D21 "https://github.com/facebookresearch/detr/blob/main/d2/README.md"
    click D22 "https://github.com/facebookresearch/detr/tree/main/d2/configs"
    click D23 "https://github.com/facebookresearch/detr/blob/main/d2/converter.py"
    click D24 "https://github.com/facebookresearch/detr/blob/main/d2/detr/__init__.py"
    click D25 "https://github.com/facebookresearch/detr/blob/main/d2/detr/config.py"

    %% Styles
    classDef data fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef model fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef training fill:#FFA07A,stroke:#000,stroke-width:1px;
    classDef utility fill:#FFD700,stroke:#000,stroke-width:1px;
    classDef deployment fill:#DDA0DD,stroke:#000,stroke-width:1px;
    classDef d2 fill:#FFB6C1,stroke:#000,stroke-width:1px;
```
