# facebookresearch/detic architecture

> Architecture diagram of the GitHub repository facebookresearch/detic, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/detic
- Repository: https://github.com/facebookresearch/detic

## Overview

##### Project Type and Purpose
- Detic is an advanced deep learning object detection system. Its main goal is to detect virtually any class (even those not present during training) by incorporating image-level supervision and CLIP-based text encoding.
- The project integrates state-of-the-art detection architectures (such as DETR and RCNN variants) alongside established third-party repositories (like CenterNet2 and Deformable-DETR) as well as custom modules for data transformation, model training, and evaluation.

##### Key Components to Highlight
A. Configuration and Initialization:
- Configuration management by showing the "configs" directory that holds many YAML and Python config files (each representing different model architectures and training configurations).
- A “cog.yaml” which may be used for deployment or workflow orchestration.

B. Data Layer:
- Custom dataset loader and augmentation modules from the “detic/data” folder (e.g., custom_build_augmentation.py, custom_dataset_dataloader.py, custom_dataset_mapper.py).

C. Model Components:
- “detic/modeling” submodules:
- Backbone modules (e.g., swintransformer.py, timm.py) that are responsible for feature extraction.
- Meta architecture components (such as custom_rcnn.py and d2_deformable_detr.py) that define the overall detection network.
- Text encoding components present in “detic/modeling/text/text_encoder.py” to integrate with CLIP for mapping text labels to feature space.

D. Training and Evaluation:
- “detic/custom_solver.py”, which represents the custom training loop solver, and the “train_net.py” script as the entry point for training.
- “detic/evaluation” directory (with files like custom_coco_eval.py and oideval.py) that is used for computing performance metrics.

E. Inference and Demo:
- How post-processing flows from the model outputs (detections) are passed to these scripts that generate final output images.
- Integration with external interfaces like the default Detectron2 demo, and mention that the project supports web demos via Gradio (as indicated in the README).

F. External and Utility Tools:
- A separate block for “tools” which includes various scripts (e.g., downloading pretrained models, dataset pre-processing scripts, merge and transformation utilities).
- A node for “third_party” libraries, linking to external dependencies like CenterNet2 and Deformable-DETR that the project builds upon.
- Demonstrate how these utilities support or interface with core modules (data processing or model evaluation).

##### Architectural Patterns and Design Principles
- Layered Architecture: The system is naturally organized into layers – configuration, data preprocessing, model building, training/evaluation, and deployment/demo.
- Modularity: The design separates fundamental concerns into discrete modules (data loaders, transformation, backbone, meta architecture, ROI heads) making clear the extension points.
- External Integrations: The architecture leverages third-party components (CLIP, Detectron2-based methods, external detection frameworks) which are explicitly incorporated via directories like “third_party” and “detic/modeling/backbone”.

##### Incorporate Specific Project Details
- That the “configs” directory allows selection among several training setups (e.g., different YOLO-like or DETR-style configurations) and that these configurations dictate the specific network architecture.
- Role of “detic/modeling/text/text_encoder.py” in connecting image features with text labels, clarifying the CLIP-based functionality mentioned in the README.
- Integration points for datasets and custom transformations which are vital for generalizing detection across multiple datasets (COCO, LVIS, OpenImages, etc.).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration and Initialization
    subgraph "Configuration and Initialization"
        config1["configs"]:::config
        cog["cog.yaml"]:::config
    end

    %% Data Layer
    subgraph "Data Layer"
        raw["External Datasets"]:::data
        datasets["datasets"]:::data
        deticData["detic/data"]:::data
        dataAug["custom_build_augmentation.py"]:::data
        dataLoader["custom_dataset_dataloader.py"]:::data
        dataMapper["custom_dataset_mapper.py"]:::data
    end

    %% Model Components
    subgraph "Model Components"
        backbone["Backbone Network"]:::model
        meta["Meta Architecture"]:::model
        roi["ROI Heads"]:::model
        textEnc["Text Encoder"]:::model
    end

    %% Training and Evaluation
    subgraph "Training and Evaluation"
        solver["Custom Solver/Training Loop"]:::training
        trainScript["train_net.py"]:::training
        evaluation["Evaluation Module"]:::training
    end

    %% Inference and Demo
    subgraph "Inference and Demo"
        demo["Demo Script"]:::inference
        predict["Predict Script"]:::inference
    end

    %% External and Utility Tools
    subgraph "External and Utility Tools"
        tools["Tools"]:::external
        thirdParty["Third Party Integrations"]:::external
    end

    %% Data Layer connections
    raw -->|"raw_data"| datasets
    datasets -->|"dataset_files"| deticData
    deticData -->|"augmentation"| dataAug
    deticData -->|"loader"| dataLoader
    deticData -->|"mapper"| dataMapper

    %% Data to Model
    dataLoader -->|"feature_input"| backbone

    %% Model internal flow
    backbone -->|"extracted_features"| meta
    backbone -->|"extracted_features"| roi
    textEnc -->|"CLIP_encoding"| meta

    %% Model to Training
    meta -->|"detection_pipeline"| solver
    meta -->|"detection_pipeline"| trainScript
    solver -->|"trained_model"| evaluation
    trainScript -->|"trained_model"| evaluation

    %% Training to Inference
    evaluation -->|"eval_results"| demo
    evaluation -->|"eval_results"| predict

    %% External integrations
    cog -->|"deployment_config"| solver
    tools -->|"utility_support"| solver
    thirdParty -->|"external_integration"| backbone

    %% Click Events
    click config1 "https://github.com/facebookresearch/detic/tree/main/configs"
    click cog "https://github.com/facebookresearch/detic/blob/main/cog.yaml"
    click datasets "https://github.com/facebookresearch/detic/tree/main/datasets"
    click deticData "https://github.com/facebookresearch/detic/tree/main/detic/data"
    click dataAug "https://github.com/facebookresearch/detic/blob/main/detic/data/custom_build_augmentation.py"
    click dataLoader "https://github.com/facebookresearch/detic/blob/main/detic/data/custom_dataset_dataloader.py"
    click dataMapper "https://github.com/facebookresearch/detic/blob/main/detic/data/custom_dataset_mapper.py"
    click backbone "https://github.com/facebookresearch/detic/tree/main/detic/modeling/backbone"
    click meta "https://github.com/facebookresearch/detic/tree/main/detic/modeling/meta_arch"
    click roi "https://github.com/facebookresearch/detic/tree/main/detic/modeling/roi_heads"
    click textEnc "https://github.com/facebookresearch/detic/blob/main/detic/modeling/text/text_encoder.py"
    click solver "https://github.com/facebookresearch/detic/blob/main/detic/custom_solver.py"
    click trainScript "https://github.com/facebookresearch/detic/blob/main/train_net.py"
    click evaluation "https://github.com/facebookresearch/detic/tree/main/detic/evaluation"
    click demo "https://github.com/facebookresearch/detic/blob/main/demo.py"
    click predict "https://github.com/facebookresearch/detic/blob/main/predict.py"
    click tools "https://github.com/facebookresearch/detic/tree/main/tools"
    click thirdParty "https://github.com/facebookresearch/detic/tree/main/third_party"

    %% Styles
    classDef config fill:#D1C4E9,stroke:#4A148C,stroke-width:2px;
    classDef data fill:#A5D6A7,stroke:#1B5E20,stroke-width:2px;
    classDef model fill:#FFF9C4,stroke:#FBC02D,stroke-width:2px;
    classDef training fill:#FFAB91,stroke:#E64A19,stroke-width:2px;
    classDef inference fill:#81D4FA,stroke:#0288D1,stroke-width:2px;
    classDef external fill:#FFCDD2,stroke:#C62828,stroke-width:2px;
```
