# foundationvision/generateu architecture

> Architecture diagram of the GitHub repository foundationvision/generateu, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/foundationvision/generateu
- Repository: https://github.com/foundationvision/generateu

## Overview

##### Project Type and Purpose
- GenerateU is an experimental research framework for **generative open-ended object detection**, built on top of the Detectron2 library. It integrates a Deformable DETR–based detection backbone (DDETRS project) with language modeling components to enable region-language pretraining without fixed category labels.

##### File-Structure Analysis
- Top-level:
- detectron2/           ← Core detection library (forked and extended)
- projects/DDETRS/      ← Project-specific code for Deformable DETR + VL pretraining
- tools/                ← Helper scripts (e.g., weight conversion, evaluation)
- launch.py             ← Distributed training entrypoint
- setup.py, requirements.txt
- detectron2/ subpackages:
- config/               ← Configuration system (defaults, instantiation)
- data/                 ← Dataset catalog, mappers, samplers, transforms
- modeling/             ← Backbones, meta-architectures, heads, proposal generators
- engine/               ← Training loop, hooks, multi-GPU launcher
- export/               ← ONNX / TorchScript / Caffe2 export utilities
- solver/, evaluation/, utils/, layers/, etc.
- projects/DDETRS/:
- cfgs/                 ← YAML config files for VG/LVIS experiments
- ddetrs/               ← Custom backbone (Swin, ViT), data augmentation, model code
- train_net.py          ← Training script wrapper for the DDETRS project

##### Key Components
a. External Dependencies
- GPU Cluster (multi-node, A100)
- PyTorch, CUDA, TorchVision, MMCV, Detectron2
- Pretrained weights (Swin, Visual Genome, LVIS embeddings)
- Datasets (VisualGenome, LVIS, optional GrIT-20M)
b. Data Ingestion & Preprocessing
- Dataset Catalog → Dataset Mapper → Augmentations → Sampler → DataLoader
c. Model Architecture
- Backbone: Swin-Transformer / ViT
- Deformable-DETR Transformer
- Region-Language Fusion: FlanT5 text encoder integration
d. Training Engine
- Detectron2 Engine: train_loop, hooks (logging, checkpointing)
- Solver: LR scheduler, optimizer
- Distributed Launcher (launch.py)
e. Evaluation & Export
- Evaluators (COCO, LVIS, semantic segmentation)
- Export modules (TorchScript, ONNX, Caffe2)

##### Relationships and Data Flow
- Data sources → Data config & registration → DataLoader → Model input
- Model forward pass → Compute detection + language losses → Engine backprop → Optimizer
- Checkpointing → Optional export → Evaluation modules consume checkpoints → Metrics

##### Architectural Patterns and Principles
- Modular plugin pattern (Detectron2’s registry-based design)
- Separation of concerns:
- “detectron2/” provides core abstractions (config, data, modeling, engine)
- “projects/DDETRS/” plugs into these abstractions for custom tasks
- Distributed data-parallel training via launch script

##### Technologies and Frameworks
- PyTorch, CUDA/C++ custom ops (ms_deform_attn), TorchScript, Caffe2 export
- Detectron2’s config system (YAML + lazy instantiation)
- Swin/ViT backbones, Deformable DETR, FlanT5 text encoder

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        GPU["GPU Cluster (A100)"]:::external
        PyTorch["PyTorch/CUDA/MMCV"]:::external
        Pretrained["Pretrained Weights"]:::external
        Datasets["Datasets (VG, LVIS, GrIT-20M)"]:::external
    end

    %% Data & Config
    subgraph "Data & Configuration" 
        direction TB
        ConfigCore["Detectron2 Config System"]:::core
        ConfigProj["DDETRS YAML Configs"]:::proj
        DataCatalog["Dataset Catalog"]:::core
        DataMapper["Dataset Mapper"]:::core
        Augment["Augmentations"]:::core
        Sampler["Sampler"]:::core
        DataLoader["DataLoader"]:::core
    end

    %% Model Architecture
    subgraph "Model" 
        direction TB
        subgraph "Backbone" 
            direction TB
            Swin["Swin Transformer"]:::proj
            ViT["ViT"]:::proj
        end
        DeformTransformer["Deformable DETR Transformer"]:::proj
        subgraph "Region-Language Fusion"
            direction TB
            FlanT5["FlanT5 Text Encoder"]:::proj
        end
        subgraph "Heads"
            direction TB
            BoxHead["Box Head"]:::core
            MaskHead["Mask Head"]:::core
            LangHead["Language Gen Head"]:::proj
        end
    end

    %% Training Engine
    subgraph "Training Engine"
        direction TB
        LaunchRoot["launch.py"]:::tools
        D2Launch["detectron2.engine.launch"]:::core
        TrainLoop["train_loop"]:::core
        Solver["Optimizer & LR Scheduler"]:::core
        Hooks["Hooks (logging, checkpoint)"]:::core
    end

    %% Evaluation & Export
    subgraph "Evaluation & Export"
        direction TB
        Evaluators["Evaluators (COCO, LVIS, SemSeg)"]:::core
        Export["Export (TorchScript, ONNX, Caffe2)"]:::core
    end

    %% Tools
    subgraph "Utilities & Scripts"
        direction TB
        ConvertSwin["convert-pretrained-swin-model-to-d2.py"]:::tools
        Logger["logger.py"]:::tools
        EvalScript["evaluate_ap_fixed.py"]:::tools
    end

    %% Relationships
    GPU -->|provides compute| PyTorch
    PyTorch -->|used by| ConfigCore
    Datasets -->|registered by| DataCatalog
    ConfigCore -->|loads yaml| ConfigProj
    DataCatalog -->|used in| DataMapper
    DataMapper -->|applies| Augment
    Augment -->|samples| Sampler
    Sampler -->|batches| DataLoader
    DataLoader -->|feeds| Swin
    DataLoader -->|feeds| ViT
    Swin -->|features| DeformTransformer
    ViT -->|features| DeformTransformer
    DeformTransformer -->|fusion input| FlanT5
    FlanT5 -->|fused features| LangHead
    DeformTransformer -->|features| BoxHead
    DeformTransformer -->|features| MaskHead
    LaunchRoot -->|spawns| D2Launch
    D2Launch -->|calls| TrainLoop
    TrainLoop -->|optimizes via| Solver
    TrainLoop -->|invokes| Hooks
    TrainLoop -->|checkpoint| Evaluators
    TrainLoop -->|checkpoint| Export
    Evaluators -->|metrics| Logger
    Export -->|artifact| ConvertSwin
    EvalScript -->|runs| Evaluators

    %% Click Events
    click ConfigCore "https://github.com/foundationvision/generateu/tree/main/detectron2/config/"
    click ConfigProj "https://github.com/foundationvision/generateu/tree/main/projects/DDETRS/configs/"
    click DataCatalog "https://github.com/foundationvision/generateu/blob/main/detectron2/data/catalog.py"
    click DataMapper "https://github.com/foundationvision/generateu/blob/main/projects/DDETRS/ddetrs/data/custom_dataset_mapper.py"
    click Augment "https://github.com/foundationvision/generateu/blob/main/projects/DDETRS/ddetrs/data/custom_build_augmentation.py"
    click Swin "https://github.com/foundationvision/generateu/blob/main/projects/DDETRS/ddetrs/backbone/swin.py"
    click ViT "https://github.com/foundationvision/generateu/blob/main/projects/DDETRS/ddetrs/backbone/vit.py"
    click DeformTransformer "https://github.com/foundationvision/generateu/blob/main/projects/DDETRS/ddetrs/models/deformable_detr/deformable_transformer.py"
    click FlanT5 "https://github.com/foundationvision/generateu/blob/main/projects/DDETRS/ddetrs/models/text/modeling_t5.py"
    click BoxHead "https://github.com/foundationvision/generateu/blob/main/detectron2/modeling/roi_heads/box_head.py"
    click MaskHead "https://github.com/foundationvision/generateu/blob/main/detectron2/modeling/roi_heads/mask_head.py"
    click LaunchRoot "https://github.com/foundationvision/generateu/blob/main/launch.py"
    click D2Launch "https://github.com/foundationvision/generateu/blob/main/detectron2/engine/launch.py"
    click TrainLoop "https://github.com/foundationvision/generateu/blob/main/detectron2/engine/train_loop.py"
    click Solver "https://github.com/foundationvision/generateu/blob/main/detectron2/solver/build.py"
    click Evaluators "https://github.com/foundationvision/generateu/tree/main/detectron2/evaluation/"
    click Export "https://github.com/foundationvision/generateu/tree/main/detectron2/export/"
    click ConvertSwin "https://github.com/foundationvision/generateu/blob/main/tools/convert-pretrained-swin-model-to-d2.py"

    %% Styles
    classDef core fill:#cfe2f3,stroke:#0b5394,stroke-width:1px
    classDef proj fill:#d9ead3,stroke:#38761d,stroke-width:1px
    classDef tools fill:#fce5cd,stroke:#b45f06,stroke-width:1px
    classDef external fill:#ececec,stroke:#666666,stroke-width:1px
```
