How foundationvision/generateu works
Project Type and Purpose
• GenerateU is an experimental research framework for generative open-ended object detection, built on top of the Detectron2 library. It integrates a Deformable DETR–based detection backbone (DDETRS project) with language modeling components to enable region-language pretraining without fixed category labels.
File-Structure Analysis
• Top-level:
• detectron2/ ← Core detection library (forked and extended)
• projects/DDETRS/ ← Project-specific code for Deformable DETR + VL pretraining
• tools/ ← Helper scripts (e.g., weight conversion, evaluation)
• launch.py ← Distributed training entrypoint
• setup.py, requirements.txt
• detectron2/ subpackages:
• config/ ← Configuration system (defaults, instantiation)
• data/ ← Dataset catalog, mappers, samplers, transforms
• modeling/ ← Backbones, meta-architectures, heads, proposal generators
• engine/ ← Training loop, hooks, multi-GPU launcher
• export/ ← ONNX / TorchScript / Caffe2 export utilities
• solver/, evaluation/, utils/, layers/, etc.
• projects/DDETRS/:
• cfgs/ ← YAML config files for VG/LVIS experiments
• ddetrs/ ← Custom backbone (Swin, ViT), data augmentation, model code
• train_net.py ← Training script wrapper for the DDETRS project
Key Components
a. External Dependencies
• GPU Cluster (multi-node, A100)
• PyTorch, CUDA, TorchVision, MMCV, Detectron2
• Pretrained weights (Swin, Visual Genome, LVIS embeddings)
• Datasets (VisualGenome, LVIS, optional GrIT-20M)
b. Data Ingestion & Preprocessing
• Dataset Catalog → Dataset Mapper → Augmentations → Sampler → DataLoader
c. Model Architecture
• Backbone: Swin-Transformer / ViT
• Deformable-DETR Transformer
• Region-Language Fusion: FlanT5 text encoder integration
d. Training Engine
• Detectron2 Engine: train_loop, hooks (logging, checkpointing)
• Solver: LR scheduler, optimizer
• Distributed Launcher (launch.py)
e. Evaluation & Export
• Evaluators (COCO, LVIS, semantic segmentation)
• Export modules (TorchScript, ONNX, Caffe2)
Relationships and Data Flow
• Data sources → Data config & registration → DataLoader → Model input
• Model forward pass → Compute detection + language losses → Engine backprop → Optimizer
• Checkpointing → Optional export → Evaluation modules consume checkpoints → Metrics
Architectural Patterns and Principles
• Modular plugin pattern (Detectron2’s registry-based design)
• Separation of concerns:
• “detectron2/” provides core abstractions (config, data, modeling, engine)
• “projects/DDETRS/” plugs into these abstractions for custom tasks
• Distributed data-parallel training via launch script
Technologies and Frameworks
• PyTorch, CUDA/C++ custom ops (ms_deform_attn), TorchScript, Caffe2 export
• Detectron2’s config system (YAML + lazy instantiation)
• Swin/ViT backbones, Deformable DETR, FlanT5 text encoder