How ruotianluo/context-aware-zsr works
Project type and purpose
• This is a specialized full-stack machine-learning pipeline for object detection and context-aware zero-shot recognition built on PyTorch and heavily based on Detectron.pytorch.
• It handles dataset preparation (Visual Genome “vg” and VOC), external proposal generation (EdgeBoxes), multiple training heads (Fast R-CNN, Mask R-CNN, Keypoint RCNN), zero-shot enhancements (word embeddings, GCN, Sync), and inference/testing.
File structure
• Top-level folders:
• configs/ – YAML config files for all baselines and custom zero-shot settings
• data/ – raw and processed datasets (vg images, annotations, proposals)
• externals/edges – MATLAB/C++ code for EdgeBoxes proposal generation
• lib/ – core Python/C++/CUDA libraries (configuration, dataset loaders, model builder, ROI layers, nn modules, utilities, zsl extensions)
• tools/ – high-level Python entry points (train_net.py, test_net.py, download scripts)
• scripts/ – bash wrappers orchestrating training/testing variants
• C++/CUDA extension sources under lib/model/csrc and lib/modeling/roi_xfrom (for ROIAlign, NMS, ROI crop).
README for additional insights
• Describes zero-shot paper, required dependencies (PyTorch, torchtext, nltk, pycocotools), data setup, build steps (bash lib/make.sh), pretrained weights download, and multiple training/testing flows (standard, WE, GCN, Sync).
Core components & their interactions
a. Configuration Layer
• configs/*.yaml + CLI overrides → feed into Tools & Scripts
b. Data Layer
• data/vg (images, annotations, proposals)
• externals/edges → raw proposals → data/vg/bansal
• lib/datasets (catalogs, JSON evaluators, VOC wrapper)
c. Core Library & Compute
• lib/core (training/test engine, config parser, RPN generator)
• lib/modeling (FPN, backbones, heads, proposal & label gen)
• lib/roi_data (batching, data loader)
• lib/nn (functional/modules, data-parallel)
• lib/utils (I/O, image ops, logging, pretrained-weights helper)
• lib/zsl (word embedding, GCN, Sync converters)
d. C++/CUDA Extensions
• lib/model/csrc*, lib/modeling/roi_xfrom/* for compute-intensive ops
e. Tools & Scripts
• tools/train_net.py → lib/core/train
• tools/test_net.py → lib/core/test
• scripts/*.sh – orchestrate configs + tools
f. External Artifacts & Services
• Pretrained weights (ImageNet, paper checkpoints) via Google Drive
• Python deps: torchtext, nltk, pycocotools
• Config YAML + CLI → tools/train_net.py
• train_net.py → lib/core/train →
– Builds model with lib/modeling/model_builder
– Registers kernels from C++/CUDA extensions
– Spins up DataLoader via lib/roi_data + lib/datasets
– Feeds minibatches to model → lib/nn for forward/backward
– Saves checkpoints (Outputs/…)
• For inference: scripts → tools/test_net.py → loads checkpoint → lib/core/test → same data pipeline → outputs detections → evaluated by lib/datasets/*_evaluator
Architectural patterns & principles
• Layered modularization: Config → Data → Model → Compute
• Plug-in extensibility: new heads (GCN, WE, iterative), new datasets
• Separation: Python control vs. low-level C++/CUDA kernels
• Config-driven design inherited from Detectron