# ruotianluo/context-aware-zsr architecture

> Architecture diagram of the GitHub repository ruotianluo/context-aware-zsr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ruotianluo/context-aware-zsr
- Repository: https://github.com/ruotianluo/context-aware-zsr

## Overview

##### Project type and purpose
- This is a specialized full-stack machine-learning pipeline for object detection and context-aware zero-shot recognition built on PyTorch and heavily based on Detectron.pytorch.
- It handles dataset preparation (Visual Genome “vg” and VOC), external proposal generation (EdgeBoxes), multiple training heads (Fast R-CNN, Mask R-CNN, Keypoint RCNN), zero-shot enhancements (word embeddings, GCN, Sync), and inference/testing.

##### File structure
- Top-level folders:
- configs/ – YAML config files for all baselines and custom zero-shot settings
- data/ – raw and processed datasets (vg images, annotations, proposals)
- externals/edges – MATLAB/C++ code for EdgeBoxes proposal generation
- lib/ – core Python/C++/CUDA libraries (configuration, dataset loaders, model builder, ROI layers, nn modules, utilities, zsl extensions)
- tools/ – high-level Python entry points (train_net.py, test_net.py, download scripts)
- scripts/ – bash wrappers orchestrating training/testing variants
- C++/CUDA extension sources under lib/model/csrc and lib/modeling/roi_xfrom (for ROIAlign, NMS, ROI crop).

##### README for additional insights
- Describes zero-shot paper, required dependencies (PyTorch, torchtext, nltk, pycocotools), data setup, build steps (`bash lib/make.sh`), pretrained weights download, and multiple training/testing flows (standard, WE, GCN, Sync).

##### Core components & their interactions
a. Configuration Layer
- configs/*.yaml + CLI overrides → feed into Tools & Scripts
b. Data Layer
- data/vg (images, annotations, proposals)
- externals/edges → raw proposals → data/vg/bansal
- lib/datasets (catalogs, JSON evaluators, VOC wrapper)
c. Core Library & Compute
- lib/core (training/test engine, config parser, RPN generator)
- lib/modeling (FPN, backbones, heads, proposal & label gen)
- lib/roi_data (batching, data loader)
- lib/nn (functional/modules, data-parallel)
- lib/utils (I/O, image ops, logging, pretrained-weights helper)
- lib/zsl (word embedding, GCN, Sync converters)
d. C++/CUDA Extensions
- lib/model/csrc*, lib/modeling/roi_xfrom/* for compute-intensive ops
e. Tools & Scripts
- tools/train_net.py → lib/core/train
- tools/test_net.py → lib/core/test
- scripts/*.sh – orchestrate configs + tools
f. External Artifacts & Services
- Pretrained weights (ImageNet, paper checkpoints) via Google Drive
- Python deps: torchtext, nltk, pycocotools

- Config YAML + CLI → tools/train_net.py
- train_net.py → lib/core/train →
– Builds model with lib/modeling/model_builder
– Registers kernels from C++/CUDA extensions
– Spins up DataLoader via lib/roi_data + lib/datasets
– Feeds minibatches to model → lib/nn for forward/backward
– Saves checkpoints (Outputs/…)
- For inference: scripts → tools/test_net.py → loads checkpoint → lib/core/test → same data pipeline → outputs detections → evaluated by lib/datasets/*_evaluator

##### Architectural patterns & principles
- Layered modularization: Config → Data → Model → Compute
- Plug-in extensibility: new heads (GCN, WE, iterative), new datasets
- Separation: Python control vs. low-level C++/CUDA kernels
- Config-driven design inherited from Detectron

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Layer
    subgraph "Configuration Layer"
        CFG["YAML Configs + CLI Overrides"]:::config
    end

    %% Data Layer
    subgraph "Data Layer"
        VG["Visual Genome Data (data/vg)"]:::data
        VOC_WRAP["VOC Wrapper & Evaluator"]:::data
        EXT_EDGES["EdgeBoxes Code"]:::data
        PROPOSAL_SCRIPT["to_raw_proposals.py"]:::data
    end

    %% Tools & Scripts
    subgraph "Tools & Scripts"
        TRAIN["train_net.py"]:::tools
        TEST["test_net.py"]:::tools
        SCRIPT["scripts/*.sh"]:::tools
    end

    %% Core Library
    subgraph "Core Library"
        CONFIG_P["config.py"]:::core
        CORE_TRAIN["train engine"]:::core
        CORE_TEST["test engine"]:::core
        RPN["rpn_generator.py"]:::core

        subgraph "Modeling"
            ALGO["model_builder.py"]:::core
            FAST_H["fast_rcnn_heads.py"]:::core
            MASK_H["mask_rcnn_heads.py"]:::core
            KP_H["keypoint_rcnn_heads.py"]:::core
            REL_H["rel_heads.py"]:::core
            RESNET["ResNet.py"]:::core
            VGG["VGG16.py"]:::core
            FPN["FPN.py"]:::core
            PROP_GEN["generate_proposals.py"]:::core
            PROP_LABEL["generate_proposal_labels.py"]:::core
            ROI_XFORM["roi_align"]:::core
        end

        DATA_LOADER["lib/roi_data"]:::core
        NN["lib/nn"]:::core
        UTILS["lib/utils"]:::core
        ZSL_GCN["convert_gcn.py"]:::core
        ZSL_SYNC["convert_sync.py"]:::core
    end

    %% C++/CUDA Extensions
    subgraph "C++/CUDA Extensions"
        CSRC["lib/model/csrc"]:::ext
        ROI_LAYERS["lib/model/roi_layers"]:::ext
        BUILD1["lib/make.sh"]:::ext
        BUILD2["lib/model/roi_crop/make.sh"]:::ext
    end

    %% External Artifacts
    PRE_WEIGHTS(("Pretrained Weights")):::external
    CHECKPOINT(("Checkpoints")):::external

    %% Connections
    CFG -->|YAML config| TRAIN
    CFG -->|YAML config| TEST
    CFG -->|YAML config| SCRIPT

    SCRIPT --> TRAIN
    SCRIPT --> TEST

    TRAIN -->|start| CORE_TRAIN
    TEST -->|start| CORE_TEST

    EXT_EDGES -->|raw proposals| VG
    PROPOSAL_SCRIPT --> VG

    VG --> DATA_LOADER
    VOC_WRAP --> DATA_LOADER
    DATA_LOADER --> CORE_TRAIN
    DATA_LOADER --> CORE_TEST

    CORE_TRAIN --> RPN
    CORE_TRAIN --> DATA_LOADER
    CORE_TRAIN --> NN
    CORE_TRAIN --> UTILS
    CORE_TRAIN --> ZSL_GCN
    CORE_TRAIN --> ZSL_SYNC
    CORE_TRAIN --> ALGO

    CORE_TEST --> UTILS
    CORE_TEST --> ALGO

    ALGO --> FAST_H
    ALGO --> MASK_H
    ALGO --> KP_H
    ALGO --> REL_H
    ALGO --> RESNET
    ALGO --> VGG
    ALGO --> FPN
    ALGO --> PROP_GEN
    ALGO --> PROP_LABEL
    ALGO --> ROI_XFORM

    ROI_XFORM --> CSRC
    ROI_XFORM --> ROI_LAYERS

    BUILD1 --> CSRC
    BUILD1 --> ROI_LAYERS
    BUILD1 --> ROI_XFORM
    BUILD2 --> ROI_LAYERS
    BUILD2 --> ROI_XFORM

    CORE_TRAIN --> CHECKPOINT
    CORE_TEST --> CHECKPOINT
    UTILS --> PRE_WEIGHTS

    %% Click Events
    click CFG "https://github.com/ruotianluo/context-aware-zsr/tree/master/configs/"
    click VG "https://github.com/ruotianluo/context-aware-zsr/tree/master/data/vg/"
    click VOC_WRAP "https://github.com/ruotianluo/context-aware-zsr/tree/master/lib/datasets/VOCdevkit-matlab-wrapper/"
    click EXT_EDGES "https://github.com/ruotianluo/context-aware-zsr/tree/master/externals/edges/"
    click PROPOSAL_SCRIPT "https://github.com/ruotianluo/context-aware-zsr/blob/master/externals/edges/to_raw_proposals.py"
    click CONFIG_P "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/core/config.py"
    click CORE_TRAIN "https://github.com/ruotianluo/context-aware-zsr/tree/master/lib/core/"
    click CORE_TEST "https://github.com/ruotianluo/context-aware-zsr/tree/master/lib/core/"
    click RPN "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/core/rpn_generator.py"
    click ALGO "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/modeling/model_builder.py"
    click FAST_H "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/modeling/fast_rcnn_heads.py"
    click MASK_H "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/modeling/mask_rcnn_heads.py"
    click KP_H "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/modeling/keypoint_rcnn_heads.py"
    click REL_H "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/modeling/rel_heads.py"
    click RESNET "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/modeling/ResNet.py"
    click VGG "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/modeling/VGG16.py"
    click FPN "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/modeling/FPN.py"
    click PROP_GEN "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/modeling/generate_proposals.py"
    click PROP_LABEL "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/modeling/generate_proposal_labels.py"
    click ROI_XFORM "https://github.com/ruotianluo/context-aware-zsr/tree/master/lib/modeling/roi_xfrom/roi_align/"
    click CSRC "https://github.com/ruotianluo/context-aware-zsr/tree/master/lib/model/csrc/"
    click ROI_LAYERS "https://github.com/ruotianluo/context-aware-zsr/tree/master/lib/model/roi_layers/"
    click BUILD1 "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/make.sh"
    click BUILD2 "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/model/roi_crop/make.sh"
    click DATA_LOADER "https://github.com/ruotianluo/context-aware-zsr/tree/master/lib/roi_data/"
    click NN "https://github.com/ruotianluo/context-aware-zsr/tree/master/lib/nn/"
    click UTILS "https://github.com/ruotianluo/context-aware-zsr/tree/master/lib/utils/"
    click ZSL_GCN "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/zsl/convert_gcn.py"
    click ZSL_SYNC "https://github.com/ruotianluo/context-aware-zsr/blob/master/lib/zsl/convert_sync.py"
    click TRAIN "https://github.com/ruotianluo/context-aware-zsr/blob/master/tools/train_net.py"
    click TEST "https://github.com/ruotianluo/context-aware-zsr/blob/master/tools/test_net.py"
    click SCRIPT "https://github.com/ruotianluo/context-aware-zsr/tree/master/scripts/"
    
    %% Styles
    classDef config fill:#B0E0E6
    classDef data fill:#ADD8E6
    classDef core fill:#90EE90
    classDef ext fill:#FFA500
    classDef tools fill:#FFFF00
    classDef external fill:#D3D3D3
    class PRE_WEIGHTS,CHECKPOINT external
```
