# fredzzhang/pvic architecture

> Architecture diagram of the GitHub repository fredzzhang/pvic, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fredzzhang/pvic
- Repository: https://github.com/fredzzhang/pvic

## Overview

##### Project type and purpose
- This is a research‐oriented PyTorch codebase (“PViC”) for Human–Object Interaction detection, built on top of DETR.
- It provides data‐loading, model definition (DETR + predicate visual context enhancements), training, evaluation and inference/visualization utilities.

##### File structure
Top‐level modules and their roles:
- Entry points
– main.py: training and evaluation driver
– inference.py: single‐image inference + visualization
- Configuration
– configs.py: central CLI/hyperparameter parsing
- Model code
– transformers.py: DETR Transformer definitions
– attention.py & attn.py: custom cross‐attention modules for PViC
– pvic.py: high‐level integration of predicate visual context into the detector
– ops.py: CUDA operators (via submodule h_detr)
- Data submodules
– hicodet/: HICO‐DET dataset download & loader
– vcoco/: V-COCO dataset download & loader
- External submodules
– h_detr/: fork of Deformable DETR with custom CUDA ops
– detr/: core DETR implementation
– pocket/: helper library installed in editable mode
- Utilities
– utils.py: I/O, logging, metric computation
– assets/: sample images and attention‐map examples
- Docs & configs
– docs.md: training/testing commands
– README.md, LICENSE, .gitignore, .gitmodules

##### README for additional insights
- Architecture is DETR + enhanced cross‐attention for predicate context
- Relies on PyTorch, torchvision, timm, wandb for logging, NumPy, SciPy, Matplotlib, Seaborn
- Data preparation scripts under hicodet/ and vcoco/ subdirs

a. Main components
1. Data Sources (HICO-DET, V-COCO)
2. Data Loader & Preprocessor (hicodet/, vcoco/, utils.py)
3. Model Definition
– Backbone Transformer (detr/, h_detr/)
– PViC Module (attention.py, attn.py, pvic.py)
4. Training Orchestrator (main.py + configs.py)
5. Loss & Optimization (built into main.py using torch.optim + criterion modules)
6. Logging & Visualization (wandb, matplotlib in utils.py + inference.py)
7. Inference Engine (inference.py + model checkpoint loading)
8. External Helpers (pocket/, ops CUDA kernels)
b. Relationships and data flow
– → Model (Backbone → PViC cross‐attention → HOI head)
– → Loss computation → Optimizer update → Model weights
– → Checkpoint saved → Inference script loads checkpoint → Visualization outputs
c. Architectural patterns / design principles
– Modular separation: data ingestion, model logic, training/inference, utilities
– Submodule reuse: h_detr and detr provide core transformer ops; pocket offers shared helper functions
– Configuration‐driven: configs.py centralizes CLI flags for reproducibility
– Extensibility: new attention modules (attention.py) plug into transformer architecture
d. Key technologies & frameworks
– PyTorch + torchvision (modeling & data pipelines)
– timm (image backbones if used)
– wandb (experiment logging)
– CUDA kernels via h_detr/models/ops
– Matplotlib / Seaborn (visualization)
– NumPy / SciPy (data ops)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources
    subgraph "Data Sources"
        direction TB
        DS1[(HICO-DET Dataset)]:::data
        DS2[(V-COCO Dataset)]:::data
    end

    %% Data Loader & Preprocessor
    subgraph "Data Loader & Preprocessor"
        direction TB
        DL1["hicodet/"]:::code
        DL2["vcoco/"]:::code
        DL3["utils.py"]:::code
    end

    %% Model Stack
    subgraph "Model Stack"
        direction TB
        subgraph "Core DETR Backbone"
            direction TB
            M1["detr/"]:::code
            M2["h_detr/"]:::code
            M3["transformers.py"]:::code
        end
        subgraph "PViC Cross-Attention Modules"
            direction TB
            M4["attention.py"]:::code
            M5["attn.py"]:::code
            M6["pvic.py"]:::code
        end
        M7["ops.py"]:::code
    end

    %% Training Orchestrator & Configuration
    subgraph "Training Orchestrator & Config"
        direction TB
        T1(["main.py"]):::process
        T2["configs.py"]:::code
    end

    %% Loss & Optimization
    LO(["Loss Computation & Optimizer"]):::process

    %% Checkpointing
    CP(["Model Checkpoint"]):::data

    %% Logging & Visualization
    subgraph "Logging & Visualization"
        direction TB
        V1(["wandb (Cloud)"]):::external
        V2["utils.py"]:::code
    end

    %% Inference Engine
    subgraph "Inference Engine"
        direction TB
        I1(["inference.py"]):::process
        I2["utils.py"]:::code
    end

    %% External Helpers
    subgraph "External Helpers & Submodules"
        direction TB
        E1["pocket/"]:::code
        E2["detr/"]:::code
        E3["h_detr/"]:::code
    end

    %% Data Flow Arrows
    DS1 -->|images| DL1
    DS2 -->|images| DL2
    DL1 -->|tensors| M1
    DL2 -->|tensors| M1
    DL3 -->|preprocessing| DL1
    DL3 -->|preprocessing| DL2

    M1 -->|features| M4
    M2 -->|features| M4
    M3 -->|architecture| M4
    M4 -->|contextual features| M6
    M5 -->|attention maps| M6
    M6 -->|enhanced tensors| M7

    M7 -->|model ops| T1

    T1 -->|backbone & PViC| M1
    T1 -->|compute loss| LO
    T2 -.->|hyperparams| DL1
    T2 -.->|hyperparams| M1
    T2 -.->|hyperparams| T1

    LO -->|gradients| LO
    LO -->|updates| T1
    T1 -->|save checkpoint| CP

    T1 -->|log metrics| V1
    V2 -->|logging & metrics| T1

    CP -->|load model| I1
    I1 -->|inference| I2
    I2 -->|visualizations| V1

    %% Configuration hooks
    T2 -.->|configs| DL3
    T2 -.->|configs| M4

    %% Click Events
    click DS1 "https://github.com/fredzzhang/pvic/tree/main/hicodet/"
    click DS2 "https://github.com/fredzzhang/pvic/tree/main/vcoco/"
    click DL1 "https://github.com/fredzzhang/pvic/tree/main/hicodet/"
    click DL2 "https://github.com/fredzzhang/pvic/tree/main/vcoco/"
    click DL3 "https://github.com/fredzzhang/pvic/blob/main/utils.py"
    click M1 "https://github.com/fredzzhang/pvic/tree/main/detr/"
    click M2 "https://github.com/fredzzhang/pvic/tree/main/h_detr/"
    click M3 "https://github.com/fredzzhang/pvic/blob/main/transformers.py"
    click M4 "https://github.com/fredzzhang/pvic/blob/main/attention.py"
    click M5 "https://github.com/fredzzhang/pvic/blob/main/attn.py"
    click M6 "https://github.com/fredzzhang/pvic/blob/main/pvic.py"
    click M7 "https://github.com/fredzzhang/pvic/blob/main/ops.py"
    click T1 "https://github.com/fredzzhang/pvic/blob/main/main.py"
    click T2 "https://github.com/fredzzhang/pvic/blob/main/configs.py"
    click V2 "https://github.com/fredzzhang/pvic/blob/main/utils.py"
    click I1 "https://github.com/fredzzhang/pvic/blob/main/inference.py"
    click I2 "https://github.com/fredzzhang/pvic/blob/main/utils.py"
    click E1 "https://github.com/fredzzhang/pvic/tree/main/pocket/"
    click E2 "https://github.com/fredzzhang/pvic/tree/main/detr/"
    click E3 "https://github.com/fredzzhang/pvic/tree/main/h_detr/"

    %% Styles
    classDef data fill:#FFCC00,stroke:#333,stroke-width:1px
    classDef code fill:#00CCFF,stroke:#333,stroke-width:1px
    classDef process fill:#00FF99,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
    classDef external fill:#CCCCCC,stroke:#333,stroke-width:1px,stroke-dasharray: 2 2
```
