How fredzzhang/pvic works
Project type and purpose
• This is a research‐oriented PyTorch codebase (“PViC”) for Human–Object Interaction detection, built on top of DETR.
• It provides data‐loading, model definition (DETR + predicate visual context enhancements), training, evaluation and inference/visualization utilities.
File structure
Top‐level modules and their roles:
• Entry points
– main.py: training and evaluation driver
– inference.py: single‐image inference + visualization
• Configuration
– configs.py: central CLI/hyperparameter parsing
• Model code
– transformers.py: DETR Transformer definitions
– attention.py & attn.py: custom cross‐attention modules for PViC
– pvic.py: high‐level integration of predicate visual context into the detector
– ops.py: CUDA operators (via submodule h_detr)
• Data submodules
– hicodet/: HICO‐DET dataset download & loader
– vcoco/: V-COCO dataset download & loader
• External submodules
– h_detr/: fork of Deformable DETR with custom CUDA ops
– detr/: core DETR implementation
– pocket/: helper library installed in editable mode
• Utilities
– utils.py: I/O, logging, metric computation
– assets/: sample images and attention‐map examples
• Docs & configs
– docs.md: training/testing commands
– README.md, LICENSE, .gitignore, .gitmodules
README for additional insights
• Architecture is DETR + enhanced cross‐attention for predicate context
• Relies on PyTorch, torchvision, timm, wandb for logging, NumPy, SciPy, Matplotlib, Seaborn
• Data preparation scripts under hicodet/ and vcoco/ subdirs
a. Main components
1. Data Sources (HICO-DET, V-COCO)
2. Data Loader & Preprocessor (hicodet/, vcoco/, utils.py)
3. Model Definition
– Backbone Transformer (detr/, h_detr/)
– PViC Module (attention.py, attn.py, pvic.py)
4. Training Orchestrator (main.py + configs.py)
5. Loss & Optimization (built into main.py using torch.optim + criterion modules)
6. Logging & Visualization (wandb, matplotlib in utils.py + inference.py)
7. Inference Engine (inference.py + model checkpoint loading)
8. External Helpers (pocket/, ops CUDA kernels)
b. Relationships and data flow
– → Model (Backbone → PViC cross‐attention → HOI head)
– → Loss computation → Optimizer update → Model weights
– → Checkpoint saved → Inference script loads checkpoint → Visualization outputs
c. Architectural patterns / design principles
– Modular separation: data ingestion, model logic, training/inference, utilities
– Submodule reuse: h_detr and detr provide core transformer ops; pocket offers shared helper functions
– Configuration‐driven: configs.py centralizes CLI flags for reproducibility
– Extensibility: new attention modules (attention.py) plug into transformer architecture
d. Key technologies & frameworks
– PyTorch + torchvision (modeling & data pipelines)
– timm (image backbones if used)
– wandb (experiment logging)
– CUDA kernels via h_detr/models/ops
– Matplotlib / Seaborn (visualization)
– NumPy / SciPy (data ops)