How chelsielei/hola works
1. This repository is a research codebase for zero-shot human-object interaction detection, centered on the HOLa model. The architecture is split into three major layers: vision-language feature adaptation, HOI reasoning, and dataset/evaluation plumbing for HICO-DET and V-COCO.
2. The core model lives in the top-level Python modules: hola_model.py, interaction_head.py, transformer_module.py, and ops.py. These implement the HOI-specific prediction path that takes image features plus person/object context and outputs interaction classes. main_tip_finetune.py and utils_tip_cache_and_union_finetune.py suggest the main training recipe is fine-tuning with cached features and union-region reasoning.
3. The CLIP subsystem is vendored locally under CLIP/ and is a key architectural dependency. It provides the image-text encoder, tokenizer, and model utilities used for feature adaptation. The README explicitly requires this local package to override any installed clip package, which means language features are a first-class input to the model rather than an optional add-on.
4. The DETR subsystem is vendored under detr/ and supplies the object-detection backbone and transformer infrastructure. It contains the DETR model, matcher, segmentation support, backbone, position encoding, and dataset mappers. In practice, this layer is the region proposal / object detection foundation that HOLa builds on before HOI interaction prediction.
5. The HOI task logic is dataset-aware and split between HICO-DET and V-COCO wrappers. hicodet/ and vcoco/ each provide dataset classes, annotation JSONs, detection preprocessing, visualization, and utility scripts. These directories are the boundary between generic model code and dataset-specific label/annotation conventions.
6. Text supervision and zero-shot semantics are stored as separate assets, not embedded in the model code. Examples include hico_text_label.py, vcoco_text_label.py, hico_list.py, vcoco_list.py, and the hoi_txtdescrip/ folder with hand-written or LLM-generated HOI descriptions. These resources feed the text side of CLIP-style adaptation and define the action vocabulary used for evaluation.
7. Data flow at runtime is: dataset image + annotations → detection backbone/DETR → region and union features → HOI interaction head → class scores for human-object pairs. On top of that, CLIP-derived text embeddings provide semantic priors or prompts for zero-shot transfer, especially for unseen HOI categories.
8. The repository includes dataset preprocessing and detection-generation stages as their own pipeline step. Under hicodet/detections/ and vcoco/detections/, scripts generate detections, preprocess them, evaluate them, and visualize results. This indicates the model is not operating directly on raw images alone; it relies on detector outputs as an intermediate artifact.
9. Evaluation is dataset-specific and script-driven. Top-level helpers like eval_vcoco.py, inference.py, tools.py, test_all.py, and the shell scripts in scripts/ orchestrate training, testing, and metric computation. The README’s hico_vitB.sh, hico_vitL.sh, and vcoco.sh are the main execution entrypoints for the full pipeline.
10. There is a clear dependency boundary around external infrastructure: pretrained checkpoints are expected from outside the repo, datasets are downloaded separately, and the project depends on modified versions of external libraries such as pocket in addition to the vendored CLIP and DETR code. This makes the repo more of an integrated research pipeline than a standalone library.
11. Supporting utilities are concentrated in utils/, detr/util/, and hicodet/utilities / vcoco/utilities. These handle box math, distributed/training helpers, sampling, scheduling, and visualization. They are not core model logic, but they are important because they connect training, evaluation, and qualitative inspection across the whole system.
12. For an architecture diagram, the most important boundaries are: input datasets and annotations, detector backbone, CLIP text/image encoder, HOI interaction head, and dataset-specific evaluation tooling. The primary cross-cutting concept is zero-shot transfer via text descriptions, which links the CLIP assets and HOI label resources to the detection-and-reasoning stack.
Main entry points: hola_model.py HOLa model · detr.py DETR model · vcoco.py V-COCO · main_tip_finetune.py Fine-tune entry
Components 22
HOLa Core
Model and training code for zero-shot HOI reasoning.
- HOLa model · Top-level model assembly for vision-language adaptation and HOI prediction.
hola_model.py - Interaction head · Predicts interaction classes for human-object pairs.
interaction_head.py - Transformer · Attention-based reasoning and feature fusion inside the HOI stack.
transformer_module.py - Ops · Core tensor and box operations used by model components.
ops.py
Feature Backbones
Vendored CLIP and DETR subsystems used for semantics and region features.
- CLIP · Vendored CLIP package providing image-text encoders and tokenization.
CLIP/clip/clip.py - CLIP model · Core CLIP model implementation and surgery variants.
CLIP/clip/model.py - DETR model · Region proposal and detection transformer foundation.
detr/models/detr.py - DETR backbone · Feature extraction backbone and positional encoding for detection.
detr/models/backbone.py - DETR transformer · Transformer blocks used by DETR and its HOI-adjacent detection stack.
detr/models/transformer.py
Task Data
Dataset wrappers, labels, and HOI text assets for HICO-DET and V-COCO.
- HICO-DET · Dataset loader and metadata for HICO-DET.
hicodet/hicodet.py - V-COCO · Dataset loader and metadata for V-COCO.
vcoco/vcoco.py - HICO labels · HICO-DET HOI label and text mapping resources.
hico_text_label.py - V-COCO labels · V-COCO HOI label and text mapping resources.
vcoco_text_label.py - HOI descriptions · Human-written and generated HOI descriptions for zero-shot semantics.
hoi_txtdescrip - HICO detections · Detection generation, preprocessing, evaluation, and visualization for HICO-DET.
hicodet/detections - V-COCO detections · Detection preprocessing and visualization for V-COCO.
vcoco/detections
Pipeline Tools
Entrypoints, scripts, and evaluation helpers that orchestrate experiments.
- Fine-tune entry · Main training recipe for cached-feature and union-region fine-tuning.
main_tip_finetune.py - Cache union utils · Training support for cached features and union-region reasoning.
utils_tip_cache_and_union_finetune.py - Inference · Runs model inference over dataset splits.
inference.py - V-COCO eval · Dataset-specific evaluation entrypoint for V-COCO.
eval_vcoco.py - Tools · Shared experiment and evaluation helpers.
tools.py - Scripts · Top-level shell launchers for the main HICO-DET and V-COCO runs.
scripts
Connections 23
- Fine-tune entry → HOLa model: trains · Fine-tuning entrypoint builds and optimizes the HOLa model.
- Fine-tune entry → Cache union utils: uses · Training recipe relies on cached features and union-region utilities.
- HOLa model → CLIP: uses · HOLa consumes CLIP image-text features for semantic adaptation.
- HOLa model → DETR model: uses · HOLa builds on DETR detection outputs and transformer features.
- HOLa model → Interaction head: predicts via · Model routes pair features into the HOI interaction head.
- HOLa model → Transformer: fuses via · Transformer module performs cross-modal and contextual reasoning.
- Interaction head → Ops: depends on · Prediction head uses shared math and box operations.
- Transformer → DETR transformer: aligns with · HOI reasoning reuses transformer-style feature processing patterns.
- CLIP → CLIP model: contains · Local CLIP package includes encoder and model code.
- DETR model → DETR backbone: built from · DETR detector is composed of backbone and transformer features.
- DETR model → DETR transformer: built from · DETR detector uses transformer blocks for detection.
- HICO-DET → HICO labels: maps labels · HICO-DET dataset wrapper aligns annotations with HOI text labels.
- HICO-DET → HICO detections: feeds · Dataset uses detection artifacts and preprocessing pipeline.
- V-COCO → V-COCO labels: maps labels · V-COCO dataset wrapper aligns annotations with HOI text labels.
- V-COCO → V-COCO detections: feeds · Dataset uses detection artifacts and preprocessing pipeline.
- HICO labels → HOI descriptions: augments · Label resources are expanded with textual HOI descriptions.
- V-COCO labels → HOI descriptions: augments · Label resources are expanded with textual HOI descriptions.
- Inference → HOLa model: runs · Inference loads the HOLa model for prediction.
- V-COCO eval → V-COCO: evaluates · V-COCO evaluation uses dataset-specific annotations and metrics.
- Tools → Inference: orchestrates · Shared tooling coordinates inference and evaluation flows.
- Tools → V-COCO eval: orchestrates · Shared tooling coordinates dataset-specific evaluation.
- Scripts → Fine-tune entry: launches · Shell scripts serve as primary experiment entrypoints.
- Scripts → Inference: launches · Shell scripts can launch inference and testing workflows.