# chelsielei/hola architecture

> Architecture diagram of the GitHub repository chelsielei/hola, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chelsielei/hola
- Repository: https://github.com/chelsielei/hola

## Overview

1. This repository is a research codebase for zero-shot human-object interaction detection, centered on the HOLa model. The architecture is split into three major layers: vision-language feature adaptation, HOI reasoning, and dataset/evaluation plumbing for HICO-DET and V-COCO.

2. The core model lives in the top-level Python modules: `hola_model.py`, `interaction_head.py`, `transformer_module.py`, and `ops.py`. These implement the HOI-specific prediction path that takes image features plus person/object context and outputs interaction classes. `main_tip_finetune.py` and `utils_tip_cache_and_union_finetune.py` suggest the main training recipe is fine-tuning with cached features and union-region reasoning.

3. The CLIP subsystem is vendored locally under `CLIP/` and is a key architectural dependency. It provides the image-text encoder, tokenizer, and model utilities used for feature adaptation. The README explicitly requires this local package to override any installed `clip` package, which means language features are a first-class input to the model rather than an optional add-on.

4. The DETR subsystem is vendored under `detr/` and supplies the object-detection backbone and transformer infrastructure. It contains the DETR model, matcher, segmentation support, backbone, position encoding, and dataset mappers. In practice, this layer is the region proposal / object detection foundation that HOLa builds on before HOI interaction prediction.

5. The HOI task logic is dataset-aware and split between HICO-DET and V-COCO wrappers. `hicodet/` and `vcoco/` each provide dataset classes, annotation JSONs, detection preprocessing, visualization, and utility scripts. These directories are the boundary between generic model code and dataset-specific label/annotation conventions.

6. Text supervision and zero-shot semantics are stored as separate assets, not embedded in the model code. Examples include `hico_text_label.py`, `vcoco_text_label.py`, `hico_list.py`, `vcoco_list.py`, and the `hoi_txtdescrip/` folder with hand-written or LLM-generated HOI descriptions. These resources feed the text side of CLIP-style adaptation and define the action vocabulary used for evaluation.

7. Data flow at runtime is: dataset image + annotations → detection backbone/DETR → region and union features → HOI interaction head → class scores for human-object pairs. On top of that, CLIP-derived text embeddings provide semantic priors or prompts for zero-shot transfer, especially for unseen HOI categories.

8. The repository includes dataset preprocessing and detection-generation stages as their own pipeline step. Under `hicodet/detections/` and `vcoco/detections/`, scripts generate detections, preprocess them, evaluate them, and visualize results. This indicates the model is not operating directly on raw images alone; it relies on detector outputs as an intermediate artifact.

9. Evaluation is dataset-specific and script-driven. Top-level helpers like `eval_vcoco.py`, `inference.py`, `tools.py`, `test_all.py`, and the shell scripts in `scripts/` orchestrate training, testing, and metric computation. The README’s `hico_vitB.sh`, `hico_vitL.sh`, and `vcoco.sh` are the main execution entrypoints for the full pipeline.

10. There is a clear dependency boundary around external infrastructure: pretrained checkpoints are expected from outside the repo, datasets are downloaded separately, and the project depends on modified versions of external libraries such as `pocket` in addition to the vendored CLIP and DETR code. This makes the repo more of an integrated research pipeline than a standalone library.

11. Supporting utilities are concentrated in `utils/`, `detr/util/`, and `hicodet/utilities` / `vcoco/utilities`. These handle box math, distributed/training helpers, sampling, scheduling, and visualization. They are not core model logic, but they are important because they connect training, evaluation, and qualitative inspection across the whole system.

12. For an architecture diagram, the most important boundaries are: input datasets and annotations, detector backbone, CLIP text/image encoder, HOI interaction head, and dataset-specific evaluation tooling. The primary cross-cutting concept is zero-shot transfer via text descriptions, which links the CLIP assets and HOI label resources to the detection-and-reasoning stack.

Main entry points: [`hola_model.py`](https://github.com/chelsielei/hola/blob/master/hola_model.py) (HOLa model), [`detr/models/detr.py`](https://github.com/chelsielei/hola/blob/master/detr/models/detr.py) (DETR model), [`vcoco/vcoco.py`](https://github.com/chelsielei/hola/blob/master/vcoco/vcoco.py) (V-COCO), [`main_tip_finetune.py`](https://github.com/chelsielei/hola/blob/master/main_tip_finetune.py) (Fine-tune entry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["HOLa Core"]
  node_hola_model["HOLa model<br/>HOI model<br/>[hola_model.py]"]
  node_interaction_head["Interaction head<br/>HOI head"]
  node_transformer_module["Transformer<br/>fusion module"]
  node_ops_py["Ops<br/>math helpers<br/>[ops.py]"]
end

subgraph group_vision["Feature Backbones"]
  node_clip_pkg["CLIP<br/>vision-language lib<br/>[clip.py]"]
  node_clip_model["CLIP model<br/>encoder<br/>[model.py]"]
  node_detr_model["DETR model<br/>detector<br/>[detr.py]"]
  node_detr_backbone["DETR backbone<br/>vision backbone<br/>[backbone.py]"]
  node_detr_transformer["DETR transformer<br/>detector transformer<br/>[transformer.py]"]
end

subgraph group_data["Task Data"]
  node_hicodet_dataset[("HICO-DET<br/>dataset wrapper<br/>[hicodet.py]")]
  node_vcoco_dataset[("V-COCO<br/>dataset wrapper<br/>[vcoco.py]")]
  node_hico_text_label["HICO labels<br/>action vocab<br/>[hico_text_label.py]"]
  node_vcoco_text_label["V-COCO labels<br/>action vocab"]
  node_hoi_txtdescrip["HOI descriptions<br/>text prompts"]
  node_hicodet_detections["HICO detections<br/>detection pipeline"]
  node_vcoco_detections["V-COCO detections<br/>detection pipeline"]
end

subgraph group_ops["Pipeline Tools"]
  node_main_tip_finetune["Fine-tune entry<br/>train script"]
  node_utils_tip_cache_and_union_finetune["Cache union utils<br/>train utilities"]
  node_inference["Inference<br/>eval runner<br/>[inference.py]"]
  node_eval_vcoco["V-COCO eval<br/>dataset eval<br/>[eval_vcoco.py]"]
  node_tools_py["Tools<br/>orchestration<br/>[tools.py]"]
  node_scripts["Scripts<br/>shell entrypoints"]
end

node_main_tip_finetune -->|"trains"| node_hola_model
node_main_tip_finetune -->|"uses"| node_utils_tip_cache_and_union_finetune
node_hola_model -->|"uses"| node_clip_pkg
node_hola_model -->|"uses"| node_detr_model
node_hola_model -->|"predicts via"| node_interaction_head
node_hola_model -->|"fuses via"| node_transformer_module
node_interaction_head -->|"depends on"| node_ops_py
node_transformer_module -.->|"aligns with"| node_detr_transformer
node_clip_pkg -->|"contains"| node_clip_model
node_detr_model -->|"built from"| node_detr_backbone
node_detr_model -->|"built from"| node_detr_transformer
node_hicodet_dataset -->|"maps labels"| node_hico_text_label
node_hicodet_dataset -->|"feeds"| node_hicodet_detections
node_vcoco_dataset -->|"maps labels"| node_vcoco_text_label
node_vcoco_dataset -->|"feeds"| node_vcoco_detections
node_hico_text_label -->|"augments"| node_hoi_txtdescrip
node_vcoco_text_label -->|"augments"| node_hoi_txtdescrip
node_inference -->|"runs"| node_hola_model
node_eval_vcoco -->|"evaluates"| node_vcoco_dataset
node_tools_py -->|"orchestrates"| node_inference
node_tools_py -->|"orchestrates"| node_eval_vcoco
node_scripts -->|"launches"| node_main_tip_finetune
node_scripts -->|"launches"| node_inference

click node_hola_model "https://github.com/chelsielei/hola/blob/master/hola_model.py"
click node_interaction_head "https://github.com/chelsielei/hola/blob/master/interaction_head.py"
click node_transformer_module "https://github.com/chelsielei/hola/blob/master/transformer_module.py"
click node_ops_py "https://github.com/chelsielei/hola/blob/master/ops.py"
click node_main_tip_finetune "https://github.com/chelsielei/hola/blob/master/main_tip_finetune.py"
click node_utils_tip_cache_and_union_finetune "https://github.com/chelsielei/hola/blob/master/utils_tip_cache_and_union_finetune.py"
click node_inference "https://github.com/chelsielei/hola/blob/master/inference.py"
click node_eval_vcoco "https://github.com/chelsielei/hola/blob/master/eval_vcoco.py"
click node_tools_py "https://github.com/chelsielei/hola/blob/master/tools.py"
click node_scripts "https://github.com/chelsielei/hola/tree/master/scripts"
click node_clip_pkg "https://github.com/chelsielei/hola/blob/master/CLIP/clip/clip.py"
click node_clip_model "https://github.com/chelsielei/hola/blob/master/CLIP/clip/model.py"
click node_detr_model "https://github.com/chelsielei/hola/blob/master/detr/models/detr.py"
click node_detr_backbone "https://github.com/chelsielei/hola/blob/master/detr/models/backbone.py"
click node_detr_transformer "https://github.com/chelsielei/hola/blob/master/detr/models/transformer.py"
click node_hicodet_dataset "https://github.com/chelsielei/hola/blob/master/hicodet/hicodet.py"
click node_vcoco_dataset "https://github.com/chelsielei/hola/blob/master/vcoco/vcoco.py"
click node_hico_text_label "https://github.com/chelsielei/hola/blob/master/hico_text_label.py"
click node_vcoco_text_label "https://github.com/chelsielei/hola/blob/master/vcoco_text_label.py"
click node_hoi_txtdescrip "https://github.com/chelsielei/hola/tree/master/hoi_txtdescrip"
click node_hicodet_detections "https://github.com/chelsielei/hola/tree/master/hicodet/detections"
click node_vcoco_detections "https://github.com/chelsielei/hola/tree/master/vcoco/detections"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_hola_model,node_interaction_head,node_transformer_module,node_ops_py toneBlue
class node_clip_pkg,node_clip_model,node_detr_model,node_detr_backbone,node_detr_transformer toneAmber
class node_hicodet_dataset,node_vcoco_dataset,node_hico_text_label,node_vcoco_text_label,node_hoi_txtdescrip,node_hicodet_detections,node_vcoco_detections toneMint
class node_main_tip_finetune,node_utils_tip_cache_and_union_finetune,node_inference,node_eval_vcoco,node_tools_py,node_scripts toneRose
```

## Components

### HOLa Core

Model and training code for zero-shot HOI reasoning.

- [HOLa model](https://github.com/chelsielei/hola/blob/master/hola_model.py): HOI model · Top-level model assembly for vision-language adaptation and HOI prediction. · `hola_model.py`
- [Interaction head](https://github.com/chelsielei/hola/blob/master/interaction_head.py): HOI head · Predicts interaction classes for human-object pairs. · `interaction_head.py`
- [Transformer](https://github.com/chelsielei/hola/blob/master/transformer_module.py): fusion module · Attention-based reasoning and feature fusion inside the HOI stack. · `transformer_module.py`
- [Ops](https://github.com/chelsielei/hola/blob/master/ops.py): math helpers · Core tensor and box operations used by model components. · `ops.py`

### Feature Backbones

Vendored CLIP and DETR subsystems used for semantics and region features.

- [CLIP](https://github.com/chelsielei/hola/blob/master/CLIP/clip/clip.py): vision-language lib · Vendored CLIP package providing image-text encoders and tokenization. · `CLIP/clip/clip.py`
- [CLIP model](https://github.com/chelsielei/hola/blob/master/CLIP/clip/model.py): encoder · Core CLIP model implementation and surgery variants. · `CLIP/clip/model.py`
- [DETR model](https://github.com/chelsielei/hola/blob/master/detr/models/detr.py): detector · Region proposal and detection transformer foundation. · `detr/models/detr.py`
- [DETR backbone](https://github.com/chelsielei/hola/blob/master/detr/models/backbone.py): vision backbone · Feature extraction backbone and positional encoding for detection. · `detr/models/backbone.py`
- [DETR transformer](https://github.com/chelsielei/hola/blob/master/detr/models/transformer.py): detector transformer · Transformer blocks used by DETR and its HOI-adjacent detection stack. · `detr/models/transformer.py`

### Task Data

Dataset wrappers, labels, and HOI text assets for HICO-DET and V-COCO.

- [HICO-DET](https://github.com/chelsielei/hola/blob/master/hicodet/hicodet.py): dataset wrapper · Dataset loader and metadata for HICO-DET. · `hicodet/hicodet.py`
- [V-COCO](https://github.com/chelsielei/hola/blob/master/vcoco/vcoco.py): dataset wrapper · Dataset loader and metadata for V-COCO. · `vcoco/vcoco.py`
- [HICO labels](https://github.com/chelsielei/hola/blob/master/hico_text_label.py): action vocab · HICO-DET HOI label and text mapping resources. · `hico_text_label.py`
- [V-COCO labels](https://github.com/chelsielei/hola/blob/master/vcoco_text_label.py): action vocab · V-COCO HOI label and text mapping resources. · `vcoco_text_label.py`
- [HOI descriptions](https://github.com/chelsielei/hola/tree/master/hoi_txtdescrip): text prompts · Human-written and generated HOI descriptions for zero-shot semantics. · `hoi_txtdescrip`
- [HICO detections](https://github.com/chelsielei/hola/tree/master/hicodet/detections): detection pipeline · Detection generation, preprocessing, evaluation, and visualization for HICO-DET. · `hicodet/detections`
- [V-COCO detections](https://github.com/chelsielei/hola/tree/master/vcoco/detections): detection pipeline · Detection preprocessing and visualization for V-COCO. · `vcoco/detections`

### Pipeline Tools

Entrypoints, scripts, and evaluation helpers that orchestrate experiments.

- [Fine-tune entry](https://github.com/chelsielei/hola/blob/master/main_tip_finetune.py): train script · Main training recipe for cached-feature and union-region fine-tuning. · `main_tip_finetune.py`
- [Cache union utils](https://github.com/chelsielei/hola/blob/master/utils_tip_cache_and_union_finetune.py): train utilities · Training support for cached features and union-region reasoning. · `utils_tip_cache_and_union_finetune.py`
- [Inference](https://github.com/chelsielei/hola/blob/master/inference.py): eval runner · Runs model inference over dataset splits. · `inference.py`
- [V-COCO eval](https://github.com/chelsielei/hola/blob/master/eval_vcoco.py): dataset eval · Dataset-specific evaluation entrypoint for V-COCO. · `eval_vcoco.py`
- [Tools](https://github.com/chelsielei/hola/blob/master/tools.py): orchestration · Shared experiment and evaluation helpers. · `tools.py`
- [Scripts](https://github.com/chelsielei/hola/tree/master/scripts): shell entrypoints · Top-level shell launchers for the main HICO-DET and V-COCO runs. · `scripts`

## Connections

- Fine-tune entry → HOLa model: trains · Fine-tuning entrypoint builds and optimizes the HOLa model.
- Fine-tune entry → Cache union utils: uses · Training recipe relies on cached features and union-region utilities.
- HOLa model → CLIP: uses · HOLa consumes CLIP image-text features for semantic adaptation.
- HOLa model → DETR model: uses · HOLa builds on DETR detection outputs and transformer features.
- HOLa model → Interaction head: predicts via · Model routes pair features into the HOI interaction head.
- HOLa model → Transformer: fuses via · Transformer module performs cross-modal and contextual reasoning.
- Interaction head → Ops: depends on · Prediction head uses shared math and box operations.
- Transformer → DETR transformer: aligns with · HOI reasoning reuses transformer-style feature processing patterns.
- CLIP → CLIP model: contains · Local CLIP package includes encoder and model code.
- DETR model → DETR backbone: built from · DETR detector is composed of backbone and transformer features.
- DETR model → DETR transformer: built from · DETR detector uses transformer blocks for detection.
- HICO-DET → HICO labels: maps labels · HICO-DET dataset wrapper aligns annotations with HOI text labels.
- HICO-DET → HICO detections: feeds · Dataset uses detection artifacts and preprocessing pipeline.
- V-COCO → V-COCO labels: maps labels · V-COCO dataset wrapper aligns annotations with HOI text labels.
- V-COCO → V-COCO detections: feeds · Dataset uses detection artifacts and preprocessing pipeline.
- HICO labels → HOI descriptions: augments · Label resources are expanded with textual HOI descriptions.
- V-COCO labels → HOI descriptions: augments · Label resources are expanded with textual HOI descriptions.
- Inference → HOLa model: runs · Inference loads the HOLa model for prediction.
- V-COCO eval → V-COCO: evaluates · V-COCO evaluation uses dataset-specific annotations and metrics.
- Tools → Inference: orchestrates · Shared tooling coordinates inference and evaluation flows.
- Tools → V-COCO eval: orchestrates · Shared tooling coordinates dataset-specific evaluation.
- Scripts → Fine-tune entry: launches · Shell scripts serve as primary experiment entrypoints.
- Scripts → Inference: launches · Shell scripts can launch inference and testing workflows.