# kaist-ami/avhbench architecture

> Architecture diagram of the GitHub repository kaist-ami/avhbench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kaist-ami/avhbench
- Repository: https://github.com/kaist-ami/avhbench

## Overview

##### Project type and purpose
- This is an open-source research pipeline and dataset release (AVHBench) for evaluating hallucinations in audio-visual LLMs.
- It bundles (a) the raw AVHBench dataset, (b) preprocessing and dataset‐construction scripts (stage1_gpt4), (c) a visual tagging submodule (“recognize-anything”), and (d) evaluation/fine-tuning code (RAM models).

##### File structure
- Top-level:
- data/ – the raw dataset (video files, QA JSON, readme).
- stage1_gpt4/ – stage-1 scripts to build dataset, generate captions and tags.
- stage1_gpt4/ contains:
- preprocess.sh – orchestration shell script.
- caption/ – sample captions JSON.
- disentangle_gpt4.py & utils – call GPT-4 API to parse or refine tag/caption pairs.
- recognize-anything/ – embedded visual tagging library (RAM models, configs, datasets, inference).
- visual_tagging/ – sample tag lists.
- Configuration and dependency files (requirements.txt, setup.py, setup.cfg).

##### Key insights from README
- The pipeline uses GPT-4 for caption disambiguation and a “Recognize-Anything” visual tagger to extract object labels.
- Outputs a benchmark dataset of {video, QA pairs} for multiple tasks.
- Provides checkpoints for AVHModel-Align-FT (based on AffectGPT) and evaluation scripts.

A. Data Sources
1. Raw Videos (from VALOR, AudioCaps)
2. QA JSON templates (downloaded)
B. Preprocessing Layer
1. preprocess.sh – invokes next modules in sequence
2. Audio extraction (ffmpeg or built-in)
3. Frame extraction (if needed)
C. Visual Tagging Module (“Recognize-Anything”)
– Reads extracted frames, applies RAM models (ViT, Swin, BERT tag2text)
– Outputs list of object tags
D. Captioning & Disentanglement Module
– caption/sample_captions.json (seed captions)
– disentangle_gpt4.py calls OpenAI GPT-4 API to refine/integrate tags & captions
– utils handle prompt formatting and JSON output
E. Dataset Assembly
– Merges tags, captions, original QA templates into final dataset JSON (AVHBench subset)
F. Benchmark Evaluation & Model Inference
1. AVHModel-Align-FT checkpoint
2. inference_tag2text_halluci.py and RAM inference
3. Metric calculation (accuracy, precision, recall, F1)
G. External Services & Dependencies
– OpenAI GPT-4 API
– PyTorch, other Python libs (requirements.txt)

##### Relationships and data flows
- Raw Data → Preprocessing → {Visual Tagger, Audio/Frame Extraction}
- Visual Tagger → Tag List → Disentanglement (GPT-4)
- Caption seeds → Disentanglement (GPT-4)
- Disentangled outputs → Dataset Assembly
- Final dataset → Benchmark Evaluation → Metrics & Leaderboard

##### Architectural patterns and principles
- Modular pipeline: each stage is a standalone component invoked by a master shell script.
- Clear separation of concerns: data ingestion, feature extraction (tags/captions), dataset assembly, evaluation.
- Reuse of external model libraries (Recognize-Anything, AffectGPT) via submodule and pip package.
- Configuration-driven (YAML/JSON configs under ram/configs).

##### Detail level
- Sub‐components inside “Recognize-Anything”:
- ram/data, ram/models, ram/inference
- That preprocess.sh orchestrates both visual_tagging/ and disentangle_gpt4.py.
- Final outputs: AVHBench JSON, metrics tables.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources
    DataSources(("Raw Data<br/>(Videos & QA JSON)")):::data
    click DataSources "https://github.com/kaist-ami/avhbench/tree/main/data/"

    %% Preprocessing Layer
    Preprocess["Preprocessing Layer<br/>(preprocess.sh)"]:::script
    click Preprocess "https://github.com/kaist-ami/avhbench/blob/main/stage1_gpt4/preprocess.sh"

    %% Visual Tagging Module
    subgraph "Visual Tagging Module<br/>(Recognize-Anything)" 
        direction TB
        RecognizeAnything["Recognize-Anything Library"]:::library
        click RecognizeAnything "https://github.com/kaist-ami/avhbench/tree/main/stage1_gpt4/recognize-anything/"
        RAMModels["RAM Models & Data"]:::library
        click RAMModels "https://github.com/kaist-ami/avhbench/tree/main/stage1_gpt4/recognize-anything/ram/"
        RAMConfigs["RAM Configs"]:::library
        click RAMConfigs "https://github.com/kaist-ami/avhbench/tree/main/stage1_gpt4/recognize-anything/ram/configs/"
        RAMInference["Inference Halluci Script"]:::script
        click RAMInference "https://github.com/kaist-ami/avhbench/blob/main/stage1_gpt4/recognize-anything/inference_tag2text_halluci.py"
    end

    %% Captioning & Disentanglement Module
    subgraph "Captioning & Disentanglement"
        direction TB
        SampleCaptions["Caption Seeds<br/>(sample_captions.json)"]:::data
        click SampleCaptions "https://github.com/kaist-ami/avhbench/blob/main/stage1_gpt4/caption/sample_captions.json"
        Disentangle["disentangle_gpt4.py"]:::script
        click Disentangle "https://github.com/kaist-ami/avhbench/blob/main/stage1_gpt4/disentangle_gpt4.py"
        Utils["Disentangle Utils"]:::script
        click Utils "https://github.com/kaist-ami/avhbench/blob/main/stage1_gpt4/disentangle_gpt4_utils.py"
        SampleTags["Sample Tag Lists"]:::data
        click SampleTags "https://github.com/kaist-ami/avhbench/blob/main/stage1_gpt4/visual_tagging/sample_tags.json"
    end

    %% Dataset Assembly
    Assemble["Dataset Assembly"]:::process

    %% Benchmark Evaluation
    Evaluate["Benchmark Evaluation"]:::process

    %% External Services & Dependencies
    ExternalLLM(("External LLM Service<br/>(GPT-4 API)")):::api
    Requirements["Pipeline Requirements"]:::config
    click Requirements "https://github.com/kaist-ami/avhbench/blob/main/stage1_gpt4/requirements.txt"
    SetupAll["Project Setup Files"]:::config
    click SetupAll "https://github.com/kaist-ami/avhbench/blob/main/requirements.txt"
    click SetupAll "https://github.com/kaist-ami/avhbench/blob/main/setup.py"
    click SetupAll "https://github.com/kaist-ami/avhbench/blob/main/setup.cfg"
    click SetupAll "https://github.com/kaist-ami/avhbench/blob/main/stage1_gpt4/recognize-anything/setup.py"
    click SetupAll "https://github.com/kaist-ami/avhbench/blob/main/stage1_gpt4/recognize-anything/setup.cfg"

    %% Flow
    DataSources -->|"raw videos & QA"| Preprocess
    Preprocess -->|"extract frames & audio"| RecognizeAnything
    RecognizeAnything -->|tags| Disentangle
    Preprocess -->|"audio/frames"| Assemble
    SampleCaptions -->|seed captions| Disentangle
    ExternalLLM -->|refinement API| Disentangle
    Disentangle -->|disentangled outputs| Assemble
    Assemble -->|final AVHBench JSON| Evaluate
    Evaluate -->|metrics| Evaluate

    %% Config linkage
    RAMConfigs --> RecognizeAnything
    Requirements --> Preprocess
    Requirements --> Disentangle

    %% Styles
    classDef data fill:#f9f,stroke:#333,stroke-width:1px,shape:cylinder
    classDef script fill:#bbf,stroke:#333,stroke-width:1px,shape:rect
    classDef library fill:#bfb,stroke:#333,stroke-width:1px,shape:rect
    classDef process fill:#ffb,stroke:#333,stroke-width:1px,shape:rect
    classDef api fill:#eef,stroke:#333,stroke-width:1px,shape:cloud
    classDef config fill:#ddd,stroke:#333,stroke-width:1px,shape:rect
```
