How kaist-ami/avhbench works
Project type and purpose
• This is an open-source research pipeline and dataset release (AVHBench) for evaluating hallucinations in audio-visual LLMs.
• It bundles (a) the raw AVHBench dataset, (b) preprocessing and dataset‐construction scripts (stage1_gpt4), (c) a visual tagging submodule (“recognize-anything”), and (d) evaluation/fine-tuning code (RAM models).
File structure
• Top-level:
• data/ – the raw dataset (video files, QA JSON, readme).
• stage1_gpt4/ – stage-1 scripts to build dataset, generate captions and tags.
• stage1_gpt4/ contains:
• preprocess.sh – orchestration shell script.
• caption/ – sample captions JSON.
• disentangle_gpt4.py & utils – call GPT-4 API to parse or refine tag/caption pairs.
• recognize-anything/ – embedded visual tagging library (RAM models, configs, datasets, inference).
• visual_tagging/ – sample tag lists.
• Configuration and dependency files (requirements.txt, setup.py, setup.cfg).
Key insights from README
• The pipeline uses GPT-4 for caption disambiguation and a “Recognize-Anything” visual tagger to extract object labels.
• Outputs a benchmark dataset of {video, QA pairs} for multiple tasks.
• Provides checkpoints for AVHModel-Align-FT (based on AffectGPT) and evaluation scripts.
A. Data Sources
1. Raw Videos (from VALOR, AudioCaps)
2. QA JSON templates (downloaded)
B. Preprocessing Layer
1. preprocess.sh – invokes next modules in sequence
2. Audio extraction (ffmpeg or built-in)
3. Frame extraction (if needed)
C. Visual Tagging Module (“Recognize-Anything”)
– Reads extracted frames, applies RAM models (ViT, Swin, BERT tag2text)
– Outputs list of object tags
D. Captioning & Disentanglement Module
– caption/sample_captions.json (seed captions)
– disentangle_gpt4.py calls OpenAI GPT-4 API to refine/integrate tags & captions
– utils handle prompt formatting and JSON output
E. Dataset Assembly
– Merges tags, captions, original QA templates into final dataset JSON (AVHBench subset)
F. Benchmark Evaluation & Model Inference
1. AVHModel-Align-FT checkpoint
2. inference_tag2text_halluci.py and RAM inference
3. Metric calculation (accuracy, precision, recall, F1)
G. External Services & Dependencies
– OpenAI GPT-4 API
– PyTorch, other Python libs (requirements.txt)
Relationships and data flows
• Raw Data → Preprocessing → {Visual Tagger, Audio/Frame Extraction}
• Visual Tagger → Tag List → Disentanglement (GPT-4)
• Caption seeds → Disentanglement (GPT-4)
• Disentangled outputs → Dataset Assembly
• Final dataset → Benchmark Evaluation → Metrics & Leaderboard
Architectural patterns and principles
• Modular pipeline: each stage is a standalone component invoked by a master shell script.
• Clear separation of concerns: data ingestion, feature extraction (tags/captions), dataset assembly, evaluation.
• Reuse of external model libraries (Recognize-Anything, AffectGPT) via submodule and pip package.
• Configuration-driven (YAML/JSON configs under ram/configs).
Detail level
• Sub‐components inside “Recognize-Anything”:
• ram/data, ram/models, ram/inference
• That preprocess.sh orchestrates both visual_tagging/ and disentangle_gpt4.py.
• Final outputs: AVHBench JSON, metrics tables.