How open-compass/vlmevalkit works
Project type and purpose
• VLMEvalKit is an open-source evaluation toolkit (Python package vlmeval) for large vision-language models (LVLMs). It provides a one-command evaluation pipeline over multiple benchmarks (image and video) by orchestrating data loading, model inference (via APIs or local calls), and metric computations.
File structure
• Top-level entrypoint:
• run.py – CLI for kicking off evaluations, uses a config system (vlmeval/config.py).
• setup.py / requirements.txt – installation and dependencies (transformers, torch, etc.).
• Core modules under vlmeval/:
• api/ – wrappers for remote/commercial model APIs (Claude, Gemini, GPT, etc.).
• vlm/ – adapters for open-source/local LVLMs (LLaVA, MiniGPT-4, InternVL, Ovis, Valley, etc.).
• dataset/ – benchmark definitions, data preprocessing, and ground-truth metric functions.
• inference.py, inference_mt.py, inference_video.py – pipeline engines for single/multi-threaded image/video inference.
• smp/ – shared multiprocessing utilities for parallel evaluation.
• utils/ & tools.py – miscellaneous helpers (matching utilities, result transfer, logging).
• Scripts and notebooks under scripts/ for custom data prep, mini-prototyping, and CI under .github/.
README for additional insights
• Describes the evaluation pipeline: data download → preprocessing → generate_inner() in VLM adapter → optional LLM-based answer extraction → metric scoring → result aggregation.
• Mentions a config system for flexible evaluation settings, and support for both exact matching and LLM-based metrics.