How scholar-wu/hermes works
1. What this repository is
HERMES is a benchmark inference/evaluation codebase for streaming video understanding. Its main job is to run video QA on several benchmarks using either LLaVA-OneVision or Qwen2.5-VL, while applying HERMES’s key idea: hierarchical KV-cache memory management for long or streaming video inputs.
2. Top-level architecture
The repo is split into three main layers:
- Data layer: benchmark annotations and local video files under data/
- Inference layer: model wrappers and HERMES memory logic under inference/ and video_qa/
- Evaluation layer: benchmark-specific scoring under eval/
A shell entrypoint in scripts/ ties them together.
3. Dataset and benchmark boundary
The data/ directory contains one JSON annotation file per benchmark:
- multiple-choice / benchmark-style sets: videomme, mvbench, egoschema, ovobench, streamingbench
- open-ended sets: rvs/ego, rvs/movie
The JSON files encode the benchmark metadata and video paths expected by the code. The actual videos are external downloads placed in the matching videos/ folders, so the repo is annotation-driven rather than containing the media itself.
4. Streaming vs offline benchmark split
The repo distinguishes two evaluation modes:
- Streaming benchmarks: StreamingBench, OVO-Bench, RVS-Ego, RVS-Movie
- Offline benchmarks: VideoMME, MVBench, EgoSchema
This split matters architecturally because streaming datasets are evaluated with the hierarchical KV-cache behavior in mind, while offline datasets are still run through the same inference pipeline but judged with different metrics and output formats.
5. Model abstraction layer
The inference/ package is the model-facing boundary. It contains:
- abstract_hermes.py: shared HERMES behavior and common interface
- llavaov_hermes.py: adapter for LLaVA-OneVision models
- qwenvl_hermes.py: adapter for Qwen2.5-VL models
This suggests the repo is designed around a model-agnostic HERMES core with family-specific backends for the two supported VLM stacks.
6. Hierarchical memory / KV-cache handling
The files reindex_1d.py and reindex_3d.py are the clearest hint at the HERMES mechanism. They likely manage how frame/token positions are remapped as video is processed in chunks, which is central to the “KV Cache as Hierarchical Memory” design.
In architectural terms, this is the main algorithmic subsystem that sits between video chunking and model generation, enabling long-context streaming without recomputing everything from scratch.
7. Video QA orchestration
The video_qa/ package is the application layer that runs end-to-end benchmark inference:
- base.py: shared dataset or task abstractions
- hermes_vqa.py: the HERMES-specific QA pipeline
- run_infer.py: command-line entrypoint that loads a benchmark, chooses a model adapter, runs inference, and writes results
This package is the central coordinator that connects annotation loading, video sampling, model execution, and result serialization.
8. Execution flow
The primary runtime flow is:
- scripts/run_infer.sh sets PYTHONPATH and selects model/dataset parameters
- video_qa/run_infer.py loads benchmark metadata and launches inference, optionally in parallel via num_chunks
- the chosen backend in inference/ processes video chunks with sample_fps and kv_size
- outputs are written into a results directory, typically as results.csv
So the repo’s core pipeline is video files + benchmark JSON → chunked model inference → predictions file.
9. Parallelism and resource tuning
The inference path is parameterized by:
- num_chunks: parallelism across processes or GPUs
- sample_fps: temporal downsampling of the input video
- kv_size: upper bound for HERMES’s KV cache memory budget
These are not just user knobs; they are architectural controls that directly shape throughput, memory footprint, and how much history the model preserves over time.
10. Evaluation subsystem
The eval/ directory is deliberately separated from inference:
- eval_multiple_choice.py handles benchmarks with discrete answer choices, plus benchmark-specific reporting such as VideoMME duration buckets and EgoSchema submission formatting
- eval_open_ended.py handles RVS-style free-form answers and uses GPT-based scoring
This separation makes the repo usable as both an inference engine and a benchmarking harness, with scoring logic isolated from model execution.
11. Model-family dependencies
The repo maintains two requirements files:
- requirements_llava.txt
- requirements_qwen.txt
That indicates the two model families have materially different runtime dependencies, while still sharing the same orchestration and evaluation layers. flash-attn is explicitly required, which matters for performance and long-context attention efficiency.
12. Output and artifact boundaries
The expected artifacts are:
- raw predictions saved under a results/ hierarchy
- CSV outputs for multiple-choice tasks
- JSON outputs for open-ended scoring
- optional temporary scoring files for GPT evaluation
This makes the repo easy to diagram as a batch pipeline: benchmark inputs in, structured result files out, followed by scoring.
13. External services and infrastructure
The only external service that materially affects architecture is GPT, used by eval_open_ended.py for answer judging. Model weights are pulled from Hugging Face, and benchmark videos are sourced from benchmark-hosted datasets. Aside from that, the system is locally executed and file-system driven.
14. Overall architectural picture
If you draw the repository, the most accurate structure is:
- Benchmark data layer on the left
- HERMES inference core in the middle, with model-specific adapters and KV-cache reindexing
- Result generation and evaluation on the right
The key design idea is that HERMES does not replace the VLMs; it wraps them with a streaming memory mechanism and a unified QA/evaluation pipeline across several video benchmarks.
Main entry points: run_infer.py Infer runner · eval_multiple_choice.py MC eval · abstract_hermes.py HERMES core · eval_open_ended.py Open-ended eval
Components 18
Data
Benchmark annotations and dataset inputs
- VideoMME · Multiple-choice benchmark annotations
data/videomme/videomme.json - MVBench · Multiple-choice benchmark annotations
data/mvbench/mvbench.json - EgoSchema · Multiple-choice benchmark annotations
data/egoschema/egoschema.json - StreamingBench · Streaming benchmark annotations
data/streamingbench/streamingbench_realtime.json - OVO-Bench · Streaming benchmark annotations
data/ovobench/ovobench_realtime_backeward.json - RVS Ego · Open-ended benchmark annotations
data/rvs/ego/ego4d_oe.json - RVS Movie · Open-ended benchmark annotations
data/rvs/movie/movienet_oe.json
Inference
HERMES model adapters and memory logic
- HERMES core · Shared HERMES behavior and common interface
inference/abstract_hermes.py - LLaVA adapter · Adapter for LLaVA-OneVision models
inference/llavaov_hermes.py - Qwen adapter · Adapter for Qwen2.5-VL models
inference/qwenvl_hermes.py - Reindex 1D · Token/position remapping for streaming memory
inference/reindex_1d.py - Reindex 3D · Frame/token remapping for video chunks
inference/reindex_3d.py
Video QA
End-to-end inference orchestration
- Base task · Shared dataset and task interfaces
video_qa/base.py - HERMES VQA · End-to-end video QA pipeline
video_qa/hermes_vqa.py - Infer runner · Loads benchmarks, runs inference, writes results
video_qa/run_infer.py
Evaluation
Benchmark scoring and reporting
- MC eval · Multiple-choice scoring and benchmark reports
eval/eval_multiple_choice.py - Open-ended eval · Free-form answer judging and scoring
eval/eval_open_ended.py
Connections 24
- Run infer → Infer runner: launches · Shell entrypoint invokes the Python runner
- Infer runner → Base task: loads · Runner uses shared dataset/task abstractions
- Infer runner → HERMES VQA: orchestrates · Runner delegates benchmark execution to the QA pipeline
- HERMES VQA → HERMES core: calls · QA pipeline uses the shared HERMES interface
- HERMES core → LLaVA adapter: adapts · Shared core connects to the LLaVA backend
- HERMES core → Qwen adapter: adapts · Shared core connects to the Qwen backend
- HERMES core → Reindex 1D: uses · Memory management relies on 1D reindexing
- HERMES core → Reindex 3D: uses · Memory management relies on 3D reindexing
- Infer runner → VideoMME: reads · Benchmark annotations drive inference
- Infer runner → MVBench: reads · Benchmark annotations drive inference
- Infer runner → EgoSchema: reads · Benchmark annotations drive inference
- Infer runner → StreamingBench: reads · Streaming benchmark annotations drive inference
- Infer runner → OVO-Bench: reads · Streaming benchmark annotations drive inference
- Infer runner → RVS Ego: reads · Open-ended benchmark annotations drive inference
- Infer runner → RVS Movie: reads · Open-ended benchmark annotations drive inference
- Infer runner → MC eval: writes · Multiple-choice runs produce CSV-style results for scoring
- Infer runner → Open-ended eval: writes · Open-ended runs produce JSON-style outputs for scoring
- MC eval → VideoMME: scores · Multiple-choice scorer handles VideoMME outputs
- MC eval → MVBench: scores · Multiple-choice scorer handles MVBench outputs
- MC eval → EgoSchema: scores · Multiple-choice scorer handles EgoSchema outputs
- MC eval → StreamingBench: reports · Includes streaming benchmark reporting
- MC eval → OVO-Bench: reports · Includes streaming benchmark reporting
- Open-ended eval → RVS Ego: judges · GPT-based scoring for RVS-Ego
- Open-ended eval → RVS Movie: judges · GPT-based scoring for RVS-Movie