# scholar-wu/hermes architecture

> Architecture diagram of the GitHub repository scholar-wu/hermes, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/scholar-wu/hermes
- Repository: https://github.com/scholar-wu/hermes

## Overview

1. **What this repository is**
   HERMES is a benchmark inference/evaluation codebase for **streaming video understanding**. Its main job is to run video QA on several benchmarks using either **LLaVA-OneVision** or **Qwen2.5-VL**, while applying HERMES’s key idea: **hierarchical KV-cache memory management** for long or streaming video inputs.

2. **Top-level architecture**
   The repo is split into three main layers:
   - **Data layer**: benchmark annotations and local video files under `data/`
   - **Inference layer**: model wrappers and HERMES memory logic under `inference/` and `video_qa/`
   - **Evaluation layer**: benchmark-specific scoring under `eval/`
   
   A shell entrypoint in `scripts/` ties them together.

3. **Dataset and benchmark boundary**
   The `data/` directory contains one JSON annotation file per benchmark:
   - multiple-choice / benchmark-style sets: `videomme`, `mvbench`, `egoschema`, `ovobench`, `streamingbench`
   - open-ended sets: `rvs/ego`, `rvs/movie`
   
   The JSON files encode the benchmark metadata and video paths expected by the code. The actual videos are external downloads placed in the matching `videos/` folders, so the repo is **annotation-driven** rather than containing the media itself.

4. **Streaming vs offline benchmark split**
   The repo distinguishes two evaluation modes:
   - **Streaming benchmarks**: `StreamingBench`, `OVO-Bench`, `RVS-Ego`, `RVS-Movie`
   - **Offline benchmarks**: `VideoMME`, `MVBench`, `EgoSchema`
   
   This split matters architecturally because streaming datasets are evaluated with the hierarchical KV-cache behavior in mind, while offline datasets are still run through the same inference pipeline but judged with different metrics and output formats.

5. **Model abstraction layer**
   The `inference/` package is the model-facing boundary. It contains:
   - `abstract_hermes.py`: shared HERMES behavior and common interface
   - `llavaov_hermes.py`: adapter for LLaVA-OneVision models
   - `qwenvl_hermes.py`: adapter for Qwen2.5-VL models
   
   This suggests the repo is designed around a **model-agnostic HERMES core** with **family-specific backends** for the two supported VLM stacks.

6. **Hierarchical memory / KV-cache handling**
   The files `reindex_1d.py` and `reindex_3d.py` are the clearest hint at the HERMES mechanism. They likely manage how frame/token positions are remapped as video is processed in chunks, which is central to the “KV Cache as Hierarchical Memory” design.
   
   In architectural terms, this is the main algorithmic subsystem that sits between video chunking and model generation, enabling long-context streaming without recomputing everything from scratch.

7. **Video QA orchestration**
   The `video_qa/` package is the application layer that runs end-to-end benchmark inference:
   - `base.py`: shared dataset or task abstractions
   - `hermes_vqa.py`: the HERMES-specific QA pipeline
   - `run_infer.py`: command-line entrypoint that loads a benchmark, chooses a model adapter, runs inference, and writes results
   
   This package is the central coordinator that connects annotation loading, video sampling, model execution, and result serialization.

8. **Execution flow**
   The primary runtime flow is:
   - `scripts/run_infer.sh` sets `PYTHONPATH` and selects model/dataset parameters
   - `video_qa/run_infer.py` loads benchmark metadata and launches inference, optionally in parallel via `num_chunks`
   - the chosen backend in `inference/` processes video chunks with `sample_fps` and `kv_size`
   - outputs are written into a results directory, typically as `results.csv`
   
   So the repo’s core pipeline is **video files + benchmark JSON → chunked model inference → predictions file**.

9. **Parallelism and resource tuning**
   The inference path is parameterized by:
   - `num_chunks`: parallelism across processes or GPUs
   - `sample_fps`: temporal downsampling of the input video
   - `kv_size`: upper bound for HERMES’s KV cache memory budget
   
   These are not just user knobs; they are architectural controls that directly shape throughput, memory footprint, and how much history the model preserves over time.

10. **Evaluation subsystem**
   The `eval/` directory is deliberately separated from inference:
   - `eval_multiple_choice.py` handles benchmarks with discrete answer choices, plus benchmark-specific reporting such as VideoMME duration buckets and EgoSchema submission formatting
   - `eval_open_ended.py` handles RVS-style free-form answers and uses GPT-based scoring
   
   This separation makes the repo usable as both an inference engine and a benchmarking harness, with scoring logic isolated from model execution.

11. **Model-family dependencies**
   The repo maintains two requirements files:
   - `requirements_llava.txt`
   - `requirements_qwen.txt`
   
   That indicates the two model families have materially different runtime dependencies, while still sharing the same orchestration and evaluation layers. `flash-attn` is explicitly required, which matters for performance and long-context attention efficiency.

12. **Output and artifact boundaries**
   The expected artifacts are:
   - raw predictions saved under a `results/` hierarchy
   - CSV outputs for multiple-choice tasks
   - JSON outputs for open-ended scoring
   - optional temporary scoring files for GPT evaluation
   
   This makes the repo easy to diagram as a **batch pipeline**: benchmark inputs in, structured result files out, followed by scoring.

13. **External services and infrastructure**
   The only external service that materially affects architecture is **GPT**, used by `eval_open_ended.py` for answer judging. Model weights are pulled from **Hugging Face**, and benchmark videos are sourced from benchmark-hosted datasets. Aside from that, the system is locally executed and file-system driven.

14. **Overall architectural picture**
   If you draw the repository, the most accurate structure is:
   - **Benchmark data layer** on the left
   - **HERMES inference core** in the middle, with model-specific adapters and KV-cache reindexing
   - **Result generation and evaluation** on the right
   
   The key design idea is that HERMES does not replace the VLMs; it wraps them with a streaming memory mechanism and a unified QA/evaluation pipeline across several video benchmarks.

Main entry points: [`video_qa/run_infer.py`](https://github.com/scholar-wu/hermes/blob/main/video_qa/run_infer.py) (Infer runner), [`eval/eval_multiple_choice.py`](https://github.com/scholar-wu/hermes/blob/main/eval/eval_multiple_choice.py) (MC eval), [`inference/abstract_hermes.py`](https://github.com/scholar-wu/hermes/blob/main/inference/abstract_hermes.py) (HERMES core), [`eval/eval_open_ended.py`](https://github.com/scholar-wu/hermes/blob/main/eval/eval_open_ended.py) (Open-ended eval)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data"]
  node_data_videomme["VideoMME<br/>mc annotations<br/>[videomme.json]"]
  node_data_mvbench["MVBench<br/>mc annotations<br/>[mvbench.json]"]
  node_data_egoschema["EgoSchema<br/>mc annotations<br/>[egoschema.json]"]
  node_data_streamingbench["StreamingBench<br/>streaming annotations"]
  node_data_ovobench["OVO-Bench<br/>streaming annotations"]
  node_data_rvs_ego["RVS Ego<br/>open-ended annotations<br/>[ego4d_oe.json]"]
  node_data_rvs_movie["RVS Movie<br/>open-ended annotations<br/>[movienet_oe.json]"]
end

subgraph group_inference["Inference"]
  node_inf_abstract["HERMES core<br/>model interface<br/>[abstract_hermes.py]"]
  node_inf_llavaov["LLaVA adapter<br/>vlm backend<br/>[llavaov_hermes.py]"]
  node_inf_qwenvl["Qwen adapter<br/>vlm backend<br/>[qwenvl_hermes.py]"]
  node_inf_reindex_1d["Reindex 1D<br/>kv-cache mapping<br/>[reindex_1d.py]"]
  node_inf_reindex_3d["Reindex 3D<br/>kv-cache mapping<br/>[reindex_3d.py]"]
end

subgraph group_video_qa["Video QA"]
  node_vq_base["Base task<br/>task abstractions<br/>[base.py]"]
  node_vq_hermes_vqa["HERMES VQA<br/>qa pipeline<br/>[hermes_vqa.py]"]
  node_vq_run_infer["Infer runner<br/>cli orchestration<br/>[run_infer.py]"]
end

subgraph group_eval["Evaluation"]
  node_eval_mc["MC eval<br/>scoring"]
  node_eval_oe["Open-ended eval<br/>gpt scoring<br/>[eval_open_ended.py]"]
end

subgraph group_scripts["Scripts"]
  node_scripts_run_infer["Run infer<br/>shell entrypoint<br/>[run_infer.sh]"]
end

node_scripts_run_infer -->|"launches"| node_vq_run_infer
node_vq_run_infer -->|"loads"| node_vq_base
node_vq_run_infer -->|"orchestrates"| node_vq_hermes_vqa
node_vq_hermes_vqa -->|"calls"| node_inf_abstract
node_inf_abstract -->|"adapts"| node_inf_llavaov
node_inf_abstract -->|"adapts"| node_inf_qwenvl
node_inf_abstract -->|"uses"| node_inf_reindex_1d
node_inf_abstract -->|"uses"| node_inf_reindex_3d
node_vq_run_infer -->|"reads"| node_data_videomme
node_vq_run_infer -->|"reads"| node_data_mvbench
node_vq_run_infer -->|"reads"| node_data_egoschema
node_vq_run_infer -->|"reads"| node_data_streamingbench
node_vq_run_infer -->|"reads"| node_data_ovobench
node_vq_run_infer -->|"reads"| node_data_rvs_ego
node_vq_run_infer -->|"reads"| node_data_rvs_movie
node_vq_run_infer -->|"writes"| node_eval_mc
node_vq_run_infer -->|"writes"| node_eval_oe
node_eval_mc -->|"scores"| node_data_videomme
node_eval_mc -->|"scores"| node_data_mvbench
node_eval_mc -->|"scores"| node_data_egoschema
node_eval_mc -->|"reports"| node_data_streamingbench
node_eval_mc -->|"reports"| node_data_ovobench
node_eval_oe -->|"judges"| node_data_rvs_ego
node_eval_oe -->|"judges"| node_data_rvs_movie

click node_data_videomme "https://github.com/scholar-wu/hermes/blob/main/data/videomme/videomme.json"
click node_data_mvbench "https://github.com/scholar-wu/hermes/blob/main/data/mvbench/mvbench.json"
click node_data_egoschema "https://github.com/scholar-wu/hermes/blob/main/data/egoschema/egoschema.json"
click node_data_streamingbench "https://github.com/scholar-wu/hermes/blob/main/data/streamingbench/streamingbench_realtime.json"
click node_data_ovobench "https://github.com/scholar-wu/hermes/blob/main/data/ovobench/ovobench_realtime_backeward.json"
click node_data_rvs_ego "https://github.com/scholar-wu/hermes/blob/main/data/rvs/ego/ego4d_oe.json"
click node_data_rvs_movie "https://github.com/scholar-wu/hermes/blob/main/data/rvs/movie/movienet_oe.json"
click node_scripts_run_infer "https://github.com/scholar-wu/hermes/blob/main/scripts/run_infer.sh"
click node_vq_base "https://github.com/scholar-wu/hermes/blob/main/video_qa/base.py"
click node_vq_hermes_vqa "https://github.com/scholar-wu/hermes/blob/main/video_qa/hermes_vqa.py"
click node_vq_run_infer "https://github.com/scholar-wu/hermes/blob/main/video_qa/run_infer.py"
click node_inf_abstract "https://github.com/scholar-wu/hermes/blob/main/inference/abstract_hermes.py"
click node_inf_llavaov "https://github.com/scholar-wu/hermes/blob/main/inference/llavaov_hermes.py"
click node_inf_qwenvl "https://github.com/scholar-wu/hermes/blob/main/inference/qwenvl_hermes.py"
click node_inf_reindex_1d "https://github.com/scholar-wu/hermes/blob/main/inference/reindex_1d.py"
click node_inf_reindex_3d "https://github.com/scholar-wu/hermes/blob/main/inference/reindex_3d.py"
click node_eval_mc "https://github.com/scholar-wu/hermes/blob/main/eval/eval_multiple_choice.py"
click node_eval_oe "https://github.com/scholar-wu/hermes/blob/main/eval/eval_open_ended.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_data_videomme,node_data_mvbench,node_data_egoschema,node_data_streamingbench,node_data_ovobench,node_data_rvs_ego,node_data_rvs_movie toneBlue
class node_inf_abstract,node_inf_llavaov,node_inf_qwenvl,node_inf_reindex_1d,node_inf_reindex_3d toneAmber
class node_vq_base,node_vq_hermes_vqa,node_vq_run_infer toneMint
class node_eval_mc,node_eval_oe toneRose
class node_scripts_run_infer toneIndigo
```

## Components

### Data

Benchmark annotations and dataset inputs

- [VideoMME](https://github.com/scholar-wu/hermes/blob/main/data/videomme/videomme.json): mc annotations · Multiple-choice benchmark annotations · `data/videomme/videomme.json`
- [MVBench](https://github.com/scholar-wu/hermes/blob/main/data/mvbench/mvbench.json): mc annotations · Multiple-choice benchmark annotations · `data/mvbench/mvbench.json`
- [EgoSchema](https://github.com/scholar-wu/hermes/blob/main/data/egoschema/egoschema.json): mc annotations · Multiple-choice benchmark annotations · `data/egoschema/egoschema.json`
- [StreamingBench](https://github.com/scholar-wu/hermes/blob/main/data/streamingbench/streamingbench_realtime.json): streaming annotations · Streaming benchmark annotations · `data/streamingbench/streamingbench_realtime.json`
- [OVO-Bench](https://github.com/scholar-wu/hermes/blob/main/data/ovobench/ovobench_realtime_backeward.json): streaming annotations · Streaming benchmark annotations · `data/ovobench/ovobench_realtime_backeward.json`
- [RVS Ego](https://github.com/scholar-wu/hermes/blob/main/data/rvs/ego/ego4d_oe.json): open-ended annotations · Open-ended benchmark annotations · `data/rvs/ego/ego4d_oe.json`
- [RVS Movie](https://github.com/scholar-wu/hermes/blob/main/data/rvs/movie/movienet_oe.json): open-ended annotations · Open-ended benchmark annotations · `data/rvs/movie/movienet_oe.json`

### Inference

HERMES model adapters and memory logic

- [HERMES core](https://github.com/scholar-wu/hermes/blob/main/inference/abstract_hermes.py): model interface · Shared HERMES behavior and common interface · `inference/abstract_hermes.py`
- [LLaVA adapter](https://github.com/scholar-wu/hermes/blob/main/inference/llavaov_hermes.py): vlm backend · Adapter for LLaVA-OneVision models · `inference/llavaov_hermes.py`
- [Qwen adapter](https://github.com/scholar-wu/hermes/blob/main/inference/qwenvl_hermes.py): vlm backend · Adapter for Qwen2.5-VL models · `inference/qwenvl_hermes.py`
- [Reindex 1D](https://github.com/scholar-wu/hermes/blob/main/inference/reindex_1d.py): kv-cache mapping · Token/position remapping for streaming memory · `inference/reindex_1d.py`
- [Reindex 3D](https://github.com/scholar-wu/hermes/blob/main/inference/reindex_3d.py): kv-cache mapping · Frame/token remapping for video chunks · `inference/reindex_3d.py`

### Video QA

End-to-end inference orchestration

- [Base task](https://github.com/scholar-wu/hermes/blob/main/video_qa/base.py): task abstractions · Shared dataset and task interfaces · `video_qa/base.py`
- [HERMES VQA](https://github.com/scholar-wu/hermes/blob/main/video_qa/hermes_vqa.py): qa pipeline · End-to-end video QA pipeline · `video_qa/hermes_vqa.py`
- [Infer runner](https://github.com/scholar-wu/hermes/blob/main/video_qa/run_infer.py): cli orchestration · Loads benchmarks, runs inference, writes results · `video_qa/run_infer.py`

### Evaluation

Benchmark scoring and reporting

- [MC eval](https://github.com/scholar-wu/hermes/blob/main/eval/eval_multiple_choice.py): scoring · Multiple-choice scoring and benchmark reports · `eval/eval_multiple_choice.py`
- [Open-ended eval](https://github.com/scholar-wu/hermes/blob/main/eval/eval_open_ended.py): gpt scoring · Free-form answer judging and scoring · `eval/eval_open_ended.py`

### Scripts

Shell entrypoints

- [Run infer](https://github.com/scholar-wu/hermes/blob/main/scripts/run_infer.sh): shell entrypoint · Benchmark inference launcher · `scripts/run_infer.sh`

## Connections

- Run infer → Infer runner: launches · Shell entrypoint invokes the Python runner
- Infer runner → Base task: loads · Runner uses shared dataset/task abstractions
- Infer runner → HERMES VQA: orchestrates · Runner delegates benchmark execution to the QA pipeline
- HERMES VQA → HERMES core: calls · QA pipeline uses the shared HERMES interface
- HERMES core → LLaVA adapter: adapts · Shared core connects to the LLaVA backend
- HERMES core → Qwen adapter: adapts · Shared core connects to the Qwen backend
- HERMES core → Reindex 1D: uses · Memory management relies on 1D reindexing
- HERMES core → Reindex 3D: uses · Memory management relies on 3D reindexing
- Infer runner → VideoMME: reads · Benchmark annotations drive inference
- Infer runner → MVBench: reads · Benchmark annotations drive inference
- Infer runner → EgoSchema: reads · Benchmark annotations drive inference
- Infer runner → StreamingBench: reads · Streaming benchmark annotations drive inference
- Infer runner → OVO-Bench: reads · Streaming benchmark annotations drive inference
- Infer runner → RVS Ego: reads · Open-ended benchmark annotations drive inference
- Infer runner → RVS Movie: reads · Open-ended benchmark annotations drive inference
- Infer runner → MC eval: writes · Multiple-choice runs produce CSV-style results for scoring
- Infer runner → Open-ended eval: writes · Open-ended runs produce JSON-style outputs for scoring
- MC eval → VideoMME: scores · Multiple-choice scorer handles VideoMME outputs
- MC eval → MVBench: scores · Multiple-choice scorer handles MVBench outputs
- MC eval → EgoSchema: scores · Multiple-choice scorer handles EgoSchema outputs
- MC eval → StreamingBench: reports · Includes streaming benchmark reporting
- MC eval → OVO-Bench: reports · Includes streaming benchmark reporting
- Open-ended eval → RVS Ego: judges · GPT-based scoring for RVS-Ego
- Open-ended eval → RVS Movie: judges · GPT-based scoring for RVS-Movie
