# gromanchenko/hydrag-benchmark architecture

> Architecture diagram of the GitHub repository gromanchenko/hydrag-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gromanchenko/hydrag-benchmark
- Repository: https://github.com/gromanchenko/hydrag-benchmark

## Overview

1. **What this repo is**
   - `hydrag-benchmark` is a **local-first benchmarking CLI** for RAG/retrieval systems.
   - It is not an app server; the core architecture is a set of **CLI commands that load a suite, process a corpus or BEIR dataset, run retrieval heads/strategies, and emit JSON metrics**.

2. **Primary entrypoint and control plane**
   - The operational boundary is `src/hydrag_benchmark/cli.py`.
   - From there, commands dispatch into separate runners for:
     - classic single-strategy benchmarking
     - doc2query prefill
     - multi-head matrix evaluation
     - BEIR evaluation
     - suite listing
   - The repository is therefore organized around **command-driven workflows**, not long-running services.

3. **Benchmark suite as the main input contract**
   - `suite.py` defines the suite format consumed across modes.
   - Suites are YAML files under `suites/` with top-level metadata plus a set of cases: `id`, `query`, `relevant_phrases`, and optional tags.
   - This is the main data model that drives evaluation; everything else is downstream of the suite definition.

4. **Classic retrieval benchmark pipeline**
   - `runner.py` and `harness.py` are the core path for the `run` command.
   - Flow is:
     - load suite
     - index or prepare the corpus from `--corpus-dir`
     - execute the selected strategy (`similarity`, `hybrid`, `crag`, `hydrag`)
     - collect ranked results
     - compute metrics
     - write a result JSON file
   - The output is a single-strategy artifact like `<suite>_<strategy>.json`.

5. **Retrieval strategies and storage boundary**
   - The `heads/` package contains the retrieval implementations and strategy variants.
   - `head_a.py` through `head_hydrag.py` represent different retrieval “heads” used in multi-head or BEIR modes, while `head_ids.py` and `head_d_chroma.py` / `head_d_surreal.py` show that some heads can target different backends.
   - The storage layer is not hardcoded to one database: the presence of Chroma and Surreal integrations indicates a boundary between **retrieval logic** and **vector/database persistence**.

6. **Corpus augmentation and cache flow**
   - `doc2query.py` and `augmentation_cache.py` implement the synthetic-question augmentation path.
   - `prefill` and parts of `multihead` populate a shared `augmentation_cache.json` so later runs can reuse generated questions.
   - `quality_filter.py` suggests a validation step that screens augmentation output before it is reused.
   - This gives the architecture a clear preprocessing lane: **corpus chunk → generated questions → cache → benchmark consumption**.

7. **Embeddings and ranking support**
   - `embedding.py` centralizes dense embedding behavior.
   - The README’s `--use-gpu` flag and the `gpu` extra imply two execution paths: a lightweight/default embedding path and a transformer/GPU-backed path for denser multi-head workflows.
   - `metrics.py` is separate from ranking logic, which keeps scoring and retrieval decoupled.

8. **Multi-head benchmark subsystem**
   - `head_matrix_runner.py` is the orchestrator for the `multihead` command.
   - It compares configurations such as `A-only`, `B-only`, `C-only`, `A+B`, and `A+B+C`, producing a matrix-style result JSON and a questions sidecar.
   - This mode is explicitly designed to compare **combinatorial head configurations**, not just a single retrieval strategy.
   - The README also shows an interpolation parameter (`--alpha`) for head C reranking, which is a key architectural knob in this path.

9. **BEIR evaluation subsystem**
   - `beir_loader.py` and `beir_runner.py` isolate benchmark execution against external BEIR datasets, independent of the local code corpus path.
   - This path adds integration points for:
     - BEIR dataset acquisition/cache
     - Ollama for generated enrichment in Head E
     - SurrealDB for one of the head backends
     - optional GPU embeddings
   - Architecturally, BEIR is a separate data source and execution mode, but it reuses the same head abstractions and metric/reporting patterns.

10. **Operational observability and latency measurement**
   - `sys_sampler.py` indicates the benchmark records system-level timing/telemetry alongside retrieval quality.
   - The README’s frozen metrics include recall, MRR, chunk overlap, and latency percentiles.
   - That means the results are not just ranking quality; they are a **paired quality/latency benchmark**.

11. **Outputs and artifact boundaries**
   - The main outputs are JSON artifacts under `results/` or a chosen `--output-dir`.
   - There are distinct schemas for the two major result types:
     - schema `0.1` for classic single-strategy runs
     - schema `0.2` for multi-head comparison runs
   - Side artifacts include the doc2query cache and, for some modes, a separate questions sidecar.

12. **Project structure and verification**
   - The codebase is a standard Python package under `src/hydrag_benchmark`, with typed package metadata (`py.typed`) and pytest coverage under `tests/`.
   - CI/publish workflows exist in `.github/workflows/`, so the repo is packaged and released as a Python CLI rather than deployed as a service.

Main entry points: [`src/hydrag_benchmark/harness.py`](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/harness.py) (Harness), [`src/hydrag_benchmark/beir_runner.py`](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/beir_runner.py) (BEIR runner), [`src/hydrag_benchmark/cli.py`](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/cli.py) (CLI), [`src/hydrag_benchmark/head_matrix_runner.py`](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/head_matrix_runner.py) (Head matrix)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Control plane"]
  node_cli["CLI<br/>command router<br/>[cli.py]"]
end

subgraph group_core["Core benchmark"]
  node_suite["Suite<br/>case model<br/>[suite.py]"]
  node_runner["Runner<br/>single-run orchestrator<br/>[runner.py]"]
  node_harness["Harness<br/>execution core<br/>[harness.py]"]
  node_metrics["Metrics<br/>scoring<br/>[metrics.py]"]
  node_sys["Sys sampler<br/>telemetry<br/>[sys_sampler.py]"]
end

subgraph group_augment["Augmentation"]
  node_doc2query["Doc2query<br/>augmentation pipeline<br/>[doc2query.py]"]
  node_cache[("Aug cache<br/>shared cache")]
  node_filter["Quality filter<br/>validation<br/>[quality_filter.py]"]
end

subgraph group_modes["Evaluation modes"]
  node_matrix["Head matrix<br/>multi-head orchestrator"]
  node_beirrun["BEIR runner<br/>external benchmark<br/>[beir_runner.py]"]
  node_beirload["BEIR loader<br/>dataset adapter<br/>[beir_loader.py]"]
end

subgraph group_heads["Retrieval heads"]
  node_headbase["Head base<br/>strategy interface<br/>[base.py]"]
  node_heada["Head A<br/>retrieval head<br/>[head_a.py]"]
  node_headb["Head B<br/>retrieval head<br/>[head_b.py]"]
  node_headc["Head C<br/>rerank head<br/>[head_c.py]"]
  node_headd["Head D<br/>backend head<br/>[head_d.py]"]
  node_heade["Head E<br/>enrichment head<br/>[head_e.py]"]
  node_hydrag{{"Hydrag<br/>composite head<br/>[head_hydrag.py]"}}
  node_ids["IDs head<br/>id lookup<br/>[head_ids.py]"]
  node_chroma[("Chroma head<br/>vector backend<br/>[head_d_chroma.py]")]
  node_surreal[("Surreal head<br/>vector backend<br/>[head_d_surreal.py]")]
end

node_embed["Embedding<br/>vectorization<br/>[embedding.py]"]

node_cli -->|"run"| node_runner
node_cli -->|"prefill"| node_doc2query
node_cli -->|"multihead"| node_matrix
node_cli -->|"beir"| node_beirrun
node_cli -->|"list/load"| node_suite
node_runner -->|"loads"| node_suite
node_runner -->|"executes"| node_harness
node_harness -->|"strategy"| node_heada
node_harness -->|"strategy"| node_headb
node_harness -->|"strategy"| node_hydrag
node_harness -->|"embeddings"| node_embed
node_harness -->|"scores"| node_metrics
node_harness -->|"telemetry"| node_sys
node_doc2query -->|"validates"| node_filter
node_doc2query -->|"writes"| node_cache
node_matrix -->|"reads/writes"| node_cache
node_matrix -->|"runs through"| node_headbase
node_matrix -->|"configures"| node_headc
node_matrix -->|"scores"| node_metrics
node_beirrun -->|"loads"| node_beirload
node_beirrun -->|"runs through"| node_headbase
node_beirrun -->|"enriches"| node_heade
node_beirrun -->|"backend"| node_surreal
node_beirrun -->|"embeddings"| node_embed
node_headbase -->|"uses"| node_ids
node_headd -->|"backend"| node_chroma
node_headd -->|"backend"| node_surreal
node_hydrag -->|"composes"| node_heada
node_hydrag -->|"composes"| node_headb
node_hydrag -->|"composes"| node_headc
node_heada -->|"uses"| node_embed
node_headb -->|"uses"| node_embed
node_headc -->|"uses"| node_embed

click node_cli "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/cli.py"
click node_suite "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/suite.py"
click node_runner "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/runner.py"
click node_harness "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/harness.py"
click node_metrics "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/metrics.py"
click node_sys "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/sys_sampler.py"
click node_doc2query "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/doc2query.py"
click node_cache "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/augmentation_cache.py"
click node_filter "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/quality_filter.py"
click node_matrix "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/head_matrix_runner.py"
click node_beirrun "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/beir_runner.py"
click node_beirload "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/beir_loader.py"
click node_headbase "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/base.py"
click node_heada "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_a.py"
click node_headb "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_b.py"
click node_headc "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_c.py"
click node_headd "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_d.py"
click node_heade "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_e.py"
click node_hydrag "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_hydrag.py"
click node_ids "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_ids.py"
click node_chroma "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_d_chroma.py"
click node_surreal "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_d_surreal.py"
click node_embed "https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/embedding.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli toneBlue
class node_suite,node_runner,node_harness,node_metrics,node_sys toneAmber
class node_doc2query,node_cache,node_filter toneMint
class node_matrix,node_beirrun,node_beirload toneRose
class node_headbase,node_heada,node_headb,node_headc,node_headd,node_heade,node_hydrag,node_ids,node_chroma,node_surreal toneIndigo
class node_embed toneNeutral
```

## Components

### Control plane

CLI entrypoints and command dispatch

- [CLI](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/cli.py): command router · Dispatches benchmark subcommands into the different runners · `src/hydrag_benchmark/cli.py`

### Core benchmark

Shared suite loading, harnessing, retrieval, and scoring

- [Suite](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/suite.py): case model · Defines the benchmark suite schema and loads suite cases · `src/hydrag_benchmark/suite.py`
- [Runner](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/runner.py): single-run orchestrator · Coordinates classic benchmark execution for one strategy · `src/hydrag_benchmark/runner.py`
- [Harness](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/harness.py): execution core · Loads corpus data, runs the selected strategy, and gathers ranked outputs · `src/hydrag_benchmark/harness.py`
- [Metrics](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/metrics.py): scoring · Computes retrieval quality and benchmark scores from results · `src/hydrag_benchmark/metrics.py`
- [Sys sampler](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/sys_sampler.py): telemetry · Collects system timing and latency signals during runs · `src/hydrag_benchmark/sys_sampler.py`

### Augmentation

Synthetic question generation and cache reuse

- [Doc2query](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/doc2query.py): augmentation pipeline · Generates synthetic questions from corpus chunks · `src/hydrag_benchmark/doc2query.py`
- [Aug cache](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/augmentation_cache.py): shared cache · Stores generated augmentation for reuse across runs · `src/hydrag_benchmark/augmentation_cache.py`
- [Quality filter](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/quality_filter.py): validation · Screens augmentation output before cache reuse · `src/hydrag_benchmark/quality_filter.py`

### Evaluation modes

Multi-head and BEIR-specific runners

- [Head matrix](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/head_matrix_runner.py): multi-head orchestrator · Runs combinatorial head comparisons and emits matrix results · `src/hydrag_benchmark/head_matrix_runner.py`
- [BEIR runner](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/beir_runner.py): external benchmark · Runs evaluation against BEIR datasets · `src/hydrag_benchmark/beir_runner.py`
- [BEIR loader](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/beir_loader.py): dataset adapter · Loads and prepares BEIR datasets for evaluation · `src/hydrag_benchmark/beir_loader.py`

### Retrieval heads

Strategy implementations and backend adapters

- [Head base](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/base.py): strategy interface · Shared retrieval head abstractions and common contracts · `src/hydrag_benchmark/heads/base.py`
- [Head A](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_a.py): retrieval head · Primary retrieval strategy variant · `src/hydrag_benchmark/heads/head_a.py`
- [Head B](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_b.py): retrieval head · Secondary retrieval strategy variant · `src/hydrag_benchmark/heads/head_b.py`
- [Head C](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_c.py): rerank head · Head variant with interpolation and reranking behavior · `src/hydrag_benchmark/heads/head_c.py`
- [Head D](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_d.py): backend head · Head implementation that bridges to backend-specific storage · `src/hydrag_benchmark/heads/head_d.py`
- [Head E](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_e.py): enrichment head · Head variant used in BEIR enrichment flows · `src/hydrag_benchmark/heads/head_e.py`
- [Hydrag](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_hydrag.py): composite head · Composite retrieval strategy used by local benchmarks and BEIR mode · `src/hydrag_benchmark/heads/head_hydrag.py`
- [IDs head](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_ids.py): id lookup · ID-based retrieval helper used by some strategies · `src/hydrag_benchmark/heads/head_ids.py`
- [Chroma head](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_d_chroma.py): vector backend · Backend adapter for Chroma-backed retrieval · `src/hydrag_benchmark/heads/head_d_chroma.py`
- [Surreal head](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/heads/head_d_surreal.py): vector backend · Backend adapter for SurrealDB-backed retrieval · `src/hydrag_benchmark/heads/head_d_surreal.py`

### Other components

- [Embedding](https://github.com/gromanchenko/hydrag-benchmark/blob/main/src/hydrag_benchmark/embedding.py): vectorization · Provides dense embedding behavior used by retrieval and reranking · `src/hydrag_benchmark/embedding.py`

## Connections

- CLI → Runner: run · Classic single-strategy benchmark command
- CLI → Doc2query: prefill · Generate augmentation cache from corpus
- CLI → Head matrix: multihead · Compare head combinations
- CLI → BEIR runner: beir · Run external dataset evaluation
- CLI → Suite: list/load · Inspect or load available suites
- Runner → Suite: loads · Reads benchmark suite definition and cases
- Runner → Harness: executes · Hands off execution for the selected strategy
- Harness → Head A: strategy · Uses a retrieval head implementation
- Harness → Head B: strategy · Uses a retrieval head implementation
- Harness → Hydrag: strategy · Uses composite Hydrag strategy
- Harness → Embedding: embeddings · Computes dense vectors for retrieval/ranking
- Harness → Metrics: scores · Computes benchmark metrics from retrieved results
- Harness → Sys sampler: telemetry · Collects latency and system signals
- Doc2query → Quality filter: validates · Screens generated questions before reuse
- Doc2query → Aug cache: writes · Persists augmentation output for later runs
- Head matrix → Aug cache: reads/writes · Shares augmentation cache across head comparisons
- Head matrix → Head base: runs through · Evaluates multiple head combinations via shared interface
- Head matrix → Head C: configures · Exercises reranking interpolation variants
- Head matrix → Metrics: scores · Produces matrix-style evaluation results
- BEIR runner → BEIR loader: loads · Prepares external BEIR datasets
- BEIR runner → Head base: runs through · Reuses shared head abstractions
- BEIR runner → Head E: enriches · Uses BEIR-specific enrichment path
- BEIR runner → Surreal head: backend · Can route to SurrealDB-backed retrieval
- BEIR runner → Embedding: embeddings · Optionally uses GPU or dense embeddings
- Head base → IDs head: uses · Shared head logic can rely on ID lookup helpers
- Head D → Chroma head: backend · Backend-specific Chroma integration
- Head D → Surreal head: backend · Backend-specific SurrealDB integration
- Hydrag → Head A: composes · Composite strategy builds on head variants
- Hydrag → Head B: composes · Composite strategy builds on head variants
- Hydrag → Head C: composes · Composite strategy builds on head variants
- Head A → Embedding: uses · Retrieval depends on embeddings
- Head B → Embedding: uses · Retrieval depends on embeddings
- Head C → Embedding: uses · Reranking and similarity use embeddings