How gromanchenko/hydrag-benchmark works
1. What this repo is
- hydrag-benchmark is a local-first benchmarking CLI for RAG/retrieval systems.
- It is not an app server; the core architecture is a set of CLI commands that load a suite, process a corpus or BEIR dataset, run retrieval heads/strategies, and emit JSON metrics.
2. Primary entrypoint and control plane
- The operational boundary is src/hydrag_benchmark/cli.py.
- From there, commands dispatch into separate runners for:
- classic single-strategy benchmarking
- doc2query prefill
- multi-head matrix evaluation
- BEIR evaluation
- suite listing
- The repository is therefore organized around command-driven workflows, not long-running services.
3. Benchmark suite as the main input contract
- suite.py defines the suite format consumed across modes.
- Suites are YAML files under suites/ with top-level metadata plus a set of cases: id, query, relevant_phrases, and optional tags.
- This is the main data model that drives evaluation; everything else is downstream of the suite definition.
4. Classic retrieval benchmark pipeline
- runner.py and harness.py are the core path for the run command.
- Flow is:
- load suite
- index or prepare the corpus from --corpus-dir
- execute the selected strategy (similarity, hybrid, crag, hydrag)
- collect ranked results
- compute metrics
- write a result JSON file
- The output is a single-strategy artifact like <suite>_<strategy>.json.
5. Retrieval strategies and storage boundary
- The heads/ package contains the retrieval implementations and strategy variants.
- head_a.py through head_hydrag.py represent different retrieval “heads” used in multi-head or BEIR modes, while head_ids.py and head_d_chroma.py / head_d_surreal.py show that some heads can target different backends.
- The storage layer is not hardcoded to one database: the presence of Chroma and Surreal integrations indicates a boundary between retrieval logic and vector/database persistence.
6. Corpus augmentation and cache flow
- doc2query.py and augmentation_cache.py implement the synthetic-question augmentation path.
- prefill and parts of multihead populate a shared augmentation_cache.json so later runs can reuse generated questions.
- quality_filter.py suggests a validation step that screens augmentation output before it is reused.
- This gives the architecture a clear preprocessing lane: corpus chunk → generated questions → cache → benchmark consumption.
7. Embeddings and ranking support
- embedding.py centralizes dense embedding behavior.
- The README’s --use-gpu flag and the gpu extra imply two execution paths: a lightweight/default embedding path and a transformer/GPU-backed path for denser multi-head workflows.
- metrics.py is separate from ranking logic, which keeps scoring and retrieval decoupled.
8. Multi-head benchmark subsystem
- head_matrix_runner.py is the orchestrator for the multihead command.
- It compares configurations such as A-only, B-only, C-only, A+B, and A+B+C, producing a matrix-style result JSON and a questions sidecar.
- This mode is explicitly designed to compare combinatorial head configurations, not just a single retrieval strategy.
- The README also shows an interpolation parameter (--alpha) for head C reranking, which is a key architectural knob in this path.
9. BEIR evaluation subsystem
- beir_loader.py and beir_runner.py isolate benchmark execution against external BEIR datasets, independent of the local code corpus path.
- This path adds integration points for:
- BEIR dataset acquisition/cache
- Ollama for generated enrichment in Head E
- SurrealDB for one of the head backends
- optional GPU embeddings
- Architecturally, BEIR is a separate data source and execution mode, but it reuses the same head abstractions and metric/reporting patterns.
10. Operational observability and latency measurement
- sys_sampler.py indicates the benchmark records system-level timing/telemetry alongside retrieval quality.
- The README’s frozen metrics include recall, MRR, chunk overlap, and latency percentiles.
- That means the results are not just ranking quality; they are a paired quality/latency benchmark.
11. Outputs and artifact boundaries
- The main outputs are JSON artifacts under results/ or a chosen --output-dir.
- There are distinct schemas for the two major result types:
- schema 0.1 for classic single-strategy runs
- schema 0.2 for multi-head comparison runs
- Side artifacts include the doc2query cache and, for some modes, a separate questions sidecar.
12. Project structure and verification
- The codebase is a standard Python package under src/hydrag_benchmark, with typed package metadata (py.typed) and pytest coverage under tests/.
- CI/publish workflows exist in .github/workflows/, so the repo is packaged and released as a Python CLI rather than deployed as a service.
Main entry points: harness.py Harness · beir_runner.py BEIR runner · cli.py CLI · head_matrix_runner.py Head matrix
Components 23
Control plane
CLI entrypoints and command dispatch
- CLI · Dispatches benchmark subcommands into the different runners
src/hydrag_benchmark/cli.py
Core benchmark
Shared suite loading, harnessing, retrieval, and scoring
- Suite · Defines the benchmark suite schema and loads suite cases
src/hydrag_benchmark/suite.py - Runner · Coordinates classic benchmark execution for one strategy
src/hydrag_benchmark/runner.py - Harness · Loads corpus data, runs the selected strategy, and gathers ranked outputs
src/hydrag_benchmark/harness.py - Metrics · Computes retrieval quality and benchmark scores from results
src/hydrag_benchmark/metrics.py - Sys sampler · Collects system timing and latency signals during runs
src/hydrag_benchmark/sys_sampler.py
Augmentation
Synthetic question generation and cache reuse
- Doc2query · Generates synthetic questions from corpus chunks
src/hydrag_benchmark/doc2query.py - Aug cache · Stores generated augmentation for reuse across runs
src/hydrag_benchmark/augmentation_cache.py - Quality filter · Screens augmentation output before cache reuse
src/hydrag_benchmark/quality_filter.py
Evaluation modes
Multi-head and BEIR-specific runners
- Head matrix · Runs combinatorial head comparisons and emits matrix results
src/hydrag_benchmark/head_matrix_runner.py - BEIR runner · Runs evaluation against BEIR datasets
src/hydrag_benchmark/beir_runner.py - BEIR loader · Loads and prepares BEIR datasets for evaluation
src/hydrag_benchmark/beir_loader.py
Retrieval heads
Strategy implementations and backend adapters
- Head base · Shared retrieval head abstractions and common contracts
src/hydrag_benchmark/heads/base.py - Head A · Primary retrieval strategy variant
src/hydrag_benchmark/heads/head_a.py - Head B · Secondary retrieval strategy variant
src/hydrag_benchmark/heads/head_b.py - Head C · Head variant with interpolation and reranking behavior
src/hydrag_benchmark/heads/head_c.py - Head D · Head implementation that bridges to backend-specific storage
src/hydrag_benchmark/heads/head_d.py - Head E · Head variant used in BEIR enrichment flows
src/hydrag_benchmark/heads/head_e.py - Hydrag · Composite retrieval strategy used by local benchmarks and BEIR mode
src/hydrag_benchmark/heads/head_hydrag.py - IDs head · ID-based retrieval helper used by some strategies
src/hydrag_benchmark/heads/head_ids.py - Chroma head · Backend adapter for Chroma-backed retrieval
src/hydrag_benchmark/heads/head_d_chroma.py - Surreal head · Backend adapter for SurrealDB-backed retrieval
src/hydrag_benchmark/heads/head_d_surreal.py
Other components
- Embedding · Provides dense embedding behavior used by retrieval and reranking
src/hydrag_benchmark/embedding.py
Connections 33
- CLI → Runner: run · Classic single-strategy benchmark command
- CLI → Doc2query: prefill · Generate augmentation cache from corpus
- CLI → Head matrix: multihead · Compare head combinations
- CLI → BEIR runner: beir · Run external dataset evaluation
- CLI → Suite: list/load · Inspect or load available suites
- Runner → Suite: loads · Reads benchmark suite definition and cases
- Runner → Harness: executes · Hands off execution for the selected strategy
- Harness → Head A: strategy · Uses a retrieval head implementation
- Harness → Head B: strategy · Uses a retrieval head implementation
- Harness → Hydrag: strategy · Uses composite Hydrag strategy
- Harness → Embedding: embeddings · Computes dense vectors for retrieval/ranking
- Harness → Metrics: scores · Computes benchmark metrics from retrieved results
- Harness → Sys sampler: telemetry · Collects latency and system signals
- Doc2query → Quality filter: validates · Screens generated questions before reuse
- Doc2query → Aug cache: writes · Persists augmentation output for later runs
- Head matrix → Aug cache: reads/writes · Shares augmentation cache across head comparisons
- Head matrix → Head base: runs through · Evaluates multiple head combinations via shared interface
- Head matrix → Head C: configures · Exercises reranking interpolation variants
- Head matrix → Metrics: scores · Produces matrix-style evaluation results
- BEIR runner → BEIR loader: loads · Prepares external BEIR datasets
- BEIR runner → Head base: runs through · Reuses shared head abstractions
- BEIR runner → Head E: enriches · Uses BEIR-specific enrichment path
- BEIR runner → Surreal head: backend · Can route to SurrealDB-backed retrieval
- BEIR runner → Embedding: embeddings · Optionally uses GPU or dense embeddings
- Head base → IDs head: uses · Shared head logic can rely on ID lookup helpers
- Head D → Chroma head: backend · Backend-specific Chroma integration
- Head D → Surreal head: backend · Backend-specific SurrealDB integration
- Hydrag → Head A: composes · Composite strategy builds on head variants
- Hydrag → Head B: composes · Composite strategy builds on head variants
- Hydrag → Head C: composes · Composite strategy builds on head variants
- Head A → Embedding: uses · Retrieval depends on embeddings
- Head B → Embedding: uses · Retrieval depends on embeddings
- Head C → Embedding: uses · Reranking and similarity use embeddings