nlpai-lab/mirage

1. Repository purpose

- This repo is a benchmark runner for MIRAGE, a retrieval-augmented generation evaluation suite.

- Its main job is to take benchmark questions plus context pools, run three modes of inference, and score them with MIRAGE-specific metrics.

2. Primary execution boundary

- main.py is the orchestration entrypoint for experiment runs.

- evaluation.py is the scoring/reporting entrypoint after predictions exist.

- This split creates a clear boundary between generation/inference and evaluation/analysis.

3. Data layer

- The mirage/ directory holds the benchmark assets:

- dataset.json for the QA examples,

- doc_pool.json for retrieval/context pool data,

- oracle.json for gold or oracle context references.

- These JSON files are the canonical inputs used across the benchmark modes.

4. Run modes

- The repo supports three experiment modes described in the README:

- LLM/Base: question only, no retrieved context.

- Oracle: question plus correct context.

- RAG/Mixed: question plus a mixture of relevant and noisy retrieved contexts.

- These modes are the main architectural branches in the pipeline.

5. Retriever subsystem

- retriever.py is the dedicated retrieval component.

- README indicates SentenceTransformer-based retrieval, with single-GPU support.

- Its role is to rank the doc_pool.json chunks and supply top-k contexts for RAG and retrieval evaluation.

6. LLM subsystem

- LLM.py isolates language-model inference.

- README says inference uses vLLM, which implies support for efficient batched generation and multi-GPU execution.

- This layer is responsible for answering in Base/Oracle/RAG settings, likely using prompts assembled from preset templates.

7. Prompting and task variants

- Preset_prompt/ contains prompt templates:

- base.json

- fixed.json

- oracle.json

- These files separate prompt behavior from code, so prompt strategy can vary without changing the orchestration logic.

8. Configuration boundary

- config.yaml centralizes experiment settings.

- README says it includes defaults for multiple LLMs and retrievers, and is meant to make runs reproducible on a single GPU setup.

- This config file likely defines model choices, shot count, retrieval parameters, and output locations.

9. Evaluation subsystem

- evaluation.py computes the benchmark outputs after inference.

- It covers three measurement layers:

- LLM metrics like EM/F1,

- Retriever metrics like precision/recall/NDCG/F1,

- MIRAGE metrics like noise vulnerability, context acceptability, context insensitivity, and context misinterpretation.

- This is the main analytical boundary separating raw model behavior from benchmark interpretation.

10. Shared utilities

- utils.py likely holds cross-cutting helpers used by both inference and evaluation.

- In a repo like this, that usually means JSON I/O, prompt assembly, batching, scoring helpers, and file-path management.

11. Experiment outputs

- The README references an Evaluation_result directory containing JSONL reports.

- That indicates a file-based pipeline: run inference, persist predictions, then evaluate into structured result files rather than using an interactive service.

12. External dependencies that shape the architecture

- The repo is Python-based and depends on:

- vLLM for LLM inference,

- SentenceTransformers for retrieval embeddings,

- likely PyYAML for config loading.

- Hugging Face is part of the data distribution path, since the dataset is also published there.

13. Overall data flow

- The typical flow is:

1. Load benchmark JSON data and config.

2. Choose a mode and prompt template.

3. Optionally retrieve top-k documents from the doc pool.

4. Send prompts to the LLM.

5. Write predictions/results.

6. Run evaluation to compute metric files and overall scores.

- An architecture diagram should show these as sequential stages with retrieval and generation as parallel model-specific subsystems.

14. Key architectural takeaway

- MIRAGE is not a serving system; it is an offline benchmark pipeline.

- The important boundaries are between static benchmark assets, retrieval, generation, and metric computation, with JSON/YAML files acting as the glue between stages.