# nlpai-lab/mirage architecture

> Architecture diagram of the GitHub repository nlpai-lab/mirage, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nlpai-lab/mirage
- Repository: https://github.com/nlpai-lab/mirage

## Overview

1. **Repository purpose**
   - This repo is a benchmark runner for MIRAGE, a retrieval-augmented generation evaluation suite.
   - Its main job is to take benchmark questions plus context pools, run three modes of inference, and score them with MIRAGE-specific metrics.

2. **Primary execution boundary**
   - `main.py` is the orchestration entrypoint for experiment runs.
   - `evaluation.py` is the scoring/reporting entrypoint after predictions exist.
   - This split creates a clear boundary between **generation/inference** and **evaluation/analysis**.

3. **Data layer**
   - The `mirage/` directory holds the benchmark assets:
     - `dataset.json` for the QA examples,
     - `doc_pool.json` for retrieval/context pool data,
     - `oracle.json` for gold or oracle context references.
   - These JSON files are the canonical inputs used across the benchmark modes.

4. **Run modes**
   - The repo supports three experiment modes described in the README:
     - **LLM/Base**: question only, no retrieved context.
     - **Oracle**: question plus correct context.
     - **RAG/Mixed**: question plus a mixture of relevant and noisy retrieved contexts.
   - These modes are the main architectural branches in the pipeline.

5. **Retriever subsystem**
   - `retriever.py` is the dedicated retrieval component.
   - README indicates SentenceTransformer-based retrieval, with single-GPU support.
   - Its role is to rank the `doc_pool.json` chunks and supply top-k contexts for RAG and retrieval evaluation.

6. **LLM subsystem**
   - `LLM.py` isolates language-model inference.
   - README says inference uses vLLM, which implies support for efficient batched generation and multi-GPU execution.
   - This layer is responsible for answering in Base/Oracle/RAG settings, likely using prompts assembled from preset templates.

7. **Prompting and task variants**
   - `Preset_prompt/` contains prompt templates:
     - `base.json`
     - `fixed.json`
     - `oracle.json`
   - These files separate prompt behavior from code, so prompt strategy can vary without changing the orchestration logic.

8. **Configuration boundary**
   - `config.yaml` centralizes experiment settings.
   - README says it includes defaults for multiple LLMs and retrievers, and is meant to make runs reproducible on a single GPU setup.
   - This config file likely defines model choices, shot count, retrieval parameters, and output locations.

9. **Evaluation subsystem**
   - `evaluation.py` computes the benchmark outputs after inference.
   - It covers three measurement layers:
     - **LLM metrics** like EM/F1,
     - **Retriever metrics** like precision/recall/NDCG/F1,
     - **MIRAGE metrics** like noise vulnerability, context acceptability, context insensitivity, and context misinterpretation.
   - This is the main analytical boundary separating raw model behavior from benchmark interpretation.

10. **Shared utilities**
   - `utils.py` likely holds cross-cutting helpers used by both inference and evaluation.
   - In a repo like this, that usually means JSON I/O, prompt assembly, batching, scoring helpers, and file-path management.

11. **Experiment outputs**
   - The README references an `Evaluation_result` directory containing JSONL reports.
   - That indicates a file-based pipeline: run inference, persist predictions, then evaluate into structured result files rather than using an interactive service.

12. **External dependencies that shape the architecture**
   - The repo is Python-based and depends on:
     - **vLLM** for LLM inference,
     - **SentenceTransformers** for retrieval embeddings,
     - likely **PyYAML** for config loading.
   - Hugging Face is part of the data distribution path, since the dataset is also published there.

13. **Overall data flow**
   - The typical flow is:
     1. Load benchmark JSON data and config.
     2. Choose a mode and prompt template.
     3. Optionally retrieve top-k documents from the doc pool.
     4. Send prompts to the LLM.
     5. Write predictions/results.
     6. Run evaluation to compute metric files and overall scores.
   - An architecture diagram should show these as sequential stages with retrieval and generation as parallel model-specific subsystems.

14. **Key architectural takeaway**
   - MIRAGE is not a serving system; it is an offline benchmark pipeline.
   - The important boundaries are between **static benchmark assets**, **retrieval**, **generation**, and **metric computation**, with JSON/YAML files acting as the glue between stages.

Main entry points: [`main.py`](https://github.com/nlpai-lab/mirage/blob/main/main.py) (main.py), [`LLM.py`](https://github.com/nlpai-lab/mirage/blob/main/LLM.py) (LLM.py), [`retriever.py`](https://github.com/nlpai-lab/mirage/blob/main/retriever.py) (retriever.py), [`config.yaml`](https://github.com/nlpai-lab/mirage/blob/main/config.yaml) (config)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g1["Benchmark assets"]
  node_n11[("dataset.json<br/>qa dataset<br/>[dataset.json]")]
  node_n12[("doc_pool.json<br/>context pool<br/>[doc_pool.json]")]
  node_n13[("oracle.json<br/>gold context<br/>[oracle.json]")]
end

subgraph group_g2["Run orchestration"]
  node_n1["main.py<br/>entrypoint<br/>[main.py]"]
  node_n3["utils.py<br/>shared helpers<br/>[utils.py]"]
  node_n4["config<br/>[config.yaml]"]
end

subgraph group_g3["Inference subsystems"]
  node_n5["Prompt presets<br/>prompt templates"]
  node_n6["base prompt<br/>template<br/>[base.json]"]
  node_n7["fixed prompt<br/>template<br/>[fixed.json]"]
  node_n8["oracle prompt<br/>template<br/>[oracle.json]"]
  node_n9["LLM.py<br/>generation<br/>[LLM.py]"]
  node_n10["retriever.py<br/>retrieval<br/>[retriever.py]"]
  node_n14(("Base mode<br/>run mode"))
  node_n15(("Oracle mode<br/>run mode"))
  node_n16(("RAG mode<br/>run mode"))
end

subgraph group_g4["Scoring"]
  node_n2["evaluation.py<br/>entrypoint<br/>[evaluation.py]"]
  node_n17{{"Metrics"}}
  node_n18[("Results<br/>jsonl output")]
end

node_n1 -->|"loads"| node_n4
node_n1 -->|"uses"| node_n11
node_n1 -->|"uses"| node_n12
node_n1 -->|"uses"| node_n13
node_n1 -->|"selects"| node_n5
node_n1 -->|"runs"| node_n10
node_n1 -->|"runs"| node_n9
node_n1 -->|"writes"| node_n18
node_n4 -->|"configures"| node_n14
node_n4 -->|"configures"| node_n15
node_n4 -->|"configures"| node_n16
node_n5 -->|"contains"| node_n6
node_n5 -->|"contains"| node_n7
node_n5 -->|"contains"| node_n8
node_n11 -->|"retrieval source"| node_n10
node_n12 -->|"indexed by"| node_n10
node_n13 -->|"feeds"| node_n15
node_n14 -->|"prompts"| node_n9
node_n15 -->|"prompts"| node_n9
node_n16 -->|"prompts"| node_n9
node_n16 -->|"depends on"| node_n10
node_n2 -->|"reads"| node_n18
node_n2 -->|"computes"| node_n17
node_n3 -.->|"supports"| node_n1
node_n3 -.->|"supports"| node_n2
node_n9 -->|"outputs"| node_n18
node_n10 -->|"outputs"| node_n18

click node_n1 "https://github.com/nlpai-lab/mirage/blob/main/main.py"
click node_n2 "https://github.com/nlpai-lab/mirage/blob/main/evaluation.py"
click node_n3 "https://github.com/nlpai-lab/mirage/blob/main/utils.py"
click node_n4 "https://github.com/nlpai-lab/mirage/blob/main/config.yaml"
click node_n5 "https://github.com/nlpai-lab/mirage/tree/main/Preset_prompt"
click node_n6 "https://github.com/nlpai-lab/mirage/blob/main/Preset_prompt/base.json"
click node_n7 "https://github.com/nlpai-lab/mirage/blob/main/Preset_prompt/fixed.json"
click node_n8 "https://github.com/nlpai-lab/mirage/blob/main/Preset_prompt/oracle.json"
click node_n9 "https://github.com/nlpai-lab/mirage/blob/main/LLM.py"
click node_n10 "https://github.com/nlpai-lab/mirage/blob/main/retriever.py"
click node_n11 "https://github.com/nlpai-lab/mirage/blob/main/mirage/dataset.json"
click node_n12 "https://github.com/nlpai-lab/mirage/blob/main/mirage/doc_pool.json"
click node_n13 "https://github.com/nlpai-lab/mirage/blob/main/mirage/oracle.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n11,node_n12,node_n13 toneBlue
class node_n1,node_n3,node_n4 toneAmber
class node_n5,node_n6,node_n7,node_n8,node_n9,node_n10,node_n14,node_n15,node_n16 toneMint
class node_n2,node_n17,node_n18 toneRose
```

## Components

### Benchmark assets

Canonical dataset, document pool, and oracle references used across runs.

- [dataset.json](https://github.com/nlpai-lab/mirage/blob/main/mirage/dataset.json): qa dataset · Benchmark questions and examples. · `mirage/dataset.json`
- [doc_pool.json](https://github.com/nlpai-lab/mirage/blob/main/mirage/doc_pool.json): context pool · Candidate documents used for retrieval and noisy context mixing. · `mirage/doc_pool.json`
- [oracle.json](https://github.com/nlpai-lab/mirage/blob/main/mirage/oracle.json): gold context · Oracle context references for the gold setting. · `mirage/oracle.json`

### Run orchestration

Entry points and shared utilities that prepare and launch experiments.

- [main.py](https://github.com/nlpai-lab/mirage/blob/main/main.py): entrypoint · Orchestrates benchmark runs and mode selection. · `main.py`
- [utils.py](https://github.com/nlpai-lab/mirage/blob/main/utils.py): shared helpers · Common helpers for I/O, formatting, and pipeline plumbing. · `utils.py`
- [config](https://github.com/nlpai-lab/mirage/blob/main/config.yaml): yaml config · Central experiment settings for models, retrieval, and outputs. · `config.yaml`

### Inference subsystems

Prompting, retrieval, and generation components used in benchmark modes.

- [Prompt presets](https://github.com/nlpai-lab/mirage/tree/main/Preset_prompt): prompt templates · Prompt variants for base, fixed, and oracle modes. · `Preset_prompt`
- [base prompt](https://github.com/nlpai-lab/mirage/blob/main/Preset_prompt/base.json): template · Prompt template for question-only inference. · `Preset_prompt/base.json`
- [fixed prompt](https://github.com/nlpai-lab/mirage/blob/main/Preset_prompt/fixed.json): template · Prompt template for mixed or fixed-context inference. · `Preset_prompt/fixed.json`
- [oracle prompt](https://github.com/nlpai-lab/mirage/blob/main/Preset_prompt/oracle.json): template · Prompt template for oracle-context inference. · `Preset_prompt/oracle.json`
- [LLM.py](https://github.com/nlpai-lab/mirage/blob/main/LLM.py): generation · Batched vLLM inference for answer generation. · `LLM.py`
- [retriever.py](https://github.com/nlpai-lab/mirage/blob/main/retriever.py): retrieval · SentenceTransformer-based ranking of document pool chunks. · `retriever.py`
- **Base mode**: run mode · Question-only inference path without retrieved context.
- **Oracle mode**: run mode · Inference path using correct context.
- **RAG mode**: run mode · Inference path using retrieved mixed context.

### Scoring

Post-run evaluation and metric computation.

- [evaluation.py](https://github.com/nlpai-lab/mirage/blob/main/evaluation.py): entrypoint · Scores predictions and produces benchmark reports. · `evaluation.py`
- **Metrics**: benchmark metrics · EM/F1, retriever metrics, and MIRAGE-specific scores.
- **Results**: jsonl output · Persisted prediction and evaluation reports.

## Connections

- main.py → config: loads · Reads experiment configuration before launching runs.
- main.py → dataset.json: uses · Consumes benchmark questions for all modes.
- main.py → doc_pool.json: uses · Accesses the document pool for retrieval-based runs.
- main.py → oracle.json: uses · Uses oracle references for gold-context runs.
- main.py → Prompt presets: selects · Chooses the prompt family for the active mode.
- main.py → retriever.py: runs · Invokes retrieval when contexts must be ranked.
- main.py → LLM.py: runs · Invokes LLM generation for answer creation.
- main.py → Results: writes · Persists predictions and run artifacts.
- config → Base mode: configures · Defines settings for base runs.
- config → Oracle mode: configures · Defines settings for oracle runs.
- config → RAG mode: configures · Defines settings for RAG runs.
- Prompt presets → base prompt: contains · Base prompt template.
- Prompt presets → fixed prompt: contains · Fixed-context prompt template.
- Prompt presets → oracle prompt: contains · Oracle prompt template.
- dataset.json → retriever.py: retrieval source · Questions are paired with document ranking against the pool.
- doc_pool.json → retriever.py: indexed by · Document chunks are ranked for top-k retrieval.
- oracle.json → Oracle mode: feeds · Supplies correct contexts to oracle mode.
- Base mode → LLM.py: prompts · Question-only prompts are sent to the generator.
- Oracle mode → LLM.py: prompts · Oracle-context prompts are sent to the generator.
- RAG mode → LLM.py: prompts · Retrieved-context prompts are sent to the generator.
- RAG mode → retriever.py: depends on · RAG mode requires retrieval results.
- evaluation.py → Results: reads · Loads prediction artifacts for scoring.
- evaluation.py → Metrics: computes · Produces benchmark metrics from predictions and references.
- utils.py → main.py: supports · Shared helpers support orchestration.
- utils.py → evaluation.py: supports · Shared helpers support evaluation and reporting.
- LLM.py → Results: outputs · Generated answers are written for later evaluation.
- retriever.py → Results: outputs · Retrieval results are stored alongside predictions.