# ibm/mt-rag-benchmark architecture

> Architecture diagram of the GitHub repository ibm/mt-rag-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ibm/mt-rag-benchmark
- Repository: https://github.com/ibm/mt-rag-benchmark

## Overview

##### Project type and purpose
- MTRAG is an open-source benchmarking toolkit and dataset for multi-turn retrieval-augmented generation (RAG).
- It bundles: four document corpora, human-annotated multi-turn dialogs, synthetic dialogs, scripts to convert dialogs into retrieval and generation tasks, and evaluation/analytics outputs.

##### File structure
- Top-level:
- corpora/ – compressed JSONL corpora for four domains (ClapNQ, Cloud, FiQA, Govt)
- human/ – human-generated conversations, retrieval tasks, generation tasks, evaluation results
- synthetic/ – synthetic conversations and generation tasks
- scripts/ – conversion tools (e.g., conversations2retrieval.py)
- README.md, LICENSE, .gitattributes
- Patterns:
- Data layer: corpora + dialogs
- Pipeline layer: scripts → retrieval tasks → generation tasks
- Evaluation layer: analytics in human/evaluations
- Two “modes”: human vs. synthetic

##### README for additional insights
- Describes dataset (110 human conversations → 842 tasks)
- Retrieval tasks follow BEIR format
- Generation tasks under three settings (Reference, Reference+RAG, Full RAG)
- Synthetic data generation pipeline and links to relevant papers and tooling

##### Elements
- Clear labels: “Corpora”, “Human Conversations”, “Synthetic Conversations”, “Task Generator”, “Retrieval Engine”, “Generation Engine”, “Analytics”
- Corpora → Retrieval Ingestion
- Conversations → Task Generator → {Retrieval, Generation}
- Retrieval → Generation
- Generation → Evaluation
- Evaluation → Visualization
- Processing steps (e.g., rectangles, green)
- External services (e.g., rounded rectangles, orange)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources
    subgraph "Data Sources"
        direction TB
        Corpora[(Document Corpora)]:::data
        subgraph "Corpora Files"
            direction TB
            ClapNQ["ClapNQ.jsonl.zip"]:::data
            Cloud["Cloud.jsonl.zip"]:::data
            FiQA["FiQA.jsonl.zip"]:::data
            Govt["Govt.jsonl.zip"]:::data
        end
        HumanConv[(Human Conversations)]:::data
        SynthConv[(Synthetic Conversations)]:::data
    end

    %% Processing / Task Generation
    subgraph "Processing"
        direction TB
        TaskGen["Task Generator\n(conversations2retrieval.py)"]:::process
        subgraph "Human Tasks"
            direction TB
            HRetrieval[(Human Retrieval Tasks)]:::storage
            HGen[(Human Generation Tasks)]:::storage
        end
        SynthGen[(Synthetic Generation Tasks)]:::storage
    end

    %% Modeling
    subgraph "Modeling"
        direction TB
        RetrievalEngine[(Retrieval Engine)]:::external
        GenerationEngine[(Generation Engine)]:::external
    end

    %% Evaluation & Analytics
    subgraph "Evaluation"
        direction TB
        subgraph "Human Evaluation"
            direction TB
            HEval[(Human Evaluations)]:::storage
        end
        SynthEval[(Synthetic Evaluation)]:::storage
    end

    %% Visualization
    InspectorUI[(InspectorRAGet UI)]:::external

    %% Data Flows
    Corpora -->|ingest| RetrievalEngine
    HumanConv -->|load| TaskGen
    SynthConv -->|load| TaskGen
    TaskGen -->|generate retrieval tasks| HRetrieval
    TaskGen -->|generate generation tasks| HGen
    TaskGen -->|generate generation tasks| SynthGen
    HRetrieval -->|input| RetrievalEngine
    RetrievalEngine -->|retrieved passages| GenerationEngine
    HGen -->|reference tasks| GenerationEngine
    SynthGen -->|synthetic tasks| GenerationEngine
    GenerationEngine -->|outputs| HEval
    GenerationEngine -->|outputs| SynthEval
    HEval -->|visualize| InspectorUI
    SynthEval -->|visualize| InspectorUI

    %% Click Events
    click Corpora "https://github.com/ibm/mt-rag-benchmark/tree/main/corpora/"
    click ClapNQ "https://github.com/ibm/mt-rag-benchmark/blob/main/corpora/clapnq.jsonl.zip"
    click Cloud "https://github.com/ibm/mt-rag-benchmark/blob/main/corpora/cloud.jsonl.zip"
    click FiQA "https://github.com/ibm/mt-rag-benchmark/blob/main/corpora/fiqa.jsonl.zip"
    click Govt "https://github.com/ibm/mt-rag-benchmark/blob/main/corpora/govt.jsonl.zip"
    click HumanConv "https://github.com/ibm/mt-rag-benchmark/blob/main/human/conversations/conversations.json"
    click SynthConv "https://github.com/ibm/mt-rag-benchmark/blob/main/synthetic/conversations/conversations.json"
    click TaskGen "https://github.com/ibm/mt-rag-benchmark/blob/main/scripts/conversations2retrieval.py"
    click HRetrieval "https://github.com/ibm/mt-rag-benchmark/tree/main/human/retrieval_tasks/"
    click HGen "https://github.com/ibm/mt-rag-benchmark/tree/main/human/generation_tasks/"
    click SynthGen "https://github.com/ibm/mt-rag-benchmark/blob/main/synthetic/generation_tasks/synthetic.jsonl"
    click HEval "https://github.com/ibm/mt-rag-benchmark/tree/main/human/evaluations/"
    click SynthEval "https://github.com/ibm/mt-rag-benchmark/blob/main/synthetic/evaluations/synthetic.json"

    %% Styles
    classDef data fill:#D0E6FF,stroke:#0284C7,color:#03416A
    classDef process fill:#D4F1C5,stroke:#5A8A24,color:#2F4F1E
    classDef external fill:#FFE3B3,stroke:#FF8C00,color:#804000
    classDef storage fill:#E0E0E0,stroke:#636363,color:#1A1A1A
```
