How ibm/mt-rag-benchmark works
Project type and purpose
• MTRAG is an open-source benchmarking toolkit and dataset for multi-turn retrieval-augmented generation (RAG).
• It bundles: four document corpora, human-annotated multi-turn dialogs, synthetic dialogs, scripts to convert dialogs into retrieval and generation tasks, and evaluation/analytics outputs.
File structure
• Top-level:
• corpora/ – compressed JSONL corpora for four domains (ClapNQ, Cloud, FiQA, Govt)
• human/ – human-generated conversations, retrieval tasks, generation tasks, evaluation results
• synthetic/ – synthetic conversations and generation tasks
• scripts/ – conversion tools (e.g., conversations2retrieval.py)
• README.md, LICENSE, .gitattributes
• Patterns:
• Data layer: corpora + dialogs
• Pipeline layer: scripts → retrieval tasks → generation tasks
• Evaluation layer: analytics in human/evaluations
• Two “modes”: human vs. synthetic
README for additional insights
• Describes dataset (110 human conversations → 842 tasks)
• Retrieval tasks follow BEIR format
• Generation tasks under three settings (Reference, Reference+RAG, Full RAG)
• Synthetic data generation pipeline and links to relevant papers and tooling
Elements
• Clear labels: “Corpora”, “Human Conversations”, “Synthetic Conversations”, “Task Generator”, “Retrieval Engine”, “Generation Engine”, “Analytics”
• Corpora → Retrieval Ingestion
• Conversations → Task Generator → {Retrieval, Generation}
• Retrieval → Generation
• Generation → Evaluation
• Evaluation → Visualization
• Processing steps (e.g., rectangles, green)
• External services (e.g., rounded rectangles, orange)