How yixuantt/multihop-rag works
Project Type and Purpose
• This is a research-oriented data pipeline and evaluation toolkit for a multi-document retrieval-augmented generation (RAG) dataset.
• It provides (a) data collection & processing scripts to build the dataset, (b) retrieval and QA sample scripts, and (c) evaluation scripts.
File Structure
Top-level components:
• dataset/ – final JSON dataset (MultiHopRAG.json, corpus.json)
• pipeline/ – ETL scripts to crawl news, filter, and construct claims
• simple_retrieval.py, qa_llama.py – demo scripts for retrieval & QA
• retrieval_evaluate.py, qa_evaluate.py – evaluation modules
• toy_data/ – example intermediate data
• util.py – shared utilities
• requirement.txt – dependency list
Key Insights from README
• The pipeline is used “just for reference” and may be reorganized later.
• The simple_retrieval.py demonstrates embedding retrieval (llama-index).
• The qa_llama.py uses Llama LLM to generate answers.
• Evaluation scripts compute metrics for retrieval and QA.
Main Components and Their Relationships
A. External Services
• News API (data source)
• Hugging Face model hub (for embeddings & LLMs)
• Llama / BAAI embeddings & LLM endpoints (or local)
B. Data Ingestion & Crawling
• pipeline/news_crawler.py → raw articles
• pipeline/collect_news_api.py → calls news_api.json configuration
C. Data Processing Pipeline (ETL)
• pipeline/filter_*.py → remove duplicates, tokenize, same-fact filters
• pipeline/generate_entity.py, generate_key.py → extract metadata & keys
• pipeline/group_by_key.py → cluster evidence docs per claim
• pipeline/prepare_claims.py → assemble final QA entries
• util.py (pipeline/util.py and root util.py) → shared helper functions
→ Output: dataset/MultiHopRAG.json + dataset/corpus.json
D. Demo & Evaluation
• simple_retrieval.py → embedding retrieval, optional rerank
• qa_llama.py → generate answers via Llama
• retrieval_evaluate.py → compute retrieval metrics from saved results
• qa_evaluate.py → compute QA metrics
Architectural Patterns & Design Principles
• Linear ETL pipeline (batch processing)
• Separation of concerns: ingestion, filtering, feature extraction, grouping, final assembly
• Modular scripts for easy extension or replacement
• Demo vs. evaluation split