# yixuantt/multihop-rag architecture

> Architecture diagram of the GitHub repository yixuantt/multihop-rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yixuantt/multihop-rag
- Repository: https://github.com/yixuantt/multihop-rag

## Overview

##### Project Type and Purpose
- This is a research-oriented data pipeline and evaluation toolkit for a multi-document retrieval-augmented generation (RAG) dataset.
- It provides (a) data collection & processing scripts to build the dataset, (b) retrieval and QA sample scripts, and (c) evaluation scripts.

##### File Structure
Top-level components:
- dataset/ – final JSON dataset (MultiHopRAG.json, corpus.json)
- pipeline/ – ETL scripts to crawl news, filter, and construct claims
- simple_retrieval.py, qa_llama.py – demo scripts for retrieval & QA
- retrieval_evaluate.py, qa_evaluate.py – evaluation modules
- toy_data/ – example intermediate data
- util.py – shared utilities
- requirement.txt – dependency list

##### Key Insights from README
- The pipeline is used “just for reference” and may be reorganized later.
- The simple_retrieval.py demonstrates embedding retrieval (llama-index).
- The qa_llama.py uses Llama LLM to generate answers.
- Evaluation scripts compute metrics for retrieval and QA.

##### Main Components and Their Relationships
A. External Services
- News API (data source)
- Hugging Face model hub (for embeddings & LLMs)
- Llama / BAAI embeddings & LLM endpoints (or local)
B. Data Ingestion & Crawling
- pipeline/news_crawler.py → raw articles
- pipeline/collect_news_api.py → calls news_api.json configuration
C. Data Processing Pipeline (ETL)
- pipeline/filter_*.py → remove duplicates, tokenize, same-fact filters
- pipeline/generate_entity.py, generate_key.py → extract metadata & keys
- pipeline/group_by_key.py → cluster evidence docs per claim
- pipeline/prepare_claims.py → assemble final QA entries
- util.py (pipeline/util.py and root util.py) → shared helper functions
→ Output: dataset/MultiHopRAG.json + dataset/corpus.json
D. Demo & Evaluation
- simple_retrieval.py → embedding retrieval, optional rerank
- qa_llama.py → generate answers via Llama
- retrieval_evaluate.py → compute retrieval metrics from saved results
- qa_evaluate.py → compute QA metrics

##### Architectural Patterns & Design Principles
- Linear ETL pipeline (batch processing)
- Separation of concerns: ingestion, filtering, feature extraction, grouping, final assembly
- Modular scripts for easy extension or replacement
- Demo vs. evaluation split

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Services
    subgraph "External Services"
        direction TB
        NewsAPI["News API"]:::external
        HFModels["Hugging Face Model Hub\n(Embeddings & LLM)"]:::external
    end

    %% Data Ingestion
    subgraph "Data Ingestion / Crawling" 
        direction TB
        Collect["collect_news_api.py"]:::ingestion
        Crawler["news_crawler.py"]:::ingestion
    end

    %% Shared Utilities
    subgraph "Shared Utilities"
        direction TB
        UtilPipeline["pipeline/util.py"]:::utility
        UtilGlobal["util.py"]:::utility
    end

    %% ETL Filters & Transformations
    subgraph "ETL Pipeline" 
        direction TB
        FilterDup["filter_news_dupicate.py"]:::etl
        FilterToken["filter_news_token.py"]:::etl
        FilterSame["filter_same_fact.py"]:::etl
        EntityExtract["generate_entity.py"]:::etl
        KeyGen["generate_key.py"]:::etl
        GroupBy["group_by_key.py"]:::etl
        PrepareClaims["prepare_claims.py"]:::etl
    end

    %% Example Data
    ToyData["toy_data/step1_data.json"]:::storage

    %% Dataset Storage
    subgraph "Dataset Storage"
        direction TB
        MultiHop["MultiHopRAG.json"]:::storage
        Corpus["corpus.json"]:::storage
    end

    %% Demos & Model Interaction
    subgraph "Demo & Model Interaction"
        direction TB
        RetrievalDemo["simple_retrieval.py"]:::demo
        QADemo["qa_llama.py"]:::demo
    end

    %% Evaluation
    subgraph "Evaluation"
        direction TB
        EvalRetrieval["retrieval_evaluate.py"]:::evaluation
        EvalQA["qa_evaluate.py"]:::evaluation
    end

    %% Resources & Docs
    Resource["resource/"]:::utility
    Req["requirement.txt"]:::utility
    PipelineDoc["pipeline/readme.md"]:::utility

    %% Data Flows
    NewsAPI -->|"raw articles"| Collect
    NewsAPI -->|"raw crawl"| Crawler
    Collect --> FilterDup
    Crawler --> FilterDup
    FilterDup --> FilterToken
    FilterToken --> FilterSame
    FilterSame --> EntityExtract
    EntityExtract --> KeyGen
    KeyGen --> GroupBy
    GroupBy --> PrepareClaims
    PrepareClaims --> MultiHop
    PrepareClaims --> Corpus
    PrepareClaims -->|"intermediate example"| ToyData

    MultiHop -->|"dataset json"| RetrievalDemo
    HFModels -->|"embeddings"| RetrievalDemo
    RetrievalDemo -->|"retrieved passages"| QADemo
    MultiHop -->|"dataset json"| QADemo
    HFModels -->|"LLM calls"| QADemo

    RetrievalDemo -->|"results"| EvalRetrieval
    QADemo -->|"answers"| EvalQA

    %% Utilities usage
    UtilPipeline -.-> Collect
    UtilPipeline -.-> Crawler
    UtilPipeline -.-> FilterDup
    UtilPipeline -.-> FilterToken
    UtilPipeline -.-> FilterSame
    UtilPipeline -.-> EntityExtract
    UtilPipeline -.-> KeyGen
    UtilPipeline -.-> GroupBy
    UtilPipeline -.-> PrepareClaims
    UtilGlobal -.-> RetrievalDemo
    UtilGlobal -.-> QADemo
    UtilGlobal -.-> EvalRetrieval
    UtilGlobal -.-> EvalQA

    %% Click Events
    click Collect "https://github.com/yixuantt/multihop-rag/blob/main/pipeline/collect_news_api.py"
    click Crawler "https://github.com/yixuantt/multihop-rag/blob/main/pipeline/news_crawler.py"
    click FilterDup "https://github.com/yixuantt/multihop-rag/blob/main/pipeline/filter_news_dupicate.py"
    click FilterToken "https://github.com/yixuantt/multihop-rag/blob/main/pipeline/filter_news_token.py"
    click FilterSame "https://github.com/yixuantt/multihop-rag/blob/main/pipeline/filter_same_fact.py"
    click EntityExtract "https://github.com/yixuantt/multihop-rag/blob/main/pipeline/generate_entity.py"
    click KeyGen "https://github.com/yixuantt/multihop-rag/blob/main/pipeline/generate_key.py"
    click GroupBy "https://github.com/yixuantt/multihop-rag/blob/main/pipeline/group_by_key.py"
    click PrepareClaims "https://github.com/yixuantt/multihop-rag/blob/main/pipeline/prepare_claims.py"
    click UtilPipeline "https://github.com/yixuantt/multihop-rag/blob/main/pipeline/util.py"
    click UtilGlobal "https://github.com/yixuantt/multihop-rag/blob/main/util.py"
    click MultiHop "https://github.com/yixuantt/multihop-rag/blob/main/dataset/MultiHopRAG.json"
    click Corpus "https://github.com/yixuantt/multihop-rag/blob/main/dataset/corpus.json"
    click RetrievalDemo "https://github.com/yixuantt/multihop-rag/blob/main/simple_retrieval.py"
    click QADemo "https://github.com/yixuantt/multihop-rag/blob/main/qa_llama.py"
    click EvalRetrieval "https://github.com/yixuantt/multihop-rag/blob/main/retrieval_evaluate.py"
    click EvalQA "https://github.com/yixuantt/multihop-rag/blob/main/qa_evaluate.py"
    click ToyData "https://github.com/yixuantt/multihop-rag/blob/main/toy_data/step1_data.json"
    click Resource "https://github.com/yixuantt/multihop-rag/tree/main/resource/"
    click Req "https://github.com/yixuantt/multihop-rag/blob/main/requirement.txt"
    click PipelineDoc "https://github.com/yixuantt/multihop-rag/blob/main/pipeline/readme.md"

    %% Styles
    classDef external fill:#eee,stroke:#999,stroke-width:1px
    classDef ingestion fill:#cde,stroke:#333,stroke-width:1px
    classDef etl fill:#acf,stroke:#333,stroke-width:1px
    classDef storage fill:#cfc,stroke:#333,stroke-width:1px
    classDef demo fill:#fc9,stroke:#333,stroke-width:1px
    classDef evaluation fill:#c9f,stroke:#333,stroke-width:1px
    classDef utility fill:#ffd,stroke:#333,stroke-width:1px
```
