# pdichone/advanced-rag-techniques architecture

> Architecture diagram of the GitHub repository pdichone/advanced-rag-techniques, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pdichone/advanced-rag-techniques
- Repository: https://github.com/pdichone/advanced-rag-techniques

## Overview

##### Project type and purpose
- This is an open-source AI tool or library module (part of “The AI Guild”) focused on retrieval-augmented QA and document processing.
- The README positions it as a code module in a community for building AI solutions—no frontend or UI, just backend Python scripts and data.

##### File structure
Top-level files and directories:
- data/ (holds raw documents; in this case, a PDF)
- dpr_technique.py (Dense Passage Retrieval setup)
- expansion_queries.py & expansion_answer.py (query & answer expansion logic)
- reranking.py (reordering retrieved passages)
- helper_utils.py (shared utilities for preprocessing, I/O, etc.)
- requirements.txt (pins dependencies)
- CLAUDE.md (notes about an LLM, likely an external service)

Patterns:
- Each major step in the retrieval-augmented QA pipeline is encapsulated in its own module.
- There is a clear separation of concerns: data ingestion, query expansion, retrieval (DPR), reranking, and answer expansion.

##### Key README insights
- External dependencies implied: deep-learning framework (e.g., PyTorch/Transformers), vector store or embeddings service, and an LLM API (e.g., Claude).

##### Main components
a. Data Ingestion & Preprocessing
- Component: helper_utils.py + data/
- Role: load PDF, clean text, chunk and index documents.

b. Embedding & Retrieval
- Component: dpr_technique.py
- Role: generate dense embeddings, query the vector store.

c. Query Expansion
- Component: expansion_queries.py
- Role: reformulate user queries for better recall.

d. Reranking
- Component: reranking.py
- Role: use a cross-encoder or LLM to reorder top-k passages.

e. Answer Expansion
- Component: expansion_answer.py
- Role: generate final answer from reranked passages via LLM.

f. External Services
- Vector database or in-memory FAISS index
- LLM API (e.g., Anthropic Claude, OpenAI)

##### Relationships & data flow
- User Query → Query Expansion → DPR Retrieval → Top-n Passages → Reranking → Best Passages → Answer Expansion → Final Response
- Data preprocessing and indexing happen upstream and feed the vector store.

##### Architectural patterns & principles
- Pipeline pattern: each script is a stage in a linear data flow.
- Separation of concerns: modular, single-purpose components.
- Extensibility points: swap out DPR with BM25, plug in different LLMs via helper_utils.

## Diagram (Mermaid)

```mermaid
graph TB
    %% External actor
    User["User Query"]:::external

    %% Data components
    DataFile["PDF Corpus"]:::data
    Preproc["Data Ingestion & Preprocessing\n(helper_utils.py)"]:::data
    VectorDB["Vector DB"]:::data

    %% ML components
    QueryExp["Query Expansion\n(expansion_queries.py)"]:::ml
    DPR["Embedding & Retrieval\n(dpr_technique.py)"]:::ml
    Rerank["Reranking\n(reranking.py)"]:::ml
    AnsExp["Answer Expansion\n(expansion_answer.py)"]:::ml

    %% External LLM
    LLM["LLM Service\n(Anthropic Claude)"]:::llm

    %% Final response
    Response["API Response"]:::data

    %% Dependency note
    Req["requirements.txt"]:::note

    %% Data flow
    DataFile -->|"load & clean text"| Preproc
    Preproc -->|"indexed embeddings"| VectorDB

    User -->|"raw query"| QueryExp
    QueryExp -->|"expanded query"| DPR
    DPR -->|"query embeddings"| VectorDB
    VectorDB -->|"top-n passages"| Rerank
    Rerank -->|"best passages"| AnsExp
    AnsExp -->|"prompt"| LLM
    LLM -->|"generated answer"| Response

    %% Optional LLM in reranking
    Rerank -->|"cross-encoder / LLM"| LLM

    %% Dependencies
    Req --> Preproc
    Req --> QueryExp
    Req --> DPR
    Req --> Rerank
    Req --> AnsExp

    %% Click events
    click DataFile "https://github.com/pdichone/advanced-rag-techniques/blob/main/data/microsoft-annual-report.pdf"
    click Preproc "https://github.com/pdichone/advanced-rag-techniques/blob/main/helper_utils.py"
    click DPR "https://github.com/pdichone/advanced-rag-techniques/blob/main/dpr_technique.py"
    click QueryExp "https://github.com/pdichone/advanced-rag-techniques/blob/main/expansion_queries.py"
    click Rerank "https://github.com/pdichone/advanced-rag-techniques/blob/main/reranking.py"
    click AnsExp "https://github.com/pdichone/advanced-rag-techniques/blob/main/expansion_answer.py"
    click LLM "https://github.com/pdichone/advanced-rag-techniques/blob/main/CLAUDE.md"
    click Req "https://github.com/pdichone/advanced-rag-techniques/blob/main/requirements.txt"

    %% Styles
    classDef data fill:#D0E6F6,stroke:#0366D6,color:#0366D6;
    classDef ml fill:#D6F6D0,stroke:#027A1F,color:#027A1F;
    classDef llm fill:#F6E2D0,stroke:#D97706,color:#D97706;
    classDef external fill:#E2E2E2,stroke:#6E6E6E,color:#6E6E6E;
    classDef note fill:#F0D0F6,stroke:#9B2FAE,color:#9B2FAE;
```
