# peishuen/rag-personal-document-assistant architecture

> Architecture diagram of the GitHub repository peishuen/rag-personal-document-assistant, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/peishuen/rag-personal-document-assistant
- Repository: https://github.com/peishuen/rag-personal-document-assistant

## Overview

This is a Streamlit personal-document RAG assistant: users upload documents, ask questions, and receive answers grounded in retrieved passages with page-level citations. The main flow covers ingestion, adaptive chunking, embedding and storage, hybrid retrieval, answer generation, grounding checks, and citation. The README also documents batch and cross-document Q&A, document management, and retrieval evaluation. Some subsystems are documented but not sampled; their inclusion and links rely on the README, with uncertain wiring omitted.

Main entry points: [`app.py`](https://github.com/peishuen/rag-personal-document-assistant/blob/main/app.py) (Streamlit app), [`src/vectorstore/chroma_store.py`](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/vectorstore/chroma_store.py) (Chroma store), [`src/ingestion/batch_loader.py`](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/ingestion/batch_loader.py) (Batch loader), [`src/retrieval/hybrid_ranker.py`](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/retrieval/hybrid_ranker.py) (Hybrid ranker)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["Assistant UI"]
  node_app["Streamlit app<br/>[app.py]"]
end

subgraph group_ingest["Ingestion and indexing"]
  node_batch["Batch loader<br/>[batch_loader.py]"]
  node_pdf["PDF loader<br/>[pdf_loader.py]"]
  node_text["Text loader<br/>[text_loader.py]"]
  node_ocr["Scanned PDF OCR<br/>[ocr_loader.py]"]
  node_chunk["Adaptive chunker<br/>[chunker.py]"]
  node_embed["Embedding service<br/>[embedder.py]"]
  node_store["Chroma store<br/>[chroma_store.py]"]
  node_chroma[("Persistent chunk index")]
end

subgraph group_retrieve["Retrieval"]
  node_crossdoc["Cross-document search<br/>[cross_document.py]"]
  node_hybrid["Hybrid ranker<br/>[hybrid_ranker.py]"]
  node_bm25["BM25 search<br/>[bm25_retriever.py]"]
  node_semantic["Semantic search"]
  node_rewriter["Query rewriter<br/>[query_rewriter.py]"]
end

subgraph group_answer["Answer and citations"]
  node_generator["Answer generator<br/>[generator.py]"]
  node_grounding["Grounding check<br/>[grounding_check.py]"]
  node_citation["Citation tracker"]
end

subgraph group_eval["Evaluation"]
  node_evaluation["Retrieval evaluation<br/>[evaluation.py]"]
  node_embedding_eval["Embedding comparison"]
end

node_user(("Document user"))
node_dashscope{{"DashScope API"}}
node_llm{{"Qwen language model"}}
node_tesseract{{"Tesseract OCR"}}

node_user -->|"uploads and asks"| node_app
node_app -->|"loads documents"| node_batch
node_batch -->|"routes PDFs"| node_pdf
node_batch -->|"routes text"| node_text
node_batch -->|"routes scans"| node_ocr
node_ocr -->|"extracts text"| node_tesseract
node_app -->|"chunks pages"| node_chunk
node_app -->|"embeds chunks"| node_embed
node_embed -->|"requests vectors"| node_dashscope
node_app -->|"stores chunks"| node_store
node_store -->|"writes and reads"| node_chroma
node_app -->|"manages sources"| node_store
node_app -->|"rewrites follow-ups"| node_rewriter
node_app -->|"searches documents"| node_crossdoc
node_crossdoc -->|"combines retrieval"| node_hybrid
node_hybrid -->|"requests keyword results"| node_bm25
node_hybrid -->|"requests semantic results"| node_semantic
node_bm25 -->|"reads chunks"| node_store
node_semantic -->|"embeds query"| node_embed
node_semantic -->|"searches vectors"| node_store
node_app -->|"generates answer"| node_generator
node_generator -->|"requests completion"| node_llm
node_app -->|"checks support"| node_grounding
node_app -->|"formats sources"| node_citation

click node_app "https://github.com/peishuen/rag-personal-document-assistant/blob/main/app.py"
click node_batch "https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/ingestion/batch_loader.py"
click node_pdf "https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/ingestion/pdf_loader.py"
click node_text "https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/ingestion/text_loader.py"
click node_ocr "https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/ingestion/ocr_loader.py"
click node_chunk "https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/chunking/chunker.py"
click node_embed "https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/embeddings/embedder.py"
click node_store "https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/vectorstore/chroma_store.py"
click node_chroma "https://github.com/peishuen/rag-personal-document-assistant/tree/main/data/chroma_db"
click node_crossdoc "https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/retrieval/cross_document.py"
click node_hybrid "https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/retrieval/hybrid_ranker.py"
click node_bm25 "https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/retrieval/bm25_retriever.py"
click node_semantic "https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/retrieval/semantic_retriever.py"
click node_rewriter "https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/generation/query_rewriter.py"
click node_generator "https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/generation/generator.py"
click node_grounding "https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/generation/grounding_check.py"
click node_citation "https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/citation/citation_tracker.py"
click node_evaluation "https://github.com/peishuen/rag-personal-document-assistant/blob/main/evaluation/evaluation.py"
click node_embedding_eval "https://github.com/peishuen/rag-personal-document-assistant/blob/main/evaluation/embedding_comparison.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_user toneBlue
class node_batch,node_pdf,node_text,node_ocr,node_chunk,node_embed,node_store,node_chroma toneAmber
class node_crossdoc,node_hybrid,node_bm25,node_semantic,node_rewriter,node_dashscope toneMint
class node_generator,node_grounding,node_citation toneRose
class node_evaluation,node_embedding_eval,node_llm,node_tesseract toneIndigo
```

## Components

### Assistant UI

- [Streamlit app](https://github.com/peishuen/rag-personal-document-assistant/blob/main/app.py): `app.py`

### Ingestion and indexing

- [Batch loader](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/ingestion/batch_loader.py): `src/ingestion/batch_loader.py`
- [PDF loader](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/ingestion/pdf_loader.py): `src/ingestion/pdf_loader.py`
- [Text loader](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/ingestion/text_loader.py): `src/ingestion/text_loader.py`
- [Scanned PDF OCR](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/ingestion/ocr_loader.py): `src/ingestion/ocr_loader.py`
- [Adaptive chunker](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/chunking/chunker.py): `src/chunking/chunker.py`
- [Embedding service](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/embeddings/embedder.py): `src/embeddings/embedder.py`
- [Chroma store](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/vectorstore/chroma_store.py): `src/vectorstore/chroma_store.py`
- [Persistent chunk index](https://github.com/peishuen/rag-personal-document-assistant/tree/main/data/chroma_db): `data/chroma_db`

### Retrieval

- [Cross-document search](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/retrieval/cross_document.py): `src/retrieval/cross_document.py`
- [Hybrid ranker](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/retrieval/hybrid_ranker.py): `src/retrieval/hybrid_ranker.py`
- [BM25 search](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/retrieval/bm25_retriever.py): `src/retrieval/bm25_retriever.py`
- [Semantic search](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/retrieval/semantic_retriever.py): `src/retrieval/semantic_retriever.py`
- [Query rewriter](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/generation/query_rewriter.py): `src/generation/query_rewriter.py`

### Answer and citations

- [Answer generator](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/generation/generator.py): `src/generation/generator.py`
- [Grounding check](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/generation/grounding_check.py): `src/generation/grounding_check.py`
- [Citation tracker](https://github.com/peishuen/rag-personal-document-assistant/blob/main/src/citation/citation_tracker.py): `src/citation/citation_tracker.py`

### Evaluation

- [Retrieval evaluation](https://github.com/peishuen/rag-personal-document-assistant/blob/main/evaluation/evaluation.py): `evaluation/evaluation.py`
- [Embedding comparison](https://github.com/peishuen/rag-personal-document-assistant/blob/main/evaluation/embedding_comparison.py): `evaluation/embedding_comparison.py`

### Other components

- **Document user**
- **DashScope API**
- **Qwen language model**
- **Tesseract OCR**

## Connections

- Document user → Streamlit app: uploads and asks
- Streamlit app → Batch loader: loads documents
- Batch loader → PDF loader: routes PDFs
- Batch loader → Text loader: routes text
- Batch loader → Scanned PDF OCR: routes scans
- Scanned PDF OCR → Tesseract OCR: extracts text
- Streamlit app → Adaptive chunker: chunks pages
- Streamlit app → Embedding service: embeds chunks
- Embedding service → DashScope API: requests vectors
- Streamlit app → Chroma store: stores chunks
- Chroma store → Persistent chunk index: writes and reads
- Streamlit app → Chroma store: manages sources
- Streamlit app → Query rewriter: rewrites follow-ups
- Streamlit app → Cross-document search: searches documents
- Cross-document search → Hybrid ranker: combines retrieval
- Hybrid ranker → BM25 search: requests keyword results
- Hybrid ranker → Semantic search: requests semantic results
- BM25 search → Chroma store: reads chunks
- Semantic search → Embedding service: embeds query
- Semantic search → Chroma store: searches vectors
- Streamlit app → Answer generator: generates answer
- Answer generator → Qwen language model: requests completion
- Streamlit app → Grounding check: checks support
- Streamlit app → Citation tracker: formats sources