# vietnh1009/rag-pipeline-visualizer architecture

> Architecture diagram of the GitHub repository vietnh1009/rag-pipeline-visualizer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vietnh1009/rag-pipeline-visualizer
- Repository: https://github.com/vietnh1009/rag-pipeline-visualizer

## Overview

RAG-pipeline-visualizer is a Streamlit application for configuring and inspecting a two-stage RAG workflow. Users configure offline document indexing, then query the resulting index and inspect the online answer-generation steps. The README documents the stage sequence and available strategies; sampled source confirms key orchestration and dispatch points. Some strategy implementations and UI wiring were not sampled, so those relationships are shown only where documented or directly evidenced.

Main entry points: [`pipeline/generation_pipeline.py`](https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/pipeline/generation_pipeline.py) (Generation pipeline), [`core/pipeline_runners.py`](https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/core/pipeline_runners.py) (Stage runners), [`ui/pages/indexing.py`](https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/ui/pages/indexing.py) (Indexing page), [`ui/pages/generation.py`](https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/ui/pages/generation.py) (Generation page)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["Streamlit interface"]
  node_app["App entry<br/>[app.py]"]
  node_index_ui["Indexing page<br/>[indexing.py]"]
  node_generation_ui["Generation page<br/>[generation.py]"]
  node_index_settings["Index settings"]
  node_result_views["Result views"]
end

subgraph group_indexing["Offline indexing"]
  node_index_pipeline["Indexing pipeline"]
  node_pipeline_runners["Stage runners"]
  node_document_loader["PDF loader<br/>[pdf_loader.py]"]
  node_chunking["Chunking strategies"]
  node_embedding["Embedding"]
  node_vector_store[("Vector database")]
end

subgraph group_generation["Online generation"]
  node_generation_pipeline["Generation pipeline"]
  node_pre_retrieval["Query transforms"]
  node_retrieval["Retrieval strategies"]
  node_post_retrieval["Rerank and filter<br/>[pipeline.py]"]
  node_prompt["Prompt builders"]
  node_generator["LLM generation"]
end

subgraph group_shared["Shared services"]
  node_cache[("Pipeline cache<br/>[pipeline_cache.py]")]
  node_llm_helper["Shared LLM calls<br/>[llm.py]"]
  node_environment["Environment helpers<br/>[env.py]"]
end

node_user(("User"))
node_answer["Answer"]

node_user -->|"opens"| node_app
node_app -->|"routes to"| node_index_ui
node_app -->|"routes to"| node_generation_ui
node_user -->|"configures"| node_index_settings
node_index_settings -->|"configures"| node_index_ui
node_index_ui -->|"runs"| node_index_pipeline
node_index_pipeline -->|"delegates to"| node_pipeline_runners
node_pipeline_runners -->|"loads documents"| node_document_loader
node_pipeline_runners -->|"chunks documents"| node_chunking
node_pipeline_runners -->|"embeds chunks"| node_embedding
node_pipeline_runners -->|"stores vectors"| node_vector_store
node_index_pipeline -->|"caches steps"| node_cache
node_index_ui -->|"renders results"| node_result_views
node_user -->|"submits query"| node_generation_ui
node_generation_ui -->|"runs"| node_generation_pipeline
node_generation_pipeline -->|"transforms query"| node_pre_retrieval
node_generation_pipeline -->|"retrieves documents"| node_retrieval
node_retrieval -->|"searches index"| node_vector_store
node_generation_pipeline -->|"processes results"| node_post_retrieval
node_post_retrieval -->|"uses LLM"| node_llm_helper
node_generation_pipeline -->|"builds prompt"| node_prompt
node_generation_pipeline -->|"generates answer"| node_generator
node_generator -->|"returns"| node_answer
node_generation_ui -->|"shows results"| node_result_views
node_document_loader -.->|"uses LLM"| node_llm_helper
node_environment -->|"supports setup"| node_pipeline_runners

click node_app "https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/app.py"
click node_index_ui "https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/ui/pages/indexing.py"
click node_generation_ui "https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/ui/pages/generation.py"
click node_index_settings "https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/ui/settings"
click node_result_views "https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/ui/results"
click node_index_pipeline "https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/pipeline/indexing_pipeline.py"
click node_pipeline_runners "https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/core/pipeline_runners.py"
click node_document_loader "https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/loader/pdf_loader.py"
click node_chunking "https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/chunking"
click node_embedding "https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/embedding"
click node_vector_store "https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/vector_db"
click node_generation_pipeline "https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/pipeline/generation_pipeline.py"
click node_pre_retrieval "https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/pre_retrieval"
click node_retrieval "https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/retrieval"
click node_post_retrieval "https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/post_retrieval/pipeline.py"
click node_prompt "https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/prompt"
click node_generator "https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/generation"
click node_cache "https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/pipeline_cache.py"
click node_llm_helper "https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/utils/llm.py"
click node_environment "https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/utils/env.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_index_ui,node_generation_ui,node_index_settings,node_result_views,node_user toneBlue
class node_index_pipeline,node_pipeline_runners,node_document_loader,node_chunking,node_embedding,node_vector_store toneAmber
class node_generation_pipeline,node_pre_retrieval,node_retrieval,node_post_retrieval,node_prompt,node_generator toneMint
class node_cache,node_llm_helper,node_environment toneRose
class node_answer toneIndigo
```

## Components

### Streamlit interface

- [App entry](https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/app.py): `app.py`
- [Indexing page](https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/ui/pages/indexing.py): `ui/pages/indexing.py`
- [Generation page](https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/ui/pages/generation.py): `ui/pages/generation.py`
- [Index settings](https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/ui/settings): `ui/settings`
- [Result views](https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/ui/results): `ui/results`

### Offline indexing

- [Indexing pipeline](https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/pipeline/indexing_pipeline.py): `pipeline/indexing_pipeline.py`
- [Stage runners](https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/core/pipeline_runners.py): `core/pipeline_runners.py`
- [PDF loader](https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/loader/pdf_loader.py): `loader/pdf_loader.py`
- [Chunking strategies](https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/chunking): `chunking`
- [Embedding](https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/embedding): `embedding`
- [Vector database](https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/vector_db): `vector_db`

### Online generation

- [Generation pipeline](https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/pipeline/generation_pipeline.py): `pipeline/generation_pipeline.py`
- [Query transforms](https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/pre_retrieval): `pre_retrieval`
- [Retrieval strategies](https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/retrieval): `retrieval`
- [Rerank and filter](https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/post_retrieval/pipeline.py): `post_retrieval/pipeline.py`
- [Prompt builders](https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/prompt): `prompt`
- [LLM generation](https://github.com/vietnh1009/rag-pipeline-visualizer/tree/main/generation): `generation`

### Shared services

- [Pipeline cache](https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/pipeline_cache.py): `pipeline_cache.py`
- [Shared LLM calls](https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/utils/llm.py): `utils/llm.py`
- [Environment helpers](https://github.com/vietnh1009/rag-pipeline-visualizer/blob/main/utils/env.py): `utils/env.py`

### Other components

- **User**
- **Answer**

## Connections

- User → App entry: opens (evidence: `README.md`)
- App entry → Indexing page: routes to (evidence: `app.py`)
- App entry → Generation page: routes to (evidence: `app.py`)
- User → Index settings: configures (evidence: `README.md`)
- Index settings → Indexing page: configures (evidence: `ui/pages/indexing.py`)
- Indexing page → Indexing pipeline: runs (evidence: `ui/pages/indexing.py`)
- Indexing pipeline → Stage runners: delegates to (evidence: `pipeline/indexing_pipeline.py`)
- Stage runners → PDF loader: loads documents (evidence: `core/pipeline_runners.py`)
- Stage runners → Chunking strategies: chunks documents (evidence: `core/pipeline_runners.py`)
- Stage runners → Embedding: embeds chunks (evidence: `core/pipeline_runners.py`)
- Stage runners → Vector database: stores vectors (evidence: `core/pipeline_runners.py`)
- Indexing pipeline → Pipeline cache: caches steps (evidence: `pipeline/indexing_pipeline.py`)
- Indexing page → Result views: renders results (evidence: `ui/pages/indexing.py`)
- User → Generation page: submits query (evidence: `README.md`)
- Generation page → Generation pipeline: runs (evidence: `README.md`)
- Generation pipeline → Query transforms: transforms query (evidence: `pipeline/generation_pipeline.py`)
- Generation pipeline → Retrieval strategies: retrieves documents (evidence: `pipeline/generation_pipeline.py`)
- Retrieval strategies → Vector database: searches index (evidence: `README.md`)
- Generation pipeline → Rerank and filter: processes results (evidence: `pipeline/generation_pipeline.py`)
- Rerank and filter → Shared LLM calls: uses LLM (evidence: `post_retrieval/pipeline.py`)
- Generation pipeline → Prompt builders: builds prompt (evidence: `pipeline/generation_pipeline.py`)
- Generation pipeline → LLM generation: generates answer (evidence: `pipeline/generation_pipeline.py`)
- LLM generation → Answer: returns (evidence: `README.md`)
- Generation page → Result views: shows results (evidence: `ui/results/vdb_results.py`)
- PDF loader → Shared LLM calls: uses LLM
- Environment helpers → Stage runners: supports setup (evidence: `core/pipeline_runners.py`)
