# guilhermevaz8/rag_taac architecture

> Architecture diagram of the GitHub repository guilhermevaz8/rag_taac, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/guilhermevaz8/rag_taac
- Repository: https://github.com/guilhermevaz8/rag_taac

## Overview

##### Project Type and Purpose
- This is a full-stack Retrieval-Augmented Generation (RAG) system written in Python.
- Purpose: Ingest user-uploaded documents, index them into vector/text stores (Qdrant, OpenSearch), then answer natural-language queries by retrieving relevant passages and calling an LLM (via LLMSherpa).

##### File-Structure Analysis
- Top-level:
- app.py – main Flask/FastAPI(?) server that drives upload, query routes and renders templates/index.html.
- templates/index.html – minimal front-end UI for document upload and Q&A.
- haystack_components/ – custom pipeline modules:
– documents_pipeline (Splitter.py, classifiers.py, save_stores.py)
– askLLM/GPT.py – wrapper for calling the LLM.
– prompt_re_eng (llm.py, new_search.py) – prompt construction and query-based re-ranking.
– pipeline.py – orchestrates ingestion & query pipelines.
- Configuration: requirements.txt; Docker invoked externally for Qdrant, OpenSearch, LLMSherpa.

##### Key Components
a. Frontend
- Web UI (templates/index.html) served by app.py
b. Backend / API Layer
- app.py exposes endpoints:
- /upload → calls ingestion pipeline
- /query → calls retrieval & generation pipeline
c. Ingestion Pipeline
- Splitter (chunks documents)
- Classifiers (metadata tagging)
- save_stores (embeddings → Qdrant, text → OpenSearch)
d. Retrieval & Generation Pipeline
- new_search (construct search queries over OpenSearch)
- vector search (Qdrant)
- prompt_re_eng.llm (assemble context + prompt)
- askLLM.GPT (call LLMSherpa service to generate answer)
e. External Services / Data Stores
- Qdrant (vector store)
- OpenSearch (text store)
- LLMSherpa (LLM inference service)
f. Orchestration
- haystack_components/pipeline.py coordinates both ingestion and query flows.

##### Component Relationships and Data Flow
- User → Frontend (upload/query) → Backend (app.py)
- Upload: Backend → Ingestion Pipeline → Qdrant & OpenSearch
- Query: Backend → Retrieval Pipeline → Qdrant + OpenSearch → Prompt Re-engineering → LLMSherpa → Backend → Frontend → User
- Docker containers host Qdrant, OpenSearch, LLMSherpa; Backend connects over HTTP/REST ports.

##### Architectural Patterns
- Pipeline pattern: clear separation between ingestion and query pipelines.
- Microservices: external services (Qdrant, OpenSearch, LLMSherpa) are containerized and communicate via REST/gRPC.
- MVC-style separation: UI (View), app.py (Controller), haystack_components & external stores (Model).

##### Final Checklist
- All components clearly named.
- Related modules (haystack_components sub-directories) within a “Pipeline” container.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% UI Layer
    User["User"]:::ui
    Browser["Browser"]:::ui
    Index["index.html"]:::ui

    User --> Browser --> Index

    %% Backend/API Layer
    App["app.py"]:::backend
    Index -->|"upload/query"| App

    %% Orchestration
    subgraph "Pipeline Orchestrator"
        Orchestrator["pipeline.py"]:::pipeline
    end
    App -->|"invoke pipelines"| Orchestrator

    %% Ingestion Pipeline
    subgraph "Ingestion Pipeline"
        Splitter["Splitter"]:::pipeline
        Classifiers["Classifiers"]:::pipeline
        SaveStores["Save Stores"]:::pipeline
    end
    Orchestrator -.->|"chunks docs"| Splitter
    Splitter -.->|"tag metadata"| Classifiers
    Classifiers -.->|"embeddings/text"| SaveStores

    %% Persistence Layer
    Qdrant["Qdrant"]:::datastore
    OpenSearch["OpenSearch"]:::datastore

    SaveStores -->|"store vector data"| Qdrant
    SaveStores -->|"store text data"| OpenSearch

    %% Retrieval Pipeline
    subgraph "Retrieval Pipeline"
        NewSearch["new_search"]:::pipeline
        LLMRe["Prompt Re-engineering"]:::pipeline
    end
    Orchestrator -.->|"build search"| NewSearch
    NewSearch -->|"text search"| OpenSearch
    OpenSearch -->|"results"| NewSearch
    NewSearch -->|"vector search"| Qdrant
    Qdrant -->|"results"| NewSearch
    NewSearch -.->|"pass context"| LLMRe

    %% Generation Component
    GenComp["LLM Caller (GPT)"]:::pipeline
    LLMRe -.->|"LLM prompt"| GenComp

    %% External LLM Service
    LLMSherpa["LLMSherpa"]:::service
    GenComp -->|"call LLM"| LLMSherpa
    LLMSherpa -->|"answer text"| GenComp

    %% Return Flow
    GenComp -->|"response"| Orchestrator
    Orchestrator --> App
    App -->|"render answer"| Index
    Index --> User

    %% Click Events
    click Index "https://github.com/guilhermevaz8/rag_taac/blob/main/templates/index.html"
    click App "https://github.com/guilhermevaz8/rag_taac/blob/main/app.py"
    click Orchestrator "https://github.com/guilhermevaz8/rag_taac/blob/main/haystack_components/pipeline.py"
    click Splitter "https://github.com/guilhermevaz8/rag_taac/blob/main/haystack_components/documents_pipeline/Splitter.py"
    click Classifiers "https://github.com/guilhermevaz8/rag_taac/blob/main/haystack_components/documents_pipeline/classifiers.py"
    click SaveStores "https://github.com/guilhermevaz8/rag_taac/blob/main/haystack_components/documents_pipeline/save_stores.py"
    click NewSearch "https://github.com/guilhermevaz8/rag_taac/blob/main/haystack_components/prompt_re_eng/new_search.py"
    click LLMRe "https://github.com/guilhermevaz8/rag_taac/blob/main/haystack_components/prompt_re_eng/llm.py"
    click GenComp "https://github.com/guilhermevaz8/rag_taac/blob/main/haystack_components/askLLM/GPT.py"

    %% Styles
    classDef ui fill:#BFDFFF,stroke:#333,stroke-width:1px
    classDef backend fill:#FFDFBF,stroke:#333,stroke-width:1px
    classDef pipeline fill:#DFFFBF,stroke:#333,stroke-width:1px
    classDef datastore fill:#FFDFFF,stroke:#333,stroke-width:1px
    classDef service fill:#F0D0FF,stroke:#333,stroke-width:1px
```
