# wterrr/rag_contextual architecture

> Architecture diagram of the GitHub repository wterrr/rag_contextual, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wterrr/rag_contextual
- Repository: https://github.com/wterrr/rag_contextual

## Overview

##### Main Components
- Backend API & Service Layer:
- Represented by app.py and the api directory (controller.py, service.py). This layer exposes endpoints (running under Uvicorn) that handle incoming requests for generating responses (e.g., chatbot assistant queries).
- Ingestion & Data Processing:
- Ingestion scripts and modules in the src/ingest directory (such as add_files.py and contextual_rag_ingest.py) handle adding and processing new data.
- The scripts folder contains shell scripts (like contextual_rag_ingest.sh) to automate the ingestion.
- File Readers:
- Located under src/readers (with specific readers for each file type, and a structured sub-module “kotaemon”) that parse various document formats (PDF, DOCX, HTML, etc.) for extraction.
- Embedding & Search Component:
- The src/embedding directory includes the RAG module and related integration, e.g., elastic_search.py. This is where the text is embedded and indexed in external services.
- The configuration for these (Qdrant and ElasticSearch endpoints) is provided via YAML files in the config directory.
- UI & Demo Applications:
- streamlit_ui.py serves as the front-end user interface.
- Demonstration scripts (in the demo folder) show example usages such as the chatbot assistant and contextual RAG.
- Configuration & Environment:
- The config directory holds configuration files (YAML and Python) that set up service endpoints, agent types, and API key configurations.
- External Dependencies & Deployment:
- docker-compose.yml is used for spinning up the necessary backend services (like the database/search engines).
- The requirements.txt file defines the technical stack (e.g., Python, Uvicorn, Streamlit, etc.).

##### Relationships and Interactions
- The UI (Streamlit) and API clients communicate with the backend (app.py) via HTTP, managed by Uvicorn.
- The API layer (in the api directory) calls service functions that delegate requests to the RAG or Chatbot assistant logic.
- When new files need to be ingested, the ingestion scripts (or continuous ingestion via src/ingest) call the file readers to parse documents.
- Processed content is handed over to the embedding components, which generate vector representations for the content and interface with Qdrant/ElasticSearch for storage and retrieval.
- Configurations in the config directory are accessed by various modules (via src/settings.py, config/config.py) to determine service URLs and operational parameters.
- Demo scripts in the demo folder provide self-contained examples showing how to invoke contextual RAG or chatbot assistant functionalities.

##### Architectural Patterns and Technologies
- Modular Design: Each functionality (ingestion, reading, embedding, API, UI) is separated into its own module or package.
- Separation of Concerns: The project clearly isolates concerns – file ingestion/reading, embedding/indexing, and user interaction.
- External Service Integration: The system integrates with external vector search engines (Qdrant, ElasticSearch) and APIs (OpenAI, LlamaParse, Cohere) via configuration.
- Use of Containerization: Docker-compose is used to manage and deploy dependent services.
- Demo & Testing Capabilities: The presence of demo scripts and a tests suite indicates a focus on ease of testing and demonstration.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction Layer
    subgraph "User Interaction"
        U1["Streamlit UI (streamlit_ui.py)"]:::ui
        U2["API Clients"]:::ui
    end

    %% Backend API & Service Layer
    subgraph "Backend API & Service Layer"
        B1["Backend Entry (app.py)"]:::backend
        B2["API Controller (api/controller.py)"]:::backend
        B3["API Service (api/service.py)"]:::backend
    end

    %% Data Processing & Ingestion Layer
    subgraph "Data Processing & Ingestion"
        I1["Add Files (src/ingest/add_files.py)"]:::ingest
        I2["Contextual RAG Ingest (src/ingest/contextual_rag_ingest.py)"]:::ingest
        I3["Ingestion Shell Script (scripts/contextual_rag_ingest.sh)"]:::ingest

        subgraph "File Readers"
            R1["File Reader (src/readers/file_reader.py)"]:::reader
            R2["Kotaemon Readers (src/readers/kotaemon)"]:::reader
            R3["Base Component (src/readers/kotaemon/base/component.py)"]:::reader
            R4["Loaders (src/readers/kotaemon/loaders/*)"]:::reader
        end
    end

    %% Embedding & Search Layer
    subgraph "Embedding & Search"
        E1["Elastic Search (src/embedding/elastic_search.py)"]:::embed
        E2["RAG Module (src/embedding/rag.py)"]:::embed
    end

    %% Configuration & Deployment Layer
    subgraph "Configuration & Deployment"
        C1["Configuration Files (config/)"]:::config
        C2["Docker Deployment (docker-compose.yml)"]:::config
        C3["Requirements (requirements.txt)"]:::config
    end

    %% Demo/Testing Layer
    subgraph "Demo/Testing"
        D1["Chatbot Demo (demo/demo_chatbot_assistant.py)"]:::demo
        D2["Contextual RAG Demo (demo/demo_contextual_rag.py)"]:::demo
    end

    %% Connections between Layers
    U1 -->|"HTTPRequest"| B1
    U2 -->|"HTTPRequest"| B1

    B1 -->|"routes"| B2
    B1 -->|"delegates"| B3

    B3 -->|"queries"| E2

    I1 -->|"parses"| R1
    I2 -->|"parses"| R1
    I3 -->|"triggers"| I2

    R1 -->|"sendsData"| E1
    R1 -->|"sendsData"| E2

    C1 ---|"configures"| B3
    C1 ---|"configures"| I2
    C1 ---|"configures"| E1
    C1 ---|"configures"| E2

    D1 -->|"invokes"| B1
    D2 -->|"invokes"| B1

    %% Styles
    classDef ui fill:#5cb85c,stroke:#333,color:#fff;
    classDef backend fill:#0275d8,stroke:#333,color:#fff;
    classDef ingest fill:#f0ad4e,stroke:#333,color:#fff;
    classDef reader fill:#f7c6a3,stroke:#333,color:#000;
    classDef embed fill:#563d7c,stroke:#333,color:#fff;
    classDef config fill:#777,stroke:#333,color:#fff;
    classDef demo fill:#5bc0de,stroke:#333,color:#fff;

    %% Click Events
    click U1 "https://github.com/wterrr/rag_contextual/blob/main/streamlit_ui.py"
    click B1 "https://github.com/wterrr/rag_contextual/blob/main/app.py"
    click B2 "https://github.com/wterrr/rag_contextual/blob/main/api/controller.py"
    click B3 "https://github.com/wterrr/rag_contextual/blob/main/api/service.py"
    click I1 "https://github.com/wterrr/rag_contextual/blob/main/src/ingest/add_files.py"
    click I2 "https://github.com/wterrr/rag_contextual/blob/main/src/ingest/contextual_rag_ingest.py"
    click I3 "https://github.com/wterrr/rag_contextual/blob/main/scripts/contextual_rag_ingest.sh"
    click R1 "https://github.com/wterrr/rag_contextual/blob/main/src/readers/file_reader.py"
    click R2 "https://github.com/wterrr/rag_contextual/tree/main/src/readers/kotaemon"
    click R3 "https://github.com/wterrr/rag_contextual/blob/main/src/readers/kotaemon/base/component.py"
    click R4 "https://github.com/wterrr/rag_contextual/tree/main/src/readers/kotaemon/loaders"
    click E1 "https://github.com/wterrr/rag_contextual/blob/main/src/embedding/elastic_search.py"
    click E2 "https://github.com/wterrr/rag_contextual/blob/main/src/embedding/rag.py"
    click C1 "https://github.com/wterrr/rag_contextual/tree/main/config/"
    click C2 "https://github.com/wterrr/rag_contextual/blob/main/docker-compose.yml"
    click C3 "https://github.com/wterrr/rag_contextual/blob/main/requirements.txt"
    click D1 "https://github.com/wterrr/rag_contextual/blob/main/demo/demo_chatbot_assistant.py"
    click D2 "https://github.com/wterrr/rag_contextual/blob/main/demo/demo_contextual_rag.py"
```
