How wterrr/rag_contextual works
Main Components
• Backend API & Service Layer:
• Represented by app.py and the api directory (controller.py, service.py). This layer exposes endpoints (running under Uvicorn) that handle incoming requests for generating responses (e.g., chatbot assistant queries).
• Ingestion & Data Processing:
• Ingestion scripts and modules in the src/ingest directory (such as add_files.py and contextual_rag_ingest.py) handle adding and processing new data.
• The scripts folder contains shell scripts (like contextual_rag_ingest.sh) to automate the ingestion.
• File Readers:
• Located under src/readers (with specific readers for each file type, and a structured sub-module “kotaemon”) that parse various document formats (PDF, DOCX, HTML, etc.) for extraction.
• Embedding & Search Component:
• The src/embedding directory includes the RAG module and related integration, e.g., elastic_search.py. This is where the text is embedded and indexed in external services.
• The configuration for these (Qdrant and ElasticSearch endpoints) is provided via YAML files in the config directory.
• UI & Demo Applications:
• streamlit_ui.py serves as the front-end user interface.
• Demonstration scripts (in the demo folder) show example usages such as the chatbot assistant and contextual RAG.
• Configuration & Environment:
• The config directory holds configuration files (YAML and Python) that set up service endpoints, agent types, and API key configurations.
• External Dependencies & Deployment:
• docker-compose.yml is used for spinning up the necessary backend services (like the database/search engines).
• The requirements.txt file defines the technical stack (e.g., Python, Uvicorn, Streamlit, etc.).
Relationships and Interactions
• The UI (Streamlit) and API clients communicate with the backend (app.py) via HTTP, managed by Uvicorn.
• The API layer (in the api directory) calls service functions that delegate requests to the RAG or Chatbot assistant logic.
• When new files need to be ingested, the ingestion scripts (or continuous ingestion via src/ingest) call the file readers to parse documents.
• Processed content is handed over to the embedding components, which generate vector representations for the content and interface with Qdrant/ElasticSearch for storage and retrieval.
• Configurations in the config directory are accessed by various modules (via src/settings.py, config/config.py) to determine service URLs and operational parameters.
• Demo scripts in the demo folder provide self-contained examples showing how to invoke contextual RAG or chatbot assistant functionalities.
Architectural Patterns and Technologies
• Modular Design: Each functionality (ingestion, reading, embedding, API, UI) is separated into its own module or package.
• Separation of Concerns: The project clearly isolates concerns – file ingestion/reading, embedding/indexing, and user interaction.
• External Service Integration: The system integrates with external vector search engines (Qdrant, ElasticSearch) and APIs (OpenAI, LlamaParse, Cohere) via configuration.
• Use of Containerization: Docker-compose is used to manage and deploy dependent services.
• Demo & Testing Capabilities: The presence of demo scripts and a tests suite indicates a focus on ease of testing and demonstration.