How 1amibrahim/doc-rag works
Project Type & Purpose
• This is a full-stack Python web application plus accompanying libraries for building Retrieval-Augmented Generation (RAG) pipelines.
• End-users interact with a Gradio-based UI to chat with their documents; developers import the kotaemon and ktem libraries to customize or embed RAG pipelines in their own projects.
High-Level File‐Structure Analysis
• Root files: app.py, st_app.py, sso_app.py → entrypoints for different deployment modes (Gradio, Streamlit, SSO demo).
• libs/kotaemon: core RAG SDK → agents, loaders, embeddings, indices, llm wrappers, pipelines, storage adapters, CLI and contribs.
• libs/ktem: application framework → database models, Streamlit pages, UI components, embeddings manager, indexing manager, migrations.
• docs, templates, scripts and CI/CD config → documentation site (MkDocs), project/ component templates, helper scripts, Dockerfile, GitHub Actions.
Key Architectural Insights from README
• Layered RAG pipeline:
• Document loading & parsing → vector + full-text indexing → hybrid retrieval → optional re-ranking → reasoning (agents, question decomposition) → final LLM generation with citations.
• Supports multiple providers: OpenAI/Azure, local LLMs (ollama, llama-cpp-python), multiple vector stores (Chroma, LanceDB, Milvus, Qdrant), full-text stores (Elasticsearch, SimpleFile).
• UI built on Gradio theming, with settings pages for collections, LLM/embeddings, pipelines, user management.
Main Components & Interactions
• Presentation Layer
– Browser UI (Gradio) served by app.py → chat tab, settings, resources, PDF viewer.
• Application Server
– Gradio app instance orchestrating requests.
– Auth & multi-user management via ktem database models (SQL via Alembic/migrations).
• RAG Pipeline Core (kotaemon)
– Loaders (PDF, HTML, Excel, OCR) → extract text + metadata.
– Indexers & Splitters → full-text indexing and chunking.
– Vectorization (embedding models).
– Document store adapters & Vector store adapters.
– Retriever (hybrid full-text + vector), Reranker, QA pipelines, Agent frameworks (ReAct, ReWOO).
• Model Providers
– External LLM/embedding APIs (OpenAI, Azure) or local servers (Ollama, llama-cpp).
• Data Storage
– Persistent relational DB (user accounts, model configs).
– Full-text store (Elasticsearch or file).
– Vector store (Chroma, LanceDB, etc.).
– Application data folder (ktem_app_data for user uploads, logs).
• Infrastructure & Ops
– Docker image variants (lite, full, ollama).
– CI/CD pipelines (GitHub Actions).
– Scripts for setup, migrations, local servers.
Architectural Patterns & Principles
• Layered architecture (UI → orchestration → pipeline → model → storage).
• Plugin-style extensibility: components (loaders, indexers, llms, agents) follow base interfaces+ registration.
• Dependency injection via flowsettings.py and .env.
• Separation of concerns: kotaemon core for pipelines, ktem for app/UI.