How umbertogriffo/rag-chatbot works
Project type and purpose
• This is a local, full-stack Python application that implements two variants of a chatbot:
a) A “Conversation-aware” Chatbot (stateful, ChatGPT-like)
b) A RAG (Retrieval-Augmented Generation) Chatbot
• It uses:
• llama-cpp-python as the local LLM backend
• Chroma as the vector store for embeddings
• Streamlit (and a CLI) for the user interface
• A custom “Memory Builder” to index Markdown docs
File structure
• Top-level scripts & apps:
• chatbot/chatbot_app.py (Streamlit GUI for conversation-only)
• chatbot/rag_chatbot_app.py (Streamlit GUI for RAG)
• chatbot/cli/{chatbot.py, rag_chatbot.py} (command-line entrypoints)
• chatbot/memory_builder.py (builds embedding index from docs/)
• Core modules under chatbot/bot/:
• client/ (llama_cpp_client.py: interfaces with llama-cpp-python; prompt templates)
• model/ (BaseModel abstraction, model registry, per-model settings)
• memory/ (embedder, vector_database wrapper for Chroma, distance metrics)
• conversation/ (chat_history, conversation_handler, context strategies)
• Data loaders: chatbot/document_loader/ (text splitting, format parsing, loader)
• Entities: chatbot/entities/document.py
• External persistence: docs/ (source Markdown), vector_store/ (Chroma persistence)
README for operational flow
• Offline “Memory Builder” loads docs/ → chunks via text_splitter → computes embeddings → stores in Chroma DB
• At runtime, user asks a question via GUI or CLI → the correct “chat app” module invokes the Bot
a) Conversation mode: uses conversation_handler + chat_history → prompts LLM via llama_cpp_client
b) RAG mode:
1. (optional) rewrite user query via LLM
2. retrieve top-k chunks from Chroma
3. synthesis strategy (“create-and-refine” or “(async) tree summarization”) → orchestrates multiple LLM calls
4. returns final answer
Main components
A. User Interface Layer
• CLI (chatbot.py, rag_chatbot.py)
• Streamlit UI (chatbot_app.py, rag_chatbot_app.py)
B. Application Layer
1. ChatRouter (entry point that picks Conversation vs. RAG)
2. Conversation Handler
• ChatHistory store (in-memory/session)
• Context Strategy (ctx_strategy.py)
3. RAG Pipeline
• Query Rewriter (LLM call)
• Retriever (Chroma wrapper)
• Response Synthesizer (create-and-refine or async-tree)
4. Memory Builder (offline batch)
• Document Loader → Text Splitter → Embedder → Chroma VectorDB
C. Model & Client Layer
• Model Registry (maps model names to BaseModel instances)
• BaseModel wrapper (loads quantized GGUF models via llama-cpp-python)
• Prompt templates (prompt.py)
D. Data & Persistence Layer
• docs/ folder (Markdown source)
• Chroma Vector Database (persistent embedding store)
• Optional: config files (pyproject.toml, Makefile)
Relationships & data flows
• User → UI → ChatRouter
• ChatRouter → Conversation Handler → Model Registry → LLM Client → Response → UI
• ChatRouter → RAG Pipeline steps in sequence (and parallel for async)
• Retriever ↔ Chroma VectorDB
• Model Registry → BaseModel → llama_cpp_python (external library)
Architectural patterns & design principles
• Layered architecture: UI / Application / Model / Data
• Modular design: separate packages for client, model, conversation, memory, loader
• Strategy pattern for context strategies and synthesis strategies
• Registry pattern for pluggable model settings