How joemattathil/private_offline_gpt works
This repository is a local, offline retrieval-augmented chatbot. Its architecture is centered on two separate flows:
1. an ingestion/indexing flow that turns documents into a searchable vector store, and
2. an interactive question-answering flow that retrieves relevant context from that store and uses a local LLM to generate replies.
There is no obvious dependence on a hosted API at runtime; the design is meant to run locally with a downloaded GPT4All/Llama model and a persisted Chroma database.
Main subsystems
• Document ingestion pipeline
- ingest.py is the entry point for building the knowledge base.
- source_documents/ is the input corpus; the README explicitly says to place user documents here.
- The repository includes source_documents/gpt-4.pdf as an example document.
- The ingestion step creates or updates the vector index stored under db/, with db/chroma.sqlite3 showing that Chroma is used as the persistence layer.
• Vector store / retrieval layer
- db/chroma.sqlite3 is the durable index backing retrieval.
- This is the boundary between offline document preparation and runtime question answering.
- The vector store is the main shared state between ingestion and the chat application.
• Chat application / UI layer
- app.py is the documented runtime entry point.
- The README says to start it with streamlit run app.py, so Streamlit is the presentation/runtime framework.
- fun_def/chat_ui.py likely contains the UI composition and chat rendering helpers.
- ui_elements/ appears to hold presentation assets or reusable UI components.
• Question answering / orchestration layer
- fun_def/question_answering.py likely contains the core RAG logic: embedding query text, retrieving relevant chunks from Chroma, and constructing the prompt sent to the local model.
- This is the application logic boundary between the UI and the model/retrieval backend.
• Configuration / shared constants
- constants.py centralizes paths or tuning values such as model location, chunking parameters, or database directories.
- KARMIK.py is likely a project-specific utility or alternate script, but the primary flow in the README points to app.py and ingest.py.
Technology and runtime choices that shape the architecture
• Streamlit for the user-facing application.
• GPT4All and Llama for local model inference.
• Chroma for vector storage and similarity search, persisted via SQLite.
• PDF/document input as the primary knowledge source, with the example document showing that the system is built for file-based ingestion.
• Fully local/offline execution after the model file has been downloaded manually.
End-to-end data flow
• Build phase
- User places documents into source_documents/.
- ingest.py reads those files, transforms them into chunks, creates embeddings, and writes the resulting index to db/chroma.sqlite3.
- This establishes the retrieval corpus used later by the chatbot.
• Query phase
- User launches app.py via Streamlit.
- The UI collects a question from the user and shows the conversation.
- fun_def/question_answering.py retrieves relevant document chunks from Chroma using the stored index.
- The selected context and user question are passed to the local GPT4All/Llama model.
- The generated answer is returned to the Streamlit UI.
Important architectural boundaries
• Offline model boundary
- The LLM is not embedded in the repo; the README instructs the user to download a model separately and place it in a models folder.
- That means the repository owns orchestration, not model weights.
• Indexing vs serving boundary
- Ingestion is a separate offline preparation step.
- The chat app assumes the vector index already exists.
• Local persistence boundary
- Chroma’s SQLite database is the durable state.
- Everything else is relatively stateless and can be recreated from documents plus the model file.
What an architecture diagram should emphasize
• A user interacts with the Streamlit app in app.py.
• The app delegates chat rendering to fun_def/chat_ui.py and retrieval/answer generation to fun_def/question_answering.py.
• The retrieval layer queries the local Chroma database in db/.
• The database is populated beforehand by ingest.py reading from source_documents/.
• The language model runs locally from a separately downloaded GPT4All/Llama model file, not from an external inference service.
• constants.py likely defines the shared paths and configuration that tie these pieces together.
In short, this is a local RAG chatbot architecture: documents are ingested once into a persistent Chroma index, and then a Streamlit chat app uses that index plus a locally hosted LLM to answer questions offline.
Main entry points: app.py app.py · question_answering.py question_answering.py · ingest.py ingest.py · README.md README
Components 15
Ingestion
Offline document preparation and index building
Chat Runtime
Interactive question answering and UI
- app.py · Starts the Streamlit chatbot application
app.py - chat_ui.py · Builds chat presentation and conversation rendering
fun_def/chat_ui.py - question_answering.py · Retrieves context and prepares answers from the local model
fun_def/question_answering.py - ui_elements · Reusable presentation components or assets
ui_elements - Local LLM · Downloaded model used for offline inference
Persistence
Durable vector store and source corpus
- Chroma DB · Persistent similarity index used for retrieval
db/chroma.sqlite3
Shared Config
Repository-wide settings and entry points
- constants.py · Central paths and tuning values shared by the app and ingestion
constants.py - KARMIK.py · Project-specific auxiliary script or alternate entry point
KARMIK.py - README · Repository usage and runtime instructions
README.md - pyproject.toml · Project packaging and tooling configuration
pyproject.toml - requirements.txt · Python dependency list
requirements.txt
Connections 17
- Documents → ingest.py: input · Documents are read by the ingestion pipeline
- gpt-4.pdf → Documents: example · Sample PDF shows the expected corpus format
- ingest.py → Chunk + Embed: prepare · Ingestion splits documents and creates embeddings
- Chunk + Embed → Chroma DB: persist · Prepared chunks and embeddings are stored in Chroma
- app.py → chat_ui.py: compose · The Streamlit app delegates presentation to the chat UI layer
- app.py → question_answering.py: delegate · The app passes user questions into the RAG orchestration layer
- chat_ui.py → ui_elements: use · UI composition can reuse shared elements and assets
- question_answering.py → Chroma DB: retrieve · Question answering queries the persisted vector store
- question_answering.py → Local LLM: generate · Retrieved context and the user question are sent to the local model
- question_answering.py → constants.py: config · Shared configuration supplies paths and runtime settings
- ingest.py → constants.py: config · Ingestion uses the same shared settings and paths
- app.py → constants.py: config · The runtime entry point relies on shared configuration
- README → app.py: run · README documents launching the Streamlit app
- README → ingest.py: build · README documents how to create or refresh the index
- README → Local LLM: download · README explains the separately downloaded model requirement
- pyproject.toml → requirements.txt: deps · Project packaging is supported by declared dependencies
- KARMIK.py → app.py: alt entry · Auxiliary script may offer an alternate way to invoke the project