# joemattathil/private_offline_gpt architecture

> Architecture diagram of the GitHub repository joemattathil/private_offline_gpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/joemattathil/private_offline_gpt
- Repository: https://github.com/joemattathil/private_offline_gpt

## Overview

This repository is a local, offline retrieval-augmented chatbot. Its architecture is centered on two separate flows:

1. an ingestion/indexing flow that turns documents into a searchable vector store, and  
2. an interactive question-answering flow that retrieves relevant context from that store and uses a local LLM to generate replies.

There is no obvious dependence on a hosted API at runtime; the design is meant to run locally with a downloaded GPT4All/Llama model and a persisted Chroma database.

Main subsystems

- Document ingestion pipeline
  - `ingest.py` is the entry point for building the knowledge base.
  - `source_documents/` is the input corpus; the README explicitly says to place user documents here.
  - The repository includes `source_documents/gpt-4.pdf` as an example document.
  - The ingestion step creates or updates the vector index stored under `db/`, with `db/chroma.sqlite3` showing that Chroma is used as the persistence layer.

- Vector store / retrieval layer
  - `db/chroma.sqlite3` is the durable index backing retrieval.
  - This is the boundary between offline document preparation and runtime question answering.
  - The vector store is the main shared state between ingestion and the chat application.

- Chat application / UI layer
  - `app.py` is the documented runtime entry point.
  - The README says to start it with `streamlit run app.py`, so Streamlit is the presentation/runtime framework.
  - `fun_def/chat_ui.py` likely contains the UI composition and chat rendering helpers.
  - `ui_elements/` appears to hold presentation assets or reusable UI components.

- Question answering / orchestration layer
  - `fun_def/question_answering.py` likely contains the core RAG logic: embedding query text, retrieving relevant chunks from Chroma, and constructing the prompt sent to the local model.
  - This is the application logic boundary between the UI and the model/retrieval backend.

- Configuration / shared constants
  - `constants.py` centralizes paths or tuning values such as model location, chunking parameters, or database directories.
  - `KARMIK.py` is likely a project-specific utility or alternate script, but the primary flow in the README points to `app.py` and `ingest.py`.

Technology and runtime choices that shape the architecture

- Streamlit for the user-facing application.
- GPT4All and Llama for local model inference.
- Chroma for vector storage and similarity search, persisted via SQLite.
- PDF/document input as the primary knowledge source, with the example document showing that the system is built for file-based ingestion.
- Fully local/offline execution after the model file has been downloaded manually.

End-to-end data flow

- Build phase
  - User places documents into `source_documents/`.
  - `ingest.py` reads those files, transforms them into chunks, creates embeddings, and writes the resulting index to `db/chroma.sqlite3`.
  - This establishes the retrieval corpus used later by the chatbot.

- Query phase
  - User launches `app.py` via Streamlit.
  - The UI collects a question from the user and shows the conversation.
  - `fun_def/question_answering.py` retrieves relevant document chunks from Chroma using the stored index.
  - The selected context and user question are passed to the local GPT4All/Llama model.
  - The generated answer is returned to the Streamlit UI.

Important architectural boundaries

- Offline model boundary
  - The LLM is not embedded in the repo; the README instructs the user to download a model separately and place it in a models folder.
  - That means the repository owns orchestration, not model weights.

- Indexing vs serving boundary
  - Ingestion is a separate offline preparation step.
  - The chat app assumes the vector index already exists.

- Local persistence boundary
  - Chroma’s SQLite database is the durable state.
  - Everything else is relatively stateless and can be recreated from documents plus the model file.

What an architecture diagram should emphasize

- A user interacts with the Streamlit app in `app.py`.
- The app delegates chat rendering to `fun_def/chat_ui.py` and retrieval/answer generation to `fun_def/question_answering.py`.
- The retrieval layer queries the local Chroma database in `db/`.
- The database is populated beforehand by `ingest.py` reading from `source_documents/`.
- The language model runs locally from a separately downloaded GPT4All/Llama model file, not from an external inference service.
- `constants.py` likely defines the shared paths and configuration that tie these pieces together.

In short, this is a local RAG chatbot architecture: documents are ingested once into a persistent Chroma index, and then a Streamlit chat app uses that index plus a locally hosted LLM to answer questions offline.

Main entry points: [`app.py`](https://github.com/joemattathil/private_offline_gpt/blob/HEAD/app.py) (app.py), [`fun_def/question_answering.py`](https://github.com/joemattathil/private_offline_gpt/blob/HEAD/fun_def/question_answering.py) (question_answering.py), [`ingest.py`](https://github.com/joemattathil/private_offline_gpt/blob/HEAD/ingest.py) (ingest.py), [`README.md`](https://github.com/joemattathil/private_offline_gpt/blob/HEAD/README.md) (README)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph "Ingestion"
  node_ingest["ingest.py<br/>pipeline entrypoint<br/>[ingest.py]"]
  node_source_docs["Documents<br/>input corpus"]
  node_pdf_example["gpt-4.pdf<br/>sample document<br/>[gpt-4.pdf]"]
  node_chunk_embed["Chunk + Embed<br/>prep stage"]
end

subgraph "Chat Runtime"
  node_app["app.py<br/>ui entrypoint<br/>[app.py]"]
  node_chat_ui["chat_ui.py<br/>ui composition<br/>[chat_ui.py]"]
  node_qa["question_answering.py<br/>rag orchestration"]
  node_ui_elements["ui_elements<br/>ui assets"]
  node_local_llm{{"Local LLM<br/>gpt4all/llama"}}
end

subgraph "Persistence"
  node_db[("Chroma DB<br/>vector store<br/>[chroma.sqlite3]")]
end

subgraph "Shared Config"
  node_constants["constants.py<br/>shared config<br/>[constants.py]"]
  node_karmik["KARMIK.py<br/>aux script<br/>[KARMIK.py]"]
  node_readme["README<br/>docs<br/>[README.md]"]
  node_pyproject["pyproject.toml<br/>build config<br/>[pyproject.toml]"]
  node_requirements["requirements.txt<br/>deps<br/>[requirements.txt]"]
end

node_source_docs -->|"input"| node_ingest
node_pdf_example -->|"example"| node_source_docs
node_ingest -->|"prepare"| node_chunk_embed
node_chunk_embed -->|"persist"| node_db
node_app -->|"compose"| node_chat_ui
node_app -->|"delegate"| node_qa
node_chat_ui -->|"use"| node_ui_elements
node_qa -->|"retrieve"| node_db
node_qa -->|"generate"| node_local_llm
node_qa -->|"config"| node_constants
node_ingest -->|"config"| node_constants
node_app -->|"config"| node_constants
node_readme -->|"run"| node_app
node_readme -->|"build"| node_ingest
node_readme -->|"download"| node_local_llm
node_pyproject -->|"deps"| node_requirements
node_karmik -.->|"alt entry"| node_app

click node_ingest "https://github.com/joemattathil/private_offline_gpt/blob/main/ingest.py"
click node_source_docs "https://github.com/joemattathil/private_offline_gpt/tree/main/source_documents"
click node_pdf_example "https://github.com/joemattathil/private_offline_gpt/blob/main/source_documents/gpt-4.pdf"
click node_db "https://github.com/joemattathil/private_offline_gpt/blob/main/db/chroma.sqlite3"
click node_app "https://github.com/joemattathil/private_offline_gpt/blob/main/app.py"
click node_chat_ui "https://github.com/joemattathil/private_offline_gpt/blob/main/fun_def/chat_ui.py"
click node_qa "https://github.com/joemattathil/private_offline_gpt/blob/main/fun_def/question_answering.py"
click node_ui_elements "https://github.com/joemattathil/private_offline_gpt/tree/main/ui_elements"
click node_constants "https://github.com/joemattathil/private_offline_gpt/blob/main/constants.py"
click node_karmik "https://github.com/joemattathil/private_offline_gpt/blob/main/KARMIK.py"
click node_readme "https://github.com/joemattathil/private_offline_gpt/blob/main/README.md"
click node_pyproject "https://github.com/joemattathil/private_offline_gpt/blob/main/pyproject.toml"
click node_requirements "https://github.com/joemattathil/private_offline_gpt/blob/main/requirements.txt"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingest,node_source_docs,node_pdf_example,node_chunk_embed toneBlue
class node_app,node_chat_ui,node_qa,node_ui_elements,node_local_llm toneAmber
class node_db toneMint
class node_constants,node_karmik,node_readme,node_pyproject,node_requirements toneRose
```

## Components

### Ingestion

Offline document preparation and index building

- [ingest.py](https://github.com/joemattathil/private_offline_gpt/blob/HEAD/ingest.py): pipeline entrypoint · Builds or refreshes the knowledge base from documents · `ingest.py`
- [Documents](https://github.com/joemattathil/private_offline_gpt/tree/HEAD/source_documents): input corpus · User-provided source files for ingestion · `source_documents`
- [gpt-4.pdf](https://github.com/joemattathil/private_offline_gpt/blob/HEAD/source_documents/gpt-4.pdf): sample document · Example input document included in the corpus · `source_documents/gpt-4.pdf`
- **Chunk + Embed**: prep stage · Transforms documents into retrievable embeddings

### Chat Runtime

Interactive question answering and UI

- [app.py](https://github.com/joemattathil/private_offline_gpt/blob/HEAD/app.py): ui entrypoint · Starts the Streamlit chatbot application · `app.py`
- [chat_ui.py](https://github.com/joemattathil/private_offline_gpt/blob/HEAD/fun_def/chat_ui.py): ui composition · Builds chat presentation and conversation rendering · `fun_def/chat_ui.py`
- [question_answering.py](https://github.com/joemattathil/private_offline_gpt/blob/HEAD/fun_def/question_answering.py): rag orchestration · Retrieves context and prepares answers from the local model · `fun_def/question_answering.py`
- [ui_elements](https://github.com/joemattathil/private_offline_gpt/tree/HEAD/ui_elements): ui assets · Reusable presentation components or assets · `ui_elements`
- **Local LLM**: gpt4all/llama · Downloaded model used for offline inference

### Persistence

Durable vector store and source corpus

- [Chroma DB](https://github.com/joemattathil/private_offline_gpt/blob/HEAD/db/chroma.sqlite3): vector store · Persistent similarity index used for retrieval · `db/chroma.sqlite3`

### Shared Config

Repository-wide settings and entry points

- [constants.py](https://github.com/joemattathil/private_offline_gpt/blob/HEAD/constants.py): shared config · Central paths and tuning values shared by the app and ingestion · `constants.py`
- [KARMIK.py](https://github.com/joemattathil/private_offline_gpt/blob/HEAD/KARMIK.py): aux script · Project-specific auxiliary script or alternate entry point · `KARMIK.py`
- [README](https://github.com/joemattathil/private_offline_gpt/blob/HEAD/README.md): docs · Repository usage and runtime instructions · `README.md`
- [pyproject.toml](https://github.com/joemattathil/private_offline_gpt/blob/HEAD/pyproject.toml): build config · Project packaging and tooling configuration · `pyproject.toml`
- [requirements.txt](https://github.com/joemattathil/private_offline_gpt/blob/HEAD/requirements.txt): deps · Python dependency list · `requirements.txt`

## Connections

- Documents → ingest.py: input · Documents are read by the ingestion pipeline
- gpt-4.pdf → Documents: example · Sample PDF shows the expected corpus format
- ingest.py → Chunk + Embed: prepare · Ingestion splits documents and creates embeddings
- Chunk + Embed → Chroma DB: persist · Prepared chunks and embeddings are stored in Chroma
- app.py → chat_ui.py: compose · The Streamlit app delegates presentation to the chat UI layer
- app.py → question_answering.py: delegate · The app passes user questions into the RAG orchestration layer
- chat_ui.py → ui_elements: use · UI composition can reuse shared elements and assets
- question_answering.py → Chroma DB: retrieve · Question answering queries the persisted vector store
- question_answering.py → Local LLM: generate · Retrieved context and the user question are sent to the local model
- question_answering.py → constants.py: config · Shared configuration supplies paths and runtime settings
- ingest.py → constants.py: config · Ingestion uses the same shared settings and paths
- app.py → constants.py: config · The runtime entry point relies on shared configuration
- README → app.py: run · README documents launching the Streamlit app
- README → ingest.py: build · README documents how to create or refresh the index
- README → Local LLM: download · README explains the separately downloaded model requirement
- pyproject.toml → requirements.txt: deps · Project packaging is supported by declared dependencies
- KARMIK.py → app.py: alt entry · Auxiliary script may offer an alternate way to invoke the project
