How promtengineer/localgpt works
Project Type & Purpose
• Project Type: Open-source, local RAG (Retrieval-Augmented Generation) tool with both CLI and web UI layers.
• Purpose: Ingest arbitrary documents, build a local vector store, run a private LLM-based Q&A pipeline without sending data off-premise, and expose both CLI and REST/UI interfaces.
File Structure
Top-level folders & files reveal these logical modules:
• SOURCE_DOCUMENTS/ ← raw inputs for ingestion
• ingest.py ← pipeline to load documents, embed them, store them in Chroma
• run_localGPT.py ← CLI tool that handles user queries via local model
• run_localGPT_API.py ← Python service (Werkzeug/Flask) exposing REST endpoints
• localGPTUI/ ← front-end web application (templates, static assets, localGPTUI.py)
• gaudi_utils/ ← hardware-specific utilities (HPU support)
• constants.py, prompt_template_utils.py, utils.py ← shared config, code templates, helpers
• Dockerfile, Dockerfile_hpu ← containerization for GPU/HPU/MPS setups
Key Components
a. Data Ingestion Pipeline
• Document Loader (LangChain loaders via constants.py)
• Embedding Model (InstructorEmbeddings)
• Vector Store (ChromaDB → local DB folder)
b. Inference Pipeline
• Retrieval (similarity search from Chroma)
• LLM Model Loader (HuggingFace or llama-cpp via load_models.py)
• Generation Engine (run_localGPT.py or run_localGPT_API.py logic)
c. Interfaces
• CLI Interface (run_localGPT.py)
• REST API Service (run_localGPT_API.py)
• Web UI (localGPTUI.py + HTML/CSS/JS under localGPTUI/static & templates)
d. External Dependencies / Infrastructure
• HuggingFace model hub or local GGML/GGUF artifacts
• LangChain library
• GPU/CPU/HPU/MPS runtimes
• Docker container (optional)
Relationships & Data Flow
• Source Documents → (ingest.py) → Embedding Model → Vector Store
• User Query (CLI or UI) → API or direct CLI script
• API/CLI → Retrieval from Vector Store → Context Chunks
• Context + Query → LLM Inference → Answer
• Answer → CLI stdout or API JSON → UI renderer
Architectural Patterns & Principles
• Layered Architecture: Data layer, Storage layer, Compute layer, Presentation layer
• Pipeline Pattern: ingestion → storage → retrieval → generation
• Adapter/Strategy Pattern: plugging in different loader models or hardware backends via gaudi_utils
• Microservices (lightweight): separate CLI, API service, and UI processes
Technologies & Frameworks to Call Out
• Python 3.10+, LangChain, ChromaDB
• HuggingFace Transformers / llama-cpp-python
• InstructorEmbeddings for vectorization
• Flask/Werkzeug for API (embedded in run_localGPT_API.py)
• Jinja2 templates + Bootstrap/jQuery for the web UI
• Docker / Dockerfile_hpu for containerized GPU or HPU execution
Important: Be Very Detailed
• Two UI options if desired: CLI vs. streamlit/web UI.
• Don’t over-simplify—capture every major Python script as its own component.