# seonglae/llama2gptq architecture

> Architecture diagram of the GitHub repository seonglae/llama2gptq, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/seonglae/llama2gptq
- Repository: https://github.com/seonglae/llama2gptq

## Overview

##### Project Type and Purpose
- The project is a local, private Chat AI designed to provide responses using reference documents. It incorporates prompt engineering over a vector database and model quantization (GPTQ) to optimize inference performance.
- It supports two interfaces: a CLI (via main.py) and a Web UI (via chat.py built with Streamlit). Its integration with Texonom (and notions like Notion CSV exports) suggests document ingestion as a core feature, alongside quantization operations.

##### Analyzing the File Structure
- Top-Level Files:
- chat.py: Implements the Streamlit-based Web UI.
- main.py: Acts as the primary command entry point for the CLI (with multiple commands including chat, document ingestion, and quantization).
- constants.py: Likely holds shared const values.
- Folder “llama2gptq”: Contains core modules for different functionalities:
- generate.py: For generating responses, likely wrapping prompt engineering and LLM engine interactions.
- ingest.py: For processing and ingesting documents (e.g., Notion CSV exports).
- qa.py: Likely dedicated to question-answer interactions using the ingested knowledge.
- quantize.py: Handles GPTQ model quantization.
- Configuration and dependency files (pyproject.toml, requirements.lock) indicate a Python-centric app with specified package management (Rye) and environment consistency.

##### Major Components
a. User Interface Layer:
- CLI Interface (main.py): Processes commands like chat, ingest, and quantize.
- Web UI (chat.py via Streamlit): Provides an interactive chat interface.
b. Command & Orchestration Module:
- main.py acts as a command router calling specific functionalities within the “llama2gptq” modules based on user input.
c. LLM and Inference Engine:
- generate.py & qa.py: These modules combine with libraries such as LangChain, Transformers, and AutoGPTQ to process prompts and generate responses.
- Quantize functionality (quantize.py): Optimizes model precision and inference performance.
d. Data Ingestion & Vector Storage:
- ingest.py: Handles importing documents (notably from Notion CSV exports) to build the knowledge base.
- Integration with external vector database systems like ChromaDB (as cited in the README) for storing and retrieving embeddings.
e. External and Supporting Services:
- Texonom integration: The system suggests related web pages via integration with Texonom.
- Underlying technologies such as CUDA (for GPU acceleration), LangChain (for prompt engineering), and Fire (for CLI implementation).

##### Relationships and Interactions
- The User interacts via either the CLI or Web UI.
- Commands received by main.py delegate tasks to the appropriate modules in “llama2gptq.”
- For a chat request, flow from the UI → main.py → qa.py/generate.py → LLM stack (using LangChain and Transformers) → response returned to the UI.
- For ingestion, the command directs to ingest.py which processes documents and communicates with the vector database backend (ChromaDB).
- The quantization process involves the quantize.py module, which takes a source model (e.g., from Facebook/Opt) and runs AutoGPTQ routines to optimize inference performance.

##### Architectural Patterns & Design Principles
- Modular design: Each major functionality (chat, ingest, quantize) is encapsulated in its own module.
- Separation of concerns: The project separates the UI (CLI, Web UI) from back-end processing (LLM inference, ingestion, quantization).
- Command-dispatch routing: main.py acts as a central router, a common pattern in CLI applications, orchestrating different tasks.
- Extensibility: The system is designed to handle multiple data flows (chat, document processing, quantization) and is prepared for future features like MPS support or stateful web applications.

- “User Interface Layer” with two sub-components: “CLI (main.py)” and “Web UI (chat.py via Streamlit).”
- “Command & Orchestration” represented by main.py.
- “LLM and Inference Engine” including sub-components “generate.py” and “qa.py” along with “LLM Stack” (LangChain, Transformers, AutoGPTQ).
- “Data Ingestion & Vector Storage” where ingest.py interacts with external vector storage (ChromaDB) and document sources.
- “Quantization Module” represented by quantize.py (using AutoGPTQ).
- “External Services” (Texonom integration for related web page suggestions).
- Invocation flow from user inputs to main.py.
- Flow from main.py to various llama2gptq modules based on functionality.
- Communication lines between the inference engine and external dependencies like the vector database.
- Visual differentiation:

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global User
    U("User"):::user

    %% User Interface Layer
    subgraph "User Interface Layer"
        UI1("CLI (main.py)"):::ui
        UI2("Web UI (chat.py via Streamlit)"):::ui
    end

    U -->|"input"| UI1
    U -->|"input"| UI2

    %% Command & Orchestration Module
    OR("Command & Orchestration (main.py)"):::orchestration
    UI1 -->|"triggers"| OR
    UI2 -->|"triggers"| OR

    %% Backend Processing
    subgraph "Backend Processing"
        %% LLM and Inference Engine Subgraph
        subgraph "LLM and Inference Engine"
            G("Response Generation (llama2gptq/generate.py)"):::backend
            QA("QA Engine (llama2gptq/qa.py)"):::backend
            LS("LLM Stack (LangChain,Transformers,AutoGPTQ)"):::backend
        end
        OR -->|"chat_request"| G
        OR -->|"chat_request"| QA
        G -->|"uses"| LS
        QA -->|"uses"| LS
        LS -->|"sends_response"| OR

        %% Data Ingestion & Vector Storage Subgraph
        subgraph "Data Ingestion & Vector Storage"
            IN("Data Ingestion (llama2gptq/ingest.py)"):::backend
            DB("Vector DB (ChromaDB)"):::db
        end
        OR -->|"ingest"| IN
        IN -->|"stores_retrieves"| DB

        %% Quantization Subgraph
        subgraph "Quantization"
            QT("Quantization Module (llama2gptq/quantize.py)"):::backend
        end
        OR -->|"quantize"| QT
    end

    %% External Services
    TEX("Texonom Integration"):::external
    OR -->|"suggest_related"| TEX
    TEX -->|"feedback"| OR

    %% Click Events
    click UI1 "https://github.com/seonglae/llama2gptq/blob/main/main.py"
    click UI2 "https://github.com/seonglae/llama2gptq/blob/main/chat.py"
    click OR "https://github.com/seonglae/llama2gptq/blob/main/main.py"
    click G "https://github.com/seonglae/llama2gptq/blob/main/llama2gptq/generate.py"
    click QA "https://github.com/seonglae/llama2gptq/blob/main/llama2gptq/qa.py"
    click IN "https://github.com/seonglae/llama2gptq/blob/main/llama2gptq/ingest.py"
    click QT "https://github.com/seonglae/llama2gptq/blob/main/llama2gptq/quantize.py"

    %% Styles
    classDef user fill:#f9e79f,stroke:#7d6608,stroke-width:2px;
    classDef ui fill:#aed6f1,stroke:#1f618d,stroke-width:2px;
    classDef orchestration fill:#f5b7b1,stroke:#922b21,stroke-width:2px;
    classDef backend fill:#d5f5e3,stroke:#1e8449,stroke-width:2px;
    classDef db fill:#fcf3cf,stroke:#7d6608,stroke-width:2px;
    classDef external fill:#e8daef,stroke:#6c3483,stroke-width:2px;
```
