# h2oai/h2ogpt architecture

> Architecture diagram of the GitHub repository h2oai/h2ogpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/h2oai/h2ogpt
- Repository: https://github.com/h2oai/h2ogpt

## Overview

##### Main Components

a) Core Components:
- Document Processing System
- LLM Inference Engine
- Database Systems (Chroma/Weaviate/FAISS)
- API Layer
- User Interface Layer (Gradio UI/CLI)

b) Integration Components:
- Inference Servers Support
- Authentication System
- Document Processing Pipeline
- Voice/Audio Processing
- Vision Processing
- Agent System

##### Key Relationships to Highlight

- Data flow for document processing
- Query/response pathways
- Authentication flows
- Model interaction patterns
- Database operations

##### Special Features to Emphasize

Create dedicated sections showing:
- HYDE (Hypothetical Document Embeddings) system
- Parallel processing capabilities
- Attention Sinks for long-context handling
- Agent system architecture
- Multi-modal processing (text, image, audio)

##### Technical Details

Annotations for:
- Supported model types (LLaMa2, Mistral, Falcon, etc.)
- Document format support
- Hardware requirements (GPU/CPU)
- Key technologies (AutoGPTQ, LORA, etc.)

##### Integration Points

Connections to:
- External inference servers
- Web search integration
- Cloud services
- Authentication providers

##### Deployment Options

A separate section showing:
- Docker deployment
- Native installation
- Cloud deployment options
- Inference server configurations

- Blue: Core components
- Green: Integration points
- Yellow: Processing pipelines
- Red: Security/authentication
- Purple: Storage systems
- Orange: External services

This architecture should emphasize h2oGPT's key strengths: privacy, offline capability, multi-modal support, and extensive integration options.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Client["Client Layer"]
        UI[Gradio UI]:::ui
        CLI[Command Line Interface]:::ui
        API["OpenAI-compatible API"]:::api
        ClientLib[Client Libraries]:::ui
    end

    subgraph Gateway["API Gateway Layer"]
        Auth[Authentication System]:::security
        Router["Request Router"]:::core
        Tools["Function Calling & Tools"]:::core
    end

    subgraph Core["Core Processing Layer"]
        DocProc["Document Processing"]:::processing
        LLMCoord["LLM Coordination"]:::core
        EmbGen["Embedding Generation"]:::processing
        AgentSys["Agent System"]:::core
    end

    subgraph Models["Model Layer"]
        LLM["LLM Models"]:::model
        EmbModel["Embedding Models"]:::model
        VisionModel["Vision Models"]:::model
        AudioModel["Audio Models"]:::model
    end

    subgraph Storage["Storage Layer"]
        VectorDB["Vector Databases"]:::storage
        DocStore["Document Store"]:::storage
        StateStore["User State Storage"]:::storage
    end

    subgraph Integration["Integration Layer"]
        InfServers["Inference Servers"]:::external
        WebSearch["Web Search"]:::external
        CloudServ["Cloud Services"]:::external
    end

    %% Connections
    UI --> Auth
    CLI --> Auth
    API --> Auth
    ClientLib --> Auth

    Auth --> Router
    Router --> Tools
    Tools --> DocProc
    Tools --> AgentSys

    DocProc --> EmbGen
    DocProc --> LLMCoord
    AgentSys --> LLMCoord
    EmbGen --> VectorDB

    LLMCoord --> LLM
    LLMCoord --> VisionModel
    LLMCoord --> AudioModel
    EmbGen --> EmbModel

    LLM --> InfServers
    LLM --> CloudServ
    DocProc --> WebSearch

    %% Styles
    classDef core fill:#2196F3,stroke:#1565C0,color:white
    classDef ui fill:#4CAF50,stroke:#2E7D32,color:white
    classDef api fill:#FFC107,stroke:#FFA000,color:black
    classDef security fill:#F44336,stroke:#D32F2F,color:white
    classDef processing fill:#FF9800,stroke:#F57C00,color:white
    classDef model fill:#9C27B0,stroke:#7B1FA2,color:white
    classDef storage fill:#795548,stroke:#5D4037,color:white
    classDef external fill:#607D8B,stroke:#455A64,color:white

    %% Click Events
    click UI "https://github.com/h2oai/h2ogpt/tree/main/gradio_utils/"
    click DocProc "https://github.com/h2oai/h2ogpt/blob/main/src/loaders.py"
    click LLM "https://github.com/h2oai/h2ogpt/tree/main/models/"
    click API "https://github.com/h2oai/h2ogpt/tree/main/openai_server/"
    click AgentSys "https://github.com/h2oai/h2ogpt/tree/main/openai_server/agent_tools/"
    click Auth "https://github.com/h2oai/h2ogpt/blob/main/gradio_utils/google_auth.py"
    click LLMCoord "https://github.com/h2oai/h2ogpt/blob/main/src/gen.py"
    click VisionModel "https://github.com/h2oai/h2ogpt/tree/main/src/vision/"
    click AudioModel "https://github.com/h2oai/h2ogpt/blob/main/src/audio_langchain.py"
    click VectorDB "https://github.com/h2oai/h2ogpt/blob/main/src/db_utils.py"
```
