# inftyai/llmlite architecture

> Architecture diagram of the GitHub repository inftyai/llmlite, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/inftyai/llmlite
- Repository: https://github.com/inftyai/llmlite

## Overview

##### Project type and purpose
- **Type**: Open-source Python library for unified LLM access
- **Purpose**: Provide a consistent API (`ChatLLM`) to interact with varied LLMs (e.g., ChatGPT, Llama-2, Baichuan) across multiple backends (Hugging Face, vLLM), handling messaging, batching, streaming, and utilities (prompting, logging).

##### File structure
- **Top-level directories**
- `llmlite/apis` – public entry points (`chatllm.py`)
- `llmlite/backends` – backend abstraction and implementations
- `llmlite/llms` – model wrappers and message definitions
- `llmlite/utils` – logging and helper functions
- `examples/` – usage scenarios
- `tests/` – unit and integration tests
- **Config/build files**: `pyproject.toml`, `requirements.txt`, `Makefile`, CI workflows in `.github/workflows`.

##### README for technical stack
- **Languages/Frameworks**: Python 3, Hugging Face Transformers, vLLM
- **Patterns**: Strategy/Plugin for backends, Factory for model selection, standardized messaging (ChatMessage).

##### Main components and their interactions
a. **API Layer**:
- `ChatLLM` (in `apis/chatllm.py`): public class users import and call
- `ChatMessage`: standardized message object
b. **Model Layer**:
- Abstract `LLM` class (`llms/llm.py`)
- Concrete subclasses (`chatgpt.py`, `llama.py`, `baichuan.py`, `chatglm.py`) that know how to build model-specific prompts and interpret outputs
c. **Backend Layer**:
- `Backend` interface (`backends/backend.py`) defines methods like `generate()`
- Hugging Face backend (`hf_backend.py`)
- vLLM backend (`vllm_backend.py`) with support for continuous batching and streaming
d. **Utilities**:
- Logging (`utils/log.py`)
- Misc helpers (`utils/util.py`)
e. **External Dependencies**:
- HF Transformers library
- vLLM engine

Interaction flow (high level):
1. User code calls `ChatLLM.completion(messages…)`.
2. `ChatLLM` uses factory logic to instantiate the correct `LLM` subclass and the chosen `Backend`.
3. Message list is transformed into a prompt via the `LLM` class.
4. `ChatLLM` invokes the backend’s `generate()` (or streaming API).
5. Backend delegates to underlying HF or vLLM engine, returns raw output.
6. Output is post-processed by `LLM` wrapper into `ChatMessage` responses.

##### Architectural patterns and design principles
- **Strategy/Plugin Pattern**: Backends are interchangeable (`huggingface`, `vllm`).
- **Factory Method**: `ChatLLM` decides at runtime which `LLM` subclass and backend to instantiate based on `model_name_or_path` and `backend` parameter.
- **Separation of Concerns**: API, model-specific logic, backend integrations, and utilities live in distinct packages.

- **Highlight core extensibility points**: where to add new backends (`backends/`) or new model wrappers (`llms/`).
- **Show integration boundaries**: external libraries (HF, vLLM) as “External Services.”
- **Omit internal test and example folders** or show them as peripheral for context only.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% API Layer
    subgraph "API Layer"
        ChatLLM["ChatLLM API"]:::api
    end

    %% Model Layer
    subgraph "Model Layer"
        ModelRegistry["LLM Registry"]:::model
        LLMAbstract["Abstract LLM"]:::model
        ChatMessage["ChatMessage"]:::model
        ChatGPT["ChatGPT Wrapper"]:::model
        Llama["Llama Wrapper"]:::model
        Baichuan["Baichuan Wrapper"]:::model
        ChatGLM["ChatGLM Wrapper"]:::model
    end

    %% Backend Layer
    subgraph "Backend Layer"
        BackendInterface["Backend Interface"]:::backend
        HFBackend["Hugging Face Backend"]:::backend
        VLLMBackend["vLLM Backend"]:::backend
    end

    %% Utilities
    subgraph "Utilities"
        UtilLog["Logging Utilities"]:::util
        UtilHelper["Misc Helpers"]:::util
        Consts["Global Constants"]:::util
    end

    %% External Services
    subgraph "External Services"
        ExternalHF["Hugging Face Lib"]:::external
        ExternalVLLM["vLLM Engine"]:::external
    end

    %% Connections
    ChatLLM -->|"completion() creates"| ModelRegistry
    ModelRegistry -->|"inherits"| LLMAbstract
    LLMAbstract -->|"implements"| ChatGPT
    LLMAbstract -->|"implements"| Llama
    LLMAbstract -->|"implements"| Baichuan
    LLMAbstract -->|"implements"| ChatGLM
    ChatLLM -->|"formats messages"| ChatMessage

    ChatLLM -->|"generate() request"| BackendInterface
    BackendInterface -->|"select backend"| HFBackend
    BackendInterface -->|"select backend"| VLLMBackend
    HFBackend -->|"calls inference"| ExternalHF
    VLLMBackend -->|"calls inference"| ExternalVLLM

    ChatLLM --> UtilLog
    ChatLLM --> UtilHelper
    ChatLLM --> Consts

    %% Click Events
    click ChatLLM "https://github.com/inftyai/llmlite/blob/main/llmlite/apis/chatllm.py"
    click ChatMessage "https://github.com/inftyai/llmlite/blob/main/llmlite/llms/messages.py"
    click LLMAbstract "https://github.com/inftyai/llmlite/blob/main/llmlite/llms/llm.py"
    click ModelRegistry "https://github.com/inftyai/llmlite/blob/main/llmlite/llms/model.py"
    click ChatGPT "https://github.com/inftyai/llmlite/blob/main/llmlite/llms/chatgpt.py"
    click Llama "https://github.com/inftyai/llmlite/blob/main/llmlite/llms/llama.py"
    click Baichuan "https://github.com/inftyai/llmlite/blob/main/llmlite/llms/baichuan.py"
    click ChatGLM "https://github.com/inftyai/llmlite/blob/main/llmlite/llms/chatglm.py"
    click BackendInterface "https://github.com/inftyai/llmlite/blob/main/llmlite/backends/backend.py"
    click HFBackend "https://github.com/inftyai/llmlite/blob/main/llmlite/backends/hf_backend.py"
    click VLLMBackend "https://github.com/inftyai/llmlite/blob/main/llmlite/backends/vllm_backend.py"
    click UtilLog "https://github.com/inftyai/llmlite/blob/main/llmlite/utils/log.py"
    click UtilHelper "https://github.com/inftyai/llmlite/blob/main/llmlite/utils/util.py"
    click Consts "https://github.com/inftyai/llmlite/blob/main/llmlite/consts.py"

    %% Styles
    classDef api fill:#D0E8FF,stroke:#0366D6,color:#0366D6;
    classDef model fill:#E1F7D5,stroke:#28A745,color:#155724;
    classDef backend fill:#FFE5B4,stroke:#FF8C00,color:#8B4513;
    classDef util fill:#F3E5F5,stroke:#9C27B0,color:#4A148C;
    classDef external fill:#E0E0E0,stroke:#757575,color:#212121;
```
