# bentoml/openllm architecture

> Architecture diagram of the GitHub repository bentoml/openllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bentoml/openllm
- Repository: https://github.com/bentoml/openllm

## Overview

##### Project type and purpose
- OpenLLM is an open-source Python CLI framework/tool for self-hosting large language models (LLMs) as OpenAI-compatible HTTP services.
- It offers a built-in chat UI, multiple inference backends, model-repository management, and seamless cloud deployment via BentoML/BentoCloud.

##### File structure
- Top-level files: pyproject.toml (build config), README.md, development scripts (release.sh, gen_readme.py), and CI/CD/workflow configs.
- src/openllm package:
- __main__.py – CLI entrypoint (“openllm serve/run/model/repo/deploy”).
- accelerator_spec.py – hardware/backend abstraction.
- analytic.py – telemetry/usage tracking.
- clean.py – temp directory & cache cleanup.
- cloud.py – “deploy” command, BentoML/BentoCloud integration.
- local.py – “serve” and “run” commands, launches local HTTP server & CLI chat.
- model.py – model metadata/schema, loading logic.
- repo.py – model-repository management (add, list, update).
- venv.py – dependency/virtual-environment management for custom models.

##### README for additional insights
- Defines OpenAI-compatible REST API endpoints at /v1, /chat UI at /chat.
- Leverages BentoML for service packaging, vllm (and others) for inference backends, chatgpt-lite for UI, and astral-sh/uv for dependency installation.
- Supports “serve” (HTTP server), “run” (CLI chat), “model” (catalog operations), “repo” (repo management), and “deploy” (cloud).

1. User Interface Layer
– CLI Client (“openllm” commands)
– Web Chat UI (chatgpt-lite served at /chat)
2. Command Dispatcher & Core Modules
– CLI Dispatcher (__main__)
– Modules: local.py, cloud.py, repo.py, model.py, analytic.py, clean.py, accelerator_spec.py, venv.py
3. Local Server & API Layer
– BentoML/FastAPI Service exposing OpenAI-compatible REST endpoints
– Request Router → ModelManager
4. Model Management & Inference Layer
– Model Loader (model.py + repo manager + HF token & cache)
– Inference Engines (vllm, PyTorch, etc., via accelerator_spec)
5. External Integrations
– Hugging Face (weights download)
– BentoML Registry & BentoCloud (cloud deployment)
– Telemetry Endpoint (analytics)
- CLI Client → CLI Dispatcher → appropriate module (serve/run/model/repo/deploy)
- “serve” module → spins up Local Server/API → listens on HTTP port
- Client (CLI run or third-party OpenAI client) ↔ HTTP API → Inference Layer → returns model responses
- Chat UI frontend → HTTP API → Inference Layer → streams tokens back to browser
- repo module → contacts remote Git repos → updates local model catalog
- cloud module → packages service with BentoML → pushes to BentoCloud → monitors deployment
- analytic module → emits usage data to telemetry endpoint
c. Architectural patterns & principles
– Layered separation: CLI layer, service/API layer, inference layer, external services.
– Plugin/adapter pattern for inference backends (accelerator_spec).
– Modular command pattern for extensibility (each CLI command in its own module).
d. Key technologies & frameworks
– Python, pyproject.toml (PEP517)
– BentoML/BentoCloud (model serving & deployment)
– FastAPI (via BentoML) for HTTP API
– vllm, PyTorch/transformers for inference
– chatgpt-lite for the chat UI
– astral-sh/uv for fast dependency installation
– pre-commit, GitHub Actions (CI/CD), Ruff (linting), pyright (type checking)

##### Details without over-splitting
- Every core module from src/openllm as part of the “Command & Core Modules” layer.
- How the model loader, accelerator_spec, and repo modules converge into the inference engine.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI & Developer Tools
    subgraph "CLI & Developer Tools"
        direction TB
        CLIClient["CLI Client\n(openllm commands)"]:::cli
        CD["CLI Dispatcher\n__main__.py"]:::cli
    end

    %% Core Modules
    subgraph "Command & Core Modules"
        direction TB
        ACC["Hardware/Backend Abstraction\naccelerator_spec.py"]:::core
        AN["Telemetry/Usage Tracking\nanalytic.py"]:::core
        CLN["Temp Dir & Cache Cleanup\nclean.py"]:::core
        CLD["Cloud Deployment\ncloud.py"]:::core
        LCL["Local Server & CLI Chat\nlocal.py"]:::core
        MDL["Model Metadata & Loader\nmodel.py"]:::core
        RPO["Model-Repo Management\nrepo.py"]:::core
        VNV["Virtual Env & Dependency\nvenv.py"]:::core
        CMN["Shared Utilities & Definitions\ncommon.py"]:::core
    end

    %% Local Server & API Layer
    subgraph "Local Server/API Layer"
        direction TB
        BM["BentoML/FastAPI Service"]:::api
        RR["Request Router\n(v1 endpoints)"]:::api
    end

    %% Model Management & Inference Layer
    subgraph "Model & Inference Layer"
        direction TB
        ML["Model Loader"]:::inference
        IE["Inference Engines\n(vllm/PyTorch/etc.)"]:::inference
    end

    %% External Integrations
    subgraph "External Integrations"
        direction TB
        HF["Hugging Face\n(weights download)"]:::ext
        BR["BentoML Registry & BentoCloud"]:::ext
        TE["Telemetry Endpoint"]:::ext
        GR["Git Repositories"]:::ext
    end

    %% Web Chat UI
    WebUI["Web Chat UI\n(chatgpt-lite at /chat)"]:::cli

    %% Connections
    CLIClient -->|"invoke"| CD
    CD -->|"dispatch"| ACC
    CD -->|"dispatch"| AN
    CD -->|"dispatch"| CLN
    CD -->|"dispatch"| CLD
    CD -->|"dispatch"| LCL
    CD -->|"dispatch"| MDL
    CD -->|"dispatch"| RPO
    CD -->|"dispatch"| VNV
    CD -->|"dispatch"| CMN

    LCL -->|"start server"| BM
    BM -->|"route requests"| RR
    RR -->|"load model"| ML
    RR -->|"infer"| IE
    RR -->|"return response"| BM

    WebUI -->|"HTTP API"| BM
    BM -->|"stream tokens"| WebUI

    RPO -->|"fetch/update"| GR
    CLD -->|"deploy service"| BR
    AN -->|"emit data"| TE
    ML -->|"download weights"| HF

    %% Click Events
    click CD "https://github.com/bentoml/openllm/blob/main/src/openllm/__main__.py"
    click ACC "https://github.com/bentoml/openllm/blob/main/src/openllm/accelerator_spec.py"
    click AN "https://github.com/bentoml/openllm/blob/main/src/openllm/analytic.py"
    click CLN "https://github.com/bentoml/openllm/blob/main/src/openllm/clean.py"
    click CLD "https://github.com/bentoml/openllm/blob/main/src/openllm/cloud.py"
    click LCL "https://github.com/bentoml/openllm/blob/main/src/openllm/local.py"
    click MDL "https://github.com/bentoml/openllm/blob/main/src/openllm/model.py"
    click RPO "https://github.com/bentoml/openllm/blob/main/src/openllm/repo.py"
    click VNV "https://github.com/bentoml/openllm/blob/main/src/openllm/venv.py"
    click CMN "https://github.com/bentoml/openllm/blob/main/src/openllm/common.py"

    %% Styles
    classDef cli fill:#fef2c0,stroke:#333,stroke-width:1px
    classDef core fill:#d0e1f9,stroke:#333,stroke-width:1px
    classDef api fill:#c6f7d1,stroke:#333,stroke-width:1px
    classDef inference fill:#f9d0e3,stroke:#333,stroke-width:1px
    classDef ext fill:#ffe4b3,stroke:#333,stroke-width:1px
```
