How bentoml/openllm works
Project type and purpose
• OpenLLM is an open-source Python CLI framework/tool for self-hosting large language models (LLMs) as OpenAI-compatible HTTP services.
• It offers a built-in chat UI, multiple inference backends, model-repository management, and seamless cloud deployment via BentoML/BentoCloud.
File structure
• Top-level files: pyproject.toml (build config), README.md, development scripts (release.sh, gen_readme.py), and CI/CD/workflow configs.
• src/openllm package:
• __main__.py – CLI entrypoint (“openllm serve/run/model/repo/deploy”).
• accelerator_spec.py – hardware/backend abstraction.
• analytic.py – telemetry/usage tracking.
• clean.py – temp directory & cache cleanup.
• cloud.py – “deploy” command, BentoML/BentoCloud integration.
• local.py – “serve” and “run” commands, launches local HTTP server & CLI chat.
• model.py – model metadata/schema, loading logic.
• repo.py – model-repository management (add, list, update).
• venv.py – dependency/virtual-environment management for custom models.
README for additional insights
• Defines OpenAI-compatible REST API endpoints at /v1, /chat UI at /chat.
• Leverages BentoML for service packaging, vllm (and others) for inference backends, chatgpt-lite for UI, and astral-sh/uv for dependency installation.
• Supports “serve” (HTTP server), “run” (CLI chat), “model” (catalog operations), “repo” (repo management), and “deploy” (cloud).
1. User Interface Layer
– CLI Client (“openllm” commands)
– Web Chat UI (chatgpt-lite served at /chat)
2. Command Dispatcher & Core Modules
– CLI Dispatcher (__main__)
– Modules: local.py, cloud.py, repo.py, model.py, analytic.py, clean.py, accelerator_spec.py, venv.py
3. Local Server & API Layer
– BentoML/FastAPI Service exposing OpenAI-compatible REST endpoints
– Request Router → ModelManager
4. Model Management & Inference Layer
– Model Loader (model.py + repo manager + HF token & cache)
– Inference Engines (vllm, PyTorch, etc., via accelerator_spec)
5. External Integrations
– Hugging Face (weights download)
– BentoML Registry & BentoCloud (cloud deployment)
– Telemetry Endpoint (analytics)
• CLI Client → CLI Dispatcher → appropriate module (serve/run/model/repo/deploy)
• “serve” module → spins up Local Server/API → listens on HTTP port
• Client (CLI run or third-party OpenAI client) ↔ HTTP API → Inference Layer → returns model responses
• Chat UI frontend → HTTP API → Inference Layer → streams tokens back to browser
• repo module → contacts remote Git repos → updates local model catalog
• cloud module → packages service with BentoML → pushes to BentoCloud → monitors deployment
• analytic module → emits usage data to telemetry endpoint
c. Architectural patterns & principles
– Layered separation: CLI layer, service/API layer, inference layer, external services.
– Plugin/adapter pattern for inference backends (accelerator_spec).
– Modular command pattern for extensibility (each CLI command in its own module).
d. Key technologies & frameworks
– Python, pyproject.toml (PEP517)
– BentoML/BentoCloud (model serving & deployment)
– FastAPI (via BentoML) for HTTP API
– vllm, PyTorch/transformers for inference
– chatgpt-lite for the chat UI
– astral-sh/uv for fast dependency installation
– pre-commit, GitHub Actions (CI/CD), Ruff (linting), pyright (type checking)
Details without over-splitting
• Every core module from src/openllm as part of the “Command & Core Modules” layer.
• How the model loader, accelerator_spec, and repo modules converge into the inference engine.