How inftyai/llmlite works
Project type and purpose
• Type: Open-source Python library for unified LLM access
• Purpose: Provide a consistent API (ChatLLM) to interact with varied LLMs (e.g., ChatGPT, Llama-2, Baichuan) across multiple backends (Hugging Face, vLLM), handling messaging, batching, streaming, and utilities (prompting, logging).
File structure
• Top-level directories
• llmlite/apis – public entry points (chatllm.py)
• llmlite/backends – backend abstraction and implementations
• llmlite/llms – model wrappers and message definitions
• llmlite/utils – logging and helper functions
• examples/ – usage scenarios
• tests/ – unit and integration tests
• Config/build files: pyproject.toml, requirements.txt, Makefile, CI workflows in .github/workflows.
README for technical stack
• Languages/Frameworks: Python 3, Hugging Face Transformers, vLLM
• Patterns: Strategy/Plugin for backends, Factory for model selection, standardized messaging (ChatMessage).
Main components and their interactions
a. API Layer:
• ChatLLM (in apis/chatllm.py): public class users import and call
• ChatMessage: standardized message object
b. Model Layer:
• Abstract LLM class (llms/llm.py)
• Concrete subclasses (chatgpt.py, llama.py, baichuan.py, chatglm.py) that know how to build model-specific prompts and interpret outputs
c. Backend Layer:
• Backend interface (backends/backend.py) defines methods like generate()
• Hugging Face backend (hf_backend.py)
• vLLM backend (vllm_backend.py) with support for continuous batching and streaming
d. Utilities:
• Logging (utils/log.py)
• Misc helpers (utils/util.py)
e. External Dependencies:
• HF Transformers library
• vLLM engine
Interaction flow (high level):
1. User code calls ChatLLM.completion(messages…).
2. ChatLLM uses factory logic to instantiate the correct LLM subclass and the chosen Backend.
3. Message list is transformed into a prompt via the LLM class.
4. ChatLLM invokes the backend’s generate() (or streaming API).
5. Backend delegates to underlying HF or vLLM engine, returns raw output.
6. Output is post-processed by LLM wrapper into ChatMessage responses.
Architectural patterns and design principles
• Strategy/Plugin Pattern: Backends are interchangeable (huggingface, vllm).
• Factory Method: ChatLLM decides at runtime which LLM subclass and backend to instantiate based on model_name_or_path and backend parameter.
• Separation of Concerns: API, model-specific logic, backend integrations, and utilities live in distinct packages.
• Highlight core extensibility points: where to add new backends (backends/) or new model wrappers (llms/).
• Show integration boundaries: external libraries (HF, vLLM) as “External Services.”
• Omit internal test and example folders or show them as peripheral for context only.