How xkiwilabs/llm-inference-hub works
Inference-hub packages vLLM model servers behind LiteLLM, exposing a shared OpenAI-compatible API and an Anthropic-compatible endpoint. The main workflow is client request → API gateway → selected model server → response. Operators set up hardware, download or switch models, start services, and manage keys and usage. Optional DecisiveLM runs beside the primary models. The tree and README document these capabilities, but no source excerpts were supplied, so internal service wiring is shown only where documented; other relationships are omitted or marked unverified.
Main entry points: config.template.yaml LiteLLM Gateway · hub Hub Commands · pull-models.sh Model Downloads · docker-compose.yml OpenAI API
Components 13
Hub Operations
- Hub Commands
hub - Workstation Setup
scripts/setup.sh - Model Downloads
scripts/pull-models.sh - Model Selection
scripts/set-model.sh
API Gateway
- LiteLLM Gateway
litellm/config.template.yaml - OpenAI API
docker-compose.yml - Anthropic API
docker-compose.yml
Model Inference
- Small vLLM Model
vllm/Dockerfile - Large vLLM Model
vllm/Dockerfile
Optional Inference
- DecisiveLM Server
Other components
- API Client
- Operator
- Hugging Face
Connections 13
- Operator → Hub Commands: runs commands
README.md - Hub Commands → Workstation Setup: sets up
README.md - Hub Commands → Model Downloads: downloads models
README.md - Hub Commands → Model Selection: switches models
README.md - Model Downloads → Hugging Face: downloads from
README.md - API Client → OpenAI API: sends requests
README.md - API Client → Anthropic API: sends requests
README.md - OpenAI API → LiteLLM Gateway: serves through
README.md - Anthropic API → LiteLLM Gateway: serves through
README.md - LiteLLM Gateway → Small vLLM Model: routes requests
README.md - LiteLLM Gateway → Large vLLM Model: routes requests
README.md - API Client → LiteLLM Gateway: calls optional route
README.md - LiteLLM Gateway → DecisiveLM Server: routes optionally
README.md