# xkiwilabs/llm-inference-hub architecture

> Architecture diagram of the GitHub repository xkiwilabs/llm-inference-hub, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xkiwilabs/llm-inference-hub
- Repository: https://github.com/xkiwilabs/llm-inference-hub

## Overview

Inference-hub packages vLLM model servers behind LiteLLM, exposing a shared OpenAI-compatible API and an Anthropic-compatible endpoint. The main workflow is client request → API gateway → selected model server → response. Operators set up hardware, download or switch models, start services, and manage keys and usage. Optional DecisiveLM runs beside the primary models. The tree and README document these capabilities, but no source excerpts were supplied, so internal service wiring is shown only where documented; other relationships are omitted or marked unverified.

Main entry points: [`litellm/config.template.yaml`](https://github.com/xkiwilabs/llm-inference-hub/blob/master/litellm/config.template.yaml) (LiteLLM Gateway), [`hub`](https://github.com/xkiwilabs/llm-inference-hub/blob/master/hub) (Hub Commands), [`scripts/pull-models.sh`](https://github.com/xkiwilabs/llm-inference-hub/blob/master/scripts/pull-models.sh) (Model Downloads), [`docker-compose.yml`](https://github.com/xkiwilabs/llm-inference-hub/blob/master/docker-compose.yml) (OpenAI API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_operations["Hub Operations"]
  node_hub["Hub Commands"]
  node_setup["Workstation Setup<br/>[setup.sh]"]
  node_model_download["Model Downloads<br/>[pull-models.sh]"]
  node_model_switch["Model Selection<br/>[set-model.sh]"]
end

subgraph group_gateway["API Gateway"]
  node_api_gateway["LiteLLM Gateway"]
  node_openai_api["OpenAI API<br/>[docker-compose.yml]"]
  node_anthropic_api["Anthropic API<br/>[docker-compose.yml]"]
end

subgraph group_inference["Model Inference"]
  node_small_model["Small vLLM Model"]
  node_large_model["Large vLLM Model"]
end

subgraph group_optional["Optional Inference"]
  node_decisivelm["DecisiveLM Server"]
end

node_client(("API Client"))
node_operator(("Operator"))
node_hf(("Hugging Face"))

node_operator -->|"runs commands"| node_hub
node_hub -->|"sets up"| node_setup
node_hub -->|"downloads models"| node_model_download
node_hub -->|"switches models"| node_model_switch
node_model_download -->|"downloads from"| node_hf
node_client -->|"sends requests"| node_openai_api
node_client -->|"sends requests"| node_anthropic_api
node_openai_api -->|"serves through"| node_api_gateway
node_anthropic_api -->|"serves through"| node_api_gateway
node_api_gateway -->|"routes requests"| node_small_model
node_api_gateway -->|"routes requests"| node_large_model
node_client -.->|"calls optional route"| node_api_gateway
node_api_gateway -.->|"routes optionally"| node_decisivelm

click node_hub "https://github.com/xkiwilabs/llm-inference-hub/blob/master/hub"
click node_setup "https://github.com/xkiwilabs/llm-inference-hub/blob/master/scripts/setup.sh"
click node_model_download "https://github.com/xkiwilabs/llm-inference-hub/blob/master/scripts/pull-models.sh"
click node_model_switch "https://github.com/xkiwilabs/llm-inference-hub/blob/master/scripts/set-model.sh"
click node_api_gateway "https://github.com/xkiwilabs/llm-inference-hub/blob/master/litellm/config.template.yaml"
click node_openai_api "https://github.com/xkiwilabs/llm-inference-hub/blob/master/docker-compose.yml"
click node_anthropic_api "https://github.com/xkiwilabs/llm-inference-hub/blob/master/docker-compose.yml"
click node_small_model "https://github.com/xkiwilabs/llm-inference-hub/blob/master/vllm/Dockerfile"
click node_large_model "https://github.com/xkiwilabs/llm-inference-hub/blob/master/vllm/Dockerfile"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_hub,node_setup,node_model_download,node_model_switch,node_client toneBlue
class node_api_gateway,node_openai_api,node_anthropic_api toneAmber
class node_small_model,node_large_model toneMint
class node_decisivelm toneRose
class node_operator,node_hf toneIndigo
```

## Components

### Hub Operations

- [Hub Commands](https://github.com/xkiwilabs/llm-inference-hub/blob/master/hub): `hub`
- [Workstation Setup](https://github.com/xkiwilabs/llm-inference-hub/blob/master/scripts/setup.sh): `scripts/setup.sh`
- [Model Downloads](https://github.com/xkiwilabs/llm-inference-hub/blob/master/scripts/pull-models.sh): `scripts/pull-models.sh`
- [Model Selection](https://github.com/xkiwilabs/llm-inference-hub/blob/master/scripts/set-model.sh): `scripts/set-model.sh`

### API Gateway

- [LiteLLM Gateway](https://github.com/xkiwilabs/llm-inference-hub/blob/master/litellm/config.template.yaml): `litellm/config.template.yaml`
- [OpenAI API](https://github.com/xkiwilabs/llm-inference-hub/blob/master/docker-compose.yml): `docker-compose.yml`
- [Anthropic API](https://github.com/xkiwilabs/llm-inference-hub/blob/master/docker-compose.yml): `docker-compose.yml`

### Model Inference

- [Small vLLM Model](https://github.com/xkiwilabs/llm-inference-hub/blob/master/vllm/Dockerfile): `vllm/Dockerfile`
- [Large vLLM Model](https://github.com/xkiwilabs/llm-inference-hub/blob/master/vllm/Dockerfile): `vllm/Dockerfile`

### Optional Inference

- **DecisiveLM Server**

### Other components

- **API Client**
- **Operator**
- **Hugging Face**

## Connections

- Operator → Hub Commands: runs commands (evidence: `README.md`)
- Hub Commands → Workstation Setup: sets up (evidence: `README.md`)
- Hub Commands → Model Downloads: downloads models (evidence: `README.md`)
- Hub Commands → Model Selection: switches models (evidence: `README.md`)
- Model Downloads → Hugging Face: downloads from (evidence: `README.md`)
- API Client → OpenAI API: sends requests (evidence: `README.md`)
- API Client → Anthropic API: sends requests (evidence: `README.md`)
- OpenAI API → LiteLLM Gateway: serves through (evidence: `README.md`)
- Anthropic API → LiteLLM Gateway: serves through (evidence: `README.md`)
- LiteLLM Gateway → Small vLLM Model: routes requests (evidence: `README.md`)
- LiteLLM Gateway → Large vLLM Model: routes requests (evidence: `README.md`)
- API Client → LiteLLM Gateway: calls optional route (evidence: `README.md`)
- LiteLLM Gateway → DecisiveLM Server: routes optionally (evidence: `README.md`)