# berriai/litellm architecture

> Architecture diagram of the GitHub repository berriai/litellm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/berriai/litellm
- Repository: https://github.com/berriai/litellm

## Overview

LiteLLM is an AI gateway and Python SDK that presents an OpenAI-compatible interface to many model providers. The principal workflow is a client request entering through the SDK or proxy, being routed to a model/provider, and returning a normalized response; the proxy also provides key management, guardrails, spend tracking, and observability. The graph preserves documented endpoint and agent/MCP capabilities. The excerpts are limited, so several subsystem connections are shown as unverified rather than asserted as inspected call paths.

Main entry points: [`litellm/proxy/proxy_server.py`](https://github.com/berriai/litellm/blob/main/litellm/proxy/proxy_server.py) (Proxy gateway), [`litellm/main.py`](https://github.com/berriai/litellm/blob/main/litellm/main.py) (Completion API), [`litellm/llms`](https://github.com/berriai/litellm/tree/main/litellm/llms) (LLM providers), [`litellm/router.py`](https://github.com/berriai/litellm/blob/main/litellm/router.py) (Model router)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Request entry"]
  node_sdk["SDK interface<br/>[__init__.py]"]
  node_completion["Completion API<br/>[main.py]"]
  node_proxy["Proxy gateway<br/>[proxy_server.py]"]
end

subgraph group_routing["Model execution"]
  node_router["Model router<br/>[router.py]"]
  node_provider["LLM providers"]
  node_embeddings["Embeddings<br/>[dispatch.py]"]
  node_responses["Responses API<br/>[main.py]"]
end

subgraph group_gateway["Gateway controls"]
  node_guardrails["Guardrails<br/>[guardrails.py]"]
  node_policy["Policy pipelines"]
  node_auth["Key authentication"]
  node_management["Admin management"]
  node_accounting["Spend accounting<br/>[application.py]"]
  node_logging["Callbacks and logging<br/>[custom_logger.py]"]
  node_cache["Response caching<br/>[caching_handler.py]"]
end

subgraph group_capabilities["Extended capabilities"]
  node_batches["Batch API<br/>[main.py]"]
  node_files["Files API<br/>[main.py]"]
  node_agents["Agent protocol<br/>[main.py]"]
  node_mcp["MCP gateway"]
  node_tracing["Trace storage<br/>[storage.py]"]
end

node_client(("API client"))
node_provider_service(("Model provider"))

node_client -->|"calls"| node_sdk
node_sdk -.->|"exposes"| node_completion
node_client -->|"sends request"| node_proxy
node_proxy -.->|"routes"| node_router
node_completion -->|"dispatches"| node_router
node_router -->|"selects model"| node_provider
node_provider -->|"calls"| node_provider_service
node_provider -.->|"returns response"| node_completion
node_completion -.->|"dispatches"| node_embeddings
node_completion -.->|"dispatches"| node_responses
node_proxy -.->|"authenticates"| node_auth
node_proxy -.->|"applies policies"| node_policy
node_policy -->|"runs guardrails"| node_guardrails
node_proxy -.->|"serves admin APIs"| node_management
node_proxy -.->|"tracks spend"| node_accounting
node_completion -->|"logs calls"| node_logging
node_completion -.->|"uses cache"| node_cache
node_client -->|"submits batches"| node_batches
node_client -->|"uses files"| node_files
node_client -->|"uses agent API"| node_agents
node_proxy -.->|"routes MCP"| node_mcp
node_proxy -.->|"stores traces"| node_tracing

click node_sdk "https://github.com/berriai/litellm/blob/main/litellm/__init__.py"
click node_completion "https://github.com/berriai/litellm/blob/main/litellm/main.py"
click node_proxy "https://github.com/berriai/litellm/blob/main/litellm/proxy/proxy_server.py"
click node_router "https://github.com/berriai/litellm/blob/main/litellm/router.py"
click node_provider "https://github.com/berriai/litellm/tree/main/litellm/llms"
click node_embeddings "https://github.com/berriai/litellm/blob/main/litellm/embeddings/dispatch.py"
click node_responses "https://github.com/berriai/litellm/blob/main/litellm/responses/main.py"
click node_guardrails "https://github.com/berriai/litellm/blob/main/litellm/guardrails.py"
click node_policy "https://github.com/berriai/litellm/blob/main/litellm/proxy/policy_engine/pipeline_executor.py"
click node_auth "https://github.com/berriai/litellm/blob/main/litellm/enterprise/litellm_enterprise/proxy/auth/user_api_key_auth.py"
click node_management "https://github.com/berriai/litellm/blob/main/litellm/enterprise/litellm_enterprise/proxy/management_endpoints/key_management_endpoints.py"
click node_accounting "https://github.com/berriai/litellm/blob/main/litellm/accounting/application.py"
click node_logging "https://github.com/berriai/litellm/blob/main/litellm/integrations/custom_logger.py"
click node_cache "https://github.com/berriai/litellm/blob/main/litellm/caching/caching_handler.py"
click node_batches "https://github.com/berriai/litellm/blob/main/litellm/batches/main.py"
click node_files "https://github.com/berriai/litellm/blob/main/litellm/files/main.py"
click node_agents "https://github.com/berriai/litellm/blob/main/litellm/a2a_protocol/main.py"
click node_mcp "https://github.com/berriai/litellm/blob/main/litellm/responses/mcp/litellm_proxy_mcp_handler.py"
click node_tracing "https://github.com/berriai/litellm/blob/main/litellm/tracing/storage.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sdk,node_completion,node_proxy,node_client toneBlue
class node_router,node_provider,node_embeddings,node_responses toneAmber
class node_guardrails,node_policy,node_auth,node_management,node_accounting,node_logging,node_cache toneMint
class node_batches,node_files,node_agents,node_mcp,node_tracing toneRose
class node_provider_service toneIndigo
```

## Components

### Request entry

- [SDK interface](https://github.com/berriai/litellm/blob/main/litellm/__init__.py): `litellm/__init__.py`
- [Completion API](https://github.com/berriai/litellm/blob/main/litellm/main.py): `litellm/main.py`
- [Proxy gateway](https://github.com/berriai/litellm/blob/main/litellm/proxy/proxy_server.py): `litellm/proxy/proxy_server.py`

### Model execution

- [Model router](https://github.com/berriai/litellm/blob/main/litellm/router.py): `litellm/router.py`
- [LLM providers](https://github.com/berriai/litellm/tree/main/litellm/llms): `litellm/llms`
- [Embeddings](https://github.com/berriai/litellm/blob/main/litellm/embeddings/dispatch.py): `litellm/embeddings/dispatch.py`
- [Responses API](https://github.com/berriai/litellm/blob/main/litellm/responses/main.py): `litellm/responses/main.py`

### Gateway controls

- [Guardrails](https://github.com/berriai/litellm/blob/main/litellm/guardrails.py): `litellm/guardrails.py`
- [Policy pipelines](https://github.com/berriai/litellm/blob/main/litellm/proxy/policy_engine/pipeline_executor.py): `litellm/proxy/policy_engine/pipeline_executor.py`
- [Key authentication](https://github.com/berriai/litellm/blob/main/litellm/enterprise/litellm_enterprise/proxy/auth/user_api_key_auth.py): `litellm/enterprise/litellm_enterprise/proxy/auth/user_api_key_auth.py`
- [Admin management](https://github.com/berriai/litellm/blob/main/litellm/enterprise/litellm_enterprise/proxy/management_endpoints/key_management_endpoints.py): `litellm/enterprise/litellm_enterprise/proxy/management_endpoints/key_management_endpoints.py`
- [Spend accounting](https://github.com/berriai/litellm/blob/main/litellm/accounting/application.py): `litellm/accounting/application.py`
- [Callbacks and logging](https://github.com/berriai/litellm/blob/main/litellm/integrations/custom_logger.py): `litellm/integrations/custom_logger.py`
- [Response caching](https://github.com/berriai/litellm/blob/main/litellm/caching/caching_handler.py): `litellm/caching/caching_handler.py`

### Extended capabilities

- [Batch API](https://github.com/berriai/litellm/blob/main/litellm/batches/main.py): `litellm/batches/main.py`
- [Files API](https://github.com/berriai/litellm/blob/main/litellm/files/main.py): `litellm/files/main.py`
- [Agent protocol](https://github.com/berriai/litellm/blob/main/litellm/a2a_protocol/main.py): `litellm/a2a_protocol/main.py`
- [MCP gateway](https://github.com/berriai/litellm/blob/main/litellm/responses/mcp/litellm_proxy_mcp_handler.py): `litellm/responses/mcp/litellm_proxy_mcp_handler.py`
- [Trace storage](https://github.com/berriai/litellm/blob/main/litellm/tracing/storage.py): `litellm/tracing/storage.py`

### Other components

- **API client**
- **Model provider**

## Connections

- API client → SDK interface: calls (evidence: `README.md`)
- SDK interface → Completion API: exposes (evidence: `litellm/main.py`)
- API client → Proxy gateway: sends request (evidence: `README.md`)
- Proxy gateway → Model router: routes
- Completion API → Model router: dispatches (evidence: `litellm/main.py`)
- Model router → LLM providers: selects model (evidence: `README.md`)
- LLM providers → Model provider: calls (evidence: `README.md`)
- LLM providers → Completion API: returns response (evidence: `litellm/main.py`)
- Completion API → Embeddings: dispatches
- Completion API → Responses API: dispatches
- Proxy gateway → Key authentication: authenticates
- Proxy gateway → Policy pipelines: applies policies
- Policy pipelines → Guardrails: runs guardrails (evidence: `litellm/proxy/policy_engine/pipeline_executor.py`)
- Proxy gateway → Admin management: serves admin APIs
- Proxy gateway → Spend accounting: tracks spend
- Completion API → Callbacks and logging: logs calls (evidence: `litellm/main.py`)
- Completion API → Response caching: uses cache
- API client → Batch API: submits batches (evidence: `README.md`)
- API client → Files API: uses files (evidence: `README.md`)
- API client → Agent protocol: uses agent API (evidence: `README.md`)
- Proxy gateway → MCP gateway: routes MCP (evidence: `litellm/responses/mcp/litellm_proxy_mcp_handler.py`)
- Proxy gateway → Trace storage: stores traces
