How berriai/litellm works
LiteLLM is an AI gateway and Python SDK that presents an OpenAI-compatible interface to many model providers. The principal workflow is a client request entering through the SDK or proxy, being routed to a model/provider, and returning a normalized response; the proxy also provides key management, guardrails, spend tracking, and observability. The graph preserves documented endpoint and agent/MCP capabilities. The excerpts are limited, so several subsystem connections are shown as unverified rather than asserted as inspected call paths.
Main entry points: proxy_server.py Proxy gateway · main.py Completion API · llms LLM providers · router.py Model router
Components 21
Request entry
- SDK interface
litellm/__init__.py - Completion API
litellm/main.py - Proxy gateway
litellm/proxy/proxy_server.py
Model execution
- Model router
litellm/router.py - LLM providers
litellm/llms - Embeddings
litellm/embeddings/dispatch.py - Responses API
litellm/responses/main.py
Gateway controls
- Guardrails
litellm/guardrails.py - Policy pipelines
litellm/proxy/policy_engine/pipeline_executor.py - Key authentication
litellm/enterprise/litellm_enterprise/proxy/auth/user_api_key_auth.py - Admin management
litellm/enterprise/litellm_enterprise/proxy/management_endpoints/key_management_endpoints.py - Spend accounting
litellm/accounting/application.py - Callbacks and logging
litellm/integrations/custom_logger.py - Response caching
litellm/caching/caching_handler.py
Extended capabilities
- Batch API
litellm/batches/main.py - Files API
litellm/files/main.py - Agent protocol
litellm/a2a_protocol/main.py - MCP gateway
litellm/responses/mcp/litellm_proxy_mcp_handler.py - Trace storage
litellm/tracing/storage.py
Other components
- API client
- Model provider
Connections 22
- API client → SDK interface: calls
README.md - SDK interface → Completion API: exposes
litellm/main.py - API client → Proxy gateway: sends request
README.md - Proxy gateway → Model router: routes
- Completion API → Model router: dispatches
litellm/main.py - Model router → LLM providers: selects model
README.md - LLM providers → Model provider: calls
README.md - LLM providers → Completion API: returns response
litellm/main.py - Completion API → Embeddings: dispatches
- Completion API → Responses API: dispatches
- Proxy gateway → Key authentication: authenticates
- Proxy gateway → Policy pipelines: applies policies
- Policy pipelines → Guardrails: runs guardrails
litellm/proxy/policy_engine/pipeline_executor.py - Proxy gateway → Admin management: serves admin APIs
- Proxy gateway → Spend accounting: tracks spend
- Completion API → Callbacks and logging: logs calls
litellm/main.py - Completion API → Response caching: uses cache
- API client → Batch API: submits batches
README.md - API client → Files API: uses files
README.md - API client → Agent protocol: uses agent API
README.md - Proxy gateway → MCP gateway: routes MCP
litellm/responses/mcp/litellm_proxy_mcp_handler.py - Proxy gateway → Trace storage: stores traces