# cyborgmarina/litellm architecture

> Architecture diagram of the GitHub repository cyborgmarina/litellm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cyborgmarina/litellm
- Repository: https://github.com/cyborgmarina/litellm

## Overview

##### High-Level Architecture Components

The system should be represented with these main components:

a) Core LLM Integration Layer
- Model Providers Integration
- Completion/Chat Handlers
- Embedding Services
- Streaming Services

b) Proxy Server Layer
- Request Routing
- Authentication
- Rate Limiting
- Caching

c) Management Layer
- User Management
- API Key Management
- Budget Controls
- Monitoring & Logging

##### Key Relationships to Highlight

**[Client Applications]**
- ↓
**[LiteLLM Proxy Server]**
- ↓
**[Router & Load Balancer]**
- ↓
**[Model Provider Handlers]**
- ↓
**[External LLM Services]**

##### Detailed Component Breakdown

a) Proxy Server Components:
**[Authentication] → [Request Validation] → [Rate Limiting]**
- ↓
**[Model Router] → [Caching Layer] → [Provider-Specific Handlers]**
- ↓
**[Response Processing] → [Logging & Monitoring]**

b) Integration Layer:
**[Model Provider Adapters]**
- OpenAI
- Azure
- Anthropic
- Gemini (Modified)
- Others

##### Cross-Cutting Concerns

As vertical layers intersecting the main flow:
- Logging & Monitoring
- Security
- Error Handling
- Cost Management

##### Specific Implementation Details

For the Gemini streaming modification:
**[Gemini Handler]**
- ↓
**[Streaming Controller]**
- ↓
**[Grounding Metadata Handler] (New Feature)**
- ↓
**[Response Formatter]**

##### Deployment View

How the system can be deployed:
**[Client Zone] → [Load Balancer] → [Application Servers]**
- ↓
**[Cache Layer] → [Database Layer]**
- ↓
**[External Services Zone]**

##### Scalability aspects

Use standardized symbols:
- Rectangles: Services/Components
- Diamonds: Decision Points
- Double Lines: Security Boundaries

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Client["Client Layer"]
        CA[Client Applications]
    end

    subgraph Proxy["Proxy Server Layer"]
        PS["Proxy Server"]:::server
        RR["Request Router"]:::router
        Auth["Authentication"]:::security
        RL["Rate Limiter"]:::security
        Cache["Caching"]:::cache
    end

    subgraph Core["Core LLM Integration Layer"]
        Router["Router & Load Balancer"]:::router
        MP["Model Providers"]:::provider
        CH["Completion Handlers"]:::handler
        ES["Embedding Services"]:::service
        SS["Streaming Services"]:::service
    end

    subgraph Management["Management Layer"]
        UM["User Management"]:::mgmt
        AK["API Key Management"]:::mgmt
        BC["Budget Controls"]:::mgmt
        ML["Monitoring & Logging"]:::monitor
    end

    subgraph External["External Services"]
        LLM["LLM Services"]:::external
    end

    %% Cross-cutting concerns
    subgraph CrossCutting["Cross-Cutting Concerns"]
        SEC["Security"]:::security
        LOG["Logging"]:::monitor
        ERR["Error Handling"]:::error
        COST["Cost Management"]:::mgmt
    end

    %% Relationships
    CA --> PS
    PS --> RR
    RR --> Auth
    Auth --> RL
    RL --> Cache
    Cache --> Router
    Router --> MP
    MP --> CH
    MP --> ES
    MP --> SS
    Router --> LLM

    %% Cross-cutting relationships
    CrossCutting -.-> Proxy
    CrossCutting -.-> Core
    CrossCutting -.-> Management

    %% Management relationships
    UM --> PS
    AK --> Auth
    BC --> COST
    ML --> LOG

    %% Styling
    classDef server fill:#6495ED
    classDef router fill:#98FB98
    classDef security fill:#FFD700
    classDef cache fill:#DDA0DD
    classDef provider fill:#20B2AA
    classDef handler fill:#87CEEB
    classDef service fill:#87CEFA
    classDef mgmt fill:#F08080
    classDef monitor fill:#FFA07A
    classDef external fill:#C0C0C0
    classDef error fill:#FF6347

    %% Click events
    click PS "https://github.com/cyborgmarina/litellm/blob/main/litellm/proxy/proxy_server.py"
    click RR "https://github.com/cyborgmarina/litellm/blob/main/litellm/proxy/route_llm_request.py"
    click Auth "https://github.com/cyborgmarina/litellm/tree/main/litellm/proxy/auth/"
    click RL "https://github.com/cyborgmarina/litellm/blob/main/litellm/proxy/hooks/dynamic_rate_limiter.py"
    click Cache "https://github.com/cyborgmarina/litellm/tree/main/litellm/caching/"
    click Router "https://github.com/cyborgmarina/litellm/blob/main/litellm/router.py"
    click MP "https://github.com/cyborgmarina/litellm/tree/main/litellm/llms/"
    click CH "https://github.com/cyborgmarina/litellm/tree/main/litellm/llms/OpenAI/chat/"
    click ES "https://github.com/cyborgmarina/litellm/tree/main/litellm/llms/bedrock/embed/"
    click SS "https://github.com/cyborgmarina/litellm/tree/main/litellm/llms/OpenAI/realtime/"
    click UM "https://github.com/cyborgmarina/litellm/blob/main/litellm/proxy/management_endpoints/internal_user_endpoints.py"
    click AK "https://github.com/cyborgmarina/litellm/blob/main/litellm/proxy/management_endpoints/key_management_endpoints.py"
    click BC "https://github.com/cyborgmarina/litellm/tree/main/litellm/proxy/spend_tracking/"
    click ML "https://github.com/cyborgmarina/litellm/tree/main/litellm/integrations/"
    click SEC "https://github.com/cyborgmarina/litellm/tree/main/litellm/proxy/auth/"
    click LOG "https://github.com/cyborgmarina/litellm/blob/main/litellm/_logging.py"
    click ERR "https://github.com/cyborgmarina/litellm/blob/main/litellm/exceptions.py"
    click COST "https://github.com/cyborgmarina/litellm/blob/main/litellm/proxy/spend_tracking/spend_management_endpoints.py"
```
