# envoyproxy/ai-gateway architecture

> Architecture diagram of the GitHub repository envoyproxy/ai-gateway, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/envoyproxy/ai-gateway
- Repository: https://github.com/envoyproxy/ai-gateway

## Overview

##### Main Components

a. Core Components:
- Envoy Gateway Integration
- Controller Component
- External Processor (ExtProc)
- API Layer (v1alpha1)
- Backend Services Integration
- Security Layer

b. Key Architectural Elements:
- Custom Resource Definitions (CRDs)
- LLM Backend Management
- LLM Route Management
- Request Translation Layer
- Authentication/Authorization System

##### Client Layer
- Incoming client requests
- Multiple client types (applications, services)

##### Gateway Layer
- Envoy Gateway component
- External Processor integration
- Route management

##### Processing Layer
- Controller component
- Request translation
- Backend authentication
- Request routing

##### Backend Integration Layer
- LLM service providers (AWS Bedrock, OpenAI)
- Security policies
- Backend configurations

Use these specific visual elements:

- Components: Use rectangles with rounded corners
- API Endpoints: Hexagons
- External Services: Clouds

- Blue: Core Envoy components
- Green: Processing components
- Yellow: Security components
- Purple: External services
- Gray: Infrastructure components

##### Key Relationships to Highlight

a. Control Plane:
- How the controller manages LLMBackends and LLMRoutes
- CRD management flow
- Configuration update paths

b. Data Plane:
- Request flow from clients through Envoy
- Translation process for different backends
- Security policy enforcement points

##### Important Implementation Details

Specific components based on the file structure:
- API Schema Translations (from internal/apischema)
- Backend Authentication Methods (from internal/extproc/backendauth)
- Request Translation Layer (from internal/extproc/translator)
- Route Management (from internal/extproc/router)

##### Specific Implementation Notes

- The separation between controller and extproc components
- The translation layer for different LLM providers
- The security policy implementation
- The custom resource definitions (from manifests/charts)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles
    classDef client fill:#f9f,stroke:#333,stroke-width:2px
    classDef gateway fill:#9cf,stroke:#333,stroke-width:2px
    classDef processor fill:#9f9,stroke:#333,stroke-width:2px
    classDef security fill:#ff9,stroke:#333,stroke-width:2px
    classDef backend fill:#c9f,stroke:#333,stroke-width:2px
    classDef api fill:#ccc,stroke:#333,stroke-width:2px

    %% Client Layer
    subgraph ClientLayer["Client Layer"]
        C1["Client Applications"]:::client
        C2["External Services"]:::client
    end

    %% Gateway Layer
    subgraph GatewayLayer["Gateway Layer"]
        EG["Envoy Gateway"]:::gateway
        EP["External Processor"]:::gateway
        click EP "https://github.com/envoyproxy/ai-gateway/tree/main/cmd/extproc/"
    end

    %% Processing Layer
    subgraph ProcessingLayer["Processing Layer"]
        CTRL["Controller"]:::processor
        click CTRL "https://github.com/envoyproxy/ai-gateway/tree/main/cmd/controller/"
        
        subgraph RequestProcessing["Request Processing"]
            RT["Request Translator"]:::processor
            click RT "https://github.com/envoyproxy/ai-gateway/tree/main/internal/extproc/translator/"
            RM["Route Manager"]:::processor
            click RM "https://github.com/envoyproxy/ai-gateway/tree/main/internal/extproc/router/"
        end
        
        subgraph SecurityLayer["Security Layer"]
            AUTH["Authentication System"]:::security
            click AUTH "https://github.com/envoyproxy/ai-gateway/tree/main/internal/extproc/backendauth/"
            SP["Security Policies"]:::security
            click SP "https://github.com/envoyproxy/ai-gateway/blob/main/manifests/charts/ai-gateway-helm/crds/aigateway.envoyproxy.io_backendsecuritypolicies.yaml"
        end
    end

    %% API Layer
    subgraph APILayer["API Layer"]
        API["API v1alpha1"]:::api
        click API "https://github.com/envoyproxy/ai-gateway/tree/main/api/v1alpha1/"
        
        subgraph CRDs["Custom Resource Definitions"]
            LB["LLM Backends"]:::api
            click LB "https://github.com/envoyproxy/ai-gateway/blob/main/manifests/charts/ai-gateway-helm/crds/aigateway.envoyproxy.io_llmbackends.yaml"
            LR["LLM Routes"]:::api
            click LR "https://github.com/envoyproxy/ai-gateway/blob/main/manifests/charts/ai-gateway-helm/crds/aigateway.envoyproxy.io_llmroutes.yaml"
        end
    end

    %% Backend Layer
    subgraph BackendLayer["Backend Integration Layer"]
        AWS["AWS Bedrock"]:::backend
        click AWS "https://github.com/envoyproxy/ai-gateway/tree/main/internal/apischema/awsbedrock/"
        OPENAI["OpenAI"]:::backend
        click OPENAI "https://github.com/envoyproxy/ai-gateway/tree/main/internal/apischema/openai/"
    end

    %% Relationships
    C1 --> EG
    C2 --> EG
    EG --> EP
    EP --> RT
    RT --> RM
    RM --> AUTH
    AUTH --> SP
    CTRL --> API
    API --> LB & LR
    AUTH --> AWS & OPENAI
    
    %% Legend
    subgraph Legend
        L1["Client Components"]:::client
        L2["Gateway Components"]:::gateway
        L3["Processing Components"]:::processor
        L4["Security Components"]:::security
        L5["Backend Components"]:::backend
        L6["API Components"]:::api
    end
```
