# azure-samples/ai-gateway architecture

> Architecture diagram of the GitHub repository azure-samples/ai-gateway, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/azure-samples/ai-gateway
- Repository: https://github.com/azure-samples/ai-gateway

## Overview

This project is an experimental repository centered on implementing and showcasing the “GenAI Gateway” pattern using Azure API Management. It’s not a traditional full‑stack application but rather a collection of interactive labs (primarily using Jupyter notebooks, Bicep templates, and API Management policies) that demonstrate how to integrate AI services (like Azure OpenAI, Azure AI Foundry, and self‑hosted models) with API management techniques. The labs cover a range of topics—from load balancing and rate limiting to content filtering, function calling, token metrics, and beyond—providing hands‑on explorations of advanced gateway scenarios for AI workloads.

##### Main Components
- API Gateway (Azure API Management): Represented centrally as the orchestrator for handling requests, enforcing policies (security, load balancing, rate limiting, logging, etc.), and routing to various backend endpoints.
- Lab Experiments: A series of labs (organized under the “labs” directory) that are structured as interactive notebooks demonstrating specific aspects of the AI Gateway. Each lab (e.g., FinOps Framework, Model Context Protocol, OpenAI Agents, Function Calling, Content Filtering, etc.) can be represented as discrete modules that integrate with the API gateway.
- Infrastructure Modules: The “modules” folder contains Bicep templates and configurations for provisioning the underlying resources (APIM, Cognitive Services components, monitoring, etc.). These reflect the “infrastructure-as-code” aspect of the architecture.
- Supporting Tools and Utilities: Tools (under “tools”), shared code (under “shared”), and developer configurations (e.g., .devcontainer, .vscode) indicate additional supporting components for testing, mocking, and deployment automation.

##### Establish Component Relationships and Interactions
- Clients (or developers) issue requests to the API Gateway.
- The API Gateway, equipped with various policies (load balancing, circuit breaking, rate limiting, token metrics, content filtering, etc.), routes these requests to one or more backend services.
- Backend services can include Azure OpenAI endpoints, Azure Functions (for function calling scenarios), mock servers (for testing), Cosmos DB (for message storing), and other Azure resources.
- Infrastructure provisioning is handled via Bicep templates (and Terraform in one lab) which ties into the setup of the API Gateway, backend services, and monitoring tools (like Application Insights).
- Logging, monitoring, and tracing tools (e.g., built‑in logging, token metrics emitting, and tracing notebooks) provide observability into the system’s operation.

##### Architectural Patterns and Design Principles
- Gateway Pattern: Azure API Management is central to the architecture, enforcing policies, aggregating service calls, and providing control over backend AI services.
- Policy‑Based Enforcement: Multiple labs demonstrate the application of policies (e.g., token limits, semantic caching, content filtering, load balancing) that reflect a decoupled and policy‑enforced design.
- Infrastructure-as-Code (IaC): The presence of Bicep and Terraform scripts shows that resource provisioning is automated and declarative.
- Modular and Experimental Lab Structure: The project is organized to allow independent experiments (labs) which can be integrated with or demonstrate improvements to the gateway pattern.

##### Technologies and Frameworks
- Azure API Management – the core “gateway” component.
- Azure OpenAI and Azure Cognitive Services – backend AI services.
- Bicep (and Terraform) for defining and deploying the cloud resources.
- Jupyter Notebooks using Python – for experimenting and documenting lab scenarios.
- Additional tools for mock servers and tracing for testing and debugging.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    Client("Client/Developer"):::client

    %% API Management Layer
    subgraph "API Management Layer"
        APIM("Azure API Management"):::apim
        OpenAIConfig("OpenAI Policy Config"):::apim
    end

    %% Experimental Labs
    subgraph "Experimental Labs"
        Labs("Lab Experiments"):::lab
    end

    %% AI & Backend Services
    subgraph "AI & Backend Services"
        AzureOpenAI("Azure OpenAI"):::backend
        AIFoundry("Azure AI Foundry"):::backend
        SelfHosted("Self-hosted SLM"):::backend
        Functions("Azure Functions"):::backend
        MockServers("Mock Servers"):::backend
        CosmosDB("Cosmos DB (Logging)"):::backend
    end

    %% Observability
    subgraph "Observability"
        AppInsights("Application Insights"):::observ
        TokenMetrics("Token Metrics"):::observ
        TracingTools("Tracing Tools"):::observ
    end

    %% Infrastructure & Tools
    subgraph "Infrastructure & Tools"
        Infra("Infrastructure Modules (IaC)"):::infrac
        Tools("Supporting Tools & Utilities"):::tools
        SharedCode("Shared Code"):::tools
    end

    %% Connections
    Client -->|"requests"| APIM
    APIM -->|"routes"| OpenAIConfig
    APIM -->|"routes to"| Labs

    APIM -->|"forwards to"| AzureOpenAI
    APIM -->|"forwards to"| AIFoundry
    APIM -->|"forwards to"| SelfHosted
    APIM -->|"forwards to"| Functions
    APIM -->|"forwards to"| MockServers
    APIM -->|"forwards to"| CosmosDB

    APIM -->|"logs to"| AppInsights
    APIM -->|"emits"| TokenMetrics
    APIM -->|"traces via"| TracingTools

    %% Infrastructure & Tools support
    Infra ---|"provisions"| APIM
    Infra ---|"provisions"| AzureOpenAI
    Infra ---|"provisions"| Functions

    Tools ---|"assists"| Labs
    SharedCode ---|"supports"| Labs

    %% Click Events
    click APIM "https://github.com/azure-samples/ai-gateway/blob/main/modules/apim/v1/apim.bicep"
    click OpenAIConfig "https://github.com/azure-samples/ai-gateway/blob/main/modules/apim/v1/openai-api.bicep"
    click Labs "https://github.com/azure-samples/ai-gateway/tree/main/labs/"
    click Infra "https://github.com/azure-samples/ai-gateway/tree/main/modules/"
    click Tools "https://github.com/azure-samples/ai-gateway/tree/main/tools/"
    click SharedCode "https://github.com/azure-samples/ai-gateway/tree/main/shared/"

    %% Styles
    classDef client fill:#ffcc00,stroke:#000,stroke-width:2px;
    classDef apim fill:#66ccff,stroke:#000,stroke-width:2px;
    classDef lab fill:#ff9999,stroke:#000,stroke-width:2px;
    classDef backend fill:#99ff99,stroke:#000,stroke-width:2px;
    classDef observ fill:#ccccff,stroke:#000,stroke-width:2px;
    classDef infrac fill:#ffccff,stroke:#000,stroke-width:2px;
    classDef tools fill:#ccffcc,stroke:#000,stroke-width:2px;
```
