How azure-samples/ai-gateway works
This project is an experimental repository centered on implementing and showcasing the “GenAI Gateway” pattern using Azure API Management. It’s not a traditional full‑stack application but rather a collection of interactive labs (primarily using Jupyter notebooks, Bicep templates, and API Management policies) that demonstrate how to integrate AI services (like Azure OpenAI, Azure AI Foundry, and self‑hosted models) with API management techniques. The labs cover a range of topics—from load balancing and rate limiting to content filtering, function calling, token metrics, and beyond—providing hands‑on explorations of advanced gateway scenarios for AI workloads.
Main Components
• API Gateway (Azure API Management): Represented centrally as the orchestrator for handling requests, enforcing policies (security, load balancing, rate limiting, logging, etc.), and routing to various backend endpoints.
• Lab Experiments: A series of labs (organized under the “labs” directory) that are structured as interactive notebooks demonstrating specific aspects of the AI Gateway. Each lab (e.g., FinOps Framework, Model Context Protocol, OpenAI Agents, Function Calling, Content Filtering, etc.) can be represented as discrete modules that integrate with the API gateway.
• Infrastructure Modules: The “modules” folder contains Bicep templates and configurations for provisioning the underlying resources (APIM, Cognitive Services components, monitoring, etc.). These reflect the “infrastructure-as-code” aspect of the architecture.
• Supporting Tools and Utilities: Tools (under “tools”), shared code (under “shared”), and developer configurations (e.g., .devcontainer, .vscode) indicate additional supporting components for testing, mocking, and deployment automation.
Establish Component Relationships and Interactions
• Clients (or developers) issue requests to the API Gateway.
• The API Gateway, equipped with various policies (load balancing, circuit breaking, rate limiting, token metrics, content filtering, etc.), routes these requests to one or more backend services.
• Backend services can include Azure OpenAI endpoints, Azure Functions (for function calling scenarios), mock servers (for testing), Cosmos DB (for message storing), and other Azure resources.
• Infrastructure provisioning is handled via Bicep templates (and Terraform in one lab) which ties into the setup of the API Gateway, backend services, and monitoring tools (like Application Insights).
• Logging, monitoring, and tracing tools (e.g., built‑in logging, token metrics emitting, and tracing notebooks) provide observability into the system’s operation.
Architectural Patterns and Design Principles
• Gateway Pattern: Azure API Management is central to the architecture, enforcing policies, aggregating service calls, and providing control over backend AI services.
• Policy‑Based Enforcement: Multiple labs demonstrate the application of policies (e.g., token limits, semantic caching, content filtering, load balancing) that reflect a decoupled and policy‑enforced design.
• Infrastructure-as-Code (IaC): The presence of Bicep and Terraform scripts shows that resource provisioning is automated and declarative.
• Modular and Experimental Lab Structure: The project is organized to allow independent experiments (labs) which can be integrated with or demonstrate improvements to the gateway pattern.
Technologies and Frameworks
• Azure API Management – the core “gateway” component.
• Azure OpenAI and Azure Cognitive Services – backend AI services.
• Bicep (and Terraform) for defining and deploying the cloud resources.
• Jupyter Notebooks using Python – for experimenting and documenting lab scenarios.
• Additional tools for mock servers and tracing for testing and debugging.