How abdoelwaly/private-llm-platform works
This repository provisions a private Azure LLM platform with two serving paths: managed Azure OpenAI behind a per-team API Management gateway, and self-hosted vLLM on an autoscaling AKS GPU pool. Applications use the relevant endpoint and team/API key; the included smoke test can exercise either path and report latency and token usage. The graph keeps provisioning, managed serving, and cluster serving distinct. No source excerpts were provided, so relationships below rely on README statements; implementation details beyond those statements are not inferred.
Main entry points: llm-gateway/main.tf APIM Gateway · deployment.yaml vLLM Deployment · dev/main.tf Dev Environment · azure-openai/main.tf Azure OpenAI
Components 20
Platform Provisioning
- Dev Environment
terraform/envs/dev/main.tf - Network and DNS
terraform/envs/dev/main.tf
Managed Model Path
- Azure OpenAI
terraform/modules/azure-openai/main.tf - APIM Gateway
terraform/modules/llm-gateway/main.tf - API Policy
terraform/modules/llm-gateway/policy.xml.tftpl - Gateway Network
terraform/modules/llm-gateway/nsg.tf - Team Quotas
terraform/modules/llm-gateway/main.tf - Usage Metrics
terraform/modules/llm-gateway/main.tf
Self-Hosted Path
- AKS GPU Pool
terraform/modules/aks-gpu-pool/main.tf - GPU and KEDA Addons
scripts/install-gpu-addons.sh - vLLM Deployment
charts/vllm/templates/deployment.yaml - vLLM Service
charts/vllm/templates/service.yaml - Queue Autoscaling
charts/vllm/templates/scaledobject.yaml - Model Cache
charts/vllm/templates/pvc.yaml - Metrics and Alerts
charts/vllm/templates/servicemonitor.yaml
Client Testing
- Smoke Test
scripts/smoke_test.py
Other components
- Company Apps
- Application Insights
- Azure Services
- Existing AKS Cluster
Connections 23
- Dev Environment → Azure OpenAI: wires module
README.md - Dev Environment → APIM Gateway: wires module
README.md - Dev Environment → AKS GPU Pool: wires module
README.md - Dev Environment → Network and DNS: configures
README.md - Azure OpenAI → Azure Services: provisions
README.md - APIM Gateway → Azure OpenAI: calls with identity
README.md - Company Apps → APIM Gateway: uses endpoint and key
README.md - APIM Gateway → API Policy: applies policy
README.md - APIM Gateway → Team Quotas: enforces team limits
README.md - APIM Gateway → Usage Metrics: emits token usage
README.md - Usage Metrics → Application Insights: sends metrics
README.md - Network and DNS → Azure OpenAI: provides private access
README.md - Network and DNS → Gateway Network: configures network
README.md - AKS GPU Pool → Existing AKS Cluster: adds GPU nodes
README.md - GPU and KEDA Addons → Existing AKS Cluster: installs drivers and KEDA
README.md - vLLM Deployment → Existing AKS Cluster: schedules on GPUs
README.md - vLLM Deployment → Model Cache: uses cache
README.md - vLLM Service → vLLM Deployment: exposes API
README.md - Queue Autoscaling → vLLM Deployment: scales on queued requests
README.md - Metrics and Alerts → vLLM Deployment: monitors time to token
README.md - Company Apps → vLLM Service: uses OpenAI-compatible API
README.md - Smoke Test → APIM Gateway: sends test request
README.md - Smoke Test → vLLM Service: sends test request
README.md