# abdoelwaly/private-llm-platform architecture

> Architecture diagram of the GitHub repository abdoelwaly/private-llm-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abdoelwaly/private-llm-platform
- Repository: https://github.com/abdoelwaly/private-llm-platform

## Overview

This repository provisions a private Azure LLM platform with two serving paths: managed Azure OpenAI behind a per-team API Management gateway, and self-hosted vLLM on an autoscaling AKS GPU pool. Applications use the relevant endpoint and team/API key; the included smoke test can exercise either path and report latency and token usage. The graph keeps provisioning, managed serving, and cluster serving distinct. No source excerpts were provided, so relationships below rely on README statements; implementation details beyond those statements are not inferred.

Main entry points: [`terraform/modules/llm-gateway/main.tf`](https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/llm-gateway/main.tf) (APIM Gateway), [`charts/vllm/templates/deployment.yaml`](https://github.com/abdoelwaly/private-llm-platform/blob/main/charts/vllm/templates/deployment.yaml) (vLLM Deployment), [`terraform/envs/dev/main.tf`](https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/envs/dev/main.tf) (Dev Environment), [`terraform/modules/azure-openai/main.tf`](https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/azure-openai/main.tf) (Azure OpenAI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_provisioning["Platform Provisioning"]
  node_dev_env["Dev Environment<br/>[main.tf]"]
  node_network_dns["Network and DNS<br/>[main.tf]"]
end

subgraph group_managed["Managed Model Path"]
  node_azure_openai["Azure OpenAI<br/>[main.tf]"]
  node_gateway["APIM Gateway<br/>[main.tf]"]
  node_gateway_policy["API Policy<br/>[policy.xml.tftpl]"]
  node_gateway_network["Gateway Network<br/>[nsg.tf]"]
  node_team_limits["Team Quotas<br/>[main.tf]"]
  node_usage_metrics["Usage Metrics<br/>[main.tf]"]
end

subgraph group_selfhosted["Self-Hosted Path"]
  node_gpu_pool["AKS GPU Pool<br/>[main.tf]"]
  node_gpu_addons["GPU and KEDA Addons"]
  node_vllm_deployment["vLLM Deployment<br/>[deployment.yaml]"]
  node_vllm_service["vLLM Service<br/>[service.yaml]"]
  node_request_scaling["Queue Autoscaling<br/>[scaledobject.yaml]"]
  node_model_cache[("Model Cache<br/>[pvc.yaml]")]
  node_vllm_monitoring["Metrics and Alerts"]
end

subgraph group_client["Client Testing"]
  node_smoke_test["Smoke Test<br/>[smoke_test.py]"]
end

node_apps(("Company Apps"))
node_application_insights["Application Insights"]
node_azure_service["Azure Services"]
node_aks_cluster["Existing AKS Cluster"]

node_dev_env -->|"wires module"| node_azure_openai
node_dev_env -->|"wires module"| node_gateway
node_dev_env -->|"wires module"| node_gpu_pool
node_dev_env -->|"configures"| node_network_dns
node_azure_openai -->|"provisions"| node_azure_service
node_gateway -->|"calls with identity"| node_azure_openai
node_apps -->|"uses endpoint and key"| node_gateway
node_gateway -->|"applies policy"| node_gateway_policy
node_gateway -->|"enforces team limits"| node_team_limits
node_gateway -->|"emits token usage"| node_usage_metrics
node_usage_metrics -->|"sends metrics"| node_application_insights
node_network_dns -->|"provides private access"| node_azure_openai
node_network_dns -->|"configures network"| node_gateway_network
node_gpu_pool -->|"adds GPU nodes"| node_aks_cluster
node_gpu_addons -->|"installs drivers and KEDA"| node_aks_cluster
node_vllm_deployment -->|"schedules on GPUs"| node_aks_cluster
node_vllm_deployment -->|"uses cache"| node_model_cache
node_vllm_service -->|"exposes API"| node_vllm_deployment
node_request_scaling -->|"scales on queued requests"| node_vllm_deployment
node_vllm_monitoring -->|"monitors time to token"| node_vllm_deployment
node_apps -->|"uses OpenAI-compatible API"| node_vllm_service
node_smoke_test -->|"sends test request"| node_gateway
node_smoke_test -->|"sends test request"| node_vllm_service

click node_dev_env "https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/envs/dev/main.tf"
click node_network_dns "https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/envs/dev/main.tf"
click node_azure_openai "https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/azure-openai/main.tf"
click node_gateway "https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/llm-gateway/main.tf"
click node_gateway_policy "https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/llm-gateway/policy.xml.tftpl"
click node_gateway_network "https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/llm-gateway/nsg.tf"
click node_team_limits "https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/llm-gateway/main.tf"
click node_usage_metrics "https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/llm-gateway/main.tf"
click node_gpu_pool "https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/aks-gpu-pool/main.tf"
click node_gpu_addons "https://github.com/abdoelwaly/private-llm-platform/blob/main/scripts/install-gpu-addons.sh"
click node_vllm_deployment "https://github.com/abdoelwaly/private-llm-platform/blob/main/charts/vllm/templates/deployment.yaml"
click node_vllm_service "https://github.com/abdoelwaly/private-llm-platform/blob/main/charts/vllm/templates/service.yaml"
click node_request_scaling "https://github.com/abdoelwaly/private-llm-platform/blob/main/charts/vllm/templates/scaledobject.yaml"
click node_model_cache "https://github.com/abdoelwaly/private-llm-platform/blob/main/charts/vllm/templates/pvc.yaml"
click node_vllm_monitoring "https://github.com/abdoelwaly/private-llm-platform/blob/main/charts/vllm/templates/servicemonitor.yaml"
click node_smoke_test "https://github.com/abdoelwaly/private-llm-platform/blob/main/scripts/smoke_test.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dev_env,node_network_dns toneBlue
class node_azure_openai,node_gateway,node_gateway_policy,node_gateway_network,node_team_limits,node_usage_metrics toneAmber
class node_gpu_pool,node_gpu_addons,node_vllm_deployment,node_vllm_service,node_request_scaling,node_model_cache,node_vllm_monitoring toneMint
class node_smoke_test toneRose
class node_apps,node_application_insights,node_azure_service,node_aks_cluster toneIndigo
```

## Components

### Platform Provisioning

- [Dev Environment](https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/envs/dev/main.tf): `terraform/envs/dev/main.tf`
- [Network and DNS](https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/envs/dev/main.tf): `terraform/envs/dev/main.tf`

### Managed Model Path

- [Azure OpenAI](https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/azure-openai/main.tf): `terraform/modules/azure-openai/main.tf`
- [APIM Gateway](https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/llm-gateway/main.tf): `terraform/modules/llm-gateway/main.tf`
- [API Policy](https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/llm-gateway/policy.xml.tftpl): `terraform/modules/llm-gateway/policy.xml.tftpl`
- [Gateway Network](https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/llm-gateway/nsg.tf): `terraform/modules/llm-gateway/nsg.tf`
- [Team Quotas](https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/llm-gateway/main.tf): `terraform/modules/llm-gateway/main.tf`
- [Usage Metrics](https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/llm-gateway/main.tf): `terraform/modules/llm-gateway/main.tf`

### Self-Hosted Path

- [AKS GPU Pool](https://github.com/abdoelwaly/private-llm-platform/blob/main/terraform/modules/aks-gpu-pool/main.tf): `terraform/modules/aks-gpu-pool/main.tf`
- [GPU and KEDA Addons](https://github.com/abdoelwaly/private-llm-platform/blob/main/scripts/install-gpu-addons.sh): `scripts/install-gpu-addons.sh`
- [vLLM Deployment](https://github.com/abdoelwaly/private-llm-platform/blob/main/charts/vllm/templates/deployment.yaml): `charts/vllm/templates/deployment.yaml`
- [vLLM Service](https://github.com/abdoelwaly/private-llm-platform/blob/main/charts/vllm/templates/service.yaml): `charts/vllm/templates/service.yaml`
- [Queue Autoscaling](https://github.com/abdoelwaly/private-llm-platform/blob/main/charts/vllm/templates/scaledobject.yaml): `charts/vllm/templates/scaledobject.yaml`
- [Model Cache](https://github.com/abdoelwaly/private-llm-platform/blob/main/charts/vllm/templates/pvc.yaml): `charts/vllm/templates/pvc.yaml`
- [Metrics and Alerts](https://github.com/abdoelwaly/private-llm-platform/blob/main/charts/vllm/templates/servicemonitor.yaml): `charts/vllm/templates/servicemonitor.yaml`

### Client Testing

- [Smoke Test](https://github.com/abdoelwaly/private-llm-platform/blob/main/scripts/smoke_test.py): `scripts/smoke_test.py`

### Other components

- **Company Apps**
- **Application Insights**
- **Azure Services**
- **Existing AKS Cluster**

## Connections

- Dev Environment → Azure OpenAI: wires module (evidence: `README.md`)
- Dev Environment → APIM Gateway: wires module (evidence: `README.md`)
- Dev Environment → AKS GPU Pool: wires module (evidence: `README.md`)
- Dev Environment → Network and DNS: configures (evidence: `README.md`)
- Azure OpenAI → Azure Services: provisions (evidence: `README.md`)
- APIM Gateway → Azure OpenAI: calls with identity (evidence: `README.md`)
- Company Apps → APIM Gateway: uses endpoint and key (evidence: `README.md`)
- APIM Gateway → API Policy: applies policy (evidence: `README.md`)
- APIM Gateway → Team Quotas: enforces team limits (evidence: `README.md`)
- APIM Gateway → Usage Metrics: emits token usage (evidence: `README.md`)
- Usage Metrics → Application Insights: sends metrics (evidence: `README.md`)
- Network and DNS → Azure OpenAI: provides private access (evidence: `README.md`)
- Network and DNS → Gateway Network: configures network (evidence: `README.md`)
- AKS GPU Pool → Existing AKS Cluster: adds GPU nodes (evidence: `README.md`)
- GPU and KEDA Addons → Existing AKS Cluster: installs drivers and KEDA (evidence: `README.md`)
- vLLM Deployment → Existing AKS Cluster: schedules on GPUs (evidence: `README.md`)
- vLLM Deployment → Model Cache: uses cache (evidence: `README.md`)
- vLLM Service → vLLM Deployment: exposes API (evidence: `README.md`)
- Queue Autoscaling → vLLM Deployment: scales on queued requests (evidence: `README.md`)
- Metrics and Alerts → vLLM Deployment: monitors time to token (evidence: `README.md`)
- Company Apps → vLLM Service: uses OpenAI-compatible API (evidence: `README.md`)
- Smoke Test → APIM Gateway: sends test request (evidence: `README.md`)
- Smoke Test → vLLM Service: sends test request (evidence: `README.md`)
