# vizuaraai/capstone-serve-glm-5-2 architecture

> Architecture diagram of the GitHub repository vizuaraai/capstone-serve-glm-5-2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vizuaraai/capstone-serve-glm-5-2
- Repository: https://github.com/vizuaraai/capstone-serve-glm-5-2

## Overview

This repository is a GLM-5.2 inference-service capstone: clients enter through an OpenAI-compatible gateway, which authenticates, rate-limits, admits or sheds, then proxies streaming or non-streaming requests with bounded retry and fallback. A vLLM engine tier serves generations; deployment manifests describe KServe/raw-vLLM and KEDA/HPA scaling. The harness measures load and SLO outcomes, then analysis computes the concurrency ceiling and goodput curve. Observability is a separate operational branch. Gateway and harness calls are supported by excerpts; several deployment and engine relationships are README-documented, while unsampled wiring is left out or marked unverified.

Main entry points: [`gateway/app.py`](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/app.py) (Gateway API), [`gateway/fallback.py`](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/fallback.py) (Retry and fallback), [`engine/serve_vllm.sh`](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/engine/serve_vllm.sh) (vLLM engine), [`harness/metrics.py`](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/harness/metrics.py) (Goodput and ceiling)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_gateway["Request gateway"]
  node_gateway_app["Gateway API<br/>[app.py]"]
  node_auth["Key authentication<br/>[auth.py]"]
  node_config["Gateway settings<br/>[config.py]"]
  node_rate_limit["Identity rate limits<br/>[ratelimit.py]"]
  node_admission["Admission shedding<br/>[shed.py]"]
  node_proxy["Retry and fallback<br/>[fallback.py]"]
  node_retry_policy["Retry policy<br/>[retries.py]"]
  node_gateway_metrics["Gateway metrics<br/>[metrics.py]"]
end

subgraph group_serving["Inference serving"]
  node_engine["vLLM engine<br/>[serve_vllm.sh]"]
  node_mock_engine["Mock engine<br/>[mock_engine.py]"]
end

subgraph group_ops["Deployment and telemetry"]
  node_deploy_manifests["Kubernetes serving"]
  node_autoscaling["Queue and KV scaling"]
  node_prometheus["Prometheus scrape<br/>[prometheus.yml]"]
  node_dashboards["Dashboards and alerts"]
end

subgraph group_testing["Load and SLO analysis"]
  node_load_client["Load client<br/>[load_client.py]"]
  node_harness_runner["Load sweep runner<br/>[run_harness.py]"]
  node_harness_metrics["Goodput and ceiling<br/>[metrics.py]"]
  node_slo["SLO targets<br/>[slo.py]"]
  node_curve["Goodput curve<br/>[goodput_curve.py]"]
  node_ceiling["Ceiling analysis<br/>[ceiling.py]"]
  node_cost_analysis["Cost per goodput"]
  node_loadtest_slo["Load-test profile<br/>[slo.yaml]"]
end

node_client(("API client"))

node_client -->|"submits chat"| node_gateway_app
node_gateway_app -->|"authenticates"| node_auth
node_gateway_app -->|"rate limits"| node_rate_limit
node_gateway_app -->|"admits or sheds"| node_admission
node_gateway_app -->|"forwards requests"| node_proxy
node_proxy -->|"checks retries"| node_retry_policy
node_gateway_app -->|"loads settings"| node_config
node_proxy -->|"streams inference"| node_engine
node_proxy -->|"records failover"| node_gateway_metrics
node_gateway_app -->|"records outcomes"| node_gateway_metrics
node_engine -.->|"deployed by"| node_deploy_manifests
node_autoscaling -->|"scales on pressure"| node_engine
node_prometheus -.->|"scrapes metrics"| node_gateway_metrics
node_prometheus -.->|"scrapes metrics"| node_engine
node_prometheus -.->|"feeds telemetry"| node_dashboards
node_load_client -->|"sends measured load"| node_gateway_app
node_harness_runner -->|"dispatches requests"| node_load_client
node_harness_runner -->|"summarizes records"| node_harness_metrics
node_harness_runner -->|"loads targets"| node_slo
node_slo -.->|"parses profile"| node_loadtest_slo
node_curve -->|"finds ceiling"| node_harness_metrics
node_curve -->|"loads targets"| node_slo
node_ceiling -->|"uses summaries"| node_harness_metrics
node_cost_analysis -.->|"estimates economics"| node_harness_metrics
node_mock_engine -.->|"serves test traffic"| node_gateway_app

click node_gateway_app "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/app.py"
click node_auth "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/auth.py"
click node_config "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/config.py"
click node_rate_limit "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/ratelimit.py"
click node_admission "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/shed.py"
click node_proxy "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/fallback.py"
click node_retry_policy "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/retries.py"
click node_gateway_metrics "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/metrics.py"
click node_engine "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/engine/serve_vllm.sh"
click node_mock_engine "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/harness/mock_engine.py"
click node_deploy_manifests "https://github.com/vizuaraai/capstone-serve-glm-5-2/tree/main/deploy"
click node_autoscaling "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/deploy/keda-scaledobject.yaml"
click node_prometheus "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/observability/prometheus.yml"
click node_dashboards "https://github.com/vizuaraai/capstone-serve-glm-5-2/tree/main/observability"
click node_load_client "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/harness/load_client.py"
click node_harness_runner "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/harness/run_harness.py"
click node_harness_metrics "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/harness/metrics.py"
click node_slo "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/harness/slo.py"
click node_curve "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/analysis/goodput_curve.py"
click node_ceiling "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/analysis/ceiling.py"
click node_cost_analysis "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/analysis/cost_per_goodput.py"
click node_loadtest_slo "https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/loadtest/slo.yaml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_gateway_app,node_auth,node_config,node_rate_limit,node_admission,node_proxy,node_retry_policy,node_gateway_metrics,node_client toneBlue
class node_engine,node_mock_engine toneAmber
class node_deploy_manifests,node_autoscaling,node_prometheus,node_dashboards toneMint
class node_load_client,node_harness_runner,node_harness_metrics,node_slo,node_curve,node_ceiling,node_cost_analysis,node_loadtest_slo toneRose
```

## Components

### Request gateway

- [Gateway API](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/app.py): `gateway/app.py`
- [Key authentication](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/auth.py): `gateway/auth.py`
- [Gateway settings](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/config.py): `gateway/config.py`
- [Identity rate limits](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/ratelimit.py): `gateway/ratelimit.py`
- [Admission shedding](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/shed.py): `gateway/shed.py`
- [Retry and fallback](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/fallback.py): `gateway/fallback.py`
- [Retry policy](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/retries.py): `gateway/retries.py`
- [Gateway metrics](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/gateway/metrics.py): `gateway/metrics.py`

### Inference serving

- [vLLM engine](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/engine/serve_vllm.sh): `engine/serve_vllm.sh`
- [Mock engine](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/harness/mock_engine.py): `harness/mock_engine.py`

### Deployment and telemetry

- [Kubernetes serving](https://github.com/vizuaraai/capstone-serve-glm-5-2/tree/main/deploy): `deploy`
- [Queue and KV scaling](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/deploy/keda-scaledobject.yaml): `deploy/keda-scaledobject.yaml`
- [Prometheus scrape](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/observability/prometheus.yml): `observability/prometheus.yml`
- [Dashboards and alerts](https://github.com/vizuaraai/capstone-serve-glm-5-2/tree/main/observability): `observability`

### Load and SLO analysis

- [Load client](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/harness/load_client.py): `harness/load_client.py`
- [Load sweep runner](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/harness/run_harness.py): `harness/run_harness.py`
- [Goodput and ceiling](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/harness/metrics.py): `harness/metrics.py`
- [SLO targets](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/harness/slo.py): `harness/slo.py`
- [Goodput curve](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/analysis/goodput_curve.py): `analysis/goodput_curve.py`
- [Ceiling analysis](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/analysis/ceiling.py): `analysis/ceiling.py`
- [Cost per goodput](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/analysis/cost_per_goodput.py): `analysis/cost_per_goodput.py`
- [Load-test profile](https://github.com/vizuaraai/capstone-serve-glm-5-2/blob/main/loadtest/slo.yaml): `loadtest/slo.yaml`

### Other components

- **API client**

## Connections

- API client → Gateway API: submits chat (evidence: `README.md`)
- Gateway API → Key authentication: authenticates (evidence: `gateway/app.py`)
- Gateway API → Identity rate limits: rate limits (evidence: `README.md`)
- Gateway API → Admission shedding: admits or sheds (evidence: `gateway/app.py`)
- Gateway API → Retry and fallback: forwards requests (evidence: `gateway/app.py`)
- Retry and fallback → Retry policy: checks retries (evidence: `gateway/fallback.py`)
- Gateway API → Gateway settings: loads settings (evidence: `gateway/app.py`)
- Retry and fallback → vLLM engine: streams inference (evidence: `README.md`)
- Retry and fallback → Gateway metrics: records failover (evidence: `gateway/fallback.py`)
- Gateway API → Gateway metrics: records outcomes (evidence: `gateway/metrics.py`)
- vLLM engine → Kubernetes serving: deployed by (evidence: `README.md`)
- Queue and KV scaling → vLLM engine: scales on pressure (evidence: `README.md`)
- Prometheus scrape → Gateway metrics: scrapes metrics (evidence: `README.md`)
- Prometheus scrape → vLLM engine: scrapes metrics (evidence: `README.md`)
- Prometheus scrape → Dashboards and alerts: feeds telemetry (evidence: `README.md`)
- Load client → Gateway API: sends measured load (evidence: `README.md`)
- Load sweep runner → Load client: dispatches requests (evidence: `harness/run_harness.py`)
- Load sweep runner → Goodput and ceiling: summarizes records (evidence: `harness/run_harness.py`)
- Load sweep runner → SLO targets: loads targets (evidence: `harness/run_harness.py`)
- SLO targets → Load-test profile: parses profile (evidence: `harness/slo.py`)
- Goodput curve → Goodput and ceiling: finds ceiling (evidence: `analysis/goodput_curve.py`)
- Goodput curve → SLO targets: loads targets (evidence: `analysis/goodput_curve.py`)
- Ceiling analysis → Goodput and ceiling: uses summaries (evidence: `analysis/ceiling.py`)
- Cost per goodput → Goodput and ceiling: estimates economics
- Mock engine → Gateway API: serves test traffic (evidence: `README.md`)
