How vizuaraai/capstone-serve-glm-5-2 works
This repository is a GLM-5.2 inference-service capstone: clients enter through an OpenAI-compatible gateway, which authenticates, rate-limits, admits or sheds, then proxies streaming or non-streaming requests with bounded retry and fallback. A vLLM engine tier serves generations; deployment manifests describe KServe/raw-vLLM and KEDA/HPA scaling. The harness measures load and SLO outcomes, then analysis computes the concurrency ceiling and goodput curve. Observability is a separate operational branch. Gateway and harness calls are supported by excerpts; several deployment and engine relationships are README-documented, while unsampled wiring is left out or marked unverified.
Main entry points: app.py Gateway API · fallback.py Retry and fallback · serve_vllm.sh vLLM engine · metrics.py Goodput and ceiling
Components 23
Request gateway
- Gateway API
gateway/app.py - Key authentication
gateway/auth.py - Gateway settings
gateway/config.py - Identity rate limits
gateway/ratelimit.py - Admission shedding
gateway/shed.py - Retry and fallback
gateway/fallback.py - Retry policy
gateway/retries.py - Gateway metrics
gateway/metrics.py
Inference serving
- vLLM engine
engine/serve_vllm.sh - Mock engine
harness/mock_engine.py
Deployment and telemetry
- Kubernetes serving
deploy - Queue and KV scaling
deploy/keda-scaledobject.yaml - Prometheus scrape
observability/prometheus.yml - Dashboards and alerts
observability
Load and SLO analysis
- Load client
harness/load_client.py - Load sweep runner
harness/run_harness.py - Goodput and ceiling
harness/metrics.py - SLO targets
harness/slo.py - Goodput curve
analysis/goodput_curve.py - Ceiling analysis
analysis/ceiling.py - Cost per goodput
analysis/cost_per_goodput.py - Load-test profile
loadtest/slo.yaml
Other components
- API client
Connections 25
- API client → Gateway API: submits chat
README.md - Gateway API → Key authentication: authenticates
gateway/app.py - Gateway API → Identity rate limits: rate limits
README.md - Gateway API → Admission shedding: admits or sheds
gateway/app.py - Gateway API → Retry and fallback: forwards requests
gateway/app.py - Retry and fallback → Retry policy: checks retries
gateway/fallback.py - Gateway API → Gateway settings: loads settings
gateway/app.py - Retry and fallback → vLLM engine: streams inference
README.md - Retry and fallback → Gateway metrics: records failover
gateway/fallback.py - Gateway API → Gateway metrics: records outcomes
gateway/metrics.py - vLLM engine → Kubernetes serving: deployed by
README.md - Queue and KV scaling → vLLM engine: scales on pressure
README.md - Prometheus scrape → Gateway metrics: scrapes metrics
README.md - Prometheus scrape → vLLM engine: scrapes metrics
README.md - Prometheus scrape → Dashboards and alerts: feeds telemetry
README.md - Load client → Gateway API: sends measured load
README.md - Load sweep runner → Load client: dispatches requests
harness/run_harness.py - Load sweep runner → Goodput and ceiling: summarizes records
harness/run_harness.py - Load sweep runner → SLO targets: loads targets
harness/run_harness.py - SLO targets → Load-test profile: parses profile
harness/slo.py - Goodput curve → Goodput and ceiling: finds ceiling
analysis/goodput_curve.py - Goodput curve → SLO targets: loads targets
analysis/goodput_curve.py - Ceiling analysis → Goodput and ceiling: uses summaries
analysis/ceiling.py - Cost per goodput → Goodput and ceiling: estimates economics
- Mock engine → Gateway API: serves test traffic
README.md