imen-source/llm-serving-lab

How imen-source/llm-serving-lab works

This repository is an LLM inference-serving optimization lab: callers submit generation requests, server variants generate responses, and a shared benchmark harness measures them. The principal variants are alternatives, not successive runtime stages: the naive and llama.cpp servers serialize requests, while the batched server schedules token steps; a round-robin proxy can dispatch to batched replicas. Shared metrics and the documented Prometheus/Grafana setup support comparison. The README documents quantization and cost analysis, but the available excerpts do not show the full benchmark or monitoring wiring.

Main entry points: server_llamacpp.py llama.cpp server · prom_metrics.py Serving metrics · client.py Load-test client · server.py Naive server

Components 14

Inference serving

Server settings src/llm_serving_lab/baseline/config.py
Naive server src/llm_serving_lab/baseline/server.py
llama.cpp server src/llm_serving_lab/baseline/server_llamacpp.py
GGUF quantization src/llm_serving_lab/baseline/server_llamacpp.py

Scheduling and scaling

Batch scheduler src/llm_serving_lab/baseline/server_batched.py
Round-robin proxy src/llm_serving_lab/baseline/load_balancer.py
Batched replicas src/llm_serving_lab/baseline/server_batched.py

Benchmarking and metrics

Load-test client src/llm_serving_lab/benchmark/client.py
Serving metrics src/llm_serving_lab/baseline/prom_metrics.py
Benchmark metrics src/llm_serving_lab/benchmark/metrics.py
Benchmark prompts src/llm_serving_lab/benchmark/prompts.py

Monitoring

Prometheus setup monitoring/prometheus/prometheus.yml
Grafana dashboard monitoring/grafana/dashboards/llm-serving-lab.json

Other components

API caller

Connections 19

API caller → Naive server: requests README.md
API caller → llama.cpp server: requests README.md
API caller → Batch scheduler: requests README.md
API caller → Round-robin proxy: requests src/llm_serving_lab/baseline/load_balancer.py
Round-robin proxy → Batched replicas: dispatches src/llm_serving_lab/baseline/load_balancer.py
Server settings → Naive server: configures src/llm_serving_lab/baseline/server.py
Server settings → llama.cpp server: configures src/llm_serving_lab/baseline/server_llamacpp.py
Server settings → Batch scheduler: configures src/llm_serving_lab/baseline/server_batched.py
Naive server → Serving metrics: updates src/llm_serving_lab/baseline/server.py
llama.cpp server → Serving metrics: updates src/llm_serving_lab/baseline/server_llamacpp.py
Batch scheduler → Serving metrics: updates src/llm_serving_lab/baseline/server_batched.py
Serving metrics → Benchmark metrics: uses src/llm_serving_lab/baseline/prom_metrics.py
Load-test client → Naive server: load tests README.md
Load-test client → llama.cpp server: load tests README.md
Load-test client → Batch scheduler: load tests README.md
Benchmark prompts → Load-test client: supplies prompts
GGUF quantization → llama.cpp server: selects precision src/llm_serving_lab/baseline/server_llamacpp.py
Prometheus setup → Serving metrics: scrapes
Grafana dashboard → Prometheus setup: visualizes

Drawn by GitDiagram on November 13, 2026 from the repository's file tree, README and source files. Read as Markdown ·