# imen-source/llm-serving-lab architecture

> Architecture diagram of the GitHub repository imen-source/llm-serving-lab, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/imen-source/llm-serving-lab
- Repository: https://github.com/imen-source/llm-serving-lab

## Overview

This repository is an LLM inference-serving optimization lab: callers submit generation requests, server variants generate responses, and a shared benchmark harness measures them. The principal variants are alternatives, not successive runtime stages: the naive and llama.cpp servers serialize requests, while the batched server schedules token steps; a round-robin proxy can dispatch to batched replicas. Shared metrics and the documented Prometheus/Grafana setup support comparison. The README documents quantization and cost analysis, but the available excerpts do not show the full benchmark or monitoring wiring.

Main entry points: [`src/llm_serving_lab/baseline/server_llamacpp.py`](https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/server_llamacpp.py) (llama.cpp server), [`src/llm_serving_lab/baseline/prom_metrics.py`](https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/prom_metrics.py) (Serving metrics), [`src/llm_serving_lab/benchmark/client.py`](https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/benchmark/client.py) (Load-test client), [`src/llm_serving_lab/baseline/server.py`](https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/server.py) (Naive server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Inference serving"]
  node_config["Server settings<br/>[config.py]"]
  node_naive["Naive server<br/>[server.py]"]
  node_llamacpp["llama.cpp server<br/>[server_llamacpp.py]"]
  node_quant["GGUF quantization<br/>[server_llamacpp.py]"]
end

subgraph group_scaling["Scheduling and scaling"]
  node_scheduler["Batch scheduler<br/>[server_batched.py]"]
  node_balancer["Round-robin proxy<br/>[load_balancer.py]"]
  node_replicas["Batched replicas<br/>[server_batched.py]"]
end

subgraph group_measurement["Benchmarking and metrics"]
  node_bench_client["Load-test client<br/>[client.py]"]
  node_prom_metrics["Serving metrics<br/>[prom_metrics.py]"]
  node_bench_metrics["Benchmark metrics<br/>[metrics.py]"]
  node_prompts["Benchmark prompts<br/>[prompts.py]"]
end

subgraph group_monitoring["Monitoring"]
  node_prometheus["Prometheus setup<br/>[prometheus.yml]"]
  node_grafana["Grafana dashboard"]
end

node_caller(("API caller"))

node_caller -->|"requests"| node_naive
node_caller -->|"requests"| node_llamacpp
node_caller -->|"requests"| node_scheduler
node_caller -->|"requests"| node_balancer
node_balancer -->|"dispatches"| node_replicas
node_config -->|"configures"| node_naive
node_config -->|"configures"| node_llamacpp
node_config -->|"configures"| node_scheduler
node_naive -->|"updates"| node_prom_metrics
node_llamacpp -->|"updates"| node_prom_metrics
node_scheduler -->|"updates"| node_prom_metrics
node_prom_metrics -->|"uses"| node_bench_metrics
node_bench_client -->|"load tests"| node_naive
node_bench_client -->|"load tests"| node_llamacpp
node_bench_client -->|"load tests"| node_scheduler
node_prompts -.->|"supplies prompts"| node_bench_client
node_quant -->|"selects precision"| node_llamacpp
node_prometheus -.->|"scrapes"| node_prom_metrics
node_grafana -.->|"visualizes"| node_prometheus

click node_config "https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/config.py"
click node_naive "https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/server.py"
click node_llamacpp "https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/server_llamacpp.py"
click node_quant "https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/server_llamacpp.py"
click node_scheduler "https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/server_batched.py"
click node_balancer "https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/load_balancer.py"
click node_replicas "https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/server_batched.py"
click node_bench_client "https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/benchmark/client.py"
click node_prom_metrics "https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/prom_metrics.py"
click node_bench_metrics "https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/benchmark/metrics.py"
click node_prompts "https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/benchmark/prompts.py"
click node_prometheus "https://github.com/imen-source/llm-serving-lab/blob/main/monitoring/prometheus/prometheus.yml"
click node_grafana "https://github.com/imen-source/llm-serving-lab/blob/main/monitoring/grafana/dashboards/llm-serving-lab.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_config,node_naive,node_llamacpp,node_quant toneBlue
class node_scheduler,node_balancer,node_replicas toneAmber
class node_bench_client,node_prom_metrics,node_bench_metrics,node_prompts,node_caller toneMint
class node_prometheus,node_grafana toneRose
```

## Components

### Inference serving

- [Server settings](https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/config.py): `src/llm_serving_lab/baseline/config.py`
- [Naive server](https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/server.py): `src/llm_serving_lab/baseline/server.py`
- [llama.cpp server](https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/server_llamacpp.py): `src/llm_serving_lab/baseline/server_llamacpp.py`
- [GGUF quantization](https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/server_llamacpp.py): `src/llm_serving_lab/baseline/server_llamacpp.py`

### Scheduling and scaling

- [Batch scheduler](https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/server_batched.py): `src/llm_serving_lab/baseline/server_batched.py`
- [Round-robin proxy](https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/load_balancer.py): `src/llm_serving_lab/baseline/load_balancer.py`
- [Batched replicas](https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/server_batched.py): `src/llm_serving_lab/baseline/server_batched.py`

### Benchmarking and metrics

- [Load-test client](https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/benchmark/client.py): `src/llm_serving_lab/benchmark/client.py`
- [Serving metrics](https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/baseline/prom_metrics.py): `src/llm_serving_lab/baseline/prom_metrics.py`
- [Benchmark metrics](https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/benchmark/metrics.py): `src/llm_serving_lab/benchmark/metrics.py`
- [Benchmark prompts](https://github.com/imen-source/llm-serving-lab/blob/main/src/llm_serving_lab/benchmark/prompts.py): `src/llm_serving_lab/benchmark/prompts.py`

### Monitoring

- [Prometheus setup](https://github.com/imen-source/llm-serving-lab/blob/main/monitoring/prometheus/prometheus.yml): `monitoring/prometheus/prometheus.yml`
- [Grafana dashboard](https://github.com/imen-source/llm-serving-lab/blob/main/monitoring/grafana/dashboards/llm-serving-lab.json): `monitoring/grafana/dashboards/llm-serving-lab.json`

### Other components

- **API caller**

## Connections

- API caller → Naive server: requests (evidence: `README.md`)
- API caller → llama.cpp server: requests (evidence: `README.md`)
- API caller → Batch scheduler: requests (evidence: `README.md`)
- API caller → Round-robin proxy: requests (evidence: `src/llm_serving_lab/baseline/load_balancer.py`)
- Round-robin proxy → Batched replicas: dispatches (evidence: `src/llm_serving_lab/baseline/load_balancer.py`)
- Server settings → Naive server: configures (evidence: `src/llm_serving_lab/baseline/server.py`)
- Server settings → llama.cpp server: configures (evidence: `src/llm_serving_lab/baseline/server_llamacpp.py`)
- Server settings → Batch scheduler: configures (evidence: `src/llm_serving_lab/baseline/server_batched.py`)
- Naive server → Serving metrics: updates (evidence: `src/llm_serving_lab/baseline/server.py`)
- llama.cpp server → Serving metrics: updates (evidence: `src/llm_serving_lab/baseline/server_llamacpp.py`)
- Batch scheduler → Serving metrics: updates (evidence: `src/llm_serving_lab/baseline/server_batched.py`)
- Serving metrics → Benchmark metrics: uses (evidence: `src/llm_serving_lab/baseline/prom_metrics.py`)
- Load-test client → Naive server: load tests (evidence: `README.md`)
- Load-test client → llama.cpp server: load tests (evidence: `README.md`)
- Load-test client → Batch scheduler: load tests (evidence: `README.md`)
- Benchmark prompts → Load-test client: supplies prompts
- GGUF quantization → llama.cpp server: selects precision (evidence: `src/llm_serving_lab/baseline/server_llamacpp.py`)
- Prometheus setup → Serving metrics: scrapes
- Grafana dashboard → Prometheus setup: visualizes