How quettaflop/quettabench works
QuettaBench benchmarks LLM inference with production-oriented single-turn, multi-turn, and stress workloads, measuring request latency and token timing against OpenAI-compatible servers. Its workload modules provide prompt/session data and distributional replay; engine adapters provide chat and completions interfaces. The README also describes opt-in GPU/live-server profiling and QuettaServe-versus-vLLM cross-validation. The listed tree does not include the documented benchmark runner or launch scripts, so their orchestration and result-writing links are omitted; profiling relationships are based on README descriptions and the sampled probe/emitter excerpts.
Main entry points: profiles.py Workload profiles · distributional.py Synthetic replay · probes Live-server probes · multi_turn.py Multi-turn mode
Components 18
Workload modeling
- Workload profiles
src/workloads/profiles.py - Prompt datasets
src/workloads/dataset.py - Arrival patterns
src/workloads/arrival.py - Trace distributions
src/workloads/trace_distributions.py - Synthetic replay
src/workloads/distributional.py - Mooncake workloads
src/workloads/mooncake.py
Benchmark interfaces
- Single-turn mode
src/modes/single_turn.py - Multi-turn mode
src/modes/multi_turn.py - Chat adapter
src/engines/openai_chat.py - Completions adapter
src/engines/openai_completions.py
Profiling tools
- GPU kernel probes
profiling/kernel_composed/probes - Live-server probes
profiling/probes - Profile emitters
profiling/emit
Cross-validation
- Cross-validation tools
scripts/crossval
Other components
- Benchmark user
- Inference server
- Profile measurements
- Cached vLLM baseline
Connections 18
- Benchmark user → Single-turn mode: selects mode
- Benchmark user → Multi-turn mode: selects mode
- Single-turn mode → Workload profiles: filters profiles
- Multi-turn mode → Workload profiles: uses profiles
- Workload profiles → Prompt datasets: selects prompts
- Workload profiles → Synthetic replay: selects replay
- Synthetic replay → Trace distributions: samples traces
- Synthetic replay → Mooncake workloads: supports replay
- Arrival patterns → Multi-turn mode: paces turns
- Chat adapter → Inference server: streams requests
- Completions adapter → Inference server: streams requests
- Benchmark user → GPU kernel probes: runs probes
- Benchmark user → Live-server probes: runs probes
- GPU kernel probes → Profile measurements: writes measurements
- Live-server probes → Inference server: probes server
- Live-server probes → Profile measurements: writes measurements
- Profile emitters → Profile measurements: curates measurements
- Cross-validation tools → Cached vLLM baseline: compares against