How picovoice/tts-latency-benchmark works
Project Type and Purpose
• This is an open-source Python benchmarking framework for measuring latency in a text-to-speech (TTS) pipeline that includes a large language model (LLM) and multiple external TTS engines.
• It simulates user–assistant interactions by sampling user queries, generating LLM responses via the OpenAI Chat Completion API, and streaming those tokens into various TTS engines. The goal is to measure “Time to First Token” (TTFT), “First Token to Speech” (FTTS), and aggregate “Voice Assistant Response Time” (VART).
File Structure Analysis
Top-level scripts:
• benchmark.py – orchestrates a full run: loads data, drives LLM, invokes TTS engines, times responses, writes JSON results to results/data.
• plot_results.py – reads JSON results, generates plots into results/plots.
Core directories:
• data/ – data/_dataset.py loads and samples user queries from Taskmaster2.
• llm/ – llm/_llm.py handles interactions with OpenAI’s chat-completion API and token streaming.
• tts/ – tts/_tts.py defines a plugin pattern for each TTS engine; tts/_timer.py wraps timing logic.
• audio/ – audio/_audio.py contains helpers for audio buffering or playback (if needed).
Support files: requirements.txt, README.md, .gitignore, LICENSE.
Additional Insights from README
• The pipeline has clearly defined stages: Data Ingestion → LLM Response → TTS Synthesis → Metrics Collection → Result Persistence → Plotting.
• External dependencies are all REST or WebSocket services (OpenAI, Amazon Polly, Azure TTS, ElevenLabs, Picovoice Orca).
• The framework is extensible: new TTS engines can be added by implementing the _tts interface.
a. Main Components
1. Data Source & Loader (data/_dataset.py)
2. Benchmark Controller (benchmark.py)
3. LLM Client (llm/_llm.py)
4. TTS Engine Plugins (tts/_tts.py + engine-specific subclasses)
5. Timer/Metrics Collector (tts/_timer.py)
6. Results Store (results/data/*.json)
7. Plotter (plot_results.py)
8. Audio Utilities (audio/_audio.py) – optional for playback
9. External Services (OpenAI, AWS Polly, Azure, ElevenLabs, Picovoice Orca)
b. Relationships & Interactions
• The Benchmark Controller invokes the Data Loader to fetch a batch of user queries.
• For each query:
– Send to LLM Client → receive streaming tokens → record TTFT.
– Pass tokens to TTS Plugin → receive streaming audio → record FTTS.
– Timer aggregates VART = TTFT + FTTS.
• The controller writes per-engine, per-query latencies to JSON.
• The Plotter reads JSON and produces comparison plots.
c. Architectural Patterns & Principles
• Pipeline pattern: sequential stages connected via streamed data.
• Plugin/strategy pattern for TTS engines: a common interface abstracts over multiple external APIs (REST vs. WebSocket).
• Single responsibility: each module handles one stage (data, LLM, TTS, timing, plotting).
• Extensibility: new TTS engines conform to the tts interface.
d. Key Technologies & Libraries
• Python 3.10
• OpenAI Python SDK for chat completion
• boto3 (for Amazon Polly)
• azure-cognitiveservices-speech
• Requests / websockets for ElevenLabs
• Picovoice Orca SDK
• matplotlib or seaborn for plotting
Essential Details to Capture
• The two timing metrics (TTFT, FTTS) and their aggregation.
• Streaming nature: token streaming into TTS streaming.
• Plugin architecture for adding new engines.
• Result persistence and plotting as separate stages.
• Configuration inputs (API keys, engine flags) passed into benchmark.py.