How collabora/whisperlive works
Project Type and Purpose
• WhisperLive is a real-time audio transcription system built around OpenAI’s Whisper models.
• It comprises:
• A server component (run_server.py + whisper_live/server.py) that hosts Whisper backends.
• A client library/CLI (whisper_live/client.py and TranscriptionClient) to stream audio and receive text.
• Browser extensions (Chrome & Firefox) for in-page audio capture.
• Docker configurations to package and deploy CPU, GPU (TensorRT), and OpenVINO variants.
File Structure
Top level:
• .github/workflows/ci.yml – CI pipeline
• docker/ – Dockerfiles for CPU, GPU, OpenVINO, TensorRT images
• run_server.py – entrypoint for the transcription server
• whisper_live/ – core Python package
– backend/ – abstraction + implementations (faster_whisper, openvino, trt)
– transcriber/ – utilities & glue for each backend
– vad.py – Voice Activity Detection
– client.py – TranscriptionClient API
– server.py – socket‐based server implementation
• Audio-Transcription-Chrome/ & Audio-Transcription-Firefox/ – browser extension source
• tests/, docs/, scripts/, requirements/, setup.py
README
• Describes three interchangeable backends: faster_whisper, TensorRT, OpenVINO.
• Outlines server flags, single‐model vs per‐connection model loading.
• Shows client parameters and supported audio sources (mic, file, RTSP, HLS).
• Mentions browser extensions that forward captured audio streams to the server.
a. Main Components
1. Audio Sources
– Local microphone
– Pre-recorded audio file
– RTSP or HLS stream
– Browser extension audio capture
2. Client Layer
– TranscriptionClient library/CLI
– Browser Extension UI & background scripts
3. Server Layer
– Connection manager (websocket/HTTP) in run_server.py & server.py
– VAD module (vad.py)
– Transcriber dispatcher (factory in backend/base.py)
– Model backends
• faster_whisper
• TensorRT (requires TensorRT-LLM & generated engines)
• OpenVINO
4. External Dependencies
– HuggingFace / OpenAI Whisper models
– NVIDIA TensorRT-LLM
– OpenVINO runtime
– PyAudio (mic capture)
5. Infrastructure & Deployment
– Docker images for CPU, GPU, OpenVINO, TensorRT
– CI/CD pipeline (ci.yml)
b. Relationships & Data Flow
– Audio Source → Client / Extension → Server API
– Server API → VAD → Raw audio chunks → Transcriber backend
– Backend → Text transcripts → Server API → Client / Extension
– Clients render/display transcripts; extensions overlay into browser UIs
– Docker images encapsulate server + model runtimes for consistent deployment
c. Architectural Patterns & Principles
– Factory pattern for backend selection (backend/base.py)
– Strategy pattern for pluggable transcribers (faster_whisper, trt, openvino)
– Single‐model vs multi‐model strategy toggled via flags
– Layered separation: transport, preprocessing (VAD), inference, postprocessing
– Containerization for environment consistency
d. Key Technologies & Frameworks
– Python 3.8+
– faster-whisper, TensorRT-LLM, OpenVINO
– PyAudio, websockets or HTTP (server-client comms)
– Browser Extension APIs (Chrome, Firefox)
Detail
• Docker build & runtime paths for each backend variant.
• Optional flows: file vs mic vs stream vs extension capture.