How ufal/whisper_streaming works
Project type and purpose
• This is a command‑line and server‑based real‑time streaming speech‑to‑text (and optional translation) system. It wraps Whisper‑family ASR models to ingest live or prerecorded audio, apply voice‐activity detection (VAD), process incremental audio chunks with a local agreement policy, and emit low‑latency partial and final transcripts.
File structure
• Top‑level scripts:
• line_packet.py – packetizes raw audio into frames for streaming
• silero_vad_iterator.py – iterates over audio frames using Silero VAD to detect speech segments
• whisper_online.py – main CLI for simulating or running real‑time streaming from file or mic
• whisper_online_server.py – server variant exposing TCP (and optionally WebSocket) endpoint for live audio
• No nested “src” or “lib” folders: logic is co‑located in these modules.
• Configuration is via CLI arguments. Dependencies are external libraries (faster‑whisper, whisper_timestamped, openai, torch, torchaudio, optional segmenters).
Key README insights
• Supports multiple backends (GPU‑accelerated, timestamped, OpenAI API, Apple‑silicon MLX).
• Buffer trimming strategies (segment vs. sentence).
• Local agreement policy confirms prefixes after N identical updates.
• CLI and server modes share core OnlineASRProcessor and ASR model wrappers.
• Example pseudocode shows OnlineASRProcessor, insert_audio_chunk, process_iter, finish, init methods.
Capturing essential architectural elements
– Streaming pipeline order (packet → VAD → buffering → inference → commit → output)
– Backend abstraction layer with four strategies
– CLI vs. server entry points
– Configuration flags affecting processing (min chunk size, buffer_trimming, VAC/VAD)
– Data types at each boundary (raw PCM frames, audio chunks, transcripts with timestamps)