# ufal/whisper_streaming architecture

> Architecture diagram of the GitHub repository ufal/whisper_streaming, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ufal/whisper_streaming
- Repository: https://github.com/ufal/whisper_streaming

## Overview

##### Project type and purpose
- This is a command‑line and server‑based real‑time streaming speech‑to‑text (and optional translation) system. It wraps Whisper‑family ASR models to ingest live or prerecorded audio, apply voice‐activity detection (VAD), process incremental audio chunks with a local agreement policy, and emit low‑latency partial and final transcripts.

##### File structure
- Top‑level scripts:
- line_packet.py – packetizes raw audio into frames for streaming
- silero_vad_iterator.py – iterates over audio frames using Silero VAD to detect speech segments
- whisper_online.py – main CLI for simulating or running real‑time streaming from file or mic
- whisper_online_server.py – server variant exposing TCP (and optionally WebSocket) endpoint for live audio
- No nested “src” or “lib” folders: logic is co‑located in these modules.
- Configuration is via CLI arguments. Dependencies are external libraries (faster‑whisper, whisper_timestamped, openai, torch, torchaudio, optional segmenters).

##### Key README insights
- Supports multiple backends (GPU‑accelerated, timestamped, OpenAI API, Apple‑silicon MLX).
- Buffer trimming strategies (segment vs. sentence).
- Local agreement policy confirms prefixes after N identical updates.
- CLI and server modes share core OnlineASRProcessor and ASR model wrappers.
- Example pseudocode shows OnlineASRProcessor, insert_audio_chunk, process_iter, finish, init methods.

##### Capturing essential architectural elements
– Streaming pipeline order (packet → VAD → buffering → inference → commit → output)
– Backend abstraction layer with four strategies
– CLI vs. server entry points
– Configuration flags affecting processing (min chunk size, buffer_trimming, VAC/VAD)
– Data types at each boundary (raw PCM frames, audio chunks, transcripts with timestamps)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Front End I/O"
        CLI["CLI Interface\n(whisper_online.py)"]:::ioComp
        Server["Server Interface\n(whisper_online_server.py)"]:::ioComp
    end

    CLI -->|raw PCM frames| Packetizer
    Server -->|audio stream| Packetizer

    Packetizer["Packetizer\n(line_packet.py)"]:::procComp
    VAD["VAD Module\n(silero_vad_iterator.py)"]:::procComp
    SentenceSegmenter["Sentence Segmenter\n(optional)"]:::optionalComp
    Processor["Online ASR Processor\n(buffering, agreement)"]:::procComp
    OutputSink["Output Sink\n(console stdout / log)"]:::outputComp

    Packetizer -->|frames| VAD
    VAD -->|speech segments| Processor

    Processor -->|chunk| SentenceSegmenter
    SentenceSegmenter -->|segmented chunk| Processor

    subgraph "ASR Backend Wrappers"
        FasterWhisper["FasterWhisperASR"]:::backendComp
        WhisperTS["WhisperTimestamped"]:::backendComp
        OpenAIAPI["OpenAI API Client"]:::backendComp
        MLXWrapper["MLX Whisper Wrapper"]:::backendComp
    end

    Processor -->|chunk| FasterWhisper
    Processor -->|chunk| WhisperTS
    Processor -->|chunk| OpenAIAPI
    Processor -->|chunk| MLXWrapper

    FasterWhisper -->|transcripts| Processor
    WhisperTS -->|transcripts| Processor
    OpenAIAPI -->|transcripts| Processor
    MLXWrapper -->|transcripts| Processor

    Processor -->|transcripts| OutputSink

    click Packetizer "https://github.com/ufal/whisper_streaming/blob/main/line_packet.py"
    click VAD "https://github.com/ufal/whisper_streaming/blob/main/silero_vad_iterator.py"
    click CLI "https://github.com/ufal/whisper_streaming/blob/main/whisper_online.py"
    click Server "https://github.com/ufal/whisper_streaming/blob/main/whisper_online_server.py"

    classDef ioComp fill:#E8F1FA,stroke:#4B8BBE,color:#073642;
    classDef procComp fill:#EDF8F5,stroke:#28A745,color:#073642;
    classDef backendComp fill:#FFF4E6,stroke:#D69E2E,color:#073642;
    classDef outputComp fill:#F0EBF4,stroke:#6A1B9A,color:#073642;
    classDef optionalComp fill:none,stroke:#999999,stroke-dasharray: 5 5,color:#999999;
```
