# collabora/whisperlive architecture

> Architecture diagram of the GitHub repository collabora/whisperlive, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/collabora/whisperlive
- Repository: https://github.com/collabora/whisperlive

## Overview

##### Project Type and Purpose
- WhisperLive is a real-time audio transcription system built around OpenAI’s Whisper models.
- It comprises:
- A server component (`run_server.py` + `whisper_live/server.py`) that hosts Whisper backends.
- A client library/CLI (`whisper_live/client.py` and `TranscriptionClient`) to stream audio and receive text.
- Browser extensions (Chrome & Firefox) for in-page audio capture.
- Docker configurations to package and deploy CPU, GPU (TensorRT), and OpenVINO variants.

##### File Structure
Top level:
- `.github/workflows/ci.yml` – CI pipeline
- `docker/` – Dockerfiles for CPU, GPU, OpenVINO, TensorRT images
- `run_server.py` – entrypoint for the transcription server
- `whisper_live/` – core Python package
– `backend/` – abstraction + implementations (faster_whisper, openvino, trt)
– `transcriber/` – utilities & glue for each backend
– `vad.py` – Voice Activity Detection
– `client.py` – `TranscriptionClient` API
– `server.py` – socket‐based server implementation
- `Audio-Transcription-Chrome/` & `Audio-Transcription-Firefox/` – browser extension source
- `tests/`, `docs/`, `scripts/`, `requirements/`, `setup.py`

##### README
- Describes three interchangeable backends: faster_whisper, TensorRT, OpenVINO.
- Outlines server flags, single‐model vs per‐connection model loading.
- Shows client parameters and supported audio sources (mic, file, RTSP, HLS).
- Mentions browser extensions that forward captured audio streams to the server.

a. Main Components
1. Audio Sources
– Local microphone
– Pre-recorded audio file
– RTSP or HLS stream
– Browser extension audio capture
2. Client Layer
– `TranscriptionClient` library/CLI
– Browser Extension UI & background scripts
3. Server Layer
– Connection manager (websocket/HTTP) in `run_server.py` & `server.py`
– VAD module (`vad.py`)
– Transcriber dispatcher (factory in `backend/base.py`)
– Model backends
- faster_whisper
- TensorRT (requires TensorRT-LLM & generated engines)
- OpenVINO
4. External Dependencies
– HuggingFace / OpenAI Whisper models
– NVIDIA TensorRT-LLM
– OpenVINO runtime
– PyAudio (mic capture)
5. Infrastructure & Deployment
– Docker images for CPU, GPU, OpenVINO, TensorRT
– CI/CD pipeline (`ci.yml`)

b. Relationships & Data Flow
– Audio Source → Client / Extension → Server API
– Server API → VAD → Raw audio chunks → Transcriber backend
– Backend → Text transcripts → Server API → Client / Extension
– Clients render/display transcripts; extensions overlay into browser UIs
– Docker images encapsulate server + model runtimes for consistent deployment

c. Architectural Patterns & Principles
– Factory pattern for backend selection (`backend/base.py`)
– Strategy pattern for pluggable transcribers (faster_whisper, trt, openvino)
– Single‐model vs multi‐model strategy toggled via flags
– Layered separation: transport, preprocessing (VAD), inference, postprocessing
– Containerization for environment consistency

d. Key Technologies & Frameworks
– Python 3.8+
– faster-whisper, TensorRT-LLM, OpenVINO
– PyAudio, websockets or HTTP (server-client comms)
– Browser Extension APIs (Chrome, Firefox)

##### Detail
- Docker build & runtime paths for each backend variant.
- Optional flows: file vs mic vs stream vs extension capture.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Audio Sources
    subgraph "Audio Sources"
        direction TB
        Mic["Local Microphone"]:::client
        File["Audio File"]:::client
        Stream["RTSP/HLS Stream"]:::client
    end

    %% Client Layer
    subgraph "Client Layer"
        direction TB
        CLI["TranscriptionClient"]:::client
        ChromeExt["Chrome Extension"]:::extension
        FirefoxExt["Firefox Extension"]:::extension
    end

    %% Server Layer
    subgraph "Server Layer"
        direction TB
        Entrypoint["run_server.py"]:::server
        Core["Server Core"]:::server
        VAD["VAD Module"]:::server
        Dispatcher["Transcriber Dispatcher"]:::server

        subgraph "Model Backends"
            direction TB
            FW_BE["faster-whisper Backend"]:::backend
            TRT_BE["TensorRT Backend"]:::backend
            OV_BE["OpenVINO Backend"]:::backend
        end
    end

    %% Infrastructure & Deployment
    subgraph "Infrastructure & Deployment"
        direction TB
        CPUImage["Docker CPU Image"]:::external
        GPUImage["Docker GPU Image"]:::external
        TRTImage["Docker TensorRT Image"]:::external
        OVImage["Docker OpenVINO Image"]:::external
        CI["CI/CD Pipeline"]:::external
    end

    %% External Services
    subgraph "External Services"
        direction TB
        HF["HuggingFace Models"]:::external
        TRT_LLM["NVIDIA TensorRT-LLM"]:::external
        OV_Runtime["OpenVINO Runtime"]:::external
        PyAudio["PyAudio"]:::external
    end

    %% Data Flow
    Mic -->|"audio" | CLI
    File -->|"audio" | CLI
    Stream -->|"audio" | CLI
    CLI -->|"stream audio/text"| Entrypoint
    ChromeExt -->|"capture→stream"| Entrypoint
    FirefoxExt -->|"capture→stream"| Entrypoint

    Entrypoint -->|"init server"| Core
    Core -->|"chunks"| VAD
    VAD -->|"vad filtered chunks"| Dispatcher
    Dispatcher -->|"select backend"| FW_BE
    Dispatcher -->|"select backend"| TRT_BE
    Dispatcher -->|"select backend"| OV_BE

    FW_BE -->|"use model"| HF
    TRT_BE -->|"use TensorRT-LLM"| TRT_LLM
    OV_BE -->|"use OpenVINO"| OV_Runtime

    CPUImage -.->|"build"| Entrypoint
    GPUImage -.->|"build"| Entrypoint
    TRTImage -.->|"build"| Entrypoint
    OVImage -.->|"build"| Entrypoint
    CI -.->|"test & deploy"| Entrypoint

    %% Click Events
    click Entrypoint "https://github.com/collabora/whisperlive/blob/main/run_server.py"
    click Core "https://github.com/collabora/whisperlive/blob/main/whisper_live/server.py"
    click CLI "https://github.com/collabora/whisperlive/blob/main/whisper_live/client.py"
    click VAD "https://github.com/collabora/whisperlive/blob/main/whisper_live/vad.py"
    click Dispatcher "https://github.com/collabora/whisperlive/blob/main/whisper_live/backend/base.py"
    click FW_BE "https://github.com/collabora/whisperlive/blob/main/whisper_live/backend/faster_whisper_backend.py"
    click TRT_BE "https://github.com/collabora/whisperlive/blob/main/whisper_live/backend/trt_backend.py"
    click OV_BE "https://github.com/collabora/whisperlive/blob/main/whisper_live/backend/openvino_backend.py"
    click ChromeExt "https://github.com/collabora/whisperlive/tree/main/Audio-Transcription-Chrome/"
    click FirefoxExt "https://github.com/collabora/whisperlive/tree/main/Audio-Transcription-Firefox/"
    click CPUImage "https://github.com/collabora/whisperlive/blob/main/docker/Dockerfile.cpu"
    click GPUImage "https://github.com/collabora/whisperlive/blob/main/docker/Dockerfile.gpu"
    click TRTImage "https://github.com/collabora/whisperlive/blob/main/docker/Dockerfile.tensorrt"
    click OVImage "https://github.com/collabora/whisperlive/blob/main/docker/Dockerfile.openvino"
    click CI "https://github.com/collabora/whisperlive/blob/main/.github/workflows/ci.yml"

    %% Styles
    classDef client fill:#D0E8FF,stroke:#333,stroke-width:1px
    classDef server fill:#D0F0D8,stroke:#333,stroke-width:1px
    classDef backend fill:#FFE8B8,stroke:#333,stroke-width:1px
    classDef extension fill:#E8D0FF,stroke:#333,stroke-width:1px
    classDef external fill:#EEEEEE,stroke:#333,stroke-width:1px
```
