# ccyrene/flash_whisper architecture

> Architecture diagram of the GitHub repository ccyrene/flash_whisper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ccyrene/flash_whisper
- Repository: https://github.com/ccyrene/flash_whisper

## Overview

##### Project Type and Purpose
- Flash Whisper is a real-time ASR system that leverages both ONNX-based execution and high-performance inference using TensorRT-LLM.
- The README shows quick-start instructions for both Docker-based server deployment (using docker-compose) and local Python invocation via a request script.
- It supports conversion processes for transformer models (using converter modules) and integrates with Triton Inference Server for serving larger models.
- Indications of using multiple client libraries (Python and Go) suggest that the system is designed to be consumed by heterogeneous clients.

##### File Structure Analysis
- The top-level files (e.g., README.md, setup.py, request.py, inspection.py) indicate entry points and setup utilities.
- The core logic resides in the src/flash_whisper directory, split mainly into:
- An “onnx” submodule containing components such as:
- config (handling generation configurations)
- extractor (for feature extraction and mel processing)
- processor (defining criteria and the processing logic)
- tokenizer (for speech-to-text token normalization and utility functions)
- a central “whisper.py” that likely ties these submodules together.
- A “serving” submodule:
- Contains Dockerfiles for different clients (Go and Python) as well as a server Dockerfile.
- Holds client implementations (both in Go and Python) using GRPC and REST patterns.
- Includes configuration for Triton Inference Server (via directories like triton/infer_bls and triton/whisper_medium).
- Has converter scripts and a “prepare.sh” to set up environments.
- A “tllm” submodule:
- Contains conversion scripts and a separate implementation of Whisper for TRT-LLM.
- Additional directories like “media” and “sample” hold assets for performance testing and example audio files.

##### Key Components
a. ASR Core Engine:
- ONNX Pipeline:
– Feature Extraction (src/flash_whisper/onnx/extractor)
– Preprocessing & Mel Processing (within the extractor)
– Processing logic (src/flash_whisper/onnx/processor)
– Tokenization (src/flash_whisper/onnx/tokenizer)
– Overall orchestration (src/flash_whisper/onnx/whisper.py)
- TensorRT-LLM Pipeline (under tllm):
– Model conversion scripts (convert_ckpt_hf_to_model.py, convert_model_to_safetensors.py, etc.)
– Inference logic (src/flash_whisper/tllm/whisper.py, tokenizer.py, fbank.py)

b. Serving Layer:
- Server Components:
– Docker Compose setup (docker-compose.yml) indicating container orchestration.
– Server (possibly a FastAPI server for Python and a Fiber/Go server for Go clients).
– Integration with Triton Inference Server for high-performance model serving.
- Client Components:
– Python client (src/flash_whisper/serving/client/python)
– Go client (src/flash_whisper/serving/client/go with its generated GRPC stubs)

c. Model Conversion / Preparation Layer:
- Converter modules under “serving/converter” and “tllm/converter” that handle converting models from Hugging Face checkpoints into ONNX or TRT-compatible formats.

d. Infrastructure / Deployment:
- Dockerfiles (multiple per client/server) and shell scripts (prepare.sh, build.sh) for setup automation.
- Configuration for different environments (e.g., onnx, Triton configurations via pbtxt files).

##### Relationships and Data Flow
- Clients (Python / Go) send audio data (currently supporting WAV) to the server endpoint.
- The server dispatches requests to the ASR core engine.
– Within the ASR engine, the audio data flows through the extractor, processor, and tokenizer for transcription.
– Depending on deployment, either the ONNX pipeline or TensorRT-LLM pipeline is used.
– There is a conversion/ preparation step (when initial setup or model update is required) that generates the required model files.
- Conversion modules interact with the model storage (directories provided by external messengers or environment variables like OUTPUT_DIR).
- The server then returns transcribed text to the client.
Client → API/Server Layer → Model Inference Pipeline → Model Conversion module (if applicable) → Back to Server & then to Client.

##### Architectural Patterns and Design Principles
- Separation of Concerns:
- Clear distinction between model inference (ONNX vs. TRT-LLM) and service layer operations.
- Dedicated modules for feature extraction, processing, and tokenization.
- Containerization:
- The Dockerfiles and docker-compose setup ensure isolated deployment of microservices.
- Extensibility:
- The system is modular, which allows for the addition or update of pipelines (e.g., new conversion modules, additional clients).
- Interoperability:
- Clients in different languages leverage similar protocols (GRPC, REST) to interact with the same service.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Client Layer
    subgraph "Client Layer"
        PC["Python Client"]:::client
        GC["Go Client"]:::client
    end

    %% Serving Layer
    subgraph "Serving Layer"
        AS["API Server"]:::server
        TR1["Triton Infer BLS"]:::external
        TR2["Triton Whisper Medium"]:::external
        PS["Preparation Script"]:::server
    end

    %% ASR Core Engine
    subgraph "ASR Core Engine"
        %% ONNX Pipeline
        subgraph "ONNX Pipeline"
            FE["ONNX Feature Extractor"]:::asr
            PR["ONNX Processor"]:::asr
            TK["ONNX Tokenizer"]:::asr
            OW["ONNX Orchestrator"]:::asr
        end
        %% TensorRT-LLM Pipeline
        subgraph "TensorRT-LLM Pipeline"
            TI["TLLM Inference"]:::asr
        end
    end

    %% Model Conversion Modules
    subgraph "Conversion Modules"
        SC["Serving Converter"]:::converter
        TLC["TLLM Converter"]:::converter
    end

    %% Relationships: Clients to API Server
    PC -->|"sends_audio"| AS
    GC -->|"sends_audio"| AS

    %% API Server interacting with Triton and ASR engine
    AS -->|"for_ONNX_inference"| OW
    AS -->|"for_TLLM_inference"| TI

    %% ONNX Pipeline internal flow
    FE -->|"extracts_features"| PR
    PR -->|"processes_signals"| TK
    TK -->|"prepares_tokens"| OW

    %% Inference results back to API Server
    OW -->|"returns_text"| AS
    TI -->|"returns_text"| AS

    %% API Server interacts with Triton Inference Server
    AS -->|"queries"| TR1
    AS -->|"queries"| TR2

    %% API Server engages conversion modules
    AS -->|"initiates"| SC
    AS -->|"initiates"| TLC
    SC -->|"supplies_models"| OW
    TLC -->|"supplies_models"| TI

    %% API Server returns responses to Clients
    AS -->|"sends_response"| PC
    AS -->|"sends_response"| GC

    %% Click Events for ASR Core Engine – ONNX Pipeline
    click FE "https://github.com/ccyrene/flash_whisper/tree/main/src/flash_whisper/onnx/extractor"
    click PR "https://github.com/ccyrene/flash_whisper/tree/main/src/flash_whisper/onnx/processor"
    click TK "https://github.com/ccyrene/flash_whisper/tree/main/src/flash_whisper/onnx/tokenizer"
    click OW "https://github.com/ccyrene/flash_whisper/blob/main/src/flash_whisper/onnx/whisper.py"

    %% Click Events for ASR Core Engine – TensorRT-LLM Pipeline
    click TI "https://github.com/ccyrene/flash_whisper/blob/main/src/flash_whisper/tllm/whisper.py"

    %% Click Events for Serving Layer – Server Components
    click AS "https://github.com/ccyrene/flash_whisper/tree/main/src/flash_whisper/serving"
    click TR1 "https://github.com/ccyrene/flash_whisper/tree/main/src/flash_whisper/serving/triton/infer_bls"
    click TR2 "https://github.com/ccyrene/flash_whisper/tree/main/src/flash_whisper/serving/triton/whisper_medium"
    click PS "https://github.com/ccyrene/flash_whisper/blob/main/src/flash_whisper/serving/prepare.sh"

    %% Click Events for Serving Layer – Client Components
    click PC "https://github.com/ccyrene/flash_whisper/tree/main/src/flash_whisper/serving/client/python"
    click GC "https://github.com/ccyrene/flash_whisper/tree/main/src/flash_whisper/serving/client/go"

    %% Click Events for Model Conversion Modules
    click SC "https://github.com/ccyrene/flash_whisper/tree/main/src/flash_whisper/serving/converter"
    click TLC "https://github.com/ccyrene/flash_whisper/tree/main/src/flash_whisper/tllm/converter"

    %% Styles
    classDef client fill:#AEDFF7,stroke:#333,stroke-width:2px;
    classDef server fill:#F7C8E0,stroke:#333,stroke-width:2px;
    classDef asr fill:#C8F7C5,stroke:#333,stroke-width:2px;
    classDef converter fill:#FFEDD5,stroke:#333,stroke-width:2px;
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:2px;
```
