# backspacetg/simul_whisper architecture

> Architecture diagram of the GitHub repository backspacetg/simul_whisper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/backspacetg/simul_whisper
- Repository: https://github.com/backspacetg/simul_whisper

## Overview

##### Project’s Main Components
- Model Checkpoints: The "cif_models" directory holds the pre-trained Whisper model checkpoint files (base, small, medium, large-v2). These are external resources loaded at runtime.
- Transcriber Module: Under "simul_whisper/transcriber", you have configuration files (config.py), latency scoring logic (latency_scorer.py), segment loading support (segment_loader.py), and a main transcriber module (simul_whisper.py). This module sets up and manages the streaming transcription process.
- Whisper Module: The "simul_whisper/whisper" directory contains the core Whisper functionality including model handling, audio processing, decoding, normalization, and utilities. It mirrors the structure of the original openai-whisper repository, with extensions for streaming features.
- Entry Point and Sample Usage: The "transcribe.py" script and the "__main__.py" file in the whisper submodule provide the user interface for initializing configurations, segmenting the audio, running the inference, decoding outputs, and managing context buffers.

##### Relationships and Data Flow
- First, the system loads the required pre-trained model checkpoint from the cif_models directory.
- The transcriber configuration settings (e.g., segment lengths, threshold parameters, context buffers) are prepared via the configuration files. This configures how the model processes segments.
- Audio is read and segmented (as depicted in the sample code and supported by the segment_loader module). The segmented audio is passed to the Whisper module where the model inference is performed.
- The latency scorer might be used to adjust or evaluate streaming decisions (i.e., truncation detection), ensuring that inference decisions are made in real time.
- Finally, after processing, the transcriber decodes the tokenized results using the tokenizer in the whisper module and outputs the final transcription.

##### Architectural Patterns and Technologies
- The project demonstrates a modular architecture that separates concerns: configuration, segmentation, core transcription (model inference/decoding), and auxiliary functions (latency scoring, normalization).
- By building on OpenAI’s Whisper structure, it inherits a layered design where the transcriber (handling streaming adjustments and buffer management) sits on top of the foundational Whisper model operations.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Resources
    subgraph "External Resources"
        MC["Model Checkpoints"]:::resource
    end

    %% Entry Points
    subgraph "Entry Points"
        EP1["Entry: transcribe.py"]:::entry
        EP2["Entry: simul_whisper/whisper/transcribe.py"]:::entry
    end

    %% Transcriber Layer
    subgraph "Transcriber Layer"
        TM["Transcriber Module"]:::transcriber
        TS["Configuration & Segmentation"]:::transcriber
        LS["Latency Scorer"]:::transcriber
    end

    %% Whisper Engine
    subgraph "Whisper Engine"
        WE["Whisper Module"]:::engine
        TP["Tokenization & Decoding"]:::engine
    end

    %% Data Flow
    EP1 -->|"init_config"| TM
    EP2 -->|"init_config"| TM
    TM -->|"setup_segments"| TS
    TM -->|"apply_latency"| LS
    TM -->|"audio_segment"| WE
    MC -->|"load_model"| WE
    WE -->|"infer_decode"| TP
    TP -->|"final_transcription"| TM

    %% Technical Stack Note
    Note1["Python 3.9.16 & PyTorch"]:::entry
    EP1 --- Note1

    %% Styles
    classDef entry fill:#f9c74f,stroke:#333,stroke-width:2px;
    classDef transcriber fill:#90be6d,stroke:#333,stroke-width:2px;
    classDef engine fill:#577590,stroke:#333,stroke-width:2px;
    classDef resource fill:#f94144,stroke:#333,stroke-width:2px;

    %% Click Events
    click MC "https://github.com/backspacetg/simul_whisper/tree/main/cif_models/"
    click TM "https://github.com/backspacetg/simul_whisper/tree/main/simul_whisper/transcriber/"
    click WE "https://github.com/backspacetg/simul_whisper/tree/main/simul_whisper/whisper/"
    click EP1 "https://github.com/backspacetg/simul_whisper/blob/main/transcribe.py"
    click EP2 "https://github.com/backspacetg/simul_whisper/blob/main/simul_whisper/whisper/transcribe.py"
```
