How backspacetg/simul_whisper works
Project’s Main Components
• Model Checkpoints: The "cif_models" directory holds the pre-trained Whisper model checkpoint files (base, small, medium, large-v2). These are external resources loaded at runtime.
• Transcriber Module: Under "simul_whisper/transcriber", you have configuration files (config.py), latency scoring logic (latency_scorer.py), segment loading support (segment_loader.py), and a main transcriber module (simul_whisper.py). This module sets up and manages the streaming transcription process.
• Whisper Module: The "simul_whisper/whisper" directory contains the core Whisper functionality including model handling, audio processing, decoding, normalization, and utilities. It mirrors the structure of the original openai-whisper repository, with extensions for streaming features.
• Entry Point and Sample Usage: The "transcribe.py" script and the "__main__.py" file in the whisper submodule provide the user interface for initializing configurations, segmenting the audio, running the inference, decoding outputs, and managing context buffers.
Relationships and Data Flow
• First, the system loads the required pre-trained model checkpoint from the cif_models directory.
• The transcriber configuration settings (e.g., segment lengths, threshold parameters, context buffers) are prepared via the configuration files. This configures how the model processes segments.
• Audio is read and segmented (as depicted in the sample code and supported by the segment_loader module). The segmented audio is passed to the Whisper module where the model inference is performed.
• The latency scorer might be used to adjust or evaluate streaming decisions (i.e., truncation detection), ensuring that inference decisions are made in real time.
• Finally, after processing, the transcriber decodes the tokenized results using the tokenizer in the whisper module and outputs the final transcription.
Architectural Patterns and Technologies
• The project demonstrates a modular architecture that separates concerns: configuration, segmentation, core transcription (model inference/decoding), and auxiliary functions (latency scoring, normalization).
• By building on OpenAI’s Whisper structure, it inherits a layered design where the transcriber (handling streaming adjustments and buffer management) sits on top of the foundational Whisper model operations.