How ccyrene/flash_whisper works
Project Type and Purpose
• Flash Whisper is a real-time ASR system that leverages both ONNX-based execution and high-performance inference using TensorRT-LLM.
• The README shows quick-start instructions for both Docker-based server deployment (using docker-compose) and local Python invocation via a request script.
• It supports conversion processes for transformer models (using converter modules) and integrates with Triton Inference Server for serving larger models.
• Indications of using multiple client libraries (Python and Go) suggest that the system is designed to be consumed by heterogeneous clients.
File Structure Analysis
• The top-level files (e.g., README.md, setup.py, request.py, inspection.py) indicate entry points and setup utilities.
• The core logic resides in the src/flash_whisper directory, split mainly into:
• An “onnx” submodule containing components such as:
• config (handling generation configurations)
• extractor (for feature extraction and mel processing)
• processor (defining criteria and the processing logic)
• tokenizer (for speech-to-text token normalization and utility functions)
• a central “whisper.py” that likely ties these submodules together.
• A “serving” submodule:
• Contains Dockerfiles for different clients (Go and Python) as well as a server Dockerfile.
• Holds client implementations (both in Go and Python) using GRPC and REST patterns.
• Includes configuration for Triton Inference Server (via directories like triton/infer_bls and triton/whisper_medium).
• Has converter scripts and a “prepare.sh” to set up environments.
• A “tllm” submodule:
• Contains conversion scripts and a separate implementation of Whisper for TRT-LLM.
• Additional directories like “media” and “sample” hold assets for performance testing and example audio files.
Key Components
a. ASR Core Engine:
• ONNX Pipeline:
– Feature Extraction (src/flash_whisper/onnx/extractor)
– Preprocessing & Mel Processing (within the extractor)
– Processing logic (src/flash_whisper/onnx/processor)
– Tokenization (src/flash_whisper/onnx/tokenizer)
– Overall orchestration (src/flash_whisper/onnx/whisper.py)
• TensorRT-LLM Pipeline (under tllm):
– Model conversion scripts (convert_ckpt_hf_to_model.py, convert_model_to_safetensors.py, etc.)
– Inference logic (src/flash_whisper/tllm/whisper.py, tokenizer.py, fbank.py)
b. Serving Layer:
• Server Components:
– Docker Compose setup (docker-compose.yml) indicating container orchestration.
– Server (possibly a FastAPI server for Python and a Fiber/Go server for Go clients).
– Integration with Triton Inference Server for high-performance model serving.
• Client Components:
– Python client (src/flash_whisper/serving/client/python)
– Go client (src/flash_whisper/serving/client/go with its generated GRPC stubs)
c. Model Conversion / Preparation Layer:
• Converter modules under “serving/converter” and “tllm/converter” that handle converting models from Hugging Face checkpoints into ONNX or TRT-compatible formats.
d. Infrastructure / Deployment:
• Dockerfiles (multiple per client/server) and shell scripts (prepare.sh, build.sh) for setup automation.
• Configuration for different environments (e.g., onnx, Triton configurations via pbtxt files).
Relationships and Data Flow
• Clients (Python / Go) send audio data (currently supporting WAV) to the server endpoint.
• The server dispatches requests to the ASR core engine.
– Within the ASR engine, the audio data flows through the extractor, processor, and tokenizer for transcription.
– Depending on deployment, either the ONNX pipeline or TensorRT-LLM pipeline is used.
– There is a conversion/ preparation step (when initial setup or model update is required) that generates the required model files.
• Conversion modules interact with the model storage (directories provided by external messengers or environment variables like OUTPUT_DIR).
• The server then returns transcribed text to the client.
Client → API/Server Layer → Model Inference Pipeline → Model Conversion module (if applicable) → Back to Server & then to Client.
Architectural Patterns and Design Principles
• Separation of Concerns:
• Clear distinction between model inference (ONNX vs. TRT-LLM) and service layer operations.
• Dedicated modules for feature extraction, processing, and tokenization.
• Containerization:
• The Dockerfiles and docker-compose setup ensure isolated deployment of microservices.
• Extensibility:
• The system is modular, which allows for the addition or update of pipelines (e.g., new conversion modules, additional clients).
• Interoperability:
• Clients in different languages leverage similar protocols (GRPC, REST) to interact with the same service.