How dimastatz/whisper-flow works
1. Repository shape
- This is a Python package plus a small service layer for real-time speech-to-text.
- The core package is whisperflow/; everything else is supporting infrastructure: tests, docs, Docker, and developer tooling.
2. Primary runtime boundary
- The main runtime is a FastAPI-based transcription service in whisperflow/fast_server.py.
- It exposes:
- a WebSocket streaming endpoint for live audio,
- a health check endpoint,
- and an HTTP batch endpoint for transcribing PCM chunks.
- This server is the integration boundary for external clients.
3. Streaming transcription pipeline
- The central flow is: audio chunks enter → buffering/session logic groups them → Whisper transcription runs → partial/final text is sent back.
- whisperflow/streaming.py appears to own the session orchestration, including incremental updates and lifecycle management.
- The README shows a callback-driven design: one callback to transcribe chunks, another to send results back to the client.
4. Model/transcription subsystem
- whisperflow/transcriber.py is the model-facing layer.
- It loads the Whisper model and performs transcription over PCM audio chunks.
- The packaged model file whisperflow/models/tiny.en.pt indicates the repository ships with a default lightweight model for local use and demos.
5. Audio input boundary
- whisperflow/audio/microphone.py is the local capture path for live microphone audio.
- The README and example script suggest it is used for interactive/demo scenarios rather than being required for server operation.
- This creates a second ingestion path distinct from network streaming: local microphone → PCM chunks → same transcription pipeline.
6. Data format contract
- The system expects PCM audio with a fixed contract: 16 kHz, mono, 16-bit signed integer.
- That format boundary is important because it simplifies the transcriber and makes the service predictable for both WebSocket clients and benchmark inputs.
7. Response semantics
- Output is not just a single transcript; the system emits partial results over time and then a final segment completion event.
- Each result includes timing/partiality metadata, which is what enables low-latency “live” UX rather than batch-only transcription.
8. Benchmarking and evaluation
- tests/benchmark/ and the README’s LibriSpeech-based benchmark establish a performance-focused subsystem.
- The main metrics are latency between partial updates and word error rate (WER).
- This is architecturally important because it validates the streaming segmentation strategy, not just raw transcription accuracy.
9. Test coverage of the core boundaries
- The tests are organized around the main modules: audio handling, streaming session behavior, transcriber behavior, chat-room/session behavior, and benchmark execution.
- This suggests the repo’s critical logic is concentrated in a few boundaries and the tests are aligned to them rather than to many small utilities.
10. Packaging and deployment
- The project is packaged for installation via setup.py and dependency pinning in requirements.txt.
- Docker support (Dockerfile, Dockerfile.test, GitHub Actions workflow) indicates the service is intended to run both locally and in containerized environments.
- run.sh is the operational entrypoint for setup, testing, server startup, benchmark runs, and container workflows.
11. Developer/runtime environment
- .devcontainer and CI workflows show the repo is designed for reproducible development.
- The practical runtime stack is Python + FastAPI + Whisper + audio I/O (PyAudio/PortAudio), with the model file bundled in-repo for convenience.
12. How to draw the architecture
- A good diagram should show two ingress paths: WebSocket/audio stream clients and local microphone/demo input.
- Both feed into the stream/session layer, which calls the Whisper transcriber using the bundled model.
- Results flow back out through the server callbacks/WebSocket, while benchmark/tests sit alongside the core pipeline as validation and performance verification.
Main entry points: streaming.py Streaming · fast_server.py Fast server · microphone.py Microphone · test_benchmark.py Benchmarks
Components 22
Core runtime
Main transcription service and pipeline
- Fast server · FastAPI boundary with WebSocket live transcription, health, and batch HTTP endpoints
whisperflow/fast_server.py - Streaming · Orchestrates buffering, incremental updates, and streaming session lifecycle
whisperflow/streaming.py - Transcriber · Loads Whisper and transcribes PCM audio chunks
whisperflow/transcriber.py - Chat room · Result delivery and session coordination abstraction used by streaming flows
whisperflow/chat_room.py - Tiny model · Default lightweight Whisper model shipped with the package
whisperflow/models/tiny.en.pt - PCM contract · 16 kHz mono 16-bit signed PCM input expected by the pipeline
Ingress paths
External and local audio entry points
- Microphone · Local live audio capture path for demos and interactive use
whisperflow/audio/microphone.py - WebSocket clients · External live transcription clients connected to the server
- HTTP clients · External batch transcription callers sending PCM chunks
- Mic users · Local users running the microphone example or interactive capture
tests/examples/mic_transcribe.py
Validation
Tests and benchmark coverage
- Audio tests · Coverage for audio capture and format handling
tests/audio/test_audio.py - Stream tests · Coverage for streaming session behavior and result flow
tests/test_streaming.py - Transcriber tests · Coverage for Whisper model loading and transcription behavior
tests/test_transcriber.py - Chat tests · Coverage for chat-room/session coordination
tests/test_chat_room.py - Benchmarks · Streaming benchmark checks latency and WER over speech samples
tests/benchmark/test_benchmark.py - Speech data · Reference audio and transcript sample used by evaluation
tests/resources/3081-166546-0000.wav
Packaging
Build, deploy, and developer tooling
- Run script · Operational script for setup, tests, server startup, and benchmark runs
run.sh - Setup · Package installation metadata and build configuration
setup.py - Requirements · Pinned Python dependencies for runtime and tooling
requirements.txt - Dockerfile · Container image for the service runtime
Dockerfile - Test image · Container image used for test execution
Dockerfile.test - CI workflow · GitHub Actions workflow for building container images
.github/workflows/docker-image.yml
Connections 24
- WebSocket clients → Fast server: stream audio · Live clients connect to the WebSocket transcription endpoint
- HTTP clients → Fast server: batch PCM · HTTP callers send chunked PCM for transcription
- Fast server → Streaming: open session · Server delegates live requests into the streaming session engine
- Streaming → Transcriber: transcribe · Session engine invokes the model layer on buffered chunks
- Transcriber → Tiny model: loads model · Whisper transcriber uses the bundled default model when available
- Streaming → Chat room: emit results · Streaming orchestrator sends partial and final results through the session/result adapter
- Chat room → Fast server: return text · Results flow back to the server for client delivery
- Microphone → PCM contract: capture PCM · Local microphone input is normalized to the repo's PCM contract
- Mic users → Microphone: use demo · Example users run the local capture path
- Microphone → Streaming: feed chunks · Captured audio enters the same streaming pipeline as network audio
- Streaming → PCM contract: expects · Streaming logic is built around the fixed PCM format boundary
- Audio tests → Microphone: verify · Tests cover audio input handling
- Stream tests → Streaming: verify · Tests cover session orchestration and incremental result behavior
- Transcriber tests → Transcriber: verify · Tests cover transcription behavior
- Chat tests → Chat room: verify · Tests cover session/result coordination
- Benchmarks → Streaming: measure · Benchmark suite evaluates the streaming segmentation strategy
- Benchmarks → Speech data: uses data · Benchmarking consumes the bundled speech sample and transcript
- Run script → Fast server: start · Operational script launches the service
- Run script → Benchmarks: run perf · Operational script runs benchmark workflows
- Setup → Requirements: installs deps · Packaging pulls in project dependencies
- Dockerfile → Fast server: package service · Container image assembles the runtime around the server
- Test image → Benchmarks: test image · Test container supports validation workflows
- CI workflow → Dockerfile: builds · CI automates container image builds
- CI workflow → Test image: builds · CI also builds the test container image