How fluidinference/fluidaudio works
FluidAudio is a Swift SDK for local audio AI on Apple devices. Applications supply audio to speech-recognition, voice-activity, and diarization APIs, or text to speech-synthesis APIs; results are transcripts, speech regions, speaker-labelled segments, embeddings, or generated audio. Its main ASR paths include batch/sliding-window and streaming variants, with additional language-specific managers. Shared audio/model utilities support these capabilities, while model inference uses Core ML and the Apple Neural Engine. README capabilities are represented at subsystem level; sampled excerpts show selected pipeline details, not every manager’s wiring.
Main entry points: AudioSourceFactory.swift Audio sources · FluidAudioCLI.swift FluidAudio CLI · DiarizerManager.swift Online diarization · SlidingWindowAsrManager.swift ASR managers
Components 21
Shared runtime
- Audio sources
Sources/FluidAudio/Shared/AudioSourceFactory.swift - Model registry
Sources/FluidAudio/ModelRegistry.swift - Model downloads
Sources/FluidAudio/Shared/Download/ModelHub.swift
Speech recognition
- ASR managers
Sources/FluidAudio/ASR/Parakeet/SlidingWindow/SlidingWindowAsrManager.swift - TDT pipeline
Sources/FluidAudio/ASR/Parakeet/SlidingWindow/TDT/AsrManager+Pipeline.swift - Streaming ASR
Sources/FluidAudio/ASR/Parakeet/Streaming/Nemotron/StreamingNemotronAsrManager+Pipeline.swift - Cohere ASR
Sources/FluidAudio/ASR/Cohere/CoherePipeline.swift - Paraformer ASR
Sources/FluidAudio/ASR/Paraformer/ParaformerManager.swift - Text normalization
Sources/FluidAudio/ITN/TextNormalizer.swift
Speech analysis
- Voice activity
Sources/FluidAudio/VAD/VadManager.swift - Online diarization
Sources/FluidAudio/Diarizer/Core/DiarizerManager.swift - Speaker tracking
Sources/FluidAudio/Diarizer/Clustering/SpeakerManager.swift - Offline diarization
Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerManager.swift - Speaker embeddings
Sources/FluidAudio/Speaker/CampPlusEmbedder.swift
Speech synthesis
- Kokoro TTS
Sources/FluidAudio/TTS/KokoroAne/KokoroAneManager.swift - Pocket TTS
Sources/FluidAudio/TTS/PocketTTS/PocketTtsManager.swift - Synthesis state
Sources/FluidAudio/TTS/PocketTTS/Pipeline/PocketTtsStateEngine.swift - TTS managers
Sources/FluidAudio/TTS/Chatterbox/ChatterboxManager.swift
Command line
- FluidAudio CLI
Sources/FluidAudioCLI/FluidAudioCLI.swift
Other components
- Application
- Core ML / ANE
Connections 33
- Application → Audio sources: provides audio
- Application → ASR managers: requests transcription
- Application → Voice activity: requests detection
- Application → Online diarization: requests diarization
- Application → Offline diarization: requests batch analysis
- Application → Speaker embeddings: requests embeddings
- Application → Kokoro TTS: provides text
- Application → Pocket TTS: provides text
- Application → TTS managers: requests synthesis
- Audio sources → ASR managers: supplies audio
- Audio sources → Voice activity: supplies audio
- Audio sources → Online diarization: supplies audio
- Audio sources → Offline diarization: supplies audio
- Model registry → Model downloads: obtains assets
- Model downloads → Model registry: returns assets
- ASR managers → TDT pipeline: runs windows
- TDT pipeline → Core ML / ANE: runs inference
- Streaming ASR → Core ML / ANE: runs inference
- Cohere ASR → Core ML / ANE: runs inference
- Paraformer ASR → Core ML / ANE: runs inference
- Voice activity → Core ML / ANE: runs inference
- Online diarization → Speaker tracking: tracks speakers
- Online diarization → Core ML / ANE: runs inference
- Offline diarization → Core ML / ANE: runs inference
- Speaker embeddings → Core ML / ANE: runs inference
- Kokoro TTS → Core ML / ANE: runs inference
- Pocket TTS → Synthesis state: uses state
- Synthesis state → Core ML / ANE: runs inference
- TTS managers → Core ML / ANE: runs inference
- FluidAudio CLI → ASR managers: dispatches commands
- FluidAudio CLI → Voice activity: dispatches commands
- FluidAudio CLI → Offline diarization: dispatches commands
- FluidAudio CLI → TTS managers: dispatches commands