How liquid4all/liquid-audio works
1. Overall shape
- This is a Python package, liquid_audio, for speech-to-speech and speech-to-text/text-to-speech inference.
- The architecture is centered on a token-based audio language model, with separate components for preprocessing/postprocessing and several model backends.
2. Public entry points
- The top-level package exposes the main user-facing pieces described in the README: LFM2AudioModel, LFM2AudioProcessor, ChatState, and LFMModality.
- These form the primary boundary between application code and model internals.
3. Tokenization and preprocessing boundary
- processor.py, detokenizer.py, and utils.py handle conversion between human data and model tokens.
- Text is tokenized/detokenized, audio input is converted into features, and generated audio tokens are turned back into waveform output.
- The README makes this split explicit: the model operates on tokens only; the processor owns data conversion.
4. Conversation/session management
- ChatState is the orchestration layer for multi-turn input construction.
- It assembles system/user/assistant turns, applies the right chat template, and packages mixed text/audio history into the tensors consumed by generation.
5. Core generation model
- src/liquid_audio/model/lfm2_audio.py and the surrounding model/ package contain the main LFM2-Audio architecture.
- The model supports two generation modes:
- Interleaved: alternates text and audio tokens for conversational, low-latency speech-to-speech output.
- Sequential: emits one modality until a switch token is produced, suited for ASR and TTS.
6. Model building blocks
- The model/ subtree includes reusable deep-learning components such as:
- transformer.py
- mlp.py
- conformer/
- The Conformer implementation includes attention, subsampling, encoder, processor, and module utilities, indicating a separate audio-encoder pathway or related acoustic front-end.
7. Audio detokenization and codebook flow
- Generated audio is represented as multiple token streams corresponding to Mimi codebooks, with the README showing 8-codebook audio tensors.
- The processor’s decode path converts these code tokens into 24 kHz waveforms, which are then written to disk or played back.
- This is a key architectural boundary: the model emits discrete audio codes; waveform synthesis happens outside the model.
8. Moshi subpackage as the low-level engine
- src/liquid_audio/moshi/ is a substantial implementation layer derived from the Moshi/Kyutai codebase.
- It contains:
- models/ for LM, TTS, compression, and loaders
- modules/ for transformer, convolution, resampling, rope, streaming, LoRA, and SEANet-related pieces
- quantization/ for vector quantization primitives
- conditioners/ for text/tensor conditioning
- This subpackage appears to provide the detailed runtime and building blocks for token generation and audio compression/decompression.
9. Runtime and serving surfaces
- moshi/run_inference.py, moshi/run_tts.py, and moshi/server.py provide executable entry points around the core model stack.
- moshi/client.py, client_gradio.py, and client_utils.py indicate both programmatic and UI-facing client paths.
- The Gradio demo in src/liquid_audio/demo/ is an optional presentation layer built on top of the same core APIs.
10. Streaming and low-latency design
- The presence of streaming.py, sampling.py, and the README’s emphasis on “real time” and “low latency” show that incremental token emission is a first-class concern.
- This affects the architecture: generation is not a single forward pass returning a full sequence, but a token stream consumed as it arrives.
11. Compression and quantization
- The moshi/quantization/ and moshi/models/compression.py layers indicate that audio is represented compactly via learned discrete codes.
- This is important for the architecture diagram because audio I/O is mediated by compression codebooks rather than raw waveform generation.
12. External dependencies and execution environment
- The runtime is PyTorch-based, with torchaudio used in examples and flash-attn optional for attention acceleration.
- The repo is packaged with pyproject.toml, uses uv.lock, and includes GitHub Actions CI in .github/workflows/check.yml.
- Pretrained model weights are fetched from Hugging Face Hub, which is an external dependency in normal usage.
13. Tests and maintainability
- There are focused module tests under moshi/modules/*_test.py, suggesting core low-level blocks are validated separately from the higher-level generation flows.
- That split mirrors the architecture: reusable primitives are tested independently of full model execution.
14. Most important diagram boundaries
- A useful architecture diagram should separate:
- Application/demo layer: README examples, Gradio demo, client scripts
- Session assembly layer: ChatState
- Processor layer: text/audio conversion and decode
- Model layer: LFM2AudioModel generation logic
- Low-level Moshi stack: transformer/compression/streaming/quantization modules
- External assets/services: pretrained weights on Hugging Face, audio files in assets/
Main entry points: processor.py Processor · lm.py Moshi LM · lfm2_audio.py LFM2AudioModel · __init__.py liquid_audio
Components 20
Application
User-facing demos and entry scripts
- Demo chat · Gradio/chat-oriented presentation layer for interactive use.
src/liquid_audio/demo/chat.py - Demo model · Demo-specific wiring around the public model API.
src/liquid_audio/demo/model.py
Core API
Public package boundary and orchestration
- liquid_audio · Top-level package exports the main public API.
src/liquid_audio/__init__.py - ChatState · Assembles multi-turn conversational history and modality-aware prompts.
src/liquid_audio/processor.py - Processor · Converts between human inputs and model token tensors; owns decode paths.
src/liquid_audio/processor.py - Detokenizer · Turns generated tokens back into text/audio outputs.
src/liquid_audio/detokenizer.py - Utils · Shared package utilities used across preprocessing and demos.
src/liquid_audio/utils.py
Model Stack
LFM2-Audio model and reusable deep-learning blocks
- LFM2AudioModel · Main token-based audio language model for interleaved and sequential generation.
src/liquid_audio/model/lfm2_audio.py - Transformer · Core transformer blocks used by the audio language model.
src/liquid_audio/model/transformer.py - MLP · Shared feedforward layers used inside model blocks.
src/liquid_audio/model/mlp.py - Conformer · Audio-side encoder path with attention, subsampling, and preprocessing utilities.
src/liquid_audio/model/conformer/encoder.py
Moshi Engine
Low-level runtime, compression, and streaming implementation
- Moshi LM · Low-level language-model runtime and loading utilities.
src/liquid_audio/moshi/models/lm.py - Moshi TTS · Speech synthesis pathway built on the Moshi stack.
src/liquid_audio/moshi/models/tts.py - Compression · Learns discrete audio codebooks and codec-style representations.
src/liquid_audio/moshi/models/compression.py - Quantization · Vector-quantization primitives backing compact audio token streams.
src/liquid_audio/moshi/quantization/vq.py - Modules · Reusable low-level components: conv, resample, rope, streaming, LoRA, SEANet.
src/liquid_audio/moshi/modules/transformer.py - Conditioners · Text and tensor conditioning layers for prompt and control signals.
src/liquid_audio/moshi/conditioners/text.py - CLI/server · Inference, TTS, server, and client entry scripts.
src/liquid_audio/moshi/server.py
Infra
Packaging, CI, and external assets
- Assets · Example audio inputs used by docs and demos.
assets/asr.wav - Model hub · Pretrained weights are fetched from Hugging Face at runtime.
Connections 23
- liquid_audio → ChatState: exports · Top-level API exposes conversational state assembly.
- liquid_audio → Processor: exports · Top-level API exposes the processor boundary.
- liquid_audio → LFM2AudioModel: exports · Top-level API exposes the main model class.
- liquid_audio → Utils: exports · Shared helpers are part of the public package surface.
- ChatState → Processor: builds inputs · Session state is converted into model-ready tensors through the processor layer.
- Processor → Detokenizer: decodes · Processor delegates token-to-output conversion to detokenization.
- Processor → Conformer: audio features · Audio input is converted into features for the model stack.
- Processor → Utils: uses · Shares tokenization and conversion helpers.
- LFM2AudioModel → Transformer: uses · Core generation relies on transformer sequence blocks.
- LFM2AudioModel → MLP: uses · Feedforward sublayers are part of model internals.
- LFM2AudioModel → Conformer: conditions on · Audio-side pathway supports speech inputs and multimodal generation.
- Demo chat → Demo model: wraps · Interactive demo binds UI to the demo model adapter.
- Demo model → LFM2AudioModel: calls · Demo code uses the public model API.
- Moshi LM → Modules: composes · LM runtime is built from shared low-level blocks.
- Moshi TTS → Compression: uses · TTS path depends on discrete audio compression codes.
- Compression → Quantization: implements · Compression layer is backed by vector quantization.
- Moshi LM → Conditioners: conditions on · Runtime incorporates text and tensor conditioning.
- CLI/server → Moshi LM: runs · Inference/server entrypoints invoke the Moshi runtime.
- CLI/server → Moshi TTS: runs · TTS entrypoints invoke the Moshi synthesis path.
- Processor → Model hub: loads weights · Model assets are retrieved from the external hub in normal usage.
- CLI/server → Assets: uses samples · Examples and demos consume bundled audio fixtures.
- Moshi LM → Compression: emits codes · Language generation operates over compact audio token streams.
- Moshi LM → Modules: streams · Low-latency generation depends on streaming and transformer utilities.