# liquid4all/liquid-audio architecture

> Architecture diagram of the GitHub repository liquid4all/liquid-audio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/liquid4all/liquid-audio
- Repository: https://github.com/liquid4all/liquid-audio

## Overview

1. **Overall shape**
   - This is a Python package, `liquid_audio`, for speech-to-speech and speech-to-text/text-to-speech inference.
   - The architecture is centered on a token-based audio language model, with separate components for preprocessing/postprocessing and several model backends.

2. **Public entry points**
   - The top-level package exposes the main user-facing pieces described in the README: `LFM2AudioModel`, `LFM2AudioProcessor`, `ChatState`, and `LFMModality`.
   - These form the primary boundary between application code and model internals.

3. **Tokenization and preprocessing boundary**
   - `processor.py`, `detokenizer.py`, and `utils.py` handle conversion between human data and model tokens.
   - Text is tokenized/detokenized, audio input is converted into features, and generated audio tokens are turned back into waveform output.
   - The README makes this split explicit: the model operates on tokens only; the processor owns data conversion.

4. **Conversation/session management**
   - `ChatState` is the orchestration layer for multi-turn input construction.
   - It assembles system/user/assistant turns, applies the right chat template, and packages mixed text/audio history into the tensors consumed by generation.

5. **Core generation model**
   - `src/liquid_audio/model/lfm2_audio.py` and the surrounding `model/` package contain the main LFM2-Audio architecture.
   - The model supports two generation modes:
     - **Interleaved**: alternates text and audio tokens for conversational, low-latency speech-to-speech output.
     - **Sequential**: emits one modality until a switch token is produced, suited for ASR and TTS.

6. **Model building blocks**
   - The `model/` subtree includes reusable deep-learning components such as:
     - `transformer.py`
     - `mlp.py`
     - `conformer/`
   - The Conformer implementation includes attention, subsampling, encoder, processor, and module utilities, indicating a separate audio-encoder pathway or related acoustic front-end.

7. **Audio detokenization and codebook flow**
   - Generated audio is represented as multiple token streams corresponding to Mimi codebooks, with the README showing 8-codebook audio tensors.
   - The processor’s `decode` path converts these code tokens into 24 kHz waveforms, which are then written to disk or played back.
   - This is a key architectural boundary: the model emits discrete audio codes; waveform synthesis happens outside the model.

8. **Moshi subpackage as the low-level engine**
   - `src/liquid_audio/moshi/` is a substantial implementation layer derived from the Moshi/Kyutai codebase.
   - It contains:
     - `models/` for LM, TTS, compression, and loaders
     - `modules/` for transformer, convolution, resampling, rope, streaming, LoRA, and SEANet-related pieces
     - `quantization/` for vector quantization primitives
     - `conditioners/` for text/tensor conditioning
   - This subpackage appears to provide the detailed runtime and building blocks for token generation and audio compression/decompression.

9. **Runtime and serving surfaces**
   - `moshi/run_inference.py`, `moshi/run_tts.py`, and `moshi/server.py` provide executable entry points around the core model stack.
   - `moshi/client.py`, `client_gradio.py`, and `client_utils.py` indicate both programmatic and UI-facing client paths.
   - The Gradio demo in `src/liquid_audio/demo/` is an optional presentation layer built on top of the same core APIs.

10. **Streaming and low-latency design**
   - The presence of `streaming.py`, `sampling.py`, and the README’s emphasis on “real time” and “low latency” show that incremental token emission is a first-class concern.
   - This affects the architecture: generation is not a single forward pass returning a full sequence, but a token stream consumed as it arrives.

11. **Compression and quantization**
   - The `moshi/quantization/` and `moshi/models/compression.py` layers indicate that audio is represented compactly via learned discrete codes.
   - This is important for the architecture diagram because audio I/O is mediated by compression codebooks rather than raw waveform generation.

12. **External dependencies and execution environment**
   - The runtime is PyTorch-based, with `torchaudio` used in examples and `flash-attn` optional for attention acceleration.
   - The repo is packaged with `pyproject.toml`, uses `uv.lock`, and includes GitHub Actions CI in `.github/workflows/check.yml`.
   - Pretrained model weights are fetched from Hugging Face Hub, which is an external dependency in normal usage.

13. **Tests and maintainability**
   - There are focused module tests under `moshi/modules/*_test.py`, suggesting core low-level blocks are validated separately from the higher-level generation flows.
   - That split mirrors the architecture: reusable primitives are tested independently of full model execution.

14. **Most important diagram boundaries**
   - A useful architecture diagram should separate:
     - **Application/demo layer**: README examples, Gradio demo, client scripts
     - **Session assembly layer**: `ChatState`
     - **Processor layer**: text/audio conversion and decode
     - **Model layer**: `LFM2AudioModel` generation logic
     - **Low-level Moshi stack**: transformer/compression/streaming/quantization modules
     - **External assets/services**: pretrained weights on Hugging Face, audio files in `assets/`

Main entry points: [`src/liquid_audio/processor.py`](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/processor.py) (Processor), [`src/liquid_audio/moshi/models/lm.py`](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/moshi/models/lm.py) (Moshi LM), [`src/liquid_audio/model/lfm2_audio.py`](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/model/lfm2_audio.py) (LFM2AudioModel), [`src/liquid_audio/__init__.py`](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/__init__.py) (liquid_audio)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_app["Application"]
  node_demo_chat["Demo chat<br/>ui demo<br/>[chat.py]"]
  node_demo_model["Demo model<br/>demo wiring<br/>[model.py]"]
end

subgraph group_core["Core API"]
  node_pkg_init["liquid_audio<br/>package root<br/>[__init__.py]"]
  node_chat_state["ChatState<br/>session builder<br/>[processor.py]"]
  node_processor["Processor<br/>audio/text I-O<br/>[processor.py]"]
  node_detokenizer["Detokenizer<br/>token decode<br/>[detokenizer.py]"]
  node_utils["Utils<br/>helpers<br/>[utils.py]"]
end

subgraph group_model["Model Stack"]
  node_lfm2_audio["LFM2AudioModel<br/>generation model<br/>[lfm2_audio.py]"]
  node_transformer["Transformer<br/>sequence core<br/>[transformer.py]"]
  node_mlp["MLP<br/>feedforward block<br/>[mlp.py]"]
  node_conformer["Conformer<br/>audio encoder<br/>[encoder.py]"]
end

subgraph group_moshi["Moshi Engine"]
  node_moshi_lm["Moshi LM<br/>lm runtime<br/>[lm.py]"]
  node_moshi_tts["Moshi TTS<br/>tts runtime<br/>[tts.py]"]
  node_compression["Compression<br/>audio codes<br/>[compression.py]"]
  node_vq["Quantization<br/>vq primitives<br/>[vq.py]"]
  node_modules["Modules<br/>runtime blocks<br/>[transformer.py]"]
  node_conditioners["Conditioners<br/>conditioning<br/>[text.py]"]
  node_cli_server["CLI/server<br/>entrypoints<br/>[server.py]"]
end

subgraph group_infra["Infra"]
  node_assets["Assets<br/>sample audio<br/>[asr.wav]"]
  node_hub(("Model hub<br/>external weights"))
end

node_pkg_init -->|"exports"| node_chat_state
node_pkg_init -->|"exports"| node_processor
node_pkg_init -->|"exports"| node_lfm2_audio
node_pkg_init -->|"exports"| node_utils
node_chat_state -->|"builds inputs"| node_processor
node_processor -->|"decodes"| node_detokenizer
node_processor -->|"audio features"| node_conformer
node_processor -->|"uses"| node_utils
node_lfm2_audio -->|"uses"| node_transformer
node_lfm2_audio -->|"uses"| node_mlp
node_lfm2_audio -->|"conditions on"| node_conformer
node_demo_chat -->|"wraps"| node_demo_model
node_demo_model -->|"calls"| node_lfm2_audio
node_moshi_lm -->|"composes"| node_modules
node_moshi_tts -->|"uses"| node_compression
node_compression -->|"implements"| node_vq
node_moshi_lm -->|"conditions on"| node_conditioners
node_cli_server -->|"runs"| node_moshi_lm
node_cli_server -->|"runs"| node_moshi_tts
node_processor -->|"loads weights"| node_hub
node_cli_server -->|"uses samples"| node_assets
node_moshi_lm -->|"emits codes"| node_compression
node_moshi_lm -->|"streams"| node_modules

click node_pkg_init "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/__init__.py"
click node_chat_state "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/processor.py"
click node_processor "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/processor.py"
click node_detokenizer "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/detokenizer.py"
click node_utils "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/utils.py"
click node_lfm2_audio "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/model/lfm2_audio.py"
click node_transformer "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/model/transformer.py"
click node_mlp "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/model/mlp.py"
click node_conformer "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/model/conformer/encoder.py"
click node_demo_chat "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/demo/chat.py"
click node_demo_model "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/demo/model.py"
click node_moshi_lm "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/moshi/models/lm.py"
click node_moshi_tts "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/moshi/models/tts.py"
click node_compression "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/moshi/models/compression.py"
click node_vq "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/moshi/quantization/vq.py"
click node_modules "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/moshi/modules/transformer.py"
click node_conditioners "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/moshi/conditioners/text.py"
click node_cli_server "https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/moshi/server.py"
click node_assets "https://github.com/liquid4all/liquid-audio/blob/main/assets/asr.wav"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_demo_chat,node_demo_model toneBlue
class node_pkg_init,node_chat_state,node_processor,node_detokenizer,node_utils toneAmber
class node_lfm2_audio,node_transformer,node_mlp,node_conformer toneMint
class node_moshi_lm,node_moshi_tts,node_compression,node_vq,node_modules,node_conditioners,node_cli_server toneRose
class node_assets,node_hub toneIndigo
```

## Components

### Application

User-facing demos and entry scripts

- [Demo chat](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/demo/chat.py): ui demo · Gradio/chat-oriented presentation layer for interactive use. · `src/liquid_audio/demo/chat.py`
- [Demo model](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/demo/model.py): demo wiring · Demo-specific wiring around the public model API. · `src/liquid_audio/demo/model.py`

### Core API

Public package boundary and orchestration

- [liquid_audio](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/__init__.py): package root · Top-level package exports the main public API. · `src/liquid_audio/__init__.py`
- [ChatState](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/processor.py): session builder · Assembles multi-turn conversational history and modality-aware prompts. · `src/liquid_audio/processor.py`
- [Processor](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/processor.py): audio/text I-O · Converts between human inputs and model token tensors; owns decode paths. · `src/liquid_audio/processor.py`
- [Detokenizer](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/detokenizer.py): token decode · Turns generated tokens back into text/audio outputs. · `src/liquid_audio/detokenizer.py`
- [Utils](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/utils.py): helpers · Shared package utilities used across preprocessing and demos. · `src/liquid_audio/utils.py`

### Model Stack

LFM2-Audio model and reusable deep-learning blocks

- [LFM2AudioModel](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/model/lfm2_audio.py): generation model · Main token-based audio language model for interleaved and sequential generation. · `src/liquid_audio/model/lfm2_audio.py`
- [Transformer](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/model/transformer.py): sequence core · Core transformer blocks used by the audio language model. · `src/liquid_audio/model/transformer.py`
- [MLP](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/model/mlp.py): feedforward block · Shared feedforward layers used inside model blocks. · `src/liquid_audio/model/mlp.py`
- [Conformer](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/model/conformer/encoder.py): audio encoder · Audio-side encoder path with attention, subsampling, and preprocessing utilities. · `src/liquid_audio/model/conformer/encoder.py`

### Moshi Engine

Low-level runtime, compression, and streaming implementation

- [Moshi LM](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/moshi/models/lm.py): lm runtime · Low-level language-model runtime and loading utilities. · `src/liquid_audio/moshi/models/lm.py`
- [Moshi TTS](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/moshi/models/tts.py): tts runtime · Speech synthesis pathway built on the Moshi stack. · `src/liquid_audio/moshi/models/tts.py`
- [Compression](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/moshi/models/compression.py): audio codes · Learns discrete audio codebooks and codec-style representations. · `src/liquid_audio/moshi/models/compression.py`
- [Quantization](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/moshi/quantization/vq.py): vq primitives · Vector-quantization primitives backing compact audio token streams. · `src/liquid_audio/moshi/quantization/vq.py`
- [Modules](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/moshi/modules/transformer.py): runtime blocks · Reusable low-level components: conv, resample, rope, streaming, LoRA, SEANet. · `src/liquid_audio/moshi/modules/transformer.py`
- [Conditioners](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/moshi/conditioners/text.py): conditioning · Text and tensor conditioning layers for prompt and control signals. · `src/liquid_audio/moshi/conditioners/text.py`
- [CLI/server](https://github.com/liquid4all/liquid-audio/blob/main/src/liquid_audio/moshi/server.py): entrypoints · Inference, TTS, server, and client entry scripts. · `src/liquid_audio/moshi/server.py`

### Infra

Packaging, CI, and external assets

- [Assets](https://github.com/liquid4all/liquid-audio/blob/main/assets/asr.wav): sample audio · Example audio inputs used by docs and demos. · `assets/asr.wav`
- **Model hub**: external weights · Pretrained weights are fetched from Hugging Face at runtime.

## Connections

- liquid_audio → ChatState: exports · Top-level API exposes conversational state assembly.
- liquid_audio → Processor: exports · Top-level API exposes the processor boundary.
- liquid_audio → LFM2AudioModel: exports · Top-level API exposes the main model class.
- liquid_audio → Utils: exports · Shared helpers are part of the public package surface.
- ChatState → Processor: builds inputs · Session state is converted into model-ready tensors through the processor layer.
- Processor → Detokenizer: decodes · Processor delegates token-to-output conversion to detokenization.
- Processor → Conformer: audio features · Audio input is converted into features for the model stack.
- Processor → Utils: uses · Shares tokenization and conversion helpers.
- LFM2AudioModel → Transformer: uses · Core generation relies on transformer sequence blocks.
- LFM2AudioModel → MLP: uses · Feedforward sublayers are part of model internals.
- LFM2AudioModel → Conformer: conditions on · Audio-side pathway supports speech inputs and multimodal generation.
- Demo chat → Demo model: wraps · Interactive demo binds UI to the demo model adapter.
- Demo model → LFM2AudioModel: calls · Demo code uses the public model API.
- Moshi LM → Modules: composes · LM runtime is built from shared low-level blocks.
- Moshi TTS → Compression: uses · TTS path depends on discrete audio compression codes.
- Compression → Quantization: implements · Compression layer is backed by vector quantization.
- Moshi LM → Conditioners: conditions on · Runtime incorporates text and tensor conditioning.
- CLI/server → Moshi LM: runs · Inference/server entrypoints invoke the Moshi runtime.
- CLI/server → Moshi TTS: runs · TTS entrypoints invoke the Moshi synthesis path.
- Processor → Model hub: loads weights · Model assets are retrieved from the external hub in normal usage.
- CLI/server → Assets: uses samples · Examples and demos consume bundled audio fixtures.
- Moshi LM → Compression: emits codes · Language generation operates over compact audio token streams.
- Moshi LM → Modules: streams · Low-latency generation depends on streaming and transformer utilities.
