How kyutai-labs/moshi works
Moshi is a real-time, full-duplex speech-and-text dialogue system, with PyTorch, MLX, and Rust inference stacks and a browser client. The main user path is joining the demo queue, entering a conversation, streaming microphone audio to an inference server, and receiving generated speech and text. Mimi handles streaming audio coding around model inference. The README also identifies TTS, ASR, local inference, and CLI use. Server excerpts show streaming routes and audio processing; details of queue wiring and several backend internals are inferred only from documented capabilities and the file tree.
Main entry points: server.py PyTorch server · main.rs Rust server · useSocket.ts Socket hook · useServerAudio.ts Server audio
Components 23
Web client
- Queue screen
client/src/pages/Queue/Queue.tsx - Queue API client
client/src/pages/Queue/api/client.ts - Conversation UI
client/src/pages/Conversation/Conversation.tsx - Socket hook
client/src/pages/Conversation/hooks/useSocket.ts - User audio
client/src/pages/Conversation/hooks/useUserAudio.ts - Server audio
client/src/pages/Conversation/hooks/useServerAudio.ts - Message protocol
client/src/protocol/encoder.ts
Serving and APIs
- PyTorch server
moshi/moshi/server.py - Rust server
rust/moshi-server/src/main.rs
PyTorch stack
- PyTorch inference
moshi/moshi/run_inference.py - PyTorch language model
moshi/moshi/models/lm.py - PyTorch Mimi codec
moshi/moshi/models/compression.py - PyTorch TTS
moshi/moshi/models/tts.py
MLX stack
- MLX inference
moshi_mlx/moshi_mlx/run_inference.py - MLX language model
moshi_mlx/moshi_mlx/models/lm.py - MLX Mimi codec
moshi_mlx/moshi_mlx/models/mimi.py
Rust stack
- Rust model core
rust/moshi-core/src/lib.rs - Rust Mimi codec
rust/moshi-core/src/mimi.rs - Rust TTS
rust/moshi-core/src/tts.rs - Rust CLI
rust/moshi-cli/src/main.rs
Other components
- User
- Queue service
- Speech output
Connections 26
- User → Queue screen: joins queue
- Queue screen → Queue API client: requests access
- Queue API client → Queue service: calls queue API
- User → Conversation UI: speaks
- Conversation UI → Socket hook: opens stream
- User audio → Socket hook: sends audio
- Socket hook → PyTorch server: streams audio
- Socket hook → Rust server: streams audio
- PyTorch server → PyTorch Mimi codec: encodes input
- PyTorch server → PyTorch language model: runs dialogue
- PyTorch language model → PyTorch Mimi codec: generates audio codes
- PyTorch inference → PyTorch language model: runs inference
- PyTorch inference → PyTorch Mimi codec: loads codec
- PyTorch server → PyTorch TTS: serves speech synthesis
- Rust server → Rust model core: runs model services
- Rust server → Rust Mimi codec: serves codec streams
- Rust server → Rust TTS: serves synthesis
- MLX inference → MLX language model: runs inference
- MLX inference → MLX Mimi codec: encodes and decodes
- Rust model core → Rust Mimi codec: uses codec
- Rust CLI → Rust model core: generates locally
- Server audio → Message protocol: decodes messages
- PyTorch server → Server audio: sends audio and text
- Rust server → Server audio: streams results
- Server audio → Speech output: plays response
- Speech output → User: returns speech