# kyutai-labs/moshi architecture

> Architecture diagram of the GitHub repository kyutai-labs/moshi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kyutai-labs/moshi
- Repository: https://github.com/kyutai-labs/moshi

## Overview

Moshi is a real-time, full-duplex speech-and-text dialogue system, with PyTorch, MLX, and Rust inference stacks and a browser client. The main user path is joining the demo queue, entering a conversation, streaming microphone audio to an inference server, and receiving generated speech and text. Mimi handles streaming audio coding around model inference. The README also identifies TTS, ASR, local inference, and CLI use. Server excerpts show streaming routes and audio processing; details of queue wiring and several backend internals are inferred only from documented capabilities and the file tree.

Main entry points: [`moshi/moshi/server.py`](https://github.com/kyutai-labs/moshi/blob/main/moshi/moshi/server.py) (PyTorch server), [`rust/moshi-server/src/main.rs`](https://github.com/kyutai-labs/moshi/blob/main/rust/moshi-server/src/main.rs) (Rust server), [`client/src/pages/Conversation/hooks/useSocket.ts`](https://github.com/kyutai-labs/moshi/blob/main/client/src/pages/Conversation/hooks/useSocket.ts) (Socket hook), [`client/src/pages/Conversation/hooks/useServerAudio.ts`](https://github.com/kyutai-labs/moshi/blob/main/client/src/pages/Conversation/hooks/useServerAudio.ts) (Server audio)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_web["Web client"]
  node_queue_ui["Queue screen<br/>[Queue.tsx]"]
  node_queue_api["Queue API client<br/>[client.ts]"]
  node_conversation["Conversation UI<br/>[Conversation.tsx]"]
  node_socket["Socket hook<br/>[useSocket.ts]"]
  node_audio_in["User audio<br/>[useUserAudio.ts]"]
  node_audio_out["Server audio<br/>[useServerAudio.ts]"]
  node_protocol["Message protocol<br/>[encoder.ts]"]
end

subgraph group_serve["Serving and APIs"]
  node_pytorch_server["PyTorch server<br/>[server.py]"]
  node_rust_server["Rust server<br/>[main.rs]"]
end

subgraph group_torch["PyTorch stack"]
  node_torch_entry["PyTorch inference<br/>[run_inference.py]"]
  node_torch_lm["PyTorch language model<br/>[lm.py]"]
  node_torch_mimi["PyTorch Mimi codec<br/>[compression.py]"]
  node_torch_tts["PyTorch TTS<br/>[tts.py]"]
end

subgraph group_mlx["MLX stack"]
  node_mlx_entry["MLX inference<br/>[run_inference.py]"]
  node_mlx_lm["MLX language model<br/>[lm.py]"]
  node_mlx_mimi["MLX Mimi codec<br/>[mimi.py]"]
end

subgraph group_rust["Rust stack"]
  node_rust_core["Rust model core<br/>[lib.rs]"]
  node_rust_mimi["Rust Mimi codec<br/>[mimi.rs]"]
  node_rust_tts["Rust TTS<br/>[tts.rs]"]
  node_cli["Rust CLI<br/>[main.rs]"]
end

node_user(("User"))
node_queue_service{{"Queue service"}}
node_speech(("Speech output"))

node_user -->|"joins queue"| node_queue_ui
node_queue_ui -->|"requests access"| node_queue_api
node_queue_api -->|"calls queue API"| node_queue_service
node_user -->|"speaks"| node_conversation
node_conversation -->|"opens stream"| node_socket
node_audio_in -->|"sends audio"| node_socket
node_socket -.->|"streams audio"| node_pytorch_server
node_socket -.->|"streams audio"| node_rust_server
node_pytorch_server -->|"encodes input"| node_torch_mimi
node_pytorch_server -->|"runs dialogue"| node_torch_lm
node_torch_lm -->|"generates audio codes"| node_torch_mimi
node_torch_entry -->|"runs inference"| node_torch_lm
node_torch_entry -->|"loads codec"| node_torch_mimi
node_pytorch_server -.->|"serves speech synthesis"| node_torch_tts
node_rust_server -->|"runs model services"| node_rust_core
node_rust_server -->|"serves codec streams"| node_rust_mimi
node_rust_server -->|"serves synthesis"| node_rust_tts
node_mlx_entry -->|"runs inference"| node_mlx_lm
node_mlx_entry -->|"encodes and decodes"| node_mlx_mimi
node_rust_core -->|"uses codec"| node_rust_mimi
node_cli -->|"generates locally"| node_rust_core
node_audio_out -->|"decodes messages"| node_protocol
node_pytorch_server -.->|"sends audio and text"| node_audio_out
node_rust_server -.->|"streams results"| node_audio_out
node_audio_out -->|"plays response"| node_speech
node_speech -->|"returns speech"| node_user

click node_queue_ui "https://github.com/kyutai-labs/moshi/blob/main/client/src/pages/Queue/Queue.tsx"
click node_queue_api "https://github.com/kyutai-labs/moshi/blob/main/client/src/pages/Queue/api/client.ts"
click node_conversation "https://github.com/kyutai-labs/moshi/blob/main/client/src/pages/Conversation/Conversation.tsx"
click node_socket "https://github.com/kyutai-labs/moshi/blob/main/client/src/pages/Conversation/hooks/useSocket.ts"
click node_audio_in "https://github.com/kyutai-labs/moshi/blob/main/client/src/pages/Conversation/hooks/useUserAudio.ts"
click node_audio_out "https://github.com/kyutai-labs/moshi/blob/main/client/src/pages/Conversation/hooks/useServerAudio.ts"
click node_protocol "https://github.com/kyutai-labs/moshi/blob/main/client/src/protocol/encoder.ts"
click node_pytorch_server "https://github.com/kyutai-labs/moshi/blob/main/moshi/moshi/server.py"
click node_rust_server "https://github.com/kyutai-labs/moshi/blob/main/rust/moshi-server/src/main.rs"
click node_torch_entry "https://github.com/kyutai-labs/moshi/blob/main/moshi/moshi/run_inference.py"
click node_torch_lm "https://github.com/kyutai-labs/moshi/blob/main/moshi/moshi/models/lm.py"
click node_torch_mimi "https://github.com/kyutai-labs/moshi/blob/main/moshi/moshi/models/compression.py"
click node_torch_tts "https://github.com/kyutai-labs/moshi/blob/main/moshi/moshi/models/tts.py"
click node_mlx_entry "https://github.com/kyutai-labs/moshi/blob/main/moshi_mlx/moshi_mlx/run_inference.py"
click node_mlx_lm "https://github.com/kyutai-labs/moshi/blob/main/moshi_mlx/moshi_mlx/models/lm.py"
click node_mlx_mimi "https://github.com/kyutai-labs/moshi/blob/main/moshi_mlx/moshi_mlx/models/mimi.py"
click node_rust_core "https://github.com/kyutai-labs/moshi/blob/main/rust/moshi-core/src/lib.rs"
click node_rust_mimi "https://github.com/kyutai-labs/moshi/blob/main/rust/moshi-core/src/mimi.rs"
click node_rust_tts "https://github.com/kyutai-labs/moshi/blob/main/rust/moshi-core/src/tts.rs"
click node_cli "https://github.com/kyutai-labs/moshi/blob/main/rust/moshi-cli/src/main.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_queue_ui,node_queue_api,node_conversation,node_socket,node_audio_in,node_audio_out,node_protocol,node_user toneBlue
class node_pytorch_server,node_rust_server toneAmber
class node_torch_entry,node_torch_lm,node_torch_mimi,node_torch_tts toneMint
class node_mlx_entry,node_mlx_lm,node_mlx_mimi,node_queue_service toneRose
class node_rust_core,node_rust_mimi,node_rust_tts,node_cli,node_speech toneIndigo
```

## Components

### Web client

- [Queue screen](https://github.com/kyutai-labs/moshi/blob/main/client/src/pages/Queue/Queue.tsx): `client/src/pages/Queue/Queue.tsx`
- [Queue API client](https://github.com/kyutai-labs/moshi/blob/main/client/src/pages/Queue/api/client.ts): `client/src/pages/Queue/api/client.ts`
- [Conversation UI](https://github.com/kyutai-labs/moshi/blob/main/client/src/pages/Conversation/Conversation.tsx): `client/src/pages/Conversation/Conversation.tsx`
- [Socket hook](https://github.com/kyutai-labs/moshi/blob/main/client/src/pages/Conversation/hooks/useSocket.ts): `client/src/pages/Conversation/hooks/useSocket.ts`
- [User audio](https://github.com/kyutai-labs/moshi/blob/main/client/src/pages/Conversation/hooks/useUserAudio.ts): `client/src/pages/Conversation/hooks/useUserAudio.ts`
- [Server audio](https://github.com/kyutai-labs/moshi/blob/main/client/src/pages/Conversation/hooks/useServerAudio.ts): `client/src/pages/Conversation/hooks/useServerAudio.ts`
- [Message protocol](https://github.com/kyutai-labs/moshi/blob/main/client/src/protocol/encoder.ts): `client/src/protocol/encoder.ts`

### Serving and APIs

- [PyTorch server](https://github.com/kyutai-labs/moshi/blob/main/moshi/moshi/server.py): `moshi/moshi/server.py`
- [Rust server](https://github.com/kyutai-labs/moshi/blob/main/rust/moshi-server/src/main.rs): `rust/moshi-server/src/main.rs`

### PyTorch stack

- [PyTorch inference](https://github.com/kyutai-labs/moshi/blob/main/moshi/moshi/run_inference.py): `moshi/moshi/run_inference.py`
- [PyTorch language model](https://github.com/kyutai-labs/moshi/blob/main/moshi/moshi/models/lm.py): `moshi/moshi/models/lm.py`
- [PyTorch Mimi codec](https://github.com/kyutai-labs/moshi/blob/main/moshi/moshi/models/compression.py): `moshi/moshi/models/compression.py`
- [PyTorch TTS](https://github.com/kyutai-labs/moshi/blob/main/moshi/moshi/models/tts.py): `moshi/moshi/models/tts.py`

### MLX stack

- [MLX inference](https://github.com/kyutai-labs/moshi/blob/main/moshi_mlx/moshi_mlx/run_inference.py): `moshi_mlx/moshi_mlx/run_inference.py`
- [MLX language model](https://github.com/kyutai-labs/moshi/blob/main/moshi_mlx/moshi_mlx/models/lm.py): `moshi_mlx/moshi_mlx/models/lm.py`
- [MLX Mimi codec](https://github.com/kyutai-labs/moshi/blob/main/moshi_mlx/moshi_mlx/models/mimi.py): `moshi_mlx/moshi_mlx/models/mimi.py`

### Rust stack

- [Rust model core](https://github.com/kyutai-labs/moshi/blob/main/rust/moshi-core/src/lib.rs): `rust/moshi-core/src/lib.rs`
- [Rust Mimi codec](https://github.com/kyutai-labs/moshi/blob/main/rust/moshi-core/src/mimi.rs): `rust/moshi-core/src/mimi.rs`
- [Rust TTS](https://github.com/kyutai-labs/moshi/blob/main/rust/moshi-core/src/tts.rs): `rust/moshi-core/src/tts.rs`
- [Rust CLI](https://github.com/kyutai-labs/moshi/blob/main/rust/moshi-cli/src/main.rs): `rust/moshi-cli/src/main.rs`

### Other components

- **User**
- **Queue service**
- **Speech output**

## Connections

- User → Queue screen: joins queue
- Queue screen → Queue API client: requests access
- Queue API client → Queue service: calls queue API
- User → Conversation UI: speaks
- Conversation UI → Socket hook: opens stream
- User audio → Socket hook: sends audio
- Socket hook → PyTorch server: streams audio
- Socket hook → Rust server: streams audio
- PyTorch server → PyTorch Mimi codec: encodes input
- PyTorch server → PyTorch language model: runs dialogue
- PyTorch language model → PyTorch Mimi codec: generates audio codes
- PyTorch inference → PyTorch language model: runs inference
- PyTorch inference → PyTorch Mimi codec: loads codec
- PyTorch server → PyTorch TTS: serves speech synthesis
- Rust server → Rust model core: runs model services
- Rust server → Rust Mimi codec: serves codec streams
- Rust server → Rust TTS: serves synthesis
- MLX inference → MLX language model: runs inference
- MLX inference → MLX Mimi codec: encodes and decodes
- Rust model core → Rust Mimi codec: uses codec
- Rust CLI → Rust model core: generates locally
- Server audio → Message protocol: decodes messages
- PyTorch server → Server audio: sends audio and text
- Rust server → Server audio: streams results
- Server audio → Speech output: plays response
- Speech output → User: returns speech