# kyutai-labs/delayed-streams-modeling architecture

> Architecture diagram of the GitHub repository kyutai-labs/delayed-streams-modeling, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kyutai-labs/delayed-streams-modeling
- Repository: https://github.com/kyutai-labs/delayed-streams-modeling

## Overview

##### Project type and purpose
- This is an open-source toolkit for running Kyutai delayed-streams speech-to-text (STT) and text-to-speech (TTS) models in multiple environments: research (PyTorch), production (Rust server), and on-device (Apple MLX). It bundles client scripts, configuration files, a standalone Rust example, and Jupyter notebooks.

##### File structure
- Top-level concerns:
- .github / CI & PR workflows
- configs/*.toml → model-server settings for STT/TTS
- scripts/ → Python entry-point scripts for STT & TTS in each environment
- stt-rs/ → standalone Rust example for STT
- stt_pytorch.ipynb, tts_pytorch.ipynb → research notebooks
- audio/ → sample assets
- Key patterns: three implementations per modality (PyTorch, Rust, MLX), each with its own client or server component.
- Configs describe model parameters (batch size, GPU/CUDA flags). Build scripts and Cargo.toml in stt-rs show Rust service packaging.

##### README for additional insights
- It explains the three implementations, how clients interact with the model (direct PyTorch calls vs. websockets to a moshi-server), and the use cases (real-time streaming, batching, on-device).
- External dependencies: Hugging Face model hub, moshi & moshi-mlx Python packages, moshi-server Rust crate, UV/UVX CLI.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Swimlanes
    subgraph "Research (PyTorch)"
        direction TB
        STT_PY1["stt_from_file_pytorch.py"]:::python
        STT_PY2["stt_from_file_with_prompt_pytorch.py"]:::python
        STT_PY3["stt_evaluate_on_dataset.py"]:::python
        NB_STT["stt_pytorch.ipynb"]:::python
        TTS_PY["tts_pytorch.py"]:::python
        NB_TTS["tts_pytorch.ipynb"]:::python
        AUDIO["Sample Audio Assets"]:::external
    end

    subgraph "Production (Rust)"
        direction TB
        RS_STT1["stt_from_file_rust_server.py"]:::python
        RS_STT2["stt_from_mic_rust_server.py"]:::python
        RS_TTS["tts_rust_server.py"]:::python
        WEBSOCKET["WebSocket API"]:::external
        SERVER["moshi-server (Rust)"]:::rust
        RS_STANDALONE["Rust STT Example (CLI)"]:::rust
    end

    subgraph "On-Device (MLX)"
        direction TB
        MLX_STT["stt_from_mic_mlx.py"]:::python
        MLX_TTS["tts_mlx.py"]:::python
    end

    subgraph "Config Service"
        direction TB
        CFG1["config-stt-en-hf.toml"]:::config
        CFG2["config-stt-en_fr-hf.toml"]:::config
        CFG3["config-tts.toml"]:::config
    end

    subgraph "External Dependencies"
        direction TB
        HF["Hugging Face Model Hub"]:::external
        GPU["GPU/CUDA Hardware"]:::external
        APPLE["Apple MLX Hardware"]:::external
    end

    subgraph "CI & Docs"
        direction TB
        CI["CI & Workflows"]:::external
        RD["README & CONTRIBUTING"]:::external
    end

    %% Data Flow
    HF -->|fetch weights| STT_PY1
    HF -->|fetch weights| STT_PY2
    HF -->|fetch weights| STT_PY3
    HF -->|fetch weights| NB_STT
    HF -->|fetch weights| TTS_PY
    HF -->|fetch weights| NB_TTS

    STT_PY1 -->|audio chunks| pythonAPI["moshi Python API"]:::python
    pythonAPI -->|inference| GPU
    pythonAPI -->|transcript| STT_PY1

    STT_PY2 --> pythonAPI
    STT_PY3 --> pythonAPI
    NB_STT --> pythonAPI

    TTS_PY --> pythonAPI
    NB_TTS --> pythonAPI

    AUDIO --> STT_PY1

    %% Production flow
    RS_STT1 -->|open| WEBSOCKET
    RS_STT2 -->|open| WEBSOCKET
    RS_TTS -->|open| WEBSOCKET
    WEBSOCKET -->|stream audio/text| SERVER
    SERVER -->|uses| CFG1
    SERVER -->|uses| CFG2
    SERVER -->|uses| CFG3
    SERVER -->|inference| GPU
    SERVER -->|stream results| WEBSOCKET
    WEBSOCKET --> RS_STT1
    WEBSOCKET --> RS_STT2
    WEBSOCKET --> RS_TTS

    RS_STANDALONE -->|CLI call| SERVER_LIB["moshi-server lib"]:::rust
    SERVER_LIB -->|inference| GPU
    RS_STANDALONE -->|transcript| RS_STANDALONE

    %% On-Device flow
    MLX_STT -->|calls| MLX_API["moshi-mlx API"]:::python
    MLX_API -->|inference| APPLE
    MLX_API -->|stream transcript| MLX_STT

    MLX_TTS -->|calls| MLX_API
    MLX_API -->|stream audio| MLX_TTS

    %% CI & Docs
    CI -.-> STT_PY1
    CI -.-> RS_STT1
    RD -.-> STT_PY1

    %% Click Events
    click STT_PY1 "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/scripts/stt_from_file_pytorch.py"
    click STT_PY2 "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/scripts/stt_from_file_with_prompt_pytorch.py"
    click STT_PY3 "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/scripts/stt_evaluate_on_dataset.py"
    click NB_STT "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/stt_pytorch.ipynb"
    click TTS_PY "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/scripts/tts_pytorch.py"
    click NB_TTS "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/tts_pytorch.ipynb"
    click RS_STT1 "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/scripts/stt_from_file_rust_server.py"
    click RS_STT2 "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/scripts/stt_from_mic_rust_server.py"
    click RS_TTS "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/scripts/tts_rust_server.py"
    click RS_STANDALONE "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/stt-rs/src/main.rs"
    click MLX_STT "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/scripts/stt_from_mic_mlx.py"
    click MLX_TTS "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/scripts/tts_mlx.py"
    click CFG1 "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/configs/config-stt-en-hf.toml"
    click CFG2 "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/configs/config-stt-en_fr-hf.toml"
    click CFG3 "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/configs/config-tts.toml"
    click AUDIO "https://github.com/kyutai-labs/delayed-streams-modeling/tree/main/audio/"
    click CI "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/.github/workflows/precommit.yml"
    click RD "https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md"

    %% Styles
    classDef python fill:#blue,stroke:#000,color:#fff
    classDef rust fill:#0f0,stroke:#000,color:#000
    classDef external fill:none,stroke-dasharray: 5 5,stroke:#888,color:#000
    classDef config fill:#faf,stroke:#000,color:#000
```
