# kyutai-labs/moshi-finetune architecture

> Architecture diagram of the GitHub repository kyutai-labs/moshi-finetune, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kyutai-labs/moshi-finetune
- Repository: https://github.com/kyutai-labs/moshi-finetune

## Overview

##### Project type and purpose
- This is an open-source Python CLI/library for fine-tuning Moshi audio-text models using LoRA adapters.
- It provides scripts (`train.py`, `annotate.py`), a modular Python package (`finetune/`), example YAML configs, and tutorials for dataset preparation, training, and inference.

##### File structure
- Top‐level scripts: `annotate.py` (dataset annotation), `train.py` (entry point for training).
- Package code under `finetune/`: subpackages for argument parsing (`args.py`), checkpointing, distributed training, data loading, model wrapping, mixed precision, loss calculation, monitoring (metrics logger for local/W&B) and utilities.
- `example/` holds example YAML configs.
- `tutorials/` contains a Jupyter notebook demo.
- CI and workflow configs in `.github/`.
- Project configuration in `pyproject.toml` and `setup.cfg`.

##### README insights
- Core workflow: clone → install → prepare YAML config → prepare dataset `.jsonl` + transcripts → run `torchrun -m train …` → monitor with W&B → run inference via `moshi.server`.
- External dependencies/services: Hugging Face Hub (model & dataset), W&B, SLURM (optional for distributed annotate), Moshi server.

##### Main components and interactions (to represent
A. **User Interfaces**
1. CLI scripts
- .py
- train.py (invoked via `torchrun`)
2. Notebook/Tutorial

B. **Configuration Loader**
- Parses YAML (`example/moshi_7B.yaml`) and CLI args (`finetune/args.py`).

C. **Data Pipeline**
- Data loader (`finetune/data/data_loader.py`) reads `.jsonl` → yields stereo audio paths + durations.
- Dataset class (`finetune/data/dataset.py`) loads audio+transcript JSON, applies interleaving (`finetune/data/interleaver.py`).
- Sharding logic for distributed annotate or training (`--shards`, torchrun/world size).

D. **Model Loading & Wrapping**
- Loads base Moshi model from Hugging Face via `moshi_paths` (`finetune/utils.py`).
- Wraps model to apply LoRA adapters (`finetune/wrapped_model.py`).

E. **Training Engine**
- Loss computation (`finetune/loss.py`), mixed precision (`finetune/mixed_precision.py`), optimizer & scheduler via PyTorch.
- Checkpointing (`finetune/checkpointing.py`) saves weights, LoRA adapters, config.
- Distributed setup (`finetune/distributed.py`) via `torchrun`/`torch.distributed`.
- Monitoring hooks: metrics logger (`finetune/monitoring/metrics_logger.py`), W&B integration.

F. **External Services**
- Hugging Face Hub (model repo, dataset snapshot).
- Weights & Biases (experiment tracking).
- SLURM (optional for `annotate.py` shards).

G. **Inference**
- Moshi server (`python -m moshi.server`) loads fine-tuned LoRA or consolidated weights for interactive chat.

##### Architecture patterns & design principles
- Modular package layout (single responsibility per module).
- Clear separation: configuration → data → model → training → monitoring.
- Distributed-first design (torchrun, sharding).
- Pluggable monitoring (local logger vs. W&B).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Frontend CLI and Tutorial
    subgraph "CLI & Tutorial Layer"
        direction TB
        AnnotateCLI["annotate.py"]:::frontend
        TrainCLI["train.py"]:::frontend
        Tutorial["Tutorial Notebook"]:::frontend
        ExampleConfig["example/moshi_7B.yaml"]:::frontend
    end

    %% Configuration Parser
    ConfigParser["Config Loader\n(finfinetune/args.py & data args)"]:::compute
    GlobalArgs["Global CLI Args"]:::compute
    DataArgs["Data-specific CLI Args"]:::compute

    %% Data Ingestion & Preprocessing
    subgraph "Data Pipeline" 
        direction TB
        DataLoader["Data Loader"]:::data
        Dataset["Dataset Class"]:::storage
        Interleaver["Interleaving Logic"]:::data
    end

    %% Model Loader & LoRA Wrapping
    subgraph "Model Loading & Wrapping"
        direction TB
        Utils["HF Model Utilities"]:::compute
        WrappedModel["LoRA Adapter Wrapper"]:::compute
    end

    %% Training Engine
    subgraph "Training Engine"
        direction TB
        Loss["Loss Computation"]:::compute
        MixedPrecision["Mixed-Precision Utilities"]:::compute
        Distributed["Distributed Setup"]:::compute
        Checkpointing["Checkpointing & Save/Load"]:::storage
        Eval["Evaluation/Inference Helper"]:::compute
    end

    %% Monitoring
    subgraph "Monitoring & Logging"
        direction TB
        MetricsLogger["Metrics Logger"]:::compute
        MonitoringUtils["Monitoring Utilities"]:::compute
    end

    %% External Services
    subgraph "External Services"
        direction TB
        HFHub["Hugging Face Hub"]:::external
        WandB["Weights & Biases"]:::external
        SLURM["SLURM Scheduler (optional)"]:::external
    end

    %% Inference
    InferenceServer["Moshi Inference Server"]:::compute

    %% Connections
    AnnotateCLI -->|executes| ConfigParser
    TrainCLI -->|executes| ConfigParser
    Tutorial -->|uses| ExampleConfig
    ConfigParser -->|parses global args| GlobalArgs
    ConfigParser -->|parses data args| DataArgs

    GlobalArgs --> DataLoader
    DataArgs --> DataLoader

    AnnotateCLI -->|annotates→ JSONL| DataLoader
    DataLoader -->|yields audio+transcript| Dataset
    Dataset --> Interleaver

    TrainCLI -->|triggers| DataLoader
    TrainCLI -->|triggers| Utils

    Interleaver -->|shards via torch.distributed| Distributed
    Utils --> WrappedModel
    WrappedModel --> Distributed

    DataLoader --> Distributed
    Distributed --> Loss
    Loss --> MixedPrecision
    MixedPrecision -->|optimizes| Distributed
    Distributed --> Eval
    Distributed --> Checkpointing
    Checkpointing -->|saves→ LoRA & config| Storage[(Checkpoint Storage)]:::storage

    Distributed --> MetricsLogger
    MetricsLogger --> WandB
    MetricsLogger --> MonitoringUtils

    HFHub --> Utils
    SLURM --> AnnotateCLI
    WandB --> MetricsLogger

    InferenceServer -->|loads checkpoint| Storage
    InferenceServer -->|serves HTTP/WebSocket| UserBrowser[(Client)]:::external

    %% Click Events
    click AnnotateCLI "https://github.com/kyutai-labs/moshi-finetune/blob/main/annotate.py"
    click TrainCLI "https://github.com/kyutai-labs/moshi-finetune/blob/main/train.py"
    click Tutorial "https://github.com/kyutai-labs/moshi-finetune/blob/main/tutorials/moshi_finetune.ipynb"
    click ExampleConfig "https://github.com/kyutai-labs/moshi-finetune/blob/main/example/moshi_7B.yaml"
    click GlobalArgs "https://github.com/kyutai-labs/moshi-finetune/blob/main/finetune/args.py"
    click DataArgs "https://github.com/kyutai-labs/moshi-finetune/blob/main/finetune/data/args.py"
    click DataLoader "https://github.com/kyutai-labs/moshi-finetune/blob/main/finetune/data/data_loader.py"
    click Dataset "https://github.com/kyutai-labs/moshi-finetune/blob/main/finetune/data/dataset.py"
    click Interleaver "https://github.com/kyutai-labs/moshi-finetune/blob/main/finetune/data/interleaver.py"
    click Utils "https://github.com/kyutai-labs/moshi-finetune/blob/main/finetune/utils.py"
    click WrappedModel "https://github.com/kyutai-labs/moshi-finetune/blob/main/finetune/wrapped_model.py"
    click Loss "https://github.com/kyutai-labs/moshi-finetune/blob/main/finetune/loss.py"
    click MixedPrecision "https://github.com/kyutai-labs/moshi-finetune/blob/main/finetune/mixed_precision.py"
    click Distributed "https://github.com/kyutai-labs/moshi-finetune/blob/main/finetune/distributed.py"
    click Checkpointing "https://github.com/kyutai-labs/moshi-finetune/blob/main/finetune/checkpointing.py"
    click Eval "https://github.com/kyutai-labs/moshi-finetune/blob/main/finetune/eval.py"
    click MetricsLogger "https://github.com/kyutai-labs/moshi-finetune/blob/main/finetune/monitoring/metrics_logger.py"
    click MonitoringUtils "https://github.com/kyutai-labs/moshi-finetune/blob/main/finetune/monitoring/utils.py"

    %% Styles
    classDef frontend fill:#ddd,stroke:#333,stroke-width:1px;
    classDef data fill:#cfc,stroke:#333,stroke-width:1px;
    classDef compute fill:#ccf,stroke:#333,stroke-width:1px;
    classDef storage fill:#cff,stroke:#333,stroke-width:1px;
    classDef external fill:#fc9,stroke:#333,stroke-width:1px;
```
