How kyutai-labs/moshi-finetune works
Project type and purpose
• This is an open-source Python CLI/library for fine-tuning Moshi audio-text models using LoRA adapters.
• It provides scripts (train.py, annotate.py), a modular Python package (finetune/), example YAML configs, and tutorials for dataset preparation, training, and inference.
File structure
• Top‐level scripts: annotate.py (dataset annotation), train.py (entry point for training).
• Package code under finetune/: subpackages for argument parsing (args.py), checkpointing, distributed training, data loading, model wrapping, mixed precision, loss calculation, monitoring (metrics logger for local/W&B) and utilities.
• example/ holds example YAML configs.
• tutorials/ contains a Jupyter notebook demo.
• CI and workflow configs in .github/.
• Project configuration in pyproject.toml and setup.cfg.
README insights
• Core workflow: clone → install → prepare YAML config → prepare dataset .jsonl + transcripts → run torchrun -m train … → monitor with W&B → run inference via moshi.server.
• External dependencies/services: Hugging Face Hub (model & dataset), W&B, SLURM (optional for distributed annotate), Moshi server.
Main components and interactions (to represent
A. User Interfaces
1. CLI scripts
• .py
• train.py (invoked via torchrun)
2. Notebook/Tutorial
B. Configuration Loader
• Parses YAML (example/moshi_7B.yaml) and CLI args (finetune/args.py).
C. Data Pipeline
• Data loader (finetune/data/data_loader.py) reads .jsonl → yields stereo audio paths + durations.
• Dataset class (finetune/data/dataset.py) loads audio+transcript JSON, applies interleaving (finetune/data/interleaver.py).
• Sharding logic for distributed annotate or training (--shards, torchrun/world size).
D. Model Loading & Wrapping
• Loads base Moshi model from Hugging Face via moshi_paths (finetune/utils.py).
• Wraps model to apply LoRA adapters (finetune/wrapped_model.py).
E. Training Engine
• Loss computation (finetune/loss.py), mixed precision (finetune/mixed_precision.py), optimizer & scheduler via PyTorch.
• Checkpointing (finetune/checkpointing.py) saves weights, LoRA adapters, config.
• Distributed setup (finetune/distributed.py) via torchrun/torch.distributed.
• Monitoring hooks: metrics logger (finetune/monitoring/metrics_logger.py), W&B integration.
F. External Services
• Hugging Face Hub (model repo, dataset snapshot).
• Weights & Biases (experiment tracking).
• SLURM (optional for annotate.py shards).
G. Inference
• Moshi server (python -m moshi.server) loads fine-tuned LoRA or consolidated weights for interactive chat.
Architecture patterns & design principles
• Modular package layout (single responsibility per module).
• Clear separation: configuration → data → model → training → monitoring.
• Distributed-first design (torchrun, sharding).
• Pluggable monitoring (local logger vs. W&B).