How wntg/llama-omni works
Project type and purpose
• This is an end-to-end speech-to-speech conversational model training & serving codebase, a reproduction of “LLaMA-Omni.”
• It implements a two-stage training pipeline (speech-to-text alignment & speech generation refinement), plus an inference/serving pipeline exposing a Gradio UI.
File structure
• Top level:
• wavs/ and data.json → raw audio and paired metadata for training
• make_data.py, 22k_to_16k.sh → data prep scripts
• omni_speech/ → main package
• requirements.txt, pyproject.toml → environment & dependencies
• omni_speech/ sub-modules:
• datasets/preprocess.py → convert raw WAVs & JSON into training examples
• model/ → builder modules for each model component:
– speech_encoder/ (audio → latent)
– language_model/ (LLama-style transformer)
– speech_projector/ (project tokens into speech space)
– speech_generator/ (vocoder for waveform synthesis)
• train/ → Stage1 (run.sh) and Stage2 (stage2.py) scripts
• infer/ → end-to-end inference pipeline, JSONL converters & helper scripts
• serve/ → a controller, Gradio web server (gradio_web_server.py), model_worker.py
Insights from README
• Two training stages:
Stage1 aligns speech to LLama tokens with bf16; Stage2 fine-tunes projector+generator using model-generated answers.
• Inference uses Whisper for ASR (external model), LLama for text generation, then projector & vocoder for speech output.
• Serving via Gradio UI, examples included under serve/examples/.
Elements
• Blue: data & preprocessing
• Green: training stages
• Purple: model components
• Orange: inference/serving
• Rectangles for processes
• Rounded rectangles for external services (Whisper, Gradio UI)
Detail & high-fidelity
• Inter-stage dependencies (Stage2 uses Stage1’s projector checkpoint)
• Configuration files (config.json) and environment setup as footnotes