# wntg/llama-omni architecture

> Architecture diagram of the GitHub repository wntg/llama-omni, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wntg/llama-omni
- Repository: https://github.com/wntg/llama-omni

## Overview

##### Project type and purpose
- This is an end-to-end speech-to-speech conversational model training & serving codebase, a reproduction of “LLaMA-Omni.”
- It implements a two-stage training pipeline (speech-to-text alignment & speech generation refinement), plus an inference/serving pipeline exposing a Gradio UI.

##### File structure
- Top level:
- `wavs/` and `data.json` → raw audio and paired metadata for training
- `make_data.py`, `22k_to_16k.sh` → data prep scripts
- `omni_speech/` → main package
- `requirements.txt`, `pyproject.toml` → environment & dependencies
- `omni_speech/` sub-modules:
- `datasets/preprocess.py` → convert raw WAVs & JSON into training examples
- `model/` → builder modules for each model component:
– `speech_encoder/` (audio → latent)
– `language_model/` (LLama-style transformer)
– `speech_projector/` (project tokens into speech space)
– `speech_generator/` (vocoder for waveform synthesis)
- `train/` → Stage1 (`run.sh`) and Stage2 (`stage2.py`) scripts
- `infer/` → end-to-end inference pipeline, JSONL converters & helper scripts
- `serve/` → a controller, Gradio web server (`gradio_web_server.py`), `model_worker.py`

##### Insights from README
- Two training stages:
Stage1 aligns speech to LLama tokens with bf16; Stage2 fine-tunes projector+generator using model-generated answers.
- Inference uses Whisper for ASR (external model), LLama for text generation, then projector & vocoder for speech output.
- Serving via Gradio UI, examples included under `serve/examples/`.

##### Elements
- Blue: data & preprocessing
- Green: training stages
- Purple: model components
- Orange: inference/serving
- Rectangles for processes
- Rounded rectangles for external services (Whisper, Gradio UI)

##### Detail & high-fidelity
- Inter-stage dependencies (Stage2 uses Stage1’s projector checkpoint)
- Configuration files (`config.json`) and environment setup as footnotes

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% Data & Preprocessing
  subgraph "Data & Preprocessing"
    raw["Raw WAVs (wavs/)"]:::data
    meta["Metadata (data.json)"]:::data
    make["Data Prep Script (make_data.py)"]:::data
    resample["Resample Script (22k_to_16k.sh)"]:::data
    preprocess["Preprocessor (preprocess.py)"]:::data
    raw --> meta --> make --> resample --> preprocess
  end

  %% Model Components
  subgraph "Model Components"
    subgraph "Assembly"
      builder["Model Builder (builder.py)"]:::model
      assemble["Assembly (omni_speech_arch.py)"]:::model
    end
    subgraph "Speech Encoder"
      enc_builder["Encoder Builder (builder.py)"]:::model
      encoder["Encoder Impl (speech_encoder.py)"]:::model
    end
    subgraph "Language Model"
      lm1["LLama Wrapper (omni_speech_llama.py)"]:::model
      lm2["LLama2s Variant (omni_speech2s_llama.py)"]:::model
    end
    subgraph "Speech Projector"
      proj_builder["Projector Builder (builder.py)"]:::model
      projector["Projector Impl (speech_projector.py)"]:::model
    end
    subgraph "Speech Generator"
      gen_builder["Generator Builder (builder.py)"]:::model
      generator["Generator Interface (speech_generator.py)"]:::model
      gen_logic["Waveform Synthesis (generation.py)"]:::model
    end
  end

  %% Training Pipeline
  subgraph "Training Pipeline"
    run1["Run Stage1 (run.sh)"]:::train
    stage1["Stage1 Training (stage1.py)"]:::train
    stage2["Stage2 Training (stage2.py)"]:::train
    preprocess --> run1 --> stage1 --> stage2
  end

  %% Model Artifacts
  subgraph "Model Artifacts"
    checkpoints["Checkpoints"]:::db
    llama_ckpt["LLaMA Pretrained"]:::db
    config["Config (config.json)"]:::db
    checkpoints --> llama_ckpt
  end
  stage1 --> checkpoints
  stage2 --> checkpoints

  %% Inference Pipeline
  subgraph "Inference Pipeline"
    user_in["User WAV"]:::serve
    whisper["Whisper ASR"]:::serve_ext
    tokens["Tokens"]:::serve
    llama_inf["LLama Inference"]:::serve
    proj_inf["Speech Projector"]:::serve
    vocoder_inf["Speech Generator"]:::serve
    output_wav["Output WAV"]:::serve
    user_in --> whisper --> tokens --> llama_inf --> proj_inf --> vocoder_inf --> output_wav
  end

  %% Serving Layer
  subgraph "Serving Layer"
    ui["Gradio UI"]:::serve_ext
    controller["Controller"]:::serve
    worker["Model Worker"]:::serve
    examples["Examples"]:::serve
    ui --> controller --> worker --> output_wav
    ui --> examples
  end

  %% Click Events
  click raw "https://github.com/wntg/llama-omni/tree/main/wavs"
  click meta "https://github.com/wntg/llama-omni/blob/main/data.json"
  click make "https://github.com/wntg/llama-omni/blob/main/make_data.py"
  click resample "https://github.com/wntg/llama-omni/blob/main/22k_to_16k.sh"
  click preprocess "https://github.com/wntg/llama-omni/blob/main/omni_speech/datasets/preprocess.py"
  click builder "https://github.com/wntg/llama-omni/blob/main/omni_speech/model/builder.py"
  click assemble "https://github.com/wntg/llama-omni/blob/main/omni_speech/model/omni_speech_arch.py"
  click enc_builder "https://github.com/wntg/llama-omni/blob/main/omni_speech/model/speech_encoder/builder.py"
  click encoder "https://github.com/wntg/llama-omni/blob/main/omni_speech/model/speech_encoder/speech_encoder.py"
  click lm1 "https://github.com/wntg/llama-omni/blob/main/omni_speech/model/language_model/omni_speech_llama.py"
  click lm2 "https://github.com/wntg/llama-omni/blob/main/omni_speech/model/language_model/omni_speech2s_llama.py"
  click proj_builder "https://github.com/wntg/llama-omni/blob/main/omni_speech/model/speech_projector/builder.py"
  click projector "https://github.com/wntg/llama-omni/blob/main/omni_speech/model/speech_projector/speech_projector.py"
  click gen_builder "https://github.com/wntg/llama-omni/blob/main/omni_speech/model/speech_generator/builder.py"
  click generator "https://github.com/wntg/llama-omni/blob/main/omni_speech/model/speech_generator/speech_generator.py"
  click gen_logic "https://github.com/wntg/llama-omni/blob/main/omni_speech/model/speech_generator/generation.py"
  click run1 "https://github.com/wntg/llama-omni/blob/main/omni_speech/train/run.sh"
  click stage1 "https://github.com/wntg/llama-omni/blob/main/omni_speech/train/stage1.py"
  click stage2 "https://github.com/wntg/llama-omni/blob/main/omni_speech/train/stage2.py"
  click config "https://github.com/wntg/llama-omni/blob/main/Llama-3.2-1B-Instruct/config.json"
  click ui "https://github.com/wntg/llama-omni/blob/main/omni_speech/serve/gradio_web_server.py"
  click controller "https://github.com/wntg/llama-omni/blob/main/omni_speech/serve/controller.py"
  click worker "https://github.com/wntg/llama-omni/blob/main/omni_speech/serve/model_worker.py"
  click examples "https://github.com/wntg/llama-omni/tree/main/omni_speech/serve/examples"
  click infer_main "https://github.com/wntg/llama-omni/blob/main/omni_speech/infer/infer.py"
  click infer_sh "https://github.com/wntg/llama-omni/blob/main/omni_speech/infer/run.sh"
  click conv "https://github.com/wntg/llama-omni/blob/main/omni_speech/infer/convert_jsonl_to_txt.py"
  click unit2wav "https://github.com/wntg/llama-omni/blob/main/omni_speech/infer/unit2wav.sh"

  %% Styles
  classDef data fill:#D0E8FF,stroke:#004080,color:#004080
  classDef model fill:#F4E1FF,stroke:#800080,color:#800080
  classDef train fill:#DFFFE0,stroke:#108000,color:#108000
  classDef db fill:#FFF2CC,stroke:#806000,color:#806000
  classDef serve fill:#FFE8CC,stroke:#FF8000,color:#FF8000
  classDef serve_ext fill:#FFE0FF,stroke:#C00080,color:#C00080
```
