# krafton-ai/raon-speech architecture

> Architecture diagram of the GitHub repository krafton-ai/raon-speech, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/krafton-ai/raon-speech
- Repository: https://github.com/krafton-ai/raon-speech

## Overview

1. **What this repo is**
   - `Raon-Speech` is a bilingual speech-language model codebase with two product lines:
     - **Raon-Speech**: offline SpeechLM for `STT`, `TTS`, `TextQA`, and `SpeechChat`
     - **Raon-SpeechChat**: full-duplex conversational speech model for offline and realtime use
   - Both share the same core package under `src/raon/`.

2. **Core model boundary**
   - The main model implementation lives in `src/raon/models/raon.py` and `src/raon/models/wrapper.py`.
   - The architecture centers on a shared LLM backbone plus speech-specific components: audio encoding/tokenization, speaker conditioning, code prediction, and audio decoding.
   - The readme describes the trainable boundary as the alignment/adaptation stack around the frozen or partially frozen speech/audio modules.

3. **Shared module stack**
   - `src/raon/modules/` contains the reusable speech building blocks:
     - `audio_encoder`, `audio_tokenizer`
     - `speaker_encoder`
     - `code_predictor`, `concurrent_audio_decoder`
     - `adaptor`, `embedding`
     - `voxtral_encoder`, `voxtral_wrapper`
   - This is the main separation between generic language modeling and speech-specific I/O.

4. **SpeechLM offline flow**
   - Offline usage flows from JSONL samples plus audio files into `src/raon/pipeline.py` or the CLI entrypoint `src/raon/generate.py`.
   - The pipeline exposes task methods such as `stt`, `tts`, `textqa`, and `speech_chat`, and can also save generated audio.
   - `config/infer.yaml` and `scripts/infer.sh` define the default inference wiring.

5. **Full-duplex flow**
   - The duplex track uses `src/raon/duplex_generate.py`, `src/raon/duplex_train.py`, and `config/duplex_infer.yaml`.
   - It consumes stereo dialogue audio plus timing annotations and produces interaction-aware output for turn-taking and interruption handling.
   - The duplex path is architecturally distinct because it needs conversation-state handling, prompt selection, and timeline-based supervision.

6. **Training subsystem**
   - `src/raon/train.py` handles SpeechLM training, while `src/raon/duplex_train.py` handles duplex training.
   - Training is launched through shell wrappers in `scripts/train.sh` and `scripts/duplex_train.sh`, typically with `torchrun` for multi-GPU execution.
   - `src/raon/utils/loss.py`, `training_callbacks.py`, `delay.py`, and `state_machine.py` support optimization and sequence/control logic.

7. **Data and supervision**
   - Sample datasets are shipped in `data/speechllm/` and `data/duplex/`.
   - SpeechLM data is JSONL-based with fields like `conversations`, `audios`, `speaker_ref_audios`, and `channel`.
   - Duplex data uses stereo wavs plus timing annotations via `turns` or `scripts`, along with metadata such as `speak_first`, `include_in_training`, `language`, and `channel`.
   - This makes the data boundary very explicit: text-audio conversation records for offline tasks, and timestamped dual-channel dialogue records for duplex.

8. **Realtime demo architecture**
   - `demo/realtime/` is a separate runtime stack for the duplex demo.
   - It is split into:
     - `api/` for the FastAPI app
     - `backends/sglang_backend.py` for model serving
     - `runtime/` for session, config, prompt mapping, and artifacts
     - `protocol/messages.py` for structured runtime messages
     - `web/` for browser-side streaming controls
   - This demo is not just a UI; it is a small serving system with session/state orchestration and backend abstraction.

9. **Serving/export boundary**
   - `scripts/export.sh` converts a Hugging Face checkpoint into an SGLang-friendly bundle for realtime serving.
   - The realtime duplex demo depends on that exported artifact, plus `sglang`, `gradio`, `fastapi`, and `uvicorn`.
   - This creates a clear deployment split between offline HF-style inference and low-latency interactive serving.

10. **External integration and runtime assumptions**
   - The repo is built around the Hugging Face ecosystem: `AutoModel.from_pretrained(..., trust_remote_code=True)` is a first-class loading path.
   - PyTorch and Torchaudio are the core runtime dependencies; CUDA and `bfloat16`/`float16` are expected for practical performance.
   - FlashAttention is optional and selected via attention backend flags, which affects training/inference performance but not the logical architecture.

11. **Utilities and preprocessing**
   - `src/raon/utils/` contains the support layer for audio I/O, mel features, data handling, prompt catalogs, special tokens, processor logic, and duplex-specific dataset helpers.
   - These utilities sit between raw files and the model, handling normalization, feature extraction, prompt construction, and output formatting.

12. **How to draw the architecture**
   - A good diagram should show four major zones:
     - **Data layer**: packaged JSONL + wav samples
     - **Core model layer**: shared Raon backbone and speech modules
     - **Workflows**: train, offline inference, duplex inference
     - **Serving/demo layer**: Gradio/FastAPI/SGLang realtime stack
   - The key architectural idea is that SpeechLM and SpeechChat are two task families built on one shared model stack, with separate training/inference/runtime paths on top.

Main entry points: [`src/raon/models/raon.py`](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/models/raon.py) (Raon model), [`demo/realtime/api/app.py`](https://github.com/krafton-ai/raon-speech/blob/main/demo/realtime/api/app.py) (Realtime API), [`src/raon/duplex_generate.py`](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/duplex_generate.py) (Duplex infer), [`src/raon/pipeline.py`](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/pipeline.py) (Offline infer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data layer"]
  node_speechllm_data[("SpeechLM data<br/>jsonl+wavs")]
  node_duplex_data[("Duplex data<br/>stereo+timing")]
end

subgraph group_core["Core model"]
  node_raon_model["Raon model<br/>shared backbone<br/>[raon.py]"]
  node_model_wrapper["Model wrapper<br/>load/export glue<br/>[wrapper.py]"]
  node_audio_stack["Audio stack<br/>encoder/tokenizer<br/>[audio_encoder.py]"]
  node_speaker_stack["Speaker encoder<br/>conditioning<br/>[speaker_encoder.py]"]
  node_prediction_stack["Code predictor<br/>speech planning<br/>[code_predictor.py]"]
  node_decoder_stack["Audio decoder<br/>speech synthesis"]
  node_adaption_stack["Adaptation stack<br/>alignment/adaptor<br/>[adaptor.py]"]
  node_voxtral_bridge["Voxtral bridge<br/>external bridge<br/>[voxtral_wrapper.py]"]
end

subgraph group_workflows["Workflows"]
  node_train_offline["Offline train<br/>speechlm trainer<br/>[train.py]"]
  node_train_duplex["Duplex train<br/>[duplex_train.py]"]
  node_infer_offline["Offline infer<br/>task pipeline<br/>[pipeline.py]"]
  node_generate_cli["Generate CLI<br/>inference entrypoint<br/>[generate.py]"]
  node_duplex_infer["Duplex infer<br/>interaction runtime<br/>[duplex_generate.py]"]
  node_export_bundle["Export bundle<br/>serving export<br/>[export.py]"]
end

subgraph group_demo["Serving demo"]
  node_realtime_api["Realtime API<br/>FastAPI app<br/>[app.py]"]
  node_sglang_backend["SGLang backend<br/>model server<br/>[sglang_backend.py]"]
  node_runtime_state["Runtime state<br/>session/runtime<br/>[session.py]"]
  node_message_proto["Message proto<br/>protocol<br/>[messages.py]"]
  node_web_controls["Web controls<br/>browser streaming<br/>[gradio_stream.js]"]
end

node_speechllm_data -->|"offline samples"| node_train_offline
node_speechllm_data -->|"eval inputs"| node_infer_offline
node_duplex_data -->|"duplex supervision"| node_train_duplex
node_duplex_data -->|"dialogue inputs"| node_duplex_infer
node_train_offline -->|"optimizes"| node_raon_model
node_train_duplex -->|"optimizes"| node_raon_model
node_infer_offline -->|"loads via"| node_model_wrapper
node_generate_cli -->|"invokes"| node_infer_offline
node_duplex_infer -->|"loads via"| node_model_wrapper
node_raon_model -->|"uses"| node_audio_stack
node_raon_model -->|"uses"| node_speaker_stack
node_raon_model -->|"uses"| node_prediction_stack
node_prediction_stack -->|"feeds"| node_decoder_stack
node_raon_model -->|"uses"| node_adaption_stack
node_raon_model -->|"integrates"| node_voxtral_bridge
node_export_bundle -->|"prepares"| node_sglang_backend
node_sglang_backend -->|"serves"| node_realtime_api
node_realtime_api -->|"orchestrates"| node_runtime_state
node_runtime_state -->|"formats"| node_message_proto
node_realtime_api -->|"streams to"| node_web_controls
node_duplex_infer -.->|"same domain"| node_realtime_api

click node_speechllm_data "https://github.com/krafton-ai/raon-speech/tree/main/data/speechllm"
click node_duplex_data "https://github.com/krafton-ai/raon-speech/tree/main/data/duplex"
click node_raon_model "https://github.com/krafton-ai/raon-speech/blob/main/src/raon/models/raon.py"
click node_model_wrapper "https://github.com/krafton-ai/raon-speech/blob/main/src/raon/models/wrapper.py"
click node_audio_stack "https://github.com/krafton-ai/raon-speech/blob/main/src/raon/modules/audio_encoder.py"
click node_speaker_stack "https://github.com/krafton-ai/raon-speech/blob/main/src/raon/modules/speaker_encoder.py"
click node_prediction_stack "https://github.com/krafton-ai/raon-speech/blob/main/src/raon/modules/code_predictor.py"
click node_decoder_stack "https://github.com/krafton-ai/raon-speech/blob/main/src/raon/modules/concurrent_audio_decoder.py"
click node_adaption_stack "https://github.com/krafton-ai/raon-speech/blob/main/src/raon/modules/adaptor.py"
click node_voxtral_bridge "https://github.com/krafton-ai/raon-speech/blob/main/src/raon/modules/voxtral_wrapper.py"
click node_train_offline "https://github.com/krafton-ai/raon-speech/blob/main/src/raon/train.py"
click node_train_duplex "https://github.com/krafton-ai/raon-speech/blob/main/src/raon/duplex_train.py"
click node_infer_offline "https://github.com/krafton-ai/raon-speech/blob/main/src/raon/pipeline.py"
click node_generate_cli "https://github.com/krafton-ai/raon-speech/blob/main/src/raon/generate.py"
click node_duplex_infer "https://github.com/krafton-ai/raon-speech/blob/main/src/raon/duplex_generate.py"
click node_export_bundle "https://github.com/krafton-ai/raon-speech/blob/main/src/raon/export.py"
click node_realtime_api "https://github.com/krafton-ai/raon-speech/blob/main/demo/realtime/api/app.py"
click node_sglang_backend "https://github.com/krafton-ai/raon-speech/blob/main/demo/realtime/backends/sglang_backend.py"
click node_runtime_state "https://github.com/krafton-ai/raon-speech/blob/main/demo/realtime/runtime/session.py"
click node_message_proto "https://github.com/krafton-ai/raon-speech/blob/main/demo/realtime/protocol/messages.py"
click node_web_controls "https://github.com/krafton-ai/raon-speech/blob/main/demo/realtime/web/gradio_stream.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_speechllm_data,node_duplex_data toneBlue
class node_raon_model,node_model_wrapper,node_audio_stack,node_speaker_stack,node_prediction_stack,node_decoder_stack,node_adaption_stack,node_voxtral_bridge toneAmber
class node_train_offline,node_train_duplex,node_infer_offline,node_generate_cli,node_duplex_infer,node_export_bundle toneMint
class node_realtime_api,node_sglang_backend,node_runtime_state,node_message_proto,node_web_controls toneRose
```

## Components

### Data layer

Packaged speech and duplex samples used for training and evaluation.

- [SpeechLM data](https://github.com/krafton-ai/raon-speech/tree/main/data/speechllm): jsonl+wavs · Offline SpeechLM training and evaluation samples for STT, TTS, TextQA, and SpeechChat. · `data/speechllm`
- [Duplex data](https://github.com/krafton-ai/raon-speech/tree/main/data/duplex): stereo+timing · Full-duplex dialogue samples with stereo audio, turn timing, and supervision metadata. · `data/duplex`

### Core model

Shared Raon backbone plus speech-specific modules and utilities.

- [Raon model](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/models/raon.py): shared backbone · Main model implementation that combines the LLM backbone with speech I/O and adaptation logic. · `src/raon/models/raon.py`
- [Model wrapper](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/models/wrapper.py): load/export glue · Wrapper around model construction, loading, and runtime-specific adaptation. · `src/raon/models/wrapper.py`
- [Audio stack](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/modules/audio_encoder.py): encoder/tokenizer · Audio encoding and tokenization boundary for speech input and representation. · `src/raon/modules/audio_encoder.py`
- [Speaker encoder](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/modules/speaker_encoder.py): conditioning · Speaker reference encoding used to condition speech generation and dialogue control. · `src/raon/modules/speaker_encoder.py`
- [Code predictor](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/modules/code_predictor.py): speech planning · Predicts intermediate speech codes and coordinates concurrent audio decoding. · `src/raon/modules/code_predictor.py`
- [Audio decoder](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/modules/concurrent_audio_decoder.py): speech synthesis · Decodes predicted codes into audio output, including concurrent decoding paths. · `src/raon/modules/concurrent_audio_decoder.py`
- [Adaptation stack](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/modules/adaptor.py): alignment/adaptor · Shared adaptation and embedding modules that bridge language and speech components. · `src/raon/modules/adaptor.py`
- [Voxtral bridge](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/modules/voxtral_wrapper.py): external bridge · Wrapper and encoder integration for Voxtral-style speech/language interoperability. · `src/raon/modules/voxtral_wrapper.py`

### Workflows

Training and inference entrypoints for offline and duplex paths.

- [Offline train](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/train.py): speechlm trainer · SpeechLM training entrypoint for the offline product line. · `src/raon/train.py`
- [Duplex train](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/duplex_train.py): duplex trainer · Training entrypoint for turn-aware duplex supervision and conversation-state learning. · `src/raon/duplex_train.py`
- [Offline infer](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/pipeline.py): task pipeline · Offline generation pipeline exposing STT, TTS, TextQA, and SpeechChat flows. · `src/raon/pipeline.py`
- [Generate CLI](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/generate.py): inference entrypoint · Command-line entrypoint for offline inference and audio generation. · `src/raon/generate.py`
- [Duplex infer](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/duplex_generate.py): interaction runtime · Full-duplex inference path with prompt selection, turn handling, and timeline-based output. · `src/raon/duplex_generate.py`
- [Export bundle](https://github.com/krafton-ai/raon-speech/blob/main/src/raon/export.py): serving export · Converts a checkpoint into a serving-friendly artifact for realtime deployment. · `src/raon/export.py`

### Serving demo

Realtime duplex serving stack with API, backend, runtime, and web controls.

- [Realtime API](https://github.com/krafton-ai/raon-speech/blob/main/demo/realtime/api/app.py): FastAPI app · HTTP service layer for the duplex realtime demo. · `demo/realtime/api/app.py`
- [SGLang backend](https://github.com/krafton-ai/raon-speech/blob/main/demo/realtime/backends/sglang_backend.py): model server · Serving backend used by the realtime duplex demo. · `demo/realtime/backends/sglang_backend.py`
- [Runtime state](https://github.com/krafton-ai/raon-speech/blob/main/demo/realtime/runtime/session.py): session/runtime · Session, config, prompt mapping, and artifact coordination for realtime interactions. · `demo/realtime/runtime/session.py`
- [Message proto](https://github.com/krafton-ai/raon-speech/blob/main/demo/realtime/protocol/messages.py): protocol · Structured runtime message definitions for streaming and orchestration. · `demo/realtime/protocol/messages.py`
- [Web controls](https://github.com/krafton-ai/raon-speech/blob/main/demo/realtime/web/gradio_stream.js): browser streaming · Browser-side streaming and stop controls for the realtime demo. · `demo/realtime/web/gradio_stream.js`

## Connections

- SpeechLM data → Offline train: offline samples · JSONL and audio samples feed SpeechLM training.
- SpeechLM data → Offline infer: eval inputs · Evaluation examples drive offline inference flows.
- Duplex data → Duplex train: duplex supervision · Stereo audio and timing annotations train the duplex model.
- Duplex data → Duplex infer: dialogue inputs · Duplex inference consumes turn-aware dialogue records and prompts.
- Offline train → Raon model: optimizes · Offline training updates the shared model boundary.
- Duplex train → Raon model: optimizes · Duplex training updates the shared model with interaction-aware supervision.
- Offline infer → Model wrapper: loads via · Offline inference uses the wrapper for model construction and runtime setup.
- Generate CLI → Offline infer: invokes · CLI entrypoint drives the offline pipeline.
- Duplex infer → Model wrapper: loads via · Duplex inference uses the same model loading boundary.
- Raon model → Audio stack: uses · The core model consumes speech audio through shared encoders and tokenizers.
- Raon model → Speaker encoder: uses · Speaker conditioning informs generation and dialogue control.
- Raon model → Code predictor: uses · Intermediate code prediction bridges language state to audio output.
- Code predictor → Audio decoder: feeds · Predicted codes are converted into audio by the decoder stack.
- Raon model → Adaptation stack: uses · Adaptation and embedding modules mediate speech-language alignment.
- Raon model → Voxtral bridge: integrates · Voxtral integration provides an alternate speech-language bridge.
- Export bundle → SGLang backend: prepares · Exported checkpoints are packaged for low-latency serving.
- SGLang backend → Realtime API: serves · The backend powers the realtime API layer.
- Realtime API → Runtime state: orchestrates · API requests are coordinated through session and runtime state.
- Runtime state → Message proto: formats · Runtime messaging uses structured protocol objects.
- Realtime API → Web controls: streams to · API responses drive browser-side streaming controls.
- Duplex infer → Realtime API: same domain · Duplex inference and realtime demo share the interaction-oriented serving boundary.