# moonshotai/kimi-audio architecture

> Architecture diagram of the GitHub repository moonshotai/kimi-audio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/moonshotai/kimi-audio
- Repository: https://github.com/moonshotai/kimi-audio

## Overview

##### Project Type and Purpose
- This is an open-source inference library for Kimi-Audio, a universal audio foundation model. It provides end-to-end audio processing: from raw audio input to text transcription, audio-text conversation, and high-fidelity audio generation.

##### Top-Level Architectural Components
a. Client / User Script
- infer.py or any Python script that uses `kimia_infer.api.kimia.KimiAudio`.
- Loads audio files (e.g., WAV), configures sampling parameters, calls `model.generate()`, and handles outputs.
b. API Layer (kimia_infer/api)
- `KimiAudio` class: orchestration, model loading (LLM + tokenizer + detokenizer), generate() method.
- `PromptManager`: builds multimodal prompts (text + audio tokens).
c. Audio Tokenizer (kimia_infer/models/tokenizer)
- Whisper-based acoustic feature extractor (continuous features).
- Vector quantizer producing discrete semantic tokens at 12.5 Hz (using GLM-4 or custom quantizer).
d. Audio LLM Core (external Hugging Face Transformers model)
- A transformer initialized from Qwen2.5-7B.
- Shared encoder layers for multimodal inputs.
- Parallel heads:
- Text head → autoregressive text tokens
- Audio-semantic head → discrete audio tokens
e. Audio Detokenizer (kimia_infer/models/detokenizer)
- Flow-matching model for chunk‐wise streaming detokenization.
- BigVGAN vocoder for waveform synthesis.
- Look-ahead buffer for low-latency output.
f. Utilities (kimia_infer/utils)
- Data conversion, sampler (temperature, top-k, penalty), special tokens.
g. External Services & Tools
- Hugging Face model hub (download LLM checkpoints, tokenizer, vocoder weights)
- Docker container (Dockerfile) for reproducible environment.

##### Component Interactions and Data Flow
1. Client loads audio file → passes to KimiAudio.generate().
2. API Layer invokes PromptManager to package:
- Text instructions (optional)
- Audio features + semantic tokens from Audio Tokenizer.
3. Multimodal embeddings into Audio LLM Core.
4. LLM Core autoregressively produces:
- Text tokens (returned to client)
- Discrete semantic audio tokens (sent to Audio Detokenizer).
5. Audio Detokenizer’s flow model + BigVGAN vocoder transform semantic tokens → waveform (streamed or full).
6. Client receives text and/or waveform → post-process/save.

##### Architectural Patterns & Principles
- Modular separation: API / Tokenizer / LLM / Detokenizer / Utils
- Streaming pipeline in detokenizer for low latency
- Adapter pattern: wrapping external models (Whisper, transformers, BigVGAN) behind a uniform API
- Clear unidirectional data flow for inference

##### Essential Details to Capture
- The three core stages: Tokenization, LLM inference, Detokenization
- The API’s role as orchestrator, using PromptManager
- External dependencies (Whisper encoder, BigVGAN, Hugging Face model hub)
- Streaming capability in the detokenizer for low-latency audio generation

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Client / User Script"
        Client["Client / User Script\n(infer.py)"]
    end

    subgraph "API Layer"
        KimiAudio["KimiAudio"]:::api
        PromptManager["PromptManager"]:::api
    end

    subgraph "Audio Tokenizer"
        WhisperFE["Whisper Feature Extractor"]:::ml
        GLMTokenizer["GLM-4 Semantic Tokenizer"]:::ml
    end

    subgraph "Audio LLM Core"
        LLMCore["Audio LLM Core\n(Transformer)"]:::ml
        TextHead["Text Head"]:::ml
        AudioHead["Audio-Semantic Head"]:::ml
    end

    subgraph "Audio Detokenizer"
        FlowModel["Flow-Matching Model"]:::ml
        BigVGANWrapper["BigVGAN Vocoder Wrapper"]:::ml
        Vocoder["BigVGAN Core & Utils"]:::ml
    end

    subgraph "Utilities"
        DataConv["Data Conversion"]:::utils
        Sampler["Sampler"]:::utils
        SpecialTokens["Special Tokens"]:::utils
    end

    subgraph "External Services & Tools"
        HF["Hugging Face Model Hub"]:::external
        Docker["Docker Environment"]:::external
    end

    Client -->|"raw waveform"| KimiAudio
    KimiAudio -->|"load audio"| WhisperFE
    WhisperFE -->|"acoustic features"| GLMTokenizer
    KimiAudio -->|"invoke"| PromptManager
    GLMTokenizer -->|"semantic tokens"| PromptManager
    PromptManager -->|"multimodal embeddings"| LLMCore
    LLMCore -->|"text tokens"| TextHead
    TextHead -->|"text output"| Client
    LLMCore -->|"audio tokens"| AudioHead
    AudioHead -->|"semantic tokens"| FlowModel
    FlowModel -->|"streaming tokens"| BigVGANWrapper
    BigVGANWrapper -->|"waveform"| Client

    DataConv --> KimiAudio
    Sampler --> KimiAudio
    SpecialTokens --> KimiAudio
    HF --> KimiAudio
    Docker --> KimiAudio

    classDef api fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef ml fill:#98FB98,stroke:#333,stroke-width:1px
    classDef utils fill:#FFD700,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px

    click Client "https://github.com/moonshotai/kimi-audio/blob/master/infer.py"
    click KimiAudio "https://github.com/moonshotai/kimi-audio/blob/master/kimia_infer/api/kimia.py"
    click PromptManager "https://github.com/moonshotai/kimi-audio/blob/master/kimia_infer/api/prompt_manager.py"
    click GLMTokenizer "https://github.com/moonshotai/kimi-audio/blob/master/kimia_infer/models/tokenizer/glm4_tokenizer.py"
    click WhisperFE "https://github.com/moonshotai/kimi-audio/blob/master/kimia_infer/models/tokenizer/whisper_Lv3/whisper.py"
    click FlowModel "https://github.com/moonshotai/kimi-audio/blob/master/kimia_infer/models/detokenizer/flow_matching/dit_block.py"
    click FlowModel "https://github.com/moonshotai/kimi-audio/blob/master/kimia_infer/models/detokenizer/flow_matching/model.py"
    click FlowModel "https://github.com/moonshotai/kimi-audio/blob/master/kimia_infer/models/detokenizer/flow_matching/ode_wrapper.py"
    click FlowModel "https://github.com/moonshotai/kimi-audio/blob/master/kimia_infer/models/detokenizer/flow_matching/scheduler.py"
    click FlowModel "https://github.com/moonshotai/kimi-audio/blob/master/kimia_infer/models/detokenizer/semantic_fm_prefix_streaming.py"
    click BigVGANWrapper "https://github.com/moonshotai/kimi-audio/blob/master/kimia_infer/models/detokenizer/bigvgan_wrapper.py"
    click Vocoder "https://github.com/moonshotai/kimi-audio/blob/master/kimia_infer/models/detokenizer/vocoder/bigvgan.py"
    click Vocoder "https://github.com/moonshotai/kimi-audio/blob/master/kimia_infer/models/detokenizer/vocoder/utils.py"
    click DataConv "https://github.com/moonshotai/kimi-audio/blob/master/kimia_infer/utils/data.py"
    click Sampler "https://github.com/moonshotai/kimi-audio/blob/master/kimia_infer/utils/sampler.py"
    click SpecialTokens "https://github.com/moonshotai/kimi-audio/blob/master/kimia_infer/utils/special_tokens.py"
    click Docker "https://github.com/moonshotai/kimi-audio/tree/master/Dockerfile"
```
