# boson-ai/higgs-audio architecture

> Architecture diagram of the GitHub repository boson-ai/higgs-audio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/boson-ai/higgs-audio
- Repository: https://github.com/boson-ai/higgs-audio

## Overview

##### Project Type and Purpose
- Higgs Audio V2 is an open-source, multimodal audio foundation model with an accompanying library and API for expressive audio generation (text-to-speech, voice cloning, multi-speaker dialogue, humming with background music).
- It combines a pretrained large language model (LLM) with a custom audio adapter (DualFFN architecture) and a unified audio tokenizer to generate waveforms given text and optional reference audio.

##### File Structure Analysis
- Top-level modules:
- boson_multimodal/audio_processing: audio tokenization, quantization, codec (DAC).
- boson_multimodal/data_collator: collates text/audio samples into model input.
- boson_multimodal/dataset: dataset abstractions (ChatML datasets).
- boson_multimodal/model/higgs_audio: model definitions (configuration, modeling, custom CUDA runner).
- boson_multimodal/serve: inference engine exposing generate() API and utilities.
- Configuration and packaging:
- pyproject.toml, setup.py, requirements.txt, setup.cfg for packaging and dependencies.
- Examples and docs:
- examples/ for usage scripts (generation.py, serve_engine examples, vLLM integration).
- tech_blogs/ for architecture and tokenizer deep dives.

##### Key Insights from README
- Core components highlighted in “Technical Details” section and architecture image:
- Automated annotation pipeline (AudioVerse) for training data.
- Unified audio tokenizer capturing semantic + acoustic features.
- DualFFN audio adapter injected into LLM to handle audio tokens.
- Support for both direct library calls (HiggsAudioServeEngine) and OpenAI-compatible HTTP API via vLLM.

1. Input Layer
– Text prompt (system & user messages)
– Optional reference audio clip
2. Audio Processing & Tokenization
– higgs_audio_tokenizer (semantic_module, quantization modules, DAC)
– Outputs discrete audio tokens + text tokens
3. Model Core
– Pretrained LLM backbone
– DualFFN audio adapter layers
– Configuration and custom CUDA Graph runner for high-throughput
4. Data Collation & Dataset (for training pipelines)
– chatml_dataset, data_collator (converts raw inputs to model-ready tensors)
5. Inference Engine
– HiggsAudioServeEngine (generate API)
– Utilities (stop strings, sampling parameters)
6. Output Layer
– Audio decoding (unquantize tokens → waveform)
– Torch-based saving (torchaudio)
7. External Services & Storage
– Hugging Face model hub (checkpoints & tokenizer)
– Docker / NGC containers (environment)
– Optional vLLM HTTP API server
b. Relationships & Interactions
– Data Collator sits between raw inputs and Model Core during training/inference.
– Serve Engine wraps Model Core and tokenization modules, exposes generate() call.
– Deployment environment: GPU/Container surrounding all inference/training components.
c. Architectural Patterns & Principles
– Modular separation: tokenization, modeling, serving, data handling.
– Pipeline architecture: clear linear flow for generation.
– Adapter pattern: Audio adapter (DualFFN) plugs into LLM without altering core.
– Abstraction of hardware acceleration: custom CUDA Graph runner for throughput.
– Extensibility: ability to swap tokenizer, adapter, or serve engine.
d. Key Technologies & Libraries
– PyTorch & TorchAudio
– CUDA (CUDA Graphs, GPU memory management)
– Hugging Face Transformers for LLM backbone
– vLLM for HTTP API compliance
– torchaudio for I/O
– Docker / NVIDIA HPC containers

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Deployment: GPU Container" 
        style Deployment fill:#f9f,stroke:#333,stroke-width:1px
        subgraph "Input Layer"
            A1["Text Prompt"]:::blue
            A2["Reference Audio"]:::blue
        end
        subgraph "Audio Processing & Tokenization"
            AP1["Unified Audio Tokenizer"]:::blue
            AP2["Semantic Feature Module"]:::blue
            AP3["Quantization Modules"]:::blue
            AP4["DAC Codec"]:::blue
        end
        subgraph "Data Handling"
            DH1["Data Collator"]:::blue
            DH2["ChatML Dataset"]:::blue
        end
        subgraph "Model Core"
            MC1["Pretrained LLM Backbone"]:::green
            MC2["DualFFN Audio Adapter"]:::green
            MC3["CUDA Graph Runner"]:::green
            MC4["Model Configuration"]:::green
            MC5["Audio Output Head"]:::green
        end
        subgraph "Inference Engine"
            IE1["HiggsAudioServeEngine"]:::orange
            IE2["Serve Utilities"]:::orange
            IE3["vLLM HTTP API"]:::orange
        end
        subgraph "Output Layer"
            O1["Audio Decoder (Unquantize)"]:::blue
            O2["Torch Save"]:::blue
        end
    end

    subgraph "External Services & Storage"
        E1["Hugging Face Hub"]:::gray
        E2["Docker/NGC Containers"]:::gray
    end

    %% Data Flow
    A1 --> AP1
    A2 --> AP1
    AP1 --> MC1
    DH2 --> DH1
    A1 --> DH1
    A2 --> DH1
    DH1 --> MC1
    MC1 --> MC2
    MC2 --> MC3
    MC3 --> MC5
    MC5 --> O1
    O1 --> O2

    %% Inference Engine Integration
    IE1 --> AP1
    IE1 --> MC1
    IE1 --> IE2
    IE3 --> IE1

    %% External Services
    MC1 --> E1
    E1 --> MC1
    E2 -.-> Deployment

    %% Click Events
    click AP1 "https://github.com/boson-ai/higgs-audio/blob/main/boson_multimodal/audio_processing/higgs_audio_tokenizer.py"
    click AP2 "https://github.com/boson-ai/higgs-audio/blob/main/boson_multimodal/audio_processing/semantic_module.py"
    click AP3 "https://github.com/boson-ai/higgs-audio/tree/main/boson_multimodal/audio_processing/quantization"
    click AP4 "https://github.com/boson-ai/higgs-audio/tree/main/boson_multimodal/audio_processing/descriptaudiocodec/dac"
    click DH1 "https://github.com/boson-ai/higgs-audio/blob/main/boson_multimodal/data_collator/higgs_audio_collator.py"
    click DH2 "https://github.com/boson-ai/higgs-audio/blob/main/boson_multimodal/dataset/chatml_dataset.py"
    click MC4 "https://github.com/boson-ai/higgs-audio/blob/main/boson_multimodal/model/higgs_audio/configuration_higgs_audio.py"
    click MC1 "https://github.com/boson-ai/higgs-audio/blob/main/boson_multimodal/model/higgs_audio/modeling_higgs_audio.py"
    click MC5 "https://github.com/boson-ai/higgs-audio/blob/main/boson_multimodal/model/higgs_audio/audio_head.py"
    click MC2 "https://github.com/boson-ai/higgs-audio/blob/main/boson_multimodal/model/higgs_audio/custom_modules.py"
    click MC3 "https://github.com/boson-ai/higgs-audio/blob/main/boson_multimodal/model/higgs_audio/cuda_graph_runner.py"
    click MC1 "https://github.com/boson-ai/higgs-audio/blob/main/boson_multimodal/model/higgs_audio/utils.py"
    click IE1 "https://github.com/boson-ai/higgs-audio/blob/main/boson_multimodal/serve/serve_engine.py"
    click IE2 "https://github.com/boson-ai/higgs-audio/blob/main/boson_multimodal/serve/utils.py"
    click IE3 "https://github.com/boson-ai/higgs-audio/blob/main/examples/vllm/run_chat_completion.py"

    classDef blue fill:#cce5ff,stroke:#004085,stroke-width:1px
    classDef green fill:#d4edda,stroke:#155724,stroke-width:1px
    classDef orange fill:#ffe5b4,stroke:#8a6d3b,stroke-width:1px
    classDef gray fill:#e2e3e5,stroke:#6c757d,stroke-width:1px
```
