# bytedance/salmonn architecture

> Architecture diagram of the GitHub repository bytedance/salmonn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bytedance/salmonn
- Repository: https://github.com/bytedance/salmonn

## Overview

##### Project type and purpose
- This is a multi‐modal Audio-LLM research codebase (“SALMONN”) integrating speech, audio events, and music with a large language model.
- It provides training scripts, CLI/web inference, quality‐assessment pipelines, and a video‐SALMONN extension.

##### File structure
- Top‐level scripts: train.py, runner.py for training; cli_inference.py, web_demo.py for inference.
- Core modules under “models/”:
- Whisper speech encoder (modeling_whisper.py)
- BEATs audio encoder (models/beats/)
- Q-Former fusion module (Qformer.py)
- LLM backbone adapters (modeling_llama.py, optims.py for LoRA)
- Data pipelines: dataset.py, plus quality_assessment/data_scripts and video_salmonn/datasets.
- Configuration: config.py and YAMLs in configs/ and video_salmonn/config/.
- Subprojects:
- quality_assessment: data generation, prompts, README.
- video_salmonn: ImageBind vision encoder, agent logic, inference, configs, prompts.

##### README insights
- Architecture: Dual audio encoders → window‐level Q-Former → LoRA adapters → LLM
- Training/inference instructions reference HuggingFace checkpoints for Whisper, BEATs, Vicuna.
- video-SALMONN adds a vision encoder (ImageBind) and multimodal agent logic.

a. Main components
1) Data Layer
– Raw audio/music inputs (resource/audio_demo, data/)
– Datasets & samplers (dataset.py, video_salmonn/datasets)
– Quality‐assessment scripts (quality_assessment/data_scripts)
2) Encoders
– Whisper speech encoder
– BEATs audio‐event encoder
– (video_salmonn) ImageBind vision encoder
3) Fusion & Adapter
– Q-Former fusion module
– LoRA adapter layers (optims.py integrations into LLM)
4) LLM Backbone
– Vicuna/LLAMA model implementation (modeling_llama.py)
5) Training & Optimization
– train.py / runner.py orchestrate loading data, forward/backward, distributed utils (dist_utils.py)
6) Inference Interfaces
– CLI (cli_inference.py)
– Web demo (web_demo.py)
– video_salmonn/inference.py (video mode)
7) Support & Config
– config management (config.py, YAML files)
– Logging (logger.py)
– Utilities (utils.py)

b. Relationships and interactions
- Data Layer → Encoders: audio/video files feed into respective encoder modules.
- Encoders → Q-Former: encoded features are concatenated and fused.
- Q-Former → LLM: fused embeddings (augmented tokens) injected via LoRA adapters.
- LLM → Output: generates text responses.
- Training loop uses datasets, encoders, fusion, LLM, optimizer modules.
- Inference scripts load pretrained checkpoints, run the same forward path, and expose CLI/web endpoints.

c. Architectural patterns / design principles
- Modular encoder–fusion–LLM pipeline (clear separation of concerns).
- Adapter‐based fine‐tuning (LoRA) to minimize LLM weight changes.
- Config‐driven orchestration for reproducibility.
- Extensible subprojects (quality_assessment, video_salmonn) adopting the same core pipeline.

d. Key technologies/frameworks
- PyTorch for model implementations.
- HuggingFace format for checkpoints (Whisper, Vicuna, BEATs).
- Hydra‐style YAML configs (via config.py).
- Gradio for web demo (optional).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Layer"
        DL1["dataset.py"]:::data
        DL2["data/example_data.json"]:::data
        DL3["resource/audio_demo/"]:::data
        DL4["quality_assessment/data_scripts/"]:::data
        DL5["video_salmonn/datasets/"]:::data
    end

    subgraph "Encoders"
        E1["Whisper Encoder (core)"]:::encoder
        E2["Whisper Encoder (video)"]:::encoder
        E3["BEATs Encoder (core)"]:::encoder
        E4["BEATs Encoder (video)"]:::encoder
        E5["ImageBind Encoder"]:::encoder
    end

    subgraph "Fusion & Adapter"
        F1["Q-Former (core)"]:::fusion
        F2["Q-Former (video)"]:::fusion
        A1["LoRA Adapter & Optimizers"]:::fusion
    end

    subgraph "LLM Backbone"
        L1["LLAMA/Vicuna (core)"]:::llm
        L2["LLAMA (video)"]:::llm
        L3["Adapter Utils"]:::llm
    end

    subgraph "Training & Optimization"
        T1["train.py"]:::script
        T2["runner.py"]:::script
        T3["dist_utils.py"]:::script
    end

    subgraph "Inference Interfaces"
        I1["cli_inference.py"]:::script
        I2["web_demo.py"]:::script
        I3["video_salmonn/inference.py"]:::script
    end

    subgraph "Config & Support"
        C1["config.py"]:::support
        C2["configs/*.yaml"]:::support
        C3["video_salmonn/config/"]:::support
        C4["logger.py"]:::support
        C5["utils.py"]:::support
    end

    subgraph "Subprojects & Prompts"
        S1["quality_assessment/"]:::support
        S2["prompts/"]:::support
        S3["video_salmonn/prompt/"]:::support
    end

    HF["HuggingFace Checkpoints"]:::external
    GPU["GPU Cluster"]:::external

    %% Data to Encoders
    DL1 --> E1
    DL1 --> E3
    DL5 --> E5
    DL4 --> DL1

    %% Encoders to Fusion
    E1 --> F1
    E3 --> F1
    E2 --> F2
    E4 --> F2
    E5 --> F2

    %% Fusion to LLM
    F1 --> L1
    F2 --> L2
    A1 --> L1
    A1 --> L2

    %% Training flow
    L1 --> T1
    L2 --> T1
    T1 --> HF
    T1 --> GPU

    %% Inference flow
    DL1 --> I1
    E1 --> I1
    F1 --> I1
    L1 --> I1
    C1 --> I1
    C2 --> I1
    C3 --> I3
    L1 --> I2
    L2 --> I3
    GPU --> I1

    %% Config to training
    C1 --> T1
    C2 --> T1
    C3 --> T1
    C4 --> T1
    C5 --> T1

    %% Subproject links
    S1 --> DL1
    S1 --> T1
    S3 --> E2
    S3 --> F2
    S3 --> L2
    S3 --> I3

    %% Click Events
    click DL1 "https://github.com/bytedance/salmonn/blob/main/dataset.py"
    click DL2 "https://github.com/bytedance/salmonn/blob/main/data/example_data.json"
    click DL3 "https://github.com/bytedance/salmonn/tree/main/resource/audio_demo/"
    click DL4 "https://github.com/bytedance/salmonn/tree/main/quality_assessment/data_scripts/"
    click DL5 "https://github.com/bytedance/salmonn/tree/main/video_salmonn/datasets/"
    click E1 "https://github.com/bytedance/salmonn/blob/main/models/modeling_whisper.py"
    click E2 "https://github.com/bytedance/salmonn/blob/main/video_salmonn/model/modeling_whisper.py"
    click E3 "https://github.com/bytedance/salmonn/tree/main/models/beats/"
    click E4 "https://github.com/bytedance/salmonn/tree/main/video_salmonn/model/beats/"
    click E5 "https://github.com/bytedance/salmonn/tree/main/video_salmonn/model/ImageBind/"
    click F1 "https://github.com/bytedance/salmonn/blob/main/models/Qformer.py"
    click F2 "https://github.com/bytedance/salmonn/blob/main/video_salmonn/model/Qformer.py"
    click A1 "https://github.com/bytedance/salmonn/blob/main/optims.py"
    click L1 "https://github.com/bytedance/salmonn/blob/main/models/modeling_llama.py"
    click L2 "https://github.com/bytedance/salmonn/blob/main/video_salmonn/model/modeling_llama.py"
    click L3 "https://github.com/bytedance/salmonn/blob/main/models/utils.py"
    click T1 "https://github.com/bytedance/salmonn/blob/main/train.py"
    click T2 "https://github.com/bytedance/salmonn/blob/main/runner.py"
    click T3 "https://github.com/bytedance/salmonn/blob/main/dist_utils.py"
    click I1 "https://github.com/bytedance/salmonn/blob/main/cli_inference.py"
    click I2 "https://github.com/bytedance/salmonn/blob/main/web_demo.py"
    click I3 "https://github.com/bytedance/salmonn/blob/main/video_salmonn/inference.py"
    click C1 "https://github.com/bytedance/salmonn/blob/main/config.py"
    click C2 "https://github.com/bytedance/salmonn/tree/main/configs/"
    click C3 "https://github.com/bytedance/salmonn/tree/main/video_salmonn/config/"
    click C4 "https://github.com/bytedance/salmonn/blob/main/logger.py"
    click C5 "https://github.com/bytedance/salmonn/blob/main/utils.py"
    click S1 "https://github.com/bytedance/salmonn/tree/main/quality_assessment/"
    click S2 "https://github.com/bytedance/salmonn/tree/main/prompts/"
    click S3 "https://github.com/bytedance/salmonn/tree/main/video_salmonn/prompt/"

    classDef data fill:#e0f2f1,stroke:#00695c
    classDef encoder fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px
    classDef fusion fill:#e1bee7,stroke:#8e24aa,stroke-width:2px
    classDef llm fill:#bbdefb,stroke:#1e88e5,stroke-width:2px
    classDef script fill:#ffe0b2,stroke:#ef6c00,stroke-width:2px
    classDef support fill:#f5f5f5,stroke:#9e9e9e
    classDef external fill:#cfd8dc,stroke:#455a64,stroke-dasharray:5 5
```
