How bytedance/salmonn works
Project type and purpose
• This is a multi‐modal Audio-LLM research codebase (“SALMONN”) integrating speech, audio events, and music with a large language model.
• It provides training scripts, CLI/web inference, quality‐assessment pipelines, and a video‐SALMONN extension.
File structure
• Top‐level scripts: train.py, runner.py for training; cli_inference.py, web_demo.py for inference.
• Core modules under “models/”:
• Whisper speech encoder (modeling_whisper.py)
• BEATs audio encoder (models/beats/)
• Q-Former fusion module (Qformer.py)
• LLM backbone adapters (modeling_llama.py, optims.py for LoRA)
• Data pipelines: dataset.py, plus quality_assessment/data_scripts and video_salmonn/datasets.
• Configuration: config.py and YAMLs in configs/ and video_salmonn/config/.
• Subprojects:
• quality_assessment: data generation, prompts, README.
• video_salmonn: ImageBind vision encoder, agent logic, inference, configs, prompts.
README insights
• Architecture: Dual audio encoders → window‐level Q-Former → LoRA adapters → LLM
• Training/inference instructions reference HuggingFace checkpoints for Whisper, BEATs, Vicuna.
• video-SALMONN adds a vision encoder (ImageBind) and multimodal agent logic.
a. Main components
1) Data Layer
– Raw audio/music inputs (resource/audio_demo, data/)
– Datasets & samplers (dataset.py, video_salmonn/datasets)
– Quality‐assessment scripts (quality_assessment/data_scripts)
2) Encoders
– Whisper speech encoder
– BEATs audio‐event encoder
– (video_salmonn) ImageBind vision encoder
3) Fusion & Adapter
– Q-Former fusion module
– LoRA adapter layers (optims.py integrations into LLM)
4) LLM Backbone
– Vicuna/LLAMA model implementation (modeling_llama.py)
5) Training & Optimization
– train.py / runner.py orchestrate loading data, forward/backward, distributed utils (dist_utils.py)
6) Inference Interfaces
– CLI (cli_inference.py)
– Web demo (web_demo.py)
– video_salmonn/inference.py (video mode)
7) Support & Config
– config management (config.py, YAML files)
– Logging (logger.py)
– Utilities (utils.py)
b. Relationships and interactions
• Data Layer → Encoders: audio/video files feed into respective encoder modules.
• Encoders → Q-Former: encoded features are concatenated and fused.
• Q-Former → LLM: fused embeddings (augmented tokens) injected via LoRA adapters.
• LLM → Output: generates text responses.
• Training loop uses datasets, encoders, fusion, LLM, optimizer modules.
• Inference scripts load pretrained checkpoints, run the same forward path, and expose CLI/web endpoints.
c. Architectural patterns / design principles
• Modular encoder–fusion–LLM pipeline (clear separation of concerns).
• Adapter‐based fine‐tuning (LoRA) to minimize LLM weight changes.
• Config‐driven orchestration for reproducibility.
• Extensible subprojects (quality_assessment, video_salmonn) adopting the same core pipeline.
d. Key technologies/frameworks
• PyTorch for model implementations.
• HuggingFace format for checkpoints (Whisper, Vicuna, BEATs).
• Hydra‐style YAML configs (via config.py).
• Gradio for web demo (optional).