# gljs/audiotoolagent architecture

> Architecture diagram of the GitHub repository gljs/audiotoolagent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gljs/audiotoolagent
- Repository: https://github.com/gljs/audiotoolagent

## Overview

1. **Overall shape**
   - This repo is an **audio-language agent system** with a thin orchestration layer on top of several specialist audio models.
   - The main production path is: **CLI / benchmark runner → agent orchestrator → tool adapters → local or remote model endpoints → aggregated answer**.

2. **Core agent boundary: `audiotoolagent/`**
   - `audiotoolagent/agent.py` is the central runtime that coordinates tool calls and response synthesis.
   - `audiotoolagent/config.py` loads YAML configuration and determines which orchestrator LLM and which tools are active.
   - `audiotoolagent/tools/` contains the tool adapters that present heterogeneous audio models behind a common interface.

3. **Orchestrator vs tools**
   - The orchestrator is a general-purpose LLM configured as either:
     - **Closed**: Gemini 3 Pro
     - **Open**: Qwen3-235B via Chutes/OpenRouter fallback
   - The orchestrator does not directly solve the audio task; it decides **which specialist tool to invoke** and then fuses the outputs into the final response.

4. **Specialist tool layer**
   - Tool adapters wrap distinct audio capabilities such as:
     - **AudioFlamingo 3**
     - **Qwen2.5 Omni**
     - **Whisper**
     - **Qwen3 Instruct** or other API-backed specialists depending on config
   - Some tools are local wrappers around model servers; others call external APIs.
   - This layer is the main abstraction boundary in the system: every specialist is treated as a callable “analysis service.”

5. **Local service boundary**
   - `audiotoolagent/apis/` contains FastAPI servers used to expose local model code over HTTP.
   - `scripts/launch_open.sh` and `scripts/launch_closed.sh` start the required local services and wire them into the agent via generated host configuration.
   - The repo therefore mixes **in-process orchestration** with **out-of-process model serving**.

6. **Model serving backends**
   - `vllm_models/` contains server entry points for serving hosted LLM/audio models through vLLM-style backends.
   - These are used especially for the Qwen2.5 Omni component and other multimodal servers.
   - The architecture is service-oriented: the agent talks to these endpoints rather than embedding all model inference in one process.

7. **AudioFlamingo 3 subsystem**
   - `af3/` is a large self-contained model package for AudioFlamingo 3, including:
     - model definitions
     - multimodal encoders and projectors
     - quantization and FP8 kernels
     - training and evaluation utilities
     - a Gradio app and CLI inference entry points
   - In the agent architecture, AF3 functions as one of the **specialist inference backends**.

8. **Other specialist model packages**
   - `stepaudio/` bundles StepAudio / CosyVoice-style components, including TTS-style generation, flow models, and a web demo.
   - `desta/` contains the DESTA25 model, trainer, and audio utilities.
   - These packages are not the orchestrator; they are **model implementations** that can be consumed directly or wrapped as tools.

9. **Benchmarking and evaluation**
   - `Evaluation/` contains separate runners for **MMAU**, **MMAR**, and **MMAUPro**, each split into open and closed configurations.
   - These scripts load benchmark data, run the agent/tool pipeline, and optionally emit outputs for scoring.
   - Evaluation is therefore a first-class subsystem, not an afterthought: it exercises the same orchestration path used in inference.

10. **Configuration and environment**
    - `configs/` defines the tool roster and orchestrator choice for each mode.
    - `.env` / environment variables supply API keys and temp paths.
    - The system is designed to be **reconfigurable by YAML**: changing the tool set or provider does not require code changes in the orchestrator.

11. **Data and assets**
    - `af3/static/` and `stepaudio/assets/` provide example audio inputs and test fixtures.
    - These assets support demos, smoke tests, and benchmark examples rather than serving as primary runtime data stores.
    - There is no central database; the system is primarily **stateless**, with audio files and model endpoints as inputs.

12. **Training / optimization code is embedded but secondary**
    - Both `af3/` and `stepaudio/` include training, quantization, and kernel code.
    - This matters architecturally because the repo is not only an agent wrapper; it also vendors the underlying specialist model stacks.
    - However, for the main AudioToolAgent flow, these packages act as **implementation backends**, not as separate runtime services.

13. **Diagram takeaway**
    - A good architecture diagram should show:
      - user/benchmark input
      - agent orchestrator
      - tool registry/adapters
      - local HTTP model servers and external API services
      - specialist model packages behind those servers
      - evaluation harness as an alternate entry path
    - The key design pattern is **LLM-orchestrated tool composition over audio-specific model services**.

Main entry points: [`audiotoolagent/agent.py`](https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/agent.py) (Agent core), [`Evaluation/_common.py`](https://github.com/gljs/audiotoolagent/blob/main/Evaluation/_common.py) (Eval common), [`audiotoolagent/config.py`](https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/config.py) (Config load), [`audiotoolagent/tools/desta25.py`](https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/tools/desta25.py) (DESTA25)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry points"]
  node_main["Main CLI<br/>entry<br/>[main.py]"]
  node_launch_open["Open launch<br/>shell launcher<br/>[launch_open.sh]"]
  node_launch_closed["Closed launch<br/>shell launcher<br/>[launch_closed.sh]"]
end

subgraph group_orchestration["Orchestration"]
  node_config["Config load<br/>yaml config<br/>[config.py]"]
  node_agent["Agent core<br/>orchestrator<br/>[agent.py]"]
  node_tool_contract["Tool contract<br/>adapter base<br/>[common.py]"]
end

subgraph group_tools["Tool adapters"]
  node_orchestrator_gemini["Gemini 3<br/>closed LLM<br/>[gemini3_pro.py]"]
  node_orchestrator_qwen["Qwen 3<br/>open LLM<br/>[qwen3_instruct.py]"]
  node_tool_af3["AudioFlamingo<br/>audio tool<br/>[audioflamingo3.py]"]
  node_tool_omni["Qwen Omni<br/>audio tool<br/>[qwen2_5omni.py]"]
  node_tool_whisper["Whisper<br/>speech tool<br/>[whisper.py]"]
  node_tool_desta["DESTA25<br/>audio tool<br/>[desta25.py]"]
end

subgraph group_services["Model services"]
  node_api_common["API base<br/>http service<br/>[common.py]"]
  node_api_desta["DESTA API<br/>http service<br/>[desta25_api.py]"]
  node_api_qwen3["Qwen3 API<br/>http service"]
  node_vllm_omni["Omni server<br/>vllm backend"]
  node_vllm_audio["Audio server<br/>vllm backend"]
end

subgraph group_models["Model stacks"]
  node_af3_core["AF3 stack<br/>model package"]
  node_stepaudio_core["StepAudio<br/>model package"]
  node_desta_core["DESTA model<br/>model package"]
end

subgraph group_evaluation["Evaluation"]
  node_eval_common["Eval common<br/>benchmark harness<br/>[_common.py]"]
  node_eval_mmau_open["MMAU open<br/>benchmark runner<br/>[MMAU_Open.py]"]
  node_eval_mmau_closed["MMAU closed<br/>benchmark runner<br/>[MMAU_Closed.py]"]
  node_eval_mmar_open["MMAR open<br/>benchmark runner<br/>[MMAR_Open.py]"]
  node_eval_mmaupro_closed["MMAUPro closed<br/>benchmark runner<br/>[MMAUPro_Closed.py]"]
end

node_main -->|"loads"| node_config
node_main -->|"starts"| node_agent
node_launch_open -->|"writes mode"| node_config
node_launch_closed -->|"writes mode"| node_config
node_config -->|"configures"| node_agent
node_agent -->|"uses"| node_tool_contract
node_agent -->|"can use"| node_orchestrator_gemini
node_agent -->|"can use"| node_orchestrator_qwen
node_agent -->|"calls"| node_tool_af3
node_agent -->|"calls"| node_tool_omni
node_agent -->|"calls"| node_tool_whisper
node_agent -->|"calls"| node_tool_desta
node_tool_desta -->|"serves via"| node_api_desta
node_orchestrator_qwen -->|"falls back to"| node_api_qwen3
node_tool_af3 -->|"backed by"| node_af3_core
node_tool_omni -->|"targets"| node_vllm_omni
node_tool_whisper -->|"targets"| node_vllm_audio
node_tool_desta -->|"wraps"| node_desta_core
node_api_common -.->|"supports"| node_api_desta
node_api_common -.->|"supports"| node_api_qwen3
node_eval_common -->|"drives"| node_agent
node_eval_mmau_open -->|"imports"| node_eval_common
node_eval_mmau_closed -->|"imports"| node_eval_common
node_eval_mmar_open -->|"imports"| node_eval_common
node_eval_mmaupro_closed -->|"imports"| node_eval_common
node_af3_core -.->|"adjacent stack"| node_stepaudio_core

click node_main "https://github.com/gljs/audiotoolagent/blob/main/main.py"
click node_launch_open "https://github.com/gljs/audiotoolagent/blob/main/scripts/launch_open.sh"
click node_launch_closed "https://github.com/gljs/audiotoolagent/blob/main/scripts/launch_closed.sh"
click node_config "https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/config.py"
click node_agent "https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/agent.py"
click node_tool_contract "https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/tools/common.py"
click node_orchestrator_gemini "https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/tools/gemini3_pro.py"
click node_orchestrator_qwen "https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/tools/qwen3_instruct.py"
click node_tool_af3 "https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/tools/audioflamingo3.py"
click node_tool_omni "https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/tools/qwen2_5omni.py"
click node_tool_whisper "https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/tools/whisper.py"
click node_tool_desta "https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/tools/desta25.py"
click node_api_common "https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/apis/common.py"
click node_api_desta "https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/apis/desta25_api.py"
click node_api_qwen3 "https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/apis/qwen3_instruct_api.py"
click node_vllm_omni "https://github.com/gljs/audiotoolagent/blob/main/vllm_models/qwen2_5omni_server.py"
click node_vllm_audio "https://github.com/gljs/audiotoolagent/blob/main/vllm_models/qwen2audio_server.py"
click node_af3_core "https://github.com/gljs/audiotoolagent/tree/main/af3/llava/model"
click node_stepaudio_core "https://github.com/gljs/audiotoolagent/tree/main/stepaudio/flashcosyvoice"
click node_desta_core "https://github.com/gljs/audiotoolagent/blob/main/desta/models/modeling_desta25.py"
click node_eval_common "https://github.com/gljs/audiotoolagent/blob/main/Evaluation/_common.py"
click node_eval_mmau_open "https://github.com/gljs/audiotoolagent/blob/main/Evaluation/MMAU_Open.py"
click node_eval_mmau_closed "https://github.com/gljs/audiotoolagent/blob/main/Evaluation/MMAU_Closed.py"
click node_eval_mmar_open "https://github.com/gljs/audiotoolagent/blob/main/Evaluation/MMAR_Open.py"
click node_eval_mmaupro_closed "https://github.com/gljs/audiotoolagent/blob/main/Evaluation/MMAUPro_Closed.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_launch_open,node_launch_closed toneBlue
class node_config,node_agent,node_tool_contract toneAmber
class node_orchestrator_gemini,node_orchestrator_qwen,node_tool_af3,node_tool_omni,node_tool_whisper,node_tool_desta toneMint
class node_api_common,node_api_desta,node_api_qwen3,node_vllm_omni,node_vllm_audio toneRose
class node_af3_core,node_stepaudio_core,node_desta_core toneIndigo
class node_eval_common,node_eval_mmau_open,node_eval_mmau_closed,node_eval_mmar_open,node_eval_mmaupro_closed toneTeal
```

## Components

### Entry points

CLI and benchmark launch paths.

- [Main CLI](https://github.com/gljs/audiotoolagent/blob/main/main.py): entry · Top-level command entry for the repo. · `main.py`
- [Open launch](https://github.com/gljs/audiotoolagent/blob/main/scripts/launch_open.sh): shell launcher · Starts the open configuration and required services. · `scripts/launch_open.sh`
- [Closed launch](https://github.com/gljs/audiotoolagent/blob/main/scripts/launch_closed.sh): shell launcher · Starts the closed configuration and required services. · `scripts/launch_closed.sh`

### Orchestration

YAML-driven agent runtime and tool selection.

- [Config load](https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/config.py): yaml config · Loads mode-specific tool and orchestrator settings. · `audiotoolagent/config.py`
- [Agent core](https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/agent.py): orchestrator · Coordinates tool calls and synthesizes the final answer. · `audiotoolagent/agent.py`
- [Tool contract](https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/tools/common.py): adapter base · Shared adapter interface and common tool plumbing. · `audiotoolagent/tools/common.py`

### Tool adapters

Callable wrappers around specialist audio/LLM capabilities.

- [Gemini 3](https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/tools/gemini3_pro.py): closed LLM · Closed orchestrator option used for planning and fusion. · `audiotoolagent/tools/gemini3_pro.py`
- [Qwen 3](https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/tools/qwen3_instruct.py): open LLM · Open orchestrator option with API fallback behavior. · `audiotoolagent/tools/qwen3_instruct.py`
- [AudioFlamingo](https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/tools/audioflamingo3.py): audio tool · Adapter for AudioFlamingo 3 specialist inference. · `audiotoolagent/tools/audioflamingo3.py`
- [Qwen Omni](https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/tools/qwen2_5omni.py): audio tool · Adapter for Qwen2.5 Omni multimodal audio analysis. · `audiotoolagent/tools/qwen2_5omni.py`
- [Whisper](https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/tools/whisper.py): speech tool · Adapter for speech recognition and transcription. · `audiotoolagent/tools/whisper.py`
- [DESTA25](https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/tools/desta25.py): audio tool · Adapter for the DESTA25 specialist model. · `audiotoolagent/tools/desta25.py`

### Model services

Out-of-process HTTP and vLLM-style serving layers.

- [API base](https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/apis/common.py): http service · Common FastAPI service helpers for local model exposure. · `audiotoolagent/apis/common.py`
- [DESTA API](https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/apis/desta25_api.py): http service · FastAPI wrapper exposing DESTA25 over HTTP. · `audiotoolagent/apis/desta25_api.py`
- [Qwen3 API](https://github.com/gljs/audiotoolagent/blob/main/audiotoolagent/apis/qwen3_instruct_api.py): http service · FastAPI wrapper exposing Qwen3 Instruct over HTTP. · `audiotoolagent/apis/qwen3_instruct_api.py`
- [Omni server](https://github.com/gljs/audiotoolagent/blob/main/vllm_models/qwen2_5omni_server.py): vllm backend · Server entry for Qwen2.5 Omni and related multimodal hosting. · `vllm_models/qwen2_5omni_server.py`
- [Audio server](https://github.com/gljs/audiotoolagent/blob/main/vllm_models/qwen2audio_server.py): vllm backend · Server entry for audio-focused vLLM model hosting. · `vllm_models/qwen2audio_server.py`

### Model stacks

Embedded specialist model implementations and backends.

- [AF3 stack](https://github.com/gljs/audiotoolagent/tree/main/af3/llava/model): model package · AudioFlamingo 3 model, encoders, quantization, and inference code. · `af3/llava/model`
- [StepAudio](https://github.com/gljs/audiotoolagent/tree/main/stepaudio/flashcosyvoice): model package · CosyVoice-style TTS and audio generation implementation. · `stepaudio/flashcosyvoice`
- [DESTA model](https://github.com/gljs/audiotoolagent/blob/main/desta/models/modeling_desta25.py): model package · DESTA25 model and training code. · `desta/models/modeling_desta25.py`

### Evaluation

Benchmark runners that exercise the same agent path.

- [Eval common](https://github.com/gljs/audiotoolagent/blob/main/Evaluation/_common.py): benchmark harness · Shared runner logic for benchmark execution. · `Evaluation/_common.py`
- [MMAU open](https://github.com/gljs/audiotoolagent/blob/main/Evaluation/MMAU_Open.py): benchmark runner · Open benchmark runner for MMAU. · `Evaluation/MMAU_Open.py`
- [MMAU closed](https://github.com/gljs/audiotoolagent/blob/main/Evaluation/MMAU_Closed.py): benchmark runner · Closed benchmark runner for MMAU. · `Evaluation/MMAU_Closed.py`
- [MMAR open](https://github.com/gljs/audiotoolagent/blob/main/Evaluation/MMAR_Open.py): benchmark runner · Open benchmark runner for MMAR. · `Evaluation/MMAR_Open.py`
- [MMAUPro closed](https://github.com/gljs/audiotoolagent/blob/main/Evaluation/MMAUPro_Closed.py): benchmark runner · Closed benchmark runner for MMAUPro. · `Evaluation/MMAUPro_Closed.py`

## Connections

- Main CLI → Config load: loads · CLI resolves the active YAML configuration.
- Main CLI → Agent core: starts · CLI invokes the orchestrator runtime.
- Open launch → Config load: writes mode · Launch script prepares open-mode host and tool settings.
- Closed launch → Config load: writes mode · Launch script prepares closed-mode host and tool settings.
- Config load → Agent core: configures · Configuration selects orchestrator and tool roster.
- Agent core → Tool contract: uses · Agent relies on the shared tool interface.
- Agent core → Gemini 3: can use · Closed-mode planner option.
- Agent core → Qwen 3: can use · Open-mode planner option.
- Agent core → AudioFlamingo: calls · Invokes AudioFlamingo for specialist analysis.
- Agent core → Qwen Omni: calls · Invokes Qwen Omni for multimodal audio reasoning.
- Agent core → Whisper: calls · Invokes speech transcription when needed.
- Agent core → DESTA25: calls · Invokes DESTA25 as a specialist backend.
- DESTA25 → DESTA API: serves via · Adapter can target the local DESTA HTTP service.
- Qwen 3 → Qwen3 API: falls back to · Open orchestrator can be backed by a local API service.
- AudioFlamingo → AF3 stack: backed by · AudioFlamingo adapter reaches the AF3 model stack.
- Qwen Omni → Omni server: targets · Qwen Omni adapter reaches the multimodal server.
- Whisper → Audio server: targets · Whisper-style audio handling can be served separately.
- DESTA25 → DESTA model: wraps · DESTA adapter maps to the underlying model package.
- API base → DESTA API: supports · Common HTTP service helpers used by model APIs.
- API base → Qwen3 API: supports · Common HTTP service helpers used by model APIs.
- Eval common → Agent core: drives · Benchmark harness executes the same agent path as inference.
- MMAU open → Eval common: imports · Open MMAU runner shares common evaluation logic.
- MMAU closed → Eval common: imports · Closed MMAU runner shares common evaluation logic.
- MMAR open → Eval common: imports · MMAR runner shares common evaluation logic.
- MMAUPro closed → Eval common: imports · MMAUPro runner shares common evaluation logic.
- AF3 stack → StepAudio: adjacent stack · Both vendor specialist audio-model implementations in the repo.
