How gljs/audiotoolagent works
1. Overall shape
- This repo is an audio-language agent system with a thin orchestration layer on top of several specialist audio models.
- The main production path is: CLI / benchmark runner → agent orchestrator → tool adapters → local or remote model endpoints → aggregated answer.
2. Core agent boundary: audiotoolagent/
- audiotoolagent/agent.py is the central runtime that coordinates tool calls and response synthesis.
- audiotoolagent/config.py loads YAML configuration and determines which orchestrator LLM and which tools are active.
- audiotoolagent/tools/ contains the tool adapters that present heterogeneous audio models behind a common interface.
3. Orchestrator vs tools
- The orchestrator is a general-purpose LLM configured as either:
- Closed: Gemini 3 Pro
- Open: Qwen3-235B via Chutes/OpenRouter fallback
- The orchestrator does not directly solve the audio task; it decides which specialist tool to invoke and then fuses the outputs into the final response.
4. Specialist tool layer
- Tool adapters wrap distinct audio capabilities such as:
- AudioFlamingo 3
- Qwen2.5 Omni
- Whisper
- Qwen3 Instruct or other API-backed specialists depending on config
- Some tools are local wrappers around model servers; others call external APIs.
- This layer is the main abstraction boundary in the system: every specialist is treated as a callable “analysis service.”
5. Local service boundary
- audiotoolagent/apis/ contains FastAPI servers used to expose local model code over HTTP.
- scripts/launch_open.sh and scripts/launch_closed.sh start the required local services and wire them into the agent via generated host configuration.
- The repo therefore mixes in-process orchestration with out-of-process model serving.
6. Model serving backends
- vllm_models/ contains server entry points for serving hosted LLM/audio models through vLLM-style backends.
- These are used especially for the Qwen2.5 Omni component and other multimodal servers.
- The architecture is service-oriented: the agent talks to these endpoints rather than embedding all model inference in one process.
7. AudioFlamingo 3 subsystem
- af3/ is a large self-contained model package for AudioFlamingo 3, including:
- model definitions
- multimodal encoders and projectors
- quantization and FP8 kernels
- training and evaluation utilities
- a Gradio app and CLI inference entry points
- In the agent architecture, AF3 functions as one of the specialist inference backends.
8. Other specialist model packages
- stepaudio/ bundles StepAudio / CosyVoice-style components, including TTS-style generation, flow models, and a web demo.
- desta/ contains the DESTA25 model, trainer, and audio utilities.
- These packages are not the orchestrator; they are model implementations that can be consumed directly or wrapped as tools.
9. Benchmarking and evaluation
- Evaluation/ contains separate runners for MMAU, MMAR, and MMAUPro, each split into open and closed configurations.
- These scripts load benchmark data, run the agent/tool pipeline, and optionally emit outputs for scoring.
- Evaluation is therefore a first-class subsystem, not an afterthought: it exercises the same orchestration path used in inference.
10. Configuration and environment
- configs/ defines the tool roster and orchestrator choice for each mode.
- .env / environment variables supply API keys and temp paths.
- The system is designed to be reconfigurable by YAML: changing the tool set or provider does not require code changes in the orchestrator.
11. Data and assets
- af3/static/ and stepaudio/assets/ provide example audio inputs and test fixtures.
- These assets support demos, smoke tests, and benchmark examples rather than serving as primary runtime data stores.
- There is no central database; the system is primarily stateless, with audio files and model endpoints as inputs.
12. Training / optimization code is embedded but secondary
- Both af3/ and stepaudio/ include training, quantization, and kernel code.
- This matters architecturally because the repo is not only an agent wrapper; it also vendors the underlying specialist model stacks.
- However, for the main AudioToolAgent flow, these packages act as implementation backends, not as separate runtime services.
13. Diagram takeaway
- A good architecture diagram should show:
- user/benchmark input
- agent orchestrator
- tool registry/adapters
- local HTTP model servers and external API services
- specialist model packages behind those servers
- evaluation harness as an alternate entry path
- The key design pattern is LLM-orchestrated tool composition over audio-specific model services.
Main entry points: agent.py Agent core · _common.py Eval common · config.py Config load · desta25.py DESTA25
Components 25
Entry points
CLI and benchmark launch paths.
- Main CLI · Top-level command entry for the repo.
main.py - Open launch · Starts the open configuration and required services.
scripts/launch_open.sh - Closed launch · Starts the closed configuration and required services.
scripts/launch_closed.sh
Orchestration
YAML-driven agent runtime and tool selection.
- Config load · Loads mode-specific tool and orchestrator settings.
audiotoolagent/config.py - Agent core · Coordinates tool calls and synthesizes the final answer.
audiotoolagent/agent.py - Tool contract · Shared adapter interface and common tool plumbing.
audiotoolagent/tools/common.py
Tool adapters
Callable wrappers around specialist audio/LLM capabilities.
- Gemini 3 · Closed orchestrator option used for planning and fusion.
audiotoolagent/tools/gemini3_pro.py - Qwen 3 · Open orchestrator option with API fallback behavior.
audiotoolagent/tools/qwen3_instruct.py - AudioFlamingo · Adapter for AudioFlamingo 3 specialist inference.
audiotoolagent/tools/audioflamingo3.py - Qwen Omni · Adapter for Qwen2.5 Omni multimodal audio analysis.
audiotoolagent/tools/qwen2_5omni.py - Whisper · Adapter for speech recognition and transcription.
audiotoolagent/tools/whisper.py - DESTA25 · Adapter for the DESTA25 specialist model.
audiotoolagent/tools/desta25.py
Model services
Out-of-process HTTP and vLLM-style serving layers.
- API base · Common FastAPI service helpers for local model exposure.
audiotoolagent/apis/common.py - DESTA API · FastAPI wrapper exposing DESTA25 over HTTP.
audiotoolagent/apis/desta25_api.py - Qwen3 API · FastAPI wrapper exposing Qwen3 Instruct over HTTP.
audiotoolagent/apis/qwen3_instruct_api.py - Omni server · Server entry for Qwen2.5 Omni and related multimodal hosting.
vllm_models/qwen2_5omni_server.py - Audio server · Server entry for audio-focused vLLM model hosting.
vllm_models/qwen2audio_server.py
Model stacks
Embedded specialist model implementations and backends.
- AF3 stack · AudioFlamingo 3 model, encoders, quantization, and inference code.
af3/llava/model - StepAudio · CosyVoice-style TTS and audio generation implementation.
stepaudio/flashcosyvoice - DESTA model · DESTA25 model and training code.
desta/models/modeling_desta25.py
Evaluation
Benchmark runners that exercise the same agent path.
- Eval common · Shared runner logic for benchmark execution.
Evaluation/_common.py - MMAU open · Open benchmark runner for MMAU.
Evaluation/MMAU_Open.py - MMAU closed · Closed benchmark runner for MMAU.
Evaluation/MMAU_Closed.py - MMAR open · Open benchmark runner for MMAR.
Evaluation/MMAR_Open.py - MMAUPro closed · Closed benchmark runner for MMAUPro.
Evaluation/MMAUPro_Closed.py
Connections 26
- Main CLI → Config load: loads · CLI resolves the active YAML configuration.
- Main CLI → Agent core: starts · CLI invokes the orchestrator runtime.
- Open launch → Config load: writes mode · Launch script prepares open-mode host and tool settings.
- Closed launch → Config load: writes mode · Launch script prepares closed-mode host and tool settings.
- Config load → Agent core: configures · Configuration selects orchestrator and tool roster.
- Agent core → Tool contract: uses · Agent relies on the shared tool interface.
- Agent core → Gemini 3: can use · Closed-mode planner option.
- Agent core → Qwen 3: can use · Open-mode planner option.
- Agent core → AudioFlamingo: calls · Invokes AudioFlamingo for specialist analysis.
- Agent core → Qwen Omni: calls · Invokes Qwen Omni for multimodal audio reasoning.
- Agent core → Whisper: calls · Invokes speech transcription when needed.
- Agent core → DESTA25: calls · Invokes DESTA25 as a specialist backend.
- DESTA25 → DESTA API: serves via · Adapter can target the local DESTA HTTP service.
- Qwen 3 → Qwen3 API: falls back to · Open orchestrator can be backed by a local API service.
- AudioFlamingo → AF3 stack: backed by · AudioFlamingo adapter reaches the AF3 model stack.
- Qwen Omni → Omni server: targets · Qwen Omni adapter reaches the multimodal server.
- Whisper → Audio server: targets · Whisper-style audio handling can be served separately.
- DESTA25 → DESTA model: wraps · DESTA adapter maps to the underlying model package.
- API base → DESTA API: supports · Common HTTP service helpers used by model APIs.
- API base → Qwen3 API: supports · Common HTTP service helpers used by model APIs.
- Eval common → Agent core: drives · Benchmark harness executes the same agent path as inference.
- MMAU open → Eval common: imports · Open MMAU runner shares common evaluation logic.
- MMAU closed → Eval common: imports · Closed MMAU runner shares common evaluation logic.
- MMAR open → Eval common: imports · MMAR runner shares common evaluation logic.
- MMAUPro closed → Eval common: imports · MMAUPro runner shares common evaluation logic.
- AF3 stack → StepAudio: adjacent stack · Both vendor specialist audio-model implementations in the repo.