# ericlbuehler/candle-vllm architecture

> Architecture diagram of the GitHub repository ericlbuehler/candle-vllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ericlbuehler/candle-vllm
- Repository: https://github.com/ericlbuehler/candle-vllm

## Overview

1. **What this repo is**
   Candle-vLLM is a Rust inference/serving system for local LLMs. It ships both a library crate (`src/lib.rs`) and a command-line server binary (`src/main.rs`) that can expose an OpenAI-compatible API, stream tokens, and optionally serve a built-in UI.

2. **Top-level runtime flow**
   The main request path is: client request → OpenAI API server (`src/openai/openai_server.rs`) → request normalization/types (`src/openai/requests.rs`, `responses.rs`, `sampling_params.rs`) → pipeline/engine orchestration (`src/openai/pipelines/*`) → scheduler (`src/scheduler/*`) → backend model execution (`src/backend/*`) → streamed or buffered response back to the client.

3. **API and compatibility boundary**
   The `openai` module is the public protocol boundary. It translates OpenAI-style chat/completion/streaming requests into internal model inputs, applies conversation state handling (`src/openai/conversation/*`), and returns OpenAI-shaped responses. This is the main integration surface for external tools, agents, and UI clients.

4. **Model serving core**
   The serving logic is split into reusable pipeline variants under `src/openai/pipelines/`: threaded, multi-process, streaming, and multi-process/distributed helpers. These components coordinate batching, token generation, and request lifecycle management through the `llm_engine` and `pipeline` abstractions.

5. **Scheduler and KV/cache management**
   The scheduler subsystem (`src/scheduler/`) is responsible for continuous batching, sequence tracking, and memory-aware execution. `block_engine.rs`, `cache_engine.rs`, `sequence.rs`, and `prefix_cache.rs` indicate that the system manages KV blocks explicitly, supports prefix reuse, and adapts to constrained memory budgets. `mamba.rs` suggests model-specific scheduling support where needed.

6. **Backend execution layer**
   The `backend` module is where model weights are loaded and executed. It contains loaders/adapters for GGUF, GPTQ/Marlin, and general graph execution (`gguf.rs`, `gptq.rs`, `graph.rs`), plus runtime services such as cache, heartbeat, and progress reporting. `backend/custom_ops/*` contains specialized kernels/operators, including MoE and sorting paths.

7. **Model family abstraction**
   `src/openai/models/*` is a large model-specific layer with implementations for many architectures: Llama, Qwen, Mistral, Gemma, DeepSeek, GLM, Phi, Yi, StableLM, MiniMax, and several vision-language variants. These modules encapsulate architecture-specific layer wiring, rotary embeddings, attention variants, MLPs, MoE, and quantized weight formats.

8. **Quantization and device-specific optimization**
   The repo supports multiple execution formats and kernels: BF16/F16, GGUF, GPTQ/Marlin, FP8 blockwise models, MXFP4/NVFP4, and “TurboQuant” KV compression. The architecture reflects this through specialized model files (`quantized_*`), layer helpers (`quantized_var_builder.rs`, `linear.rs`, `rotary_emb.rs`), and backend features for CUDA, Metal, flash attention, FlashInfer, and CUDA graphs.

9. **Multimodal path**
   Multimodal support is handled through dedicated model subtrees like `gemma3_vl`, `llama4`, `mistral3_vl`, and `qwen3_vl`, plus `src/openai/multimodal.rs`. These modules add vision input/config/vision-tower logic while still flowing through the same serving pipeline and response surface as text-only models.

10. **Tool calling and MCP**
   Tool integration is a separate subsystem under `src/tools/*` and `src/mcp/*`. The tools package parses and validates model-emitted tool calls, while the MCP modules implement client/server/transport/manager pieces for Model Context Protocol integration. This sits between generation output parsing and downstream tool execution.

11. **Streaming and output post-processing**
   Token streaming is explicitly supported via `src/openai/streaming.rs`, `pipelines/streaming.rs`, and `communicator.rs`. After model decoding, output can be incrementally parsed/logged, including tool-call extraction (`src/tools/stream_parser.rs`) and model-specific logits/postprocessing (`logits_processor.rs`, `logger.rs`).

12. **Distributed and multi-node execution**
   The repo includes explicit distributed inference support in `src/openai/distributed.rs` and `src/backend/heartbeat.rs`, with CLI flags for multi-GPU and TCP-based multi-node coordination. The architecture assumes local weight availability per node and uses NCCL/bootstrap-style coordination rather than MPI.

13. **Packaging and deployment surfaces**
   The repo is packaged for Rust/Cargo, Docker, and Python-based conversion/utilities. `build.rs`, `Dockerfile`, `build_docker.sh`, `.github/workflows/*`, `setup.py`, and the `examples/` directory support release builds, container deployment, format conversion, benchmarking, and client-side usage examples.

14. **How to draw the architecture**
   A useful diagram would show three boundaries: external clients/UI/MCP tools at the top, the OpenAI-compatible server and pipelines in the middle, and the scheduler/backend/model kernels at the bottom. Side paths should show optional multimodal input, tool-calling/MCP, prefix/KV cache services, and distributed execution across multiple devices or nodes.

Main entry points: [`src/openai/openai_server.rs`](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/openai_server.rs) (OpenAI server), [`src/openai/pipelines/mod.rs`](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/pipelines/mod.rs) (Pipelines), [`src/scheduler/mod.rs`](https://github.com/ericlbuehler/candle-vllm/blob/master/src/scheduler/mod.rs) (Scheduler), [`src/backend/mod.rs`](https://github.com/ericlbuehler/candle-vllm/blob/master/src/backend/mod.rs) (Backend core)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_clients["Clients & entrypoints"]
  node_main["Server binary<br/>rust bin<br/>[main.rs]"]
  node_lib["Library crate<br/>rust lib<br/>[lib.rs]"]
end

subgraph group_openai["OpenAI boundary"]
  node_api["API surface<br/>rust module<br/>[api.rs]"]
  node_openai_server["OpenAI server<br/>[openai_server.rs]"]
  node_requests["Request types<br/>protocol types<br/>[requests.rs]"]
  node_responses["Response types<br/>protocol types<br/>[responses.rs]"]
  node_pipelines["Pipelines<br/>orchestration<br/>[mod.rs]"]
  node_llm_engine["LLM engine<br/>runtime core<br/>[llm_engine.rs]"]
  node_threaded["Threaded pipe<br/>pipeline variant<br/>[threaded.rs]"]
  node_multiprocess["Multi-process pipe<br/>pipeline variant<br/>[multiprocess.rs]"]
  node_streaming["Streaming pipe<br/>pipeline variant<br/>[streaming.rs]"]
  node_conversation["Conversation state<br/>dialog state<br/>[mod.rs]"]
  node_distributed["Distributed mode<br/>coordination<br/>[distributed.rs]"]
  node_multimodal["Multimodal glue<br/>input adapter<br/>[multimodal.rs]"]
  node_streaming_api["Streaming output<br/>[streaming.rs]"]
end

subgraph group_scheduler["Scheduling & cache"]
  node_scheduler_core["Scheduler<br/>batching core<br/>[mod.rs]"]
  node_block_engine[("Block engine<br/>kv blocks<br/>[block_engine.rs]")]
  node_cache_engine[("Cache engine<br/>cache manager<br/>[cache_engine.rs]")]
  node_sequence["Sequence tracking<br/>sequence state<br/>[sequence.rs]"]
  node_prefix_cache[("Prefix cache<br/>reuse cache<br/>[prefix_cache.rs]")]
end

subgraph group_backend["Model backend"]
  node_backend_core["Backend core<br/>model runtime<br/>[mod.rs]"]
  node_gguf["GGUF loader<br/>weights loader<br/>[gguf.rs]"]
  node_gptq["GPTQ/Marlin<br/>quantized loader<br/>[gptq.rs]"]
  node_graph["Graph runtime<br/>execution graph<br/>[graph.rs]"]
  node_custom_ops["Custom ops<br/>kernels<br/>[mod.rs]"]
end

subgraph group_models["Model families"]
  node_layers["Shared layers<br/>model layers<br/>[mod.rs]"]
  node_llama["Llama family<br/>model family<br/>[llama.rs]"]
  node_qwen_vl["Qwen VL<br/>multimodal model<br/>[mod.rs]"]
  node_gemma_vl["Gemma VL<br/>multimodal model<br/>[mod.rs]"]
end

subgraph group_integration["Tools & MCP"]
  node_tools["Tool parsing<br/>tooling<br/>[mod.rs]"]
  node_mcp["MCP stack<br/>protocol integration<br/>[mod.rs]"]
end

node_main -->|"starts"| node_openai_server
node_lib -->|"exports"| node_api
node_api -->|"wires"| node_openai_server
node_openai_server -->|"parses"| node_requests
node_openai_server -->|"returns"| node_responses
node_openai_server -->|"dispatches"| node_pipelines
node_openai_server -->|"state"| node_conversation
node_openai_server -->|"optional input"| node_multimodal
node_openai_server -->|"streams"| node_streaming_api
node_pipelines -->|"hosts"| node_llm_engine
node_pipelines -->|"variant"| node_threaded
node_pipelines -->|"variant"| node_multiprocess
node_pipelines -->|"variant"| node_streaming
node_pipelines -->|"optional"| node_distributed
node_llm_engine -->|"drives"| node_scheduler_core
node_scheduler_core -->|"allocates"| node_block_engine
node_scheduler_core -->|"reuses"| node_cache_engine
node_scheduler_core -->|"tracks"| node_sequence
node_scheduler_core -->|"optimizes"| node_prefix_cache
node_scheduler_core -->|"executes via"| node_backend_core
node_backend_core -->|"loads"| node_gguf
node_backend_core -->|"loads"| node_gptq
node_backend_core -->|"runs"| node_graph
node_backend_core -->|"uses"| node_custom_ops
node_graph -->|"composes"| node_layers
node_layers -->|"used by"| node_llama
node_layers -->|"used by"| node_qwen_vl
node_layers -->|"used by"| node_gemma_vl
node_qwen_vl -->|"ingests"| node_multimodal
node_gemma_vl -->|"ingests"| node_multimodal
node_openai_server -->|"tool calls"| node_tools
node_tools -->|"bridges"| node_mcp

click node_main "https://github.com/ericlbuehler/candle-vllm/blob/master/src/main.rs"
click node_lib "https://github.com/ericlbuehler/candle-vllm/blob/master/src/lib.rs"
click node_api "https://github.com/ericlbuehler/candle-vllm/blob/master/src/api.rs"
click node_openai_server "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/openai_server.rs"
click node_requests "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/requests.rs"
click node_responses "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/responses.rs"
click node_pipelines "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/pipelines/mod.rs"
click node_llm_engine "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/pipelines/llm_engine.rs"
click node_threaded "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/pipelines/threaded.rs"
click node_multiprocess "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/pipelines/multiprocess.rs"
click node_streaming "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/pipelines/streaming.rs"
click node_conversation "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/conversation/mod.rs"
click node_distributed "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/distributed.rs"
click node_scheduler_core "https://github.com/ericlbuehler/candle-vllm/blob/master/src/scheduler/mod.rs"
click node_block_engine "https://github.com/ericlbuehler/candle-vllm/blob/master/src/scheduler/block_engine.rs"
click node_cache_engine "https://github.com/ericlbuehler/candle-vllm/blob/master/src/scheduler/cache_engine.rs"
click node_sequence "https://github.com/ericlbuehler/candle-vllm/blob/master/src/scheduler/sequence.rs"
click node_prefix_cache "https://github.com/ericlbuehler/candle-vllm/blob/master/src/scheduler/prefix_cache.rs"
click node_backend_core "https://github.com/ericlbuehler/candle-vllm/blob/master/src/backend/mod.rs"
click node_gguf "https://github.com/ericlbuehler/candle-vllm/blob/master/src/backend/gguf.rs"
click node_gptq "https://github.com/ericlbuehler/candle-vllm/blob/master/src/backend/gptq.rs"
click node_graph "https://github.com/ericlbuehler/candle-vllm/blob/master/src/backend/graph.rs"
click node_custom_ops "https://github.com/ericlbuehler/candle-vllm/blob/master/src/backend/custom_ops/mod.rs"
click node_layers "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/models/layers/mod.rs"
click node_llama "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/models/llama.rs"
click node_qwen_vl "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/models/qwen3_vl/mod.rs"
click node_gemma_vl "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/models/gemma3_vl/mod.rs"
click node_tools "https://github.com/ericlbuehler/candle-vllm/blob/master/src/tools/mod.rs"
click node_mcp "https://github.com/ericlbuehler/candle-vllm/blob/master/src/mcp/mod.rs"
click node_multimodal "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/multimodal.rs"
click node_streaming_api "https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/streaming.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_lib toneBlue
class node_api,node_openai_server,node_requests,node_responses,node_pipelines,node_llm_engine,node_threaded,node_multiprocess,node_streaming,node_conversation,node_distributed,node_multimodal,node_streaming_api toneAmber
class node_scheduler_core,node_block_engine,node_cache_engine,node_sequence,node_prefix_cache toneMint
class node_backend_core,node_gguf,node_gptq,node_graph,node_custom_ops toneRose
class node_layers,node_llama,node_qwen_vl,node_gemma_vl toneIndigo
class node_tools,node_mcp toneTeal
```

## Components

### Clients & entrypoints

External-facing entrypoints and optional UI/deployment surfaces.

- [Server binary](https://github.com/ericlbuehler/candle-vllm/blob/master/src/main.rs): rust bin · CLI entrypoint that starts serving and runtime coordination. · `src/main.rs`
- [Library crate](https://github.com/ericlbuehler/candle-vllm/blob/master/src/lib.rs): rust lib · Reusable crate surface exported to binaries and downstream users. · `src/lib.rs`

### OpenAI boundary

Protocol translation, request handling, pipelines, and serving orchestration.

- [API surface](https://github.com/ericlbuehler/candle-vllm/blob/master/src/api.rs): rust module · High-level API glue and public-facing server wiring. · `src/api.rs`
- [OpenAI server](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/openai_server.rs): server · OpenAI-compatible request handling and response delivery. · `src/openai/openai_server.rs`
- [Request types](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/requests.rs): protocol types · Chat/completion request normalization and sampling inputs. · `src/openai/requests.rs`
- [Response types](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/responses.rs): protocol types · OpenAI-shaped response structures and streaming payloads. · `src/openai/responses.rs`
- [Pipelines](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/pipelines/mod.rs): orchestration · Serving pipeline variants and engine coordination layer. · `src/openai/pipelines/mod.rs`
- [LLM engine](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/pipelines/llm_engine.rs): runtime core · Shared execution engine used by pipeline variants. · `src/openai/pipelines/llm_engine.rs`
- [Threaded pipe](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/pipelines/threaded.rs): pipeline variant · Single-process threaded serving path. · `src/openai/pipelines/threaded.rs`
- [Multi-process pipe](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/pipelines/multiprocess.rs): pipeline variant · Process-separated serving path for scaling and isolation. · `src/openai/pipelines/multiprocess.rs`
- [Streaming pipe](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/pipelines/streaming.rs): pipeline variant · Incremental token delivery and stream orchestration. · `src/openai/pipelines/streaming.rs`
- [Conversation state](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/conversation/mod.rs): dialog state · Conversation assembly and default prompt/state handling. · `src/openai/conversation/mod.rs`
- [Distributed mode](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/distributed.rs): coordination · Multi-node / multi-GPU serving coordination hooks. · `src/openai/distributed.rs`
- [Multimodal glue](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/multimodal.rs): input adapter · Bridges image and other non-text inputs into the OpenAI pipeline. · `src/openai/multimodal.rs`
- [Streaming output](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/streaming.rs): streaming · Token streaming and incremental response formatting. · `src/openai/streaming.rs`

### Scheduling & cache

Sequence tracking, batching, and KV/cache-aware execution.

- [Scheduler](https://github.com/ericlbuehler/candle-vllm/blob/master/src/scheduler/mod.rs): batching core · Sequence scheduling, continuous batching, and memory-aware execution. · `src/scheduler/mod.rs`
- [Block engine](https://github.com/ericlbuehler/candle-vllm/blob/master/src/scheduler/block_engine.rs): kv blocks · Explicit KV block management and allocation strategy. · `src/scheduler/block_engine.rs`
- [Cache engine](https://github.com/ericlbuehler/candle-vllm/blob/master/src/scheduler/cache_engine.rs): cache manager · Cache lifecycle and reuse management for inference. · `src/scheduler/cache_engine.rs`
- [Sequence tracking](https://github.com/ericlbuehler/candle-vllm/blob/master/src/scheduler/sequence.rs): sequence state · Per-request token/sequence bookkeeping and scheduling state. · `src/scheduler/sequence.rs`
- [Prefix cache](https://github.com/ericlbuehler/candle-vllm/blob/master/src/scheduler/prefix_cache.rs): reuse cache · Prefix reuse and prompt cache acceleration path. · `src/scheduler/prefix_cache.rs`

### Model backend

Weight loading, graph execution, runtime services, and custom kernels.

- [Backend core](https://github.com/ericlbuehler/candle-vllm/blob/master/src/backend/mod.rs): model runtime · Model loading, execution, graph runtime, and backend services. · `src/backend/mod.rs`
- [GGUF loader](https://github.com/ericlbuehler/candle-vllm/blob/master/src/backend/gguf.rs): weights loader · GGUF model format loading and backend adaptation. · `src/backend/gguf.rs`
- [GPTQ/Marlin](https://github.com/ericlbuehler/candle-vllm/blob/master/src/backend/gptq.rs): quantized loader · GPTQ and Marlin quantized execution support. · `src/backend/gptq.rs`
- [Graph runtime](https://github.com/ericlbuehler/candle-vllm/blob/master/src/backend/graph.rs): execution graph · Core graph execution and model runtime wiring. · `src/backend/graph.rs`
- [Custom ops](https://github.com/ericlbuehler/candle-vllm/blob/master/src/backend/custom_ops/mod.rs): kernels · Specialized backend kernels and operators. · `src/backend/custom_ops/mod.rs`

### Model families

Architecture-specific model implementations and shared layer helpers.

- [Shared layers](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/models/layers/mod.rs): model layers · Reusable attention, MLP, norm, rotary, and quantized layer helpers. · `src/openai/models/layers/mod.rs`
- [Llama family](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/models/llama.rs): model family · Representative decoder-only model implementation and variants. · `src/openai/models/llama.rs`
- [Qwen VL](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/models/qwen3_vl/mod.rs): multimodal model · Vision-language model subtree for Qwen-style multimodal inference. · `src/openai/models/qwen3_vl/mod.rs`
- [Gemma VL](https://github.com/ericlbuehler/candle-vllm/blob/master/src/openai/models/gemma3_vl/mod.rs): multimodal model · Vision-language model subtree for Gemma-style multimodal inference. · `src/openai/models/gemma3_vl/mod.rs`

### Tools & MCP

Tool parsing/execution integration and MCP client/server plumbing.

- [Tool parsing](https://github.com/ericlbuehler/candle-vllm/blob/master/src/tools/mod.rs): tooling · Tool-call parsing, validation, and schema helpers. · `src/tools/mod.rs`
- [MCP stack](https://github.com/ericlbuehler/candle-vllm/blob/master/src/mcp/mod.rs): protocol integration · Model Context Protocol client/server/transport/manager plumbing. · `src/mcp/mod.rs`

## Connections

- Server binary → OpenAI server: starts · Binary launches the OpenAI-compatible server.
- Library crate → API surface: exports · Library crate exposes the API/server modules.
- API surface → OpenAI server: wires · Public API connects to serving entrypoints.
- OpenAI server → Request types: parses · Server normalizes incoming OpenAI-style requests.
- OpenAI server → Response types: returns · Server emits OpenAI-shaped responses and streams.
- OpenAI server → Pipelines: dispatches · Request handling delegates into serving pipelines.
- OpenAI server → Conversation state: state · Conversation context is assembled before generation.
- OpenAI server → Multimodal glue: optional input · Vision and other non-text inputs are adapted here.
- OpenAI server → Streaming output: streams · Streaming responses are surfaced to clients.
- Pipelines → LLM engine: hosts · Pipelines share the common engine abstraction.
- Pipelines → Threaded pipe: variant · Single-process serving path.
- Pipelines → Multi-process pipe: variant · Multi-process serving path.
- Pipelines → Streaming pipe: variant · Incremental decoding path.
- Pipelines → Distributed mode: optional · Distributed coordination is enabled when configured.
- LLM engine → Scheduler: drives · Engine feeds work into the scheduler.
- Scheduler → Block engine: allocates · Scheduler manages KV block placement.
- Scheduler → Cache engine: reuses · Scheduler consults cache-aware execution state.
- Scheduler → Sequence tracking: tracks · Per-request sequence state is maintained.
- Scheduler → Prefix cache: optimizes · Prefix reuse accelerates repeated prompts.
- Scheduler → Backend core: executes via · Scheduled work is handed to backend runtime.
- Backend core → GGUF loader: loads · GGUF weights are adapted into runtime form.
- Backend core → GPTQ/Marlin: loads · Quantized model formats are supported.
- Backend core → Graph runtime: runs · Graph execution performs model forward passes.
- Backend core → Custom ops: uses · Specialized kernels accelerate model execution.
- Graph runtime → Shared layers: composes · Graph execution relies on shared layer building blocks.
- Shared layers → Llama family: used by · Representative model family built on shared layers.
- Shared layers → Qwen VL: used by · Multimodal Qwen subtree builds on shared layers.
- Shared layers → Gemma VL: used by · Multimodal Gemma subtree builds on shared layers.
- Qwen VL → Multimodal glue: ingests · Vision-language inputs are normalized into model inputs.
- Gemma VL → Multimodal glue: ingests · Vision-language inputs are normalized into model inputs.
- OpenAI server → Tool parsing: tool calls · Model-emitted tool calls are parsed and validated.
- Tool parsing → MCP stack: bridges · Tool integration can hand off to MCP clients/servers.
