# nvidia-ai-blueprints/llm-router architecture

> Architecture diagram of the GitHub repository nvidia-ai-blueprints/llm-router, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia-ai-blueprints/llm-router
- Repository: https://github.com/nvidia-ai-blueprints/llm-router

## Overview

This experimental NeMo Agent Toolkit product classifies text or multimodal chat requests and returns a recommended model name; unlike the documented v1, v2 does not proxy requests to that model. The main path is chat request → router wrapper → selected intent or neural-network objective → model recommendation. The neural-network branch can use CLIP embeddings and a trained router; training utilities prepare data, generate embeddings, and train/evaluate that router. The README also documents an interactive demo and health check. Some integration links are README-level or unverified because the relevant implementation was not sampled.

Main entry points: [`src/nat_sfc_router/functions/router_fn.py`](https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/functions/router_fn.py) (Router Wrapper), [`src/nat_sfc_router/schema/openai_chat_request.py`](https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/schema/openai_chat_request.py) (Chat Request Schema), [`src/nat_sfc_router/functions/hf_intent_objective_fn.py`](https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/functions/hf_intent_objective_fn.py) (Intent Objective), [`src/nat_sfc_router/functions/nn_objective_fn.py`](https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/functions/nn_objective_fn.py) (Neural Objective)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Request Interface"]
  node_chat_request["Chat Request Schema"]
  node_router_fn["Router Wrapper<br/>[router_fn.py]"]
  node_healthcheck["Health Check<br/>[healthcheck_fn.py]"]
end

subgraph group_routing["Routing Strategies"]
  node_intent["Intent Objective"]
  node_nn_objective["Neural Objective<br/>[nn_objective_fn.py]"]
  node_model_router["Model Router<br/>[model_router.py]"]
  node_nn_model["Neural Router Model<br/>[nn_router.py]"]
end

subgraph group_training["Router Training"]
  node_hf_data["Evaluation Data Prep<br/>[prepare_hf_data.py]"]
  node_embedding_gen["Embedding Generation"]
  node_training_code["Router Training<br/>[nn_router.py]"]
end

subgraph group_demo["Demo Experience"]
  node_demo_app["Interactive Demo<br/>[app.py]"]
end

node_user(("Chat User"))
node_qwen["Qwen Intent Server"]
node_clip["CLIP Embedding Service"]
node_model_providers["Model Providers"]

node_user -.->|"submits"| node_chat_request
node_chat_request -.->|"enters routing"| node_router_fn
node_router_fn -->|"accepts request"| node_chat_request
node_router_fn -->|"invokes configured objective"| node_intent
node_router_fn -.->|"invokes configured objective"| node_nn_objective
node_intent -->|"processes request"| node_chat_request
node_intent -->|"requests classification"| node_qwen
node_nn_objective -->|"processes request"| node_chat_request
node_nn_objective -->|"delegates routing"| node_model_router
node_model_router -.->|"generates embeddings"| node_clip
node_model_router -->|"routes embeddings"| node_nn_model
node_nn_model -.->|"returns model choice"| node_model_providers
node_demo_app -.->|"uses routing service"| node_router_fn
node_hf_data -.->|"prepares training inputs"| node_embedding_gen
node_embedding_gen -.->|"generates embeddings"| node_clip
node_embedding_gen -.->|"supplies embeddings"| node_training_code
node_training_code -->|"trains router"| node_nn_model

click node_chat_request "https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/schema/openai_chat_request.py"
click node_router_fn "https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/functions/router_fn.py"
click node_healthcheck "https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/functions/healthcheck_fn.py"
click node_intent "https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/functions/hf_intent_objective_fn.py"
click node_nn_objective "https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/functions/nn_objective_fn.py"
click node_model_router "https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/training/model_router.py"
click node_nn_model "https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/training/nn_router.py"
click node_demo_app "https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/demo/app.py"
click node_hf_data "https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/training/prepare_hf_data.py"
click node_embedding_gen "https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/training/generate_embeddings.py"
click node_training_code "https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/training/nn_router.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_chat_request,node_router_fn,node_healthcheck,node_user toneBlue
class node_intent,node_nn_objective,node_model_router,node_nn_model toneAmber
class node_hf_data,node_embedding_gen,node_training_code,node_qwen toneMint
class node_demo_app toneRose
class node_clip,node_model_providers toneIndigo
```

## Components

### Request Interface

- [Chat Request Schema](https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/schema/openai_chat_request.py): `src/nat_sfc_router/schema/openai_chat_request.py`
- [Router Wrapper](https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/functions/router_fn.py): `src/nat_sfc_router/functions/router_fn.py`
- [Health Check](https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/functions/healthcheck_fn.py): `src/nat_sfc_router/functions/healthcheck_fn.py`

### Routing Strategies

- [Intent Objective](https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/functions/hf_intent_objective_fn.py): `src/nat_sfc_router/functions/hf_intent_objective_fn.py`
- [Neural Objective](https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/functions/nn_objective_fn.py): `src/nat_sfc_router/functions/nn_objective_fn.py`
- [Model Router](https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/training/model_router.py): `src/nat_sfc_router/training/model_router.py`
- [Neural Router Model](https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/training/nn_router.py): `src/nat_sfc_router/training/nn_router.py`

### Router Training

- [Evaluation Data Prep](https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/training/prepare_hf_data.py): `src/nat_sfc_router/training/prepare_hf_data.py`
- [Embedding Generation](https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/training/generate_embeddings.py): `src/nat_sfc_router/training/generate_embeddings.py`
- [Router Training](https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/src/nat_sfc_router/training/nn_router.py): `src/nat_sfc_router/training/nn_router.py`

### Demo Experience

- [Interactive Demo](https://github.com/nvidia-ai-blueprints/llm-router/blob/experimental/demo/app.py): `demo/app.py`

### Other components

- **Chat User**
- **Qwen Intent Server**
- **CLIP Embedding Service**
- **Model Providers**

## Connections

- Chat User → Chat Request Schema: submits (evidence: `README.md`)
- Chat Request Schema → Router Wrapper: enters routing (evidence: `README.md`)
- Router Wrapper → Chat Request Schema: accepts request (evidence: `src/nat_sfc_router/functions/router_fn.py`)
- Router Wrapper → Intent Objective: invokes configured objective (evidence: `src/nat_sfc_router/functions/router_fn.py`)
- Router Wrapper → Neural Objective: invokes configured objective (evidence: `README.md`)
- Intent Objective → Chat Request Schema: processes request (evidence: `src/nat_sfc_router/functions/hf_intent_objective_fn.py`)
- Intent Objective → Qwen Intent Server: requests classification (evidence: `src/nat_sfc_router/functions/hf_intent_objective_fn.py`)
- Neural Objective → Chat Request Schema: processes request (evidence: `src/nat_sfc_router/functions/nn_objective_fn.py`)
- Neural Objective → Model Router: delegates routing (evidence: `src/nat_sfc_router/functions/nn_objective_fn.py`)
- Model Router → CLIP Embedding Service: generates embeddings (evidence: `README.md`)
- Model Router → Neural Router Model: routes embeddings (evidence: `src/nat_sfc_router/training/model_router.py`)
- Neural Router Model → Model Providers: returns model choice (evidence: `README.md`)
- Interactive Demo → Router Wrapper: uses routing service (evidence: `README.md`)
- Evaluation Data Prep → Embedding Generation: prepares training inputs (evidence: `README.md`)
- Embedding Generation → CLIP Embedding Service: generates embeddings (evidence: `README.md`)
- Embedding Generation → Router Training: supplies embeddings (evidence: `README.md`)
- Router Training → Neural Router Model: trains router (evidence: `src/nat_sfc_router/training/nn_router.py`)
