How nvidia-ai-blueprints/llm-router works
This experimental NeMo Agent Toolkit product classifies text or multimodal chat requests and returns a recommended model name; unlike the documented v1, v2 does not proxy requests to that model. The main path is chat request → router wrapper → selected intent or neural-network objective → model recommendation. The neural-network branch can use CLIP embeddings and a trained router; training utilities prepare data, generate embeddings, and train/evaluate that router. The README also documents an interactive demo and health check. Some integration links are README-level or unverified because the relevant implementation was not sampled.
Main entry points: router_fn.py Router Wrapper · openai_chat_request.py Chat Request Schema · hf_intent_objective_fn.py Intent Objective · nn_objective_fn.py Neural Objective
Components 15
Request Interface
- Chat Request Schema
src/nat_sfc_router/schema/openai_chat_request.py - Router Wrapper
src/nat_sfc_router/functions/router_fn.py - Health Check
src/nat_sfc_router/functions/healthcheck_fn.py
Routing Strategies
- Intent Objective
src/nat_sfc_router/functions/hf_intent_objective_fn.py - Neural Objective
src/nat_sfc_router/functions/nn_objective_fn.py - Model Router
src/nat_sfc_router/training/model_router.py - Neural Router Model
src/nat_sfc_router/training/nn_router.py
Router Training
- Evaluation Data Prep
src/nat_sfc_router/training/prepare_hf_data.py - Embedding Generation
src/nat_sfc_router/training/generate_embeddings.py - Router Training
src/nat_sfc_router/training/nn_router.py
Demo Experience
- Interactive Demo
demo/app.py
Other components
- Chat User
- Qwen Intent Server
- CLIP Embedding Service
- Model Providers
Connections 17
- Chat User → Chat Request Schema: submits
README.md - Chat Request Schema → Router Wrapper: enters routing
README.md - Router Wrapper → Chat Request Schema: accepts request
src/nat_sfc_router/functions/router_fn.py - Router Wrapper → Intent Objective: invokes configured objective
src/nat_sfc_router/functions/router_fn.py - Router Wrapper → Neural Objective: invokes configured objective
README.md - Intent Objective → Chat Request Schema: processes request
src/nat_sfc_router/functions/hf_intent_objective_fn.py - Intent Objective → Qwen Intent Server: requests classification
src/nat_sfc_router/functions/hf_intent_objective_fn.py - Neural Objective → Chat Request Schema: processes request
src/nat_sfc_router/functions/nn_objective_fn.py - Neural Objective → Model Router: delegates routing
src/nat_sfc_router/functions/nn_objective_fn.py - Model Router → CLIP Embedding Service: generates embeddings
README.md - Model Router → Neural Router Model: routes embeddings
src/nat_sfc_router/training/model_router.py - Neural Router Model → Model Providers: returns model choice
README.md - Interactive Demo → Router Wrapper: uses routing service
README.md - Evaluation Data Prep → Embedding Generation: prepares training inputs
README.md - Embedding Generation → CLIP Embedding Service: generates embeddings
README.md - Embedding Generation → Router Training: supplies embeddings
README.md - Router Training → Neural Router Model: trains router
src/nat_sfc_router/training/nn_router.py