How anishakalivarapu/llm-router works
Smart LLM Router is a FastAPI service that classifies query complexity, selects a model according to cost/latency/quality preference, and executes an escalating OpenAI/Gemini model chain. The main path is API request → router → classifier and model selection → executor → provider → response, with metrics recorded for successful routes. The README also documents classify, model-list, health, and metrics endpoints. The supplied excerpts show the route, classifier, executor, and provider wiring; some endpoint and OpenAI implementation details are not excerpted, so those links are omitted where unsupported.
Main entry points: router.py Query router · main.py App and dependency setup · executor.py Retries and escalation · registry.py Provider registry
Components 19
HTTP API
- FastAPI endpoints
app/api.py - Request and response schemas
app/schemas.py - App and dependency setup
app/main.py
Classification and Routing
- Query router
app/router.py - Heuristic classifier
app/classifiers/heuristic.py - Hybrid classifier
app/classifiers/hybrid.py - LLM classifier
app/classifiers/llm.py
Model Execution
- Retries and escalation
app/executor.py - Circuit breaker
app/executor.py
Provider Adapters
- Provider protocol and HTTP
app/providers/base.py - Provider registry
app/providers/registry.py - OpenAI adapter
app/providers/openai_provider.py - Gemini adapter
app/providers/gemini_provider.py
Configuration and Metrics
- Settings and model ladder
app/config.py - In-memory metrics
app/metrics.py
Other components
- API client
- OpenAI API
- Gemini API
- HTTPX client
Connections 29
- API client → FastAPI endpoints: sends requests
README.md - FastAPI endpoints → Request and response schemas: validates and returns
app/api.py - FastAPI endpoints → Query router: dispatches routes
app/api.py - App and dependency setup → FastAPI endpoints: registers endpoints
app/main.py - App and dependency setup → Query router: builds router
app/main.py - App and dependency setup → Retries and escalation: builds executor
app/main.py - App and dependency setup → In-memory metrics: creates metrics
app/main.py - App and dependency setup → Provider registry: builds providers
app/main.py - Query router → Heuristic classifier: uses fallback
app/router.py - Query router → Retries and escalation: executes chain
app/router.py - Query router → In-memory metrics: records outcomes
app/router.py - Query router → Settings and model ladder: selects models
app/router.py - Query router → Hybrid classifier: dispatches classifier
- Hybrid classifier → Heuristic classifier: scores first
app/classifiers/hybrid.py - Hybrid classifier → LLM classifier: asks when uncertain
app/classifiers/hybrid.py - LLM classifier → Retries and escalation: labels with model
app/classifiers/llm.py - Retries and escalation → Settings and model ladder: uses retry settings
app/executor.py - Retries and escalation → Provider protocol and HTTP: calls provider protocol
app/executor.py - Retries and escalation → Circuit breaker: checks and updates
app/executor.py - Provider registry → Settings and model ladder: reads credentials
app/providers/registry.py - Provider registry → OpenAI adapter: registers adapter
app/providers/registry.py - Provider registry → Gemini adapter: registers adapter
app/providers/registry.py - OpenAI adapter → Provider protocol and HTTP: posts and maps errors
app/providers/openai_provider.py - Gemini adapter → Provider protocol and HTTP: posts and maps errors
app/providers/gemini_provider.py - OpenAI adapter → OpenAI API: requests completions
- Gemini adapter → Gemini API: requests completions
app/providers/gemini_provider.py - App and dependency setup → HTTPX client: creates HTTP client
app/main.py - Provider registry → HTTPX client: configures adapters
app/providers/registry.py - In-memory metrics → Settings and model ladder: prices baseline
app/metrics.py