How kuadrant/inferno works
Project Type & Purpose
• Inferno is a backend microservices-style application providing three Envoy ext_proc filters for LLM requests:
• Semantic Cache
• Prompt Guard
• Token Usage Metrics
• It runs as a single Go binary exposing three gRPC endpoints, intended to be wired into Envoy’s external processing filter chain.
High-Level Architectural Components
a. Client(s)
• Any HTTP/HTTPS client issuing OpenAI-style requests (completions, chat, embeddings).
b. Envoy Proxy
• Configured via envoy/envoy.yaml and filter.yaml to intercept LLM API calls.
• Chains through three ext_proc gRPC services.
c. Inferno Service
• Single Go process (main.go → internal/server/server.go) listening on three ports:
• 50051 – Semantic Cache
• 50052 – Prompt Guard
• 50053 – Token Metrics
• Each ext_proc handler lives under internal/ext_proc.
d. External Dependencies
• Embedding Model Server (for semantic cache embedding lookups)
• Risk Assessment API (for prompt guard)
• LLM API (OpenAI) proxied by Envoy
• (Optional) Kubernetes CRDs under config/crd/bases if deployed in K8s
e. Infrastructure
• Docker Compose for local demo (docker-compose.yaml)
• Dockerfile + Makefile for builds
• CI flows in .github/workflows
Relationships & Data Flow
1. Client → Envoy (port 10000)
2. Envoy matches v1/chat or /v1/embeddings → invokes ext_proc filter chain
3. Envoy ext_proc call #1 → gRPC → Inferno.SemanticCache
• Calls Embedding Model Server → caches vectors
4. Envoy ext_proc call #2 → gRPC → Inferno.PromptGuard
• Calls external Risk API
5. Envoy ext_proc call #3 → gRPC → Inferno.TokenMetrics
• Parses tokens for monitoring
6. Envoy forwards modified request to upstream LLM API
7. Responses reverse through filter chain (Prompt Guard response check, etc.)
8. Envoy returns final response to Client
Architectural Patterns & Principles
• Micro-filter pipeline via Envoy ext_proc (chain of responsibility)
• Single Go binary hosting multiple gRPC services
• Configuration-driven ports and endpoints via environment variables
• Decoupling of concerns: caching, risk checking, metrics
• Pluggable external dependencies (pluggable model and risk servers)