# kuadrant/inferno architecture

> Architecture diagram of the GitHub repository kuadrant/inferno, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kuadrant/inferno
- Repository: https://github.com/kuadrant/inferno

## Overview

##### Project Type & Purpose
- Inferno is a backend microservices-style application providing three Envoy ext_proc filters for LLM requests:
- Semantic Cache
- Prompt Guard
- Token Usage Metrics
- It runs as a single Go binary exposing three gRPC endpoints, intended to be wired into Envoy’s external processing filter chain.

##### High-Level Architectural Components
a. Client(s)
- Any HTTP/HTTPS client issuing OpenAI-style requests (completions, chat, embeddings).
b. Envoy Proxy
- Configured via `envoy/envoy.yaml` and `filter.yaml` to intercept LLM API calls.
- Chains through three ext_proc gRPC services.
c. Inferno Service
- Single Go process (`main.go` → `internal/server/server.go`) listening on three ports:
- 50051 – Semantic Cache
- 50052 – Prompt Guard
- 50053 – Token Metrics
- Each ext_proc handler lives under `internal/ext_proc`.
d. External Dependencies
- Embedding Model Server (for semantic cache embedding lookups)
- Risk Assessment API (for prompt guard)
- LLM API (OpenAI) proxied by Envoy
- (Optional) Kubernetes CRDs under `config/crd/bases` if deployed in K8s
e. Infrastructure
- Docker Compose for local demo (`docker-compose.yaml`)
- Dockerfile + Makefile for builds
- CI flows in `.github/workflows`

##### Relationships & Data Flow
1. Client → Envoy (port 10000)
2. Envoy matches v1/chat or /v1/embeddings → invokes ext_proc filter chain
3. Envoy ext_proc call #1 → gRPC → Inferno.SemanticCache
- Calls Embedding Model Server → caches vectors
4. Envoy ext_proc call #2 → gRPC → Inferno.PromptGuard
- Calls external Risk API
5. Envoy ext_proc call #3 → gRPC → Inferno.TokenMetrics
- Parses tokens for monitoring
6. Envoy forwards modified request to upstream LLM API
7. Responses reverse through filter chain (Prompt Guard response check, etc.)
8. Envoy returns final response to Client

##### Architectural Patterns & Principles
- Micro-filter pipeline via Envoy ext_proc (chain of responsibility)
- Single Go binary hosting multiple gRPC services
- Configuration-driven ports and endpoints via environment variables
- Decoupling of concerns: caching, risk checking, metrics
- Pluggable external dependencies (pluggable model and risk servers)

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Client
    Client["Client\n(HTTP/HTTPS)"]:::external

    %% Envoy Proxy
    Envoy["Envoy Proxy\n(port 10000)"]:::infra

    %% Inferno Service
    subgraph "Inferno Service" 
        direction TB
        Main["main.go\nEntry Point"]:::internal
        ServerSetup["Server Setup\n(internal/server)"]:::internal
        Config["Config Loader\n(internal/server/config.go)"]:::internal
        SemanticCache["Inferno.SemanticCache\n(port 50051)"]:::internal
        PromptGuard["Inferno.PromptGuard\n(port 50052)"]:::internal
        TokenMetrics["Inferno.TokenMetrics\n(port 50053)"]:::internal
    end

    %% External Dependencies
    EmbeddingServer["Embedding Model Server"]:::external
    RiskAPI["Risk Assessment API"]:::external
    OpenAI["LLM API (OpenAI)"]:::external

    %% Infrastructure & CI
    subgraph "Infra & Config" 
        direction TB
        EnvoyConfig["Envoy Config\n(envoy/envoy.yaml, filter.yaml)"]:::infra
        DockerCompose["Docker Compose\n(docker-compose.yaml)"]:::infra
        DockerBuild["Dockerfile & Makefile"]:::infra
        CI["CI Workflows\n(.github/workflows)"]:::infra
        KubernetesCRD["Kubernetes CRDs\n(config/crd/bases)"]:::infra
    end

    %% Data Flows
    Client -->|HTTP/HTTPS| Envoy
    Envoy -->|matches LLM API\r\ninvokes filter chain| SemanticCache
    SemanticCache -->|embedding lookup| EmbeddingServer
    EmbeddingServer -->|vector cache\r\nreturn| SemanticCache
    SemanticCache -->|response| Envoy

    Envoy -->|ext_proc call #2| PromptGuard
    PromptGuard -->|risk API call| RiskAPI
    RiskAPI -->|risk decision| PromptGuard
    PromptGuard -->|response| Envoy

    Envoy -->|ext_proc call #3| TokenMetrics
    TokenMetrics -->|token parse & metrics| TokenMetrics
    TokenMetrics -->|response| Envoy

    Envoy -->|forward modified request| OpenAI
    OpenAI -->|LLM response| Envoy
    Envoy -->|final response| Client

    %% Connections within Inferno
    EnvoyConfig -.-> Main
    Main --> ServerSetup
    ServerSetup --> Config
    ServerSetup --> SemanticCache
    ServerSetup --> PromptGuard
    ServerSetup --> TokenMetrics

    DockerCompose & DockerBuild & CI & KubernetesCRD -.-> Main

    %% Click Events
    click EnvoyConfig "https://github.com/kuadrant/inferno/blob/main/envoy/envoy.yaml"
    click Main "https://github.com/kuadrant/inferno/blob/main/main.go"
    click ServerSetup "https://github.com/kuadrant/inferno/blob/main/internal/server/server.go"
    click Config "https://github.com/kuadrant/inferno/blob/main/internal/server/config.go"
    click SemanticCache "https://github.com/kuadrant/inferno/blob/main/internal/ext_proc/semantic_cache.go"
    click PromptGuard "https://github.com/kuadrant/inferno/blob/main/internal/ext_proc/prompt_guard.go"
    click TokenMetrics "https://github.com/kuadrant/inferno/blob/main/internal/ext_proc/token_usage_metrics.go"
    click DockerCompose "https://github.com/kuadrant/inferno/blob/main/docker-compose.yaml"
    click DockerBuild "https://github.com/kuadrant/inferno/tree/main/Dockerfile"
    click CI "https://github.com/kuadrant/inferno/blob/main/.github/workflows/build-image.yaml"
    click KubernetesCRD "https://github.com/kuadrant/inferno/blob/main/config/crd/bases/kuadrant.io_infernos.yaml"

    %% Styles
    classDef external fill:#e3f2fd,stroke:#90caf9
    classDef internal fill:#e8f5e9,stroke:#66bb6a
    classDef infra fill:#fff3e0,stroke:#ffa726
```
