# mixaill76/auto_ai_router architecture

> Architecture diagram of the GitHub repository mixaill76/auto_ai_router, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mixaill76/auto_ai_router
- Repository: https://github.com/mixaill76/auto_ai_router

## Overview

#### Purpose and Entry Point
Auto AI Router is a Go LLM API proxy/router. It accepts compatible client requests, selects and calls configured upstream LLM providers or proxy chains, and adds balancing, throttling, failure isolation, observability, and optional LiteLLM billing/authentication. The executable starts at `cmd/server/main.go`; its runtime configuration model is in `internal/config/config.go`.

#### HTTP Ingress and Routing
The HTTP server assembles API, health, logging, tracing, and LiteLLM-related routes in `internal/router/router.go`, with specialized route setup in `internal/router/router_litellm.go` and `internal/router/router_trace.go`. Requests enter the proxy layer rather than a browser application; the embedded Web UI is operational/diagnostic support under `internal/proxy/webui/webui.go`.

#### Proxy Orchestration
`internal/proxy/proxy.go` is the primary request-processing boundary. `internal/proxy/orchestrator.go` coordinates model resolution, upstream attempts, retries/fallbacks, chain propagation, response handling, and accounting hooks. Streaming and non-streaming response paths are materially distinct (`internal/proxy/stream.go`, `internal/proxy/response_writer.go`), preserving SSE behavior while extracting usage and terminal errors.

#### Model Selection, Provider Dispatch, and Resilience
Configured model/credential definitions are managed by `internal/models/manager.go`, including pricing through `internal/models/price_calculator.go`. The proxy applies round-robin or weighted credential selection from `internal/balancer/roundrobin.go` and `internal/balancer/weighted.go`. Provider failures feed fail2ban protection (`internal/fail2ban/fail2ban.go`) and proxy-health tracking (`internal/proxyhealth/tracker.go`), allowing retries/fallbacks without exposing provider-specific mechanics to clients.

#### Protocol Conversion Boundary
The converter subsystem translates request and response schemas among OpenAI-compatible, Anthropic, Vertex/Gemini, Bedrock, and Responses APIs. The central dispatch layer is `internal/converter/converter.go`; provider-specific transformations live in paths such as `internal/converter/anthropic/messages.go`, `internal/converter/vertex/request.go`, and `internal/converter/responses/converter.go`. It also owns SSE/event-stream conversion (`internal/converter/eventstream.go`), keeping protocol normalization separate from transport orchestration.

#### Policy, Identity, and Stateful Controls
The proxy enforces scopes, client authorization, budget policy, sessions, and rate limits around upstream dispatch. Core boundaries include `internal/proxy/scope.go`, `internal/proxy/budget_enforcement.go`, and `internal/ratelimit/backend.go`. Rate limiting supports local, Redis, and hybrid backends (`internal/ratelimit/redis_backend.go`, `internal/ratelimit/hybrid_backend.go`); response persistence can use Redis via `internal/responsestore/redis_store.go`.

#### Persistence, Logging, and Operations
Optional LiteLLM DB integration supplies API-key auth, credential/model data, budgets, and spend persistence through `internal/litellmdb/manager.go`, `internal/litellmdb/auth/auth.go`, and `internal/litellmdb/spendlog/logger.go`. Kafka can receive spend-log events (`internal/kafkalog/manager.go`). ClickHouse initialization/migrations support spend-log storage (`clickhouse/init/01_spend_logs.sql`, `clickhouse/migrations/002_cache_web_search_columns.sql`). Prometheus metrics and telemetry are emitted from `internal/monitoring/metrics.go` and `internal/telemetry/telemetry.go`; health checks include DB monitoring in `internal/health/monitor.go`.

#### Runtime and Infrastructure
The main runtime is a Go server, deployable directly, in Docker (`Dockerfile`), or via Compose. `docker-compose.yml` defines the base deployment; `docker-compose.kafka.yml` and `docker-compose.chain.yml` provide Kafka and chained-router topologies. External upstreams include OpenAI, Vertex AI/Gemini, Anthropic, Bedrock, Comet API, and other router instances; optional infrastructure includes Redis, Kafka, ClickHouse, and a LiteLLM-compatible database.

Main entry points: [`internal/proxy/proxy.go`](https://github.com/mixaill76/auto_ai_router/blob/main/internal/proxy/proxy.go) (Proxy boundary), [`internal/proxy/orchestrator.go`](https://github.com/mixaill76/auto_ai_router/blob/main/internal/proxy/orchestrator.go) (Request orchestration), [`internal/converter/converter.go`](https://github.com/mixaill76/auto_ai_router/blob/main/internal/converter/converter.go) (Protocol converter), [`cmd/server/main.go`](https://github.com/mixaill76/auto_ai_router/blob/main/cmd/server/main.go) (Server entrypoint)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Go Runtime"]
  node_main["Server entrypoint<br/>Go executable<br/>[main.go]"]
  node_config["Runtime configuration<br/>config model<br/>[config.go]"]
  node_router["HTTP routes<br/>HTTP router<br/>[router.go]"]
end

subgraph group_proxy["Proxy Pipeline"]
  node_proxy["Proxy boundary<br/>request handler<br/>[proxy.go]"]
  node_orchestrator["Request orchestration<br/>attempt coordinator<br/>[orchestrator.go]"]
  node_responses["Streaming &amp; responses<br/>response pipeline<br/>[stream.go]"]
  node_converter["Protocol converter<br/>schema translation<br/>[converter.go]"]
  node_models["Model &amp; pricing manager<br/>model registry<br/>[manager.go]"]
end

subgraph group_control["Policy &amp; Resilience"]
  node_balancer["Credential balancing<br/>selection policy<br/>[roundrobin.go]"]
  node_policy["Scopes &amp; budgets<br/>authorization policy<br/>[scope.go]"]
  node_rate_limits["Rate-limit backends<br/>stateful control<br/>[backend.go]"]
  node_failure_controls["Failure isolation<br/>resilience tracking<br/>[fail2ban.go]"]
end

subgraph group_integrations["Optional Integrations"]
  node_litellm[("LiteLLM DB integration<br/>identity and spend data<br/>[manager.go]")]
  node_response_store[("Response store<br/>Redis persistence<br/>[redis_store.go]")]
  node_spend_log["Spend logging<br/>event publisher<br/>[manager.go]"]
  node_observability["Metrics &amp; telemetry<br/>operations signals<br/>[metrics.go]"]
end

subgraph group_external["External Systems"]
  node_upstreams{{"LLM providers &amp; router chains<br/>upstream APIs"}}
  node_redis[("Redis<br/>external state store")]
  node_kafka["Kafka<br/>event broker"]
  node_clickhouse[("ClickHouse<br/>spend-log database<br/>[01_spend_logs.sql]")]
end

node_main -->|"loads"| node_config
node_main -->|"starts"| node_router
node_router -->|"proxy requests"| node_proxy
node_proxy -->|"enforces"| node_policy
node_proxy -->|"checks"| node_rate_limits
node_proxy -->|"dispatches"| node_orchestrator
node_orchestrator -->|"resolves model"| node_models
node_orchestrator -->|"selects credential"| node_balancer
node_orchestrator -->|"normalizes protocol"| node_converter
node_orchestrator -->|"calls and retries"| node_upstreams
node_upstreams -->|"provider responses"| node_converter
node_converter -->|"normalized stream/response"| node_responses
node_responses -->|"returns client response"| node_proxy
node_orchestrator -->|"reports failures"| node_failure_controls
node_proxy -.->|"auth, budgets, accounting"| node_litellm
node_proxy -.->|"persists responses"| node_response_store
node_rate_limits -.->|"distributed state"| node_redis
node_response_store -->|"stores"| node_redis
node_orchestrator -.->|"emits spend events"| node_spend_log
node_spend_log -->|"publishes"| node_kafka
node_kafka -.->|"spend-log storage"| node_clickhouse
node_proxy -->|"emits signals"| node_observability

click node_main "https://github.com/mixaill76/auto_ai_router/blob/main/cmd/server/main.go"
click node_config "https://github.com/mixaill76/auto_ai_router/blob/main/internal/config/config.go"
click node_router "https://github.com/mixaill76/auto_ai_router/blob/main/internal/router/router.go"
click node_proxy "https://github.com/mixaill76/auto_ai_router/blob/main/internal/proxy/proxy.go"
click node_orchestrator "https://github.com/mixaill76/auto_ai_router/blob/main/internal/proxy/orchestrator.go"
click node_responses "https://github.com/mixaill76/auto_ai_router/blob/main/internal/proxy/stream.go"
click node_converter "https://github.com/mixaill76/auto_ai_router/blob/main/internal/converter/converter.go"
click node_models "https://github.com/mixaill76/auto_ai_router/blob/main/internal/models/manager.go"
click node_balancer "https://github.com/mixaill76/auto_ai_router/blob/main/internal/balancer/roundrobin.go"
click node_policy "https://github.com/mixaill76/auto_ai_router/blob/main/internal/proxy/scope.go"
click node_rate_limits "https://github.com/mixaill76/auto_ai_router/blob/main/internal/ratelimit/backend.go"
click node_failure_controls "https://github.com/mixaill76/auto_ai_router/blob/main/internal/fail2ban/fail2ban.go"
click node_litellm "https://github.com/mixaill76/auto_ai_router/blob/main/internal/litellmdb/manager.go"
click node_response_store "https://github.com/mixaill76/auto_ai_router/blob/main/internal/responsestore/redis_store.go"
click node_spend_log "https://github.com/mixaill76/auto_ai_router/blob/main/internal/kafkalog/manager.go"
click node_observability "https://github.com/mixaill76/auto_ai_router/blob/main/internal/monitoring/metrics.go"
click node_clickhouse "https://github.com/mixaill76/auto_ai_router/blob/main/clickhouse/init/01_spend_logs.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_config,node_router toneBlue
class node_proxy,node_orchestrator,node_responses,node_converter,node_models toneAmber
class node_balancer,node_policy,node_rate_limits,node_failure_controls toneMint
class node_litellm,node_response_store,node_spend_log,node_observability toneRose
class node_upstreams,node_redis,node_kafka,node_clickhouse toneIndigo
```

## Components

### Go Runtime

- [Server entrypoint](https://github.com/mixaill76/auto_ai_router/blob/main/cmd/server/main.go): Go executable · `cmd/server/main.go`
- [Runtime configuration](https://github.com/mixaill76/auto_ai_router/blob/main/internal/config/config.go): config model · `internal/config/config.go`
- [HTTP routes](https://github.com/mixaill76/auto_ai_router/blob/main/internal/router/router.go): HTTP router · API, health, tracing, LiteLLM, and operational UI routes. · `internal/router/router.go`

### Proxy Pipeline

- [Proxy boundary](https://github.com/mixaill76/auto_ai_router/blob/main/internal/proxy/proxy.go): request handler · `internal/proxy/proxy.go`
- [Request orchestration](https://github.com/mixaill76/auto_ai_router/blob/main/internal/proxy/orchestrator.go): attempt coordinator · Resolves models and coordinates retries, fallbacks, chains, and accounting. · `internal/proxy/orchestrator.go`
- [Streaming & responses](https://github.com/mixaill76/auto_ai_router/blob/main/internal/proxy/stream.go): response pipeline · Maintains SSE and extracts terminal usage or errors. · `internal/proxy/stream.go`
- [Protocol converter](https://github.com/mixaill76/auto_ai_router/blob/main/internal/converter/converter.go): schema translation · Normalizes provider APIs and event streams. · `internal/converter/converter.go`
- [Model & pricing manager](https://github.com/mixaill76/auto_ai_router/blob/main/internal/models/manager.go): model registry · `internal/models/manager.go`

### Policy & Resilience

- [Credential balancing](https://github.com/mixaill76/auto_ai_router/blob/main/internal/balancer/roundrobin.go): selection policy · Round-robin or weighted upstream credential selection. · `internal/balancer/roundrobin.go`
- [Scopes & budgets](https://github.com/mixaill76/auto_ai_router/blob/main/internal/proxy/scope.go): authorization policy · `internal/proxy/scope.go`
- [Rate-limit backends](https://github.com/mixaill76/auto_ai_router/blob/main/internal/ratelimit/backend.go): stateful control · Supports local, Redis, and hybrid limits. · `internal/ratelimit/backend.go`
- [Failure isolation](https://github.com/mixaill76/auto_ai_router/blob/main/internal/fail2ban/fail2ban.go): resilience tracking · Provider failures drive bans and proxy-health tracking. · `internal/fail2ban/fail2ban.go`

### Optional Integrations

- [LiteLLM DB integration](https://github.com/mixaill76/auto_ai_router/blob/main/internal/litellmdb/manager.go): identity and spend data · Provides API-key auth, models, budgets, and spend persistence. · `internal/litellmdb/manager.go`
- [Response store](https://github.com/mixaill76/auto_ai_router/blob/main/internal/responsestore/redis_store.go): Redis persistence · `internal/responsestore/redis_store.go`
- [Spend logging](https://github.com/mixaill76/auto_ai_router/blob/main/internal/kafkalog/manager.go): event publisher · `internal/kafkalog/manager.go`
- [Metrics & telemetry](https://github.com/mixaill76/auto_ai_router/blob/main/internal/monitoring/metrics.go): operations signals · `internal/monitoring/metrics.go`

### External Systems

- **LLM providers & router chains**: upstream APIs · OpenAI-compatible, Anthropic, Vertex/Gemini, Bedrock, Comet, and router instances.
- **Redis**: external state store
- **Kafka**: event broker
- [ClickHouse](https://github.com/mixaill76/auto_ai_router/blob/main/clickhouse/init/01_spend_logs.sql): spend-log database · `clickhouse/init/01_spend_logs.sql`

## Connections

- Server entrypoint → Runtime configuration: loads
- Server entrypoint → HTTP routes: starts
- HTTP routes → Proxy boundary: proxy requests
- Proxy boundary → Scopes & budgets: enforces
- Proxy boundary → Rate-limit backends: checks
- Proxy boundary → Request orchestration: dispatches
- Request orchestration → Model & pricing manager: resolves model
- Request orchestration → Credential balancing: selects credential
- Request orchestration → Protocol converter: normalizes protocol
- Request orchestration → LLM providers & router chains: calls and retries
- LLM providers & router chains → Protocol converter: provider responses
- Protocol converter → Streaming & responses: normalized stream/response
- Streaming & responses → Proxy boundary: returns client response
- Request orchestration → Failure isolation: reports failures
- Proxy boundary → LiteLLM DB integration: auth, budgets, accounting
- Proxy boundary → Response store: persists responses
- Rate-limit backends → Redis: distributed state
- Response store → Redis: stores
- Request orchestration → Spend logging: emits spend events
- Spend logging → Kafka: publishes
- Kafka → ClickHouse: spend-log storage
- Proxy boundary → Metrics & telemetry: emits signals