mixaill76/auto_ai_router

Purpose and Entry Point

Auto AI Router is a Go LLM API proxy/router. It accepts compatible client requests, selects and calls configured upstream LLM providers or proxy chains, and adds balancing, throttling, failure isolation, observability, and optional LiteLLM billing/authentication. The executable starts at cmd/server/main.go; its runtime configuration model is in internal/config/config.go.

HTTP Ingress and Routing

The HTTP server assembles API, health, logging, tracing, and LiteLLM-related routes in internal/router/router.go, with specialized route setup in internal/router/router_litellm.go and internal/router/router_trace.go. Requests enter the proxy layer rather than a browser application; the embedded Web UI is operational/diagnostic support under internal/proxy/webui/webui.go.

Proxy Orchestration

internal/proxy/proxy.go is the primary request-processing boundary. internal/proxy/orchestrator.go coordinates model resolution, upstream attempts, retries/fallbacks, chain propagation, response handling, and accounting hooks. Streaming and non-streaming response paths are materially distinct (internal/proxy/stream.go, internal/proxy/response_writer.go), preserving SSE behavior while extracting usage and terminal errors.

Model Selection, Provider Dispatch, and Resilience

Configured model/credential definitions are managed by internal/models/manager.go, including pricing through internal/models/price_calculator.go. The proxy applies round-robin or weighted credential selection from internal/balancer/roundrobin.go and internal/balancer/weighted.go. Provider failures feed fail2ban protection (internal/fail2ban/fail2ban.go) and proxy-health tracking (internal/proxyhealth/tracker.go), allowing retries/fallbacks without exposing provider-specific mechanics to clients.

Protocol Conversion Boundary

The converter subsystem translates request and response schemas among OpenAI-compatible, Anthropic, Vertex/Gemini, Bedrock, and Responses APIs. The central dispatch layer is internal/converter/converter.go; provider-specific transformations live in paths such as internal/converter/anthropic/messages.go, internal/converter/vertex/request.go, and internal/converter/responses/converter.go. It also owns SSE/event-stream conversion (internal/converter/eventstream.go), keeping protocol normalization separate from transport orchestration.

Policy, Identity, and Stateful Controls

The proxy enforces scopes, client authorization, budget policy, sessions, and rate limits around upstream dispatch. Core boundaries include internal/proxy/scope.go, internal/proxy/budget_enforcement.go, and internal/ratelimit/backend.go. Rate limiting supports local, Redis, and hybrid backends (internal/ratelimit/redis_backend.go, internal/ratelimit/hybrid_backend.go); response persistence can use Redis via internal/responsestore/redis_store.go.

Persistence, Logging, and Operations

Optional LiteLLM DB integration supplies API-key auth, credential/model data, budgets, and spend persistence through internal/litellmdb/manager.go, internal/litellmdb/auth/auth.go, and internal/litellmdb/spendlog/logger.go. Kafka can receive spend-log events (internal/kafkalog/manager.go). ClickHouse initialization/migrations support spend-log storage (clickhouse/init/01_spend_logs.sql, clickhouse/migrations/002_cache_web_search_columns.sql). Prometheus metrics and telemetry are emitted from internal/monitoring/metrics.go and internal/telemetry/telemetry.go; health checks include DB monitoring in internal/health/monitor.go.

Runtime and Infrastructure

The main runtime is a Go server, deployable directly, in Docker (Dockerfile), or via Compose. docker-compose.yml defines the base deployment; docker-compose.kafka.yml and docker-compose.chain.yml provide Kafka and chained-router topologies. External upstreams include OpenAI, Vertex AI/Gemini, Anthropic, Bedrock, Comet API, and other router instances; optional infrastructure includes Redis, Kafka, ClickHouse, and a LiteLLM-compatible database.