billjr99/llmproxy

1. llmproxy is a Python HTTP proxy that presents a single OpenAI-compatible endpoint while multiplexing many upstream LLM providers. The core runtime is a Flask app, optionally served by gunicorn; the same process also hosts admin UI/API and usage endpoints.

2. The main request path is: client request in → model name parsing/routing in server.py → provider selection from providers.json and live config → outbound call to the chosen upstream → response translation back to the client’s dialect. The proxy strips the provider prefix from the model id before forwarding, and it can retry/fail over across multiple candidate backends for virtual models.

3. Model identity is central to the architecture. Real models are addressed as provider/upstream_model, while synthetic virtual models live under the reserved llmproxy namespace. Virtual routing is the main abstraction layer: it groups models by free tier, locality, reasoning level, capability, provider, or fusion behavior, then cycles through candidates until one succeeds.

4. Configuration is file-based and hot-reloaded. ~/.config/llmproxy/config.json is the live runtime config, while llmproxy/providers.json is the bundled sidecar and single source of truth for provider templates, free-tier metadata, reasoning tags, capability tags, and pricing snapshots. Startup sync can reconcile the user config from the bundled sidecar without requiring network access.

5. Provider discovery is mixed-mode. Some providers are discovered from standard OpenAI-style /models endpoints; others use provider-specific model catalog URLs or a manual model_filter allowlist when discovery is broken or unavailable. This means the provider layer is not just credentials: it also carries discovery strategy and per-provider protocol selection.

6. Outbound protocol translation is an explicit subsystem. The proxy can speak OpenAI, Anthropic, or Gemini upstream protocols depending on a provider’s protocol field, while still accepting OpenAI, Anthropic, and Gemini inbound APIs from clients. Internally, everything is normalized around the chat-completions shape so routing, capability checks, and usage accounting can work uniformly.

7. Routing logic has multiple scoring layers. Capacity-aware free-tier models are preferred using free_limits; locality is detected from loopback and Docker host aliases; reasoning and capability tags influence candidate order; and llmproxy__loadbalanced adds a cost waterfall that prefers free, then local, then cheapest paid models. The failover engine treats HTTP errors, timeouts, malformed 200 bodies, and unmet forced capabilities as fallback conditions.

8. The fusion path is a separate orchestration subsystem. Instead of selecting one backend, llmproxy__fusion fans the prompt out to a panel of models, uses a judge model to compare answers, then synthesizes a final response. It emits provenance metadata in the response body and an X-LLMProxy-Fusion header, and it degrades gracefully if judge or synthesis steps fail.

9. Usage and cost accounting are first-class. usage.py tracks token counts and estimated dollar cost per model, exposing them through GET /v1/usage and POST /v1/usage/reset. Costs come either from provider-reported usage cost or from the bundled pricing snapshot, so the accounting layer depends on both live responses and the sidecar metadata.

10. The admin subsystem is a separate boundary from normal proxy traffic. /admin serves a static single-page frontend, while /admin/api/* mutates config and provider state. It is loopback-only unless an admin token is configured, which keeps the secret-editing surface distinct from the public proxy surface.

11. setup_wizard.py and run.py are the operational entry points. The wizard writes the initial config, adds provider templates, and tags models; run.py starts the proxy or invokes setup/list/version flows. A small llmproxy_test_client.py exists for live integration checks against a running instance.

12. Free-tier maintenance is automated outside the request path. scripts/update_free_models.py scrapes multiple sources—provider docs, model catalogs, OpenRouter, LiteLLM cost data, community lists, and optional live probes—to refresh providers.json. GitHub Actions also runs this scraper on a schedule and can open PRs, while the server can optionally run the same updater on startup and even create a PR through the GitHub API.

13. Deployment is container-friendly but not container-specific. The Docker image bundles the app and scraper, uses a non-root user, and expects config to be mounted from /config. The architecture assumes config persistence is external to the image, while the provider sidecar may be regenerated by CI or by a live deployment depending on how the instance is managed.

14. The test surface is split by concern: unit/integration pytest tests cover routing, config, dialect translation, admin auth, usage, and scraper behavior; CI enforces linting and config-example regeneration drift. That separation reflects the repository’s core boundaries: request routing, config/data maintenance, and operational tooling are deliberately decoupled.