# billjr99/llmproxy architecture

> Architecture diagram of the GitHub repository billjr99/llmproxy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/billjr99/llmproxy
- Repository: https://github.com/billjr99/llmproxy

## Overview

1. llmproxy is a Python HTTP proxy that presents a single OpenAI-compatible endpoint while multiplexing many upstream LLM providers. The core runtime is a Flask app, optionally served by gunicorn; the same process also hosts admin UI/API and usage endpoints.

2. The main request path is: client request in → model name parsing/routing in `server.py` → provider selection from `providers.json` and live config → outbound call to the chosen upstream → response translation back to the client’s dialect. The proxy strips the provider prefix from the model id before forwarding, and it can retry/fail over across multiple candidate backends for virtual models.

3. Model identity is central to the architecture. Real models are addressed as `provider/upstream_model`, while synthetic virtual models live under the reserved `llmproxy` namespace. Virtual routing is the main abstraction layer: it groups models by free tier, locality, reasoning level, capability, provider, or fusion behavior, then cycles through candidates until one succeeds.

4. Configuration is file-based and hot-reloaded. `~/.config/llmproxy/config.json` is the live runtime config, while `llmproxy/providers.json` is the bundled sidecar and single source of truth for provider templates, free-tier metadata, reasoning tags, capability tags, and pricing snapshots. Startup sync can reconcile the user config from the bundled sidecar without requiring network access.

5. Provider discovery is mixed-mode. Some providers are discovered from standard OpenAI-style `/models` endpoints; others use provider-specific model catalog URLs or a manual `model_filter` allowlist when discovery is broken or unavailable. This means the provider layer is not just credentials: it also carries discovery strategy and per-provider protocol selection.

6. Outbound protocol translation is an explicit subsystem. The proxy can speak OpenAI, Anthropic, or Gemini upstream protocols depending on a provider’s `protocol` field, while still accepting OpenAI, Anthropic, and Gemini inbound APIs from clients. Internally, everything is normalized around the chat-completions shape so routing, capability checks, and usage accounting can work uniformly.

7. Routing logic has multiple scoring layers. Capacity-aware free-tier models are preferred using `free_limits`; locality is detected from loopback and Docker host aliases; reasoning and capability tags influence candidate order; and `llmproxy__loadbalanced` adds a cost waterfall that prefers free, then local, then cheapest paid models. The failover engine treats HTTP errors, timeouts, malformed 200 bodies, and unmet forced capabilities as fallback conditions.

8. The fusion path is a separate orchestration subsystem. Instead of selecting one backend, `llmproxy__fusion` fans the prompt out to a panel of models, uses a judge model to compare answers, then synthesizes a final response. It emits provenance metadata in the response body and an `X-LLMProxy-Fusion` header, and it degrades gracefully if judge or synthesis steps fail.

9. Usage and cost accounting are first-class. `usage.py` tracks token counts and estimated dollar cost per model, exposing them through `GET /v1/usage` and `POST /v1/usage/reset`. Costs come either from provider-reported usage cost or from the bundled pricing snapshot, so the accounting layer depends on both live responses and the sidecar metadata.

10. The admin subsystem is a separate boundary from normal proxy traffic. `/admin` serves a static single-page frontend, while `/admin/api/*` mutates config and provider state. It is loopback-only unless an admin token is configured, which keeps the secret-editing surface distinct from the public proxy surface.

11. `setup_wizard.py` and `run.py` are the operational entry points. The wizard writes the initial config, adds provider templates, and tags models; `run.py` starts the proxy or invokes setup/list/version flows. A small `llmproxy_test_client.py` exists for live integration checks against a running instance.

12. Free-tier maintenance is automated outside the request path. `scripts/update_free_models.py` scrapes multiple sources—provider docs, model catalogs, OpenRouter, LiteLLM cost data, community lists, and optional live probes—to refresh `providers.json`. GitHub Actions also runs this scraper on a schedule and can open PRs, while the server can optionally run the same updater on startup and even create a PR through the GitHub API.

13. Deployment is container-friendly but not container-specific. The Docker image bundles the app and scraper, uses a non-root user, and expects config to be mounted from `/config`. The architecture assumes config persistence is external to the image, while the provider sidecar may be regenerated by CI or by a live deployment depending on how the instance is managed.

14. The test surface is split by concern: unit/integration pytest tests cover routing, config, dialect translation, admin auth, usage, and scraper behavior; CI enforces linting and config-example regeneration drift. That separation reflects the repository’s core boundaries: request routing, config/data maintenance, and operational tooling are deliberately decoupled.

Main entry points: [`llmproxy/server.py`](https://github.com/billjr99/llmproxy/blob/main/llmproxy/server.py) (Proxy server), [`llmproxy/dialects/base.py`](https://github.com/billjr99/llmproxy/blob/main/llmproxy/dialects/base.py) (Dialect core), [`scripts/update_free_models.py`](https://github.com/billjr99/llmproxy/blob/main/scripts/update_free_models.py) (Catalog scraper), [`llmproxy/setup_wizard.py`](https://github.com/billjr99/llmproxy/blob/main/llmproxy/setup_wizard.py) (Setup wizard)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Proxy runtime"]
  node_server["Proxy server<br/>flask app<br/>[server.py]"]
  node_config["Config<br/>[config.py]"]
  node_providers["Provider registry<br/>catalog/routing<br/>[providers.py]"]
  node_dialect_base["Dialect core<br/>protocol normalize<br/>[base.py]"]
  node_dialect_openai["OpenAI dialect<br/>protocol adapter<br/>[openai.py]"]
  node_dialect_anthropic["Anthropic dialect<br/>protocol adapter<br/>[anthropic.py]"]
  node_dialect_gemini["Gemini dialect<br/>protocol adapter<br/>[gemini.py]"]
  node_usage[("Usage meter<br/>tokens/cost<br/>[usage.py]")]
  node_fusion{{"Fusion engine<br/>multi-model orchestration<br/>[fusion.py]"}}
end

subgraph group_control["Control plane"]
  node_run_py["run.py<br/>entrypoint<br/>[run.py]"]
  node_main_mod["__main__<br/>module entry<br/>[__main__.py]"]
  node_admin["Admin API<br/>admin surface<br/>[admin.py]"]
  node_wizard["Setup wizard<br/>bootstrap<br/>[setup_wizard.py]"]
  node_setup_mod["Setup helpers<br/>installer logic<br/>[setup.py]"]
end

subgraph group_maintenance["Maintenance"]
  node_provider_sync["Catalog scraper<br/>refresh pipeline"]
  node_github_pr["PR automation<br/>github integration<br/>[github_pr.py]"]
  node_sources["Scraper sources<br/>data collectors<br/>[base.py]"]
end

subgraph group_deployment["Deployment"]
  node_dockerfile["Docker image<br/>container build"]
  node_compose["Compose setup<br/>deployment spec<br/>[docker-compose.yml]"]
end

node_run_py -->|"starts"| node_server
node_run_py -->|"bootstraps"| node_wizard
node_main_mod -->|"delegates"| node_run_py
node_server -->|"reads"| node_config
node_server -->|"routes with"| node_providers
node_server -->|"normalizes via"| node_dialect_base
node_server -->|"records"| node_usage
node_server -->|"dispatches"| node_fusion
node_server -->|"mounts"| node_admin
node_dialect_base -->|"adapts"| node_dialect_openai
node_dialect_base -->|"adapts"| node_dialect_anthropic
node_dialect_base -->|"adapts"| node_dialect_gemini
node_fusion -->|"selects models"| node_providers
node_fusion -->|"charges"| node_usage
node_admin -->|"mutates"| node_config
node_wizard -->|"writes"| node_config
node_wizard -->|"seeds"| node_providers
node_provider_sync -->|"collects from"| node_sources
node_provider_sync -->|"regenerates"| node_providers
node_provider_sync -->|"opens PRs"| node_github_pr
node_github_pr -.->|"supports"| node_provider_sync
node_dockerfile -->|"packages"| node_server
node_compose -->|"deploys"| node_dockerfile
node_setup_mod -.->|"supports"| node_wizard

click node_run_py "https://github.com/billjr99/llmproxy/blob/main/run.py"
click node_main_mod "https://github.com/billjr99/llmproxy/blob/main/llmproxy/__main__.py"
click node_server "https://github.com/billjr99/llmproxy/blob/main/llmproxy/server.py"
click node_config "https://github.com/billjr99/llmproxy/blob/main/llmproxy/config.py"
click node_providers "https://github.com/billjr99/llmproxy/blob/main/llmproxy/providers.py"
click node_dialect_base "https://github.com/billjr99/llmproxy/blob/main/llmproxy/dialects/base.py"
click node_dialect_openai "https://github.com/billjr99/llmproxy/blob/main/llmproxy/dialects/openai.py"
click node_dialect_anthropic "https://github.com/billjr99/llmproxy/blob/main/llmproxy/dialects/anthropic.py"
click node_dialect_gemini "https://github.com/billjr99/llmproxy/blob/main/llmproxy/dialects/gemini.py"
click node_usage "https://github.com/billjr99/llmproxy/blob/main/llmproxy/usage.py"
click node_fusion "https://github.com/billjr99/llmproxy/blob/main/llmproxy/fusion.py"
click node_admin "https://github.com/billjr99/llmproxy/blob/main/llmproxy/admin.py"
click node_wizard "https://github.com/billjr99/llmproxy/blob/main/llmproxy/setup_wizard.py"
click node_setup_mod "https://github.com/billjr99/llmproxy/blob/main/llmproxy/setup.py"
click node_provider_sync "https://github.com/billjr99/llmproxy/blob/main/scripts/update_free_models.py"
click node_github_pr "https://github.com/billjr99/llmproxy/blob/main/llmproxy/github_pr.py"
click node_sources "https://github.com/billjr99/llmproxy/blob/main/scripts/sources/base.py"
click node_dockerfile "https://github.com/billjr99/llmproxy/tree/main/Dockerfile"
click node_compose "https://github.com/billjr99/llmproxy/blob/main/llmproxy/docker-compose.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_server,node_config,node_providers,node_dialect_base,node_dialect_openai,node_dialect_anthropic,node_dialect_gemini,node_usage,node_fusion toneBlue
class node_run_py,node_main_mod,node_admin,node_wizard,node_setup_mod toneAmber
class node_provider_sync,node_github_pr,node_sources toneMint
class node_dockerfile,node_compose toneRose
```

## Components

### Proxy runtime

Live request handling, routing, translation, and accounting.

- [Proxy server](https://github.com/billjr99/llmproxy/blob/main/llmproxy/server.py): flask app · Main HTTP surface for OpenAI-compatible proxying, routing, and request handling. · `llmproxy/server.py`
- [Config](https://github.com/billjr99/llmproxy/blob/main/llmproxy/config.py): runtime config · Loads and hot-reloads live configuration state. · `llmproxy/config.py`
- [Provider registry](https://github.com/billjr99/llmproxy/blob/main/llmproxy/providers.py): catalog/routing · Bundled provider templates, metadata, pricing snapshots, and discovery strategy. · `llmproxy/providers.py`
- [Dialect core](https://github.com/billjr99/llmproxy/blob/main/llmproxy/dialects/base.py): protocol normalize · Shared normalization layer for inbound and outbound chat-completions translation. · `llmproxy/dialects/base.py`
- [OpenAI dialect](https://github.com/billjr99/llmproxy/blob/main/llmproxy/dialects/openai.py): protocol adapter · OpenAI-style request and response translation. · `llmproxy/dialects/openai.py`
- [Anthropic dialect](https://github.com/billjr99/llmproxy/blob/main/llmproxy/dialects/anthropic.py): protocol adapter · Anthropic-style request and response translation. · `llmproxy/dialects/anthropic.py`
- [Gemini dialect](https://github.com/billjr99/llmproxy/blob/main/llmproxy/dialects/gemini.py): protocol adapter · Gemini-style request and response translation. · `llmproxy/dialects/gemini.py`
- [Usage meter](https://github.com/billjr99/llmproxy/blob/main/llmproxy/usage.py): tokens/cost · Tracks token counts and estimated cost, and exposes usage reset endpoints. · `llmproxy/usage.py`
- [Fusion engine](https://github.com/billjr99/llmproxy/blob/main/llmproxy/fusion.py): multi-model orchestration · Fans out to multiple models, judges outputs, and synthesizes a final answer. · `llmproxy/fusion.py`

### Control plane

Admin and startup flows that configure or launch the service.

- [run.py](https://github.com/billjr99/llmproxy/blob/main/run.py): entrypoint · CLI entrypoint for starting the proxy or invoking setup/list/version flows. · `run.py`
- [__main__](https://github.com/billjr99/llmproxy/blob/main/llmproxy/__main__.py): module entry · Package entrypoint for python -m execution. · `llmproxy/__main__.py`
- [Admin API](https://github.com/billjr99/llmproxy/blob/main/llmproxy/admin.py): admin surface · Loopback-protected admin UI and API for editing config and provider state. · `llmproxy/admin.py`
- [Setup wizard](https://github.com/billjr99/llmproxy/blob/main/llmproxy/setup_wizard.py): bootstrap · Interactive bootstrap that writes initial config and seeds provider templates. · `llmproxy/setup_wizard.py`
- [Setup helpers](https://github.com/billjr99/llmproxy/blob/main/llmproxy/setup.py): installer logic · Support code for initial installation and environment setup. · `llmproxy/setup.py`

### Maintenance

Provider catalog refresh, PR automation, and external data ingestion.

- [Catalog scraper](https://github.com/billjr99/llmproxy/blob/main/scripts/update_free_models.py): refresh pipeline · Scrapes docs, catalogs, pricing, and probes to refresh bundled provider data. · `scripts/update_free_models.py`
- [PR automation](https://github.com/billjr99/llmproxy/blob/main/llmproxy/github_pr.py): github integration · Creates or manages update pull requests from refreshed provider data. · `llmproxy/github_pr.py`
- [Scraper sources](https://github.com/billjr99/llmproxy/blob/main/scripts/sources/base.py): data collectors · Provider-specific docs, catalogs, community lists, APIs, and probes used for refreshes. · `scripts/sources/base.py`

### Deployment

Container and packaging entrypoints.

- [Docker image](https://github.com/billjr99/llmproxy/tree/main/Dockerfile): container build · Container packaging for the app and scraper with mounted external config. · `Dockerfile`
- [Compose setup](https://github.com/billjr99/llmproxy/blob/main/llmproxy/docker-compose.yml): deployment spec · Local container orchestration and runtime wiring. · `llmproxy/docker-compose.yml`

## Connections

- run.py → Proxy server: starts · Launches the proxy runtime.
- run.py → Setup wizard: bootstraps · Runs initial setup flows.
- __main__ → run.py: delegates · Module entry forwards to the CLI runner.
- Proxy server → Config: reads · Loads live runtime configuration and hot-reload state.
- Proxy server → Provider registry: routes with · Uses provider templates, metadata, and pricing for model selection.
- Proxy server → Dialect core: normalizes via · Converts inbound and outbound payloads to a shared chat shape.
- Proxy server → Usage meter: records · Accounts for tokens and estimated cost.
- Proxy server → Fusion engine: dispatches · Invokes fusion orchestration for special virtual models.
- Proxy server → Admin API: mounts · Exposes the admin UI and API alongside proxy traffic.
- Dialect core → OpenAI dialect: adapts · Implements OpenAI-compatible translation.
- Dialect core → Anthropic dialect: adapts · Implements Anthropic-compatible translation.
- Dialect core → Gemini dialect: adapts · Implements Gemini-compatible translation.
- Fusion engine → Provider registry: selects models · Pulls candidate panel, judge, and synthesis backends from provider metadata.
- Fusion engine → Usage meter: charges · Feeds usage and cost accounting for fused responses.
- Admin API → Config: mutates · Edits config and provider state through administrative endpoints.
- Setup wizard → Config: writes · Creates the initial runtime configuration.
- Setup wizard → Provider registry: seeds · Adds provider templates and model tags during bootstrap.
- Catalog scraper → Scraper sources: collects from · Aggregates docs, catalogs, community lists, APIs, and probes.
- Catalog scraper → Provider registry: regenerates · Refreshes the bundled provider sidecar.
- Catalog scraper → PR automation: opens PRs · Can publish refresh results as a pull request.
- PR automation → Catalog scraper: supports · Supports automated update workflows.
- Docker image → Proxy server: packages · Builds the container image that runs the proxy.
- Compose setup → Docker image: deploys · Uses the container image for local orchestration.
- Setup helpers → Setup wizard: supports · Provides installer and environment setup helpers.