# loseall/openai-http architecture

> Architecture diagram of the GitHub repository loseall/openai-http, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/loseall/openai-http
- Repository: https://github.com/loseall/openai-http

## Overview

1. **Top-level shape**
   - This is a Python 3.12+ service package, not just a library: it exposes an OpenAI v1-compatible HTTP API and can also be embedded as a server via `run_server()`.
   - The main runtime is FastAPI/Uvicorn-style ASGI, with the package centered under `openai_http/`.

2. **Core request path**
   - Requests enter FastAPI, pass through middleware for request IDs and structured logging, then land in route handlers under `openai_http/routers/`.
   - The handlers translate OpenAI-style HTTP payloads into backend calls, then map backend output back into OpenAI v1 response shapes defined in `openai_http/schemas/`.

3. **API surface and routing**
   - Implemented endpoints are the core OpenAI compatibility set: `/v1/models`, `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, and `/health`.
   - Audio and image routes are already registered, but intentionally return 501 stubs.
   - The router layout mirrors the product surface, so an architecture diagram should show separate endpoint groups rather than a single monolithic API layer.

4. **Backend abstraction boundary**
   - The most important architectural seam is `openai_http/backends/base.py`, which defines `BackendBase`.
   - The server is backend-agnostic: it expects a backend object that can generate completions, stream tokens, list/get models, and optionally implement embeddings and tool/function calls.
   - Default behavior for unsupported capabilities is 501, which makes backend support explicit rather than implicit.

5. **Pluggable inference implementations**
   - The repository ships with a `mock_backend` for local testing and compatibility validation.
   - The example backend in `examples/transformers-backend/` demonstrates a real model-backed implementation using Hugging Face Transformers, including tool calling support for chat.
   - This means the architecture is “HTTP facade + pluggable inference engine,” not a fixed model-serving stack.

6. **Streaming and SSE**
   - Chat/completions support both normal and streaming responses.
   - Streaming is implemented as a first-class path, likely using Server-Sent Events semantics from the router layer down through backend `generate_stream()` yielding chunks.
   - In a diagram, streaming should be shown as a separate response mode from standard request/response.

7. **Queueing and GPU serialization**
   - `openai_http/queue.py` provides request throttling/serialization, described as a semaphore-based queue that prevents concurrent GPU access.
   - This is a key operational boundary: route handlers acquire queue capacity before invoking backend inference.
   - When the queue depth is exceeded, the service returns 429, so the queue acts both as protection and backpressure.

8. **Configuration and runtime control**
   - `openai_http/config.py` loads TOML configuration with environment variable overrides using the `OPENAI_HTTP__...` convention.
   - Server host/port, auth, queue depth, and observability settings are all runtime-configurable, so deployment concerns are externalized from code.
   - For architecture purposes, config sits beside the server bootstrap layer, not inside request handlers.

9. **Authentication and error normalization**
   - `openai_http/auth.py` handles API-key style authorization when enabled.
   - `openai_http/errors.py` centralizes OpenAI-shaped error responses, so failures are normalized to the API contract rather than leaking raw framework errors.
   - This creates a boundary between internal exceptions and externally visible API semantics.

10. **Schemas and contract stability**
   - `openai_http/schemas/` contains the Pydantic models for chat, completions, embeddings, common objects, and model metadata.
   - These schemas are the contract layer between routers and backends, and they are what keep the service aligned with the OpenAI v1 format.
   - If drawing the architecture, put schemas as the shared “data model” block spanning handlers and backend adapters.

11. **Observability**
   - Logging is structured JSON/text with request IDs, and there is an observability package for metrics integration.
   - README indicates OpenTelemetry metrics exposed via a Prometheus endpoint, so there are two notable observability outputs: logs and metrics.
   - This is operationally important because it means request tracing and service health are designed into the server, not added externally.

12. **Health and lifecycle**
   - There is a dedicated health route and a server lifecycle path in `app.py` / `_server.py`.
   - The server likely performs backend setup/teardown during lifespan events, which is important for model loading or resource cleanup in real backends.
   - This is the place to show startup/shutdown behavior in a diagram.

13. **Testing and compatibility validation**
   - Testing is stratified into unit, integration, contract, and SDK compatibility tests.
   - The SDK suite is especially important: it runs the official `openai` Python client against the server, validating real-world API compatibility.
   - This repo is therefore organized around contract conformance, not only internal correctness.

14. **Specs and examples as architecture documentation**
   - `specs/` contains explicit feature specs, API contracts, plans, and data models, which act as design-time documentation for implemented and planned capabilities.
   - `examples/transformers-backend/` is the canonical real-world extension example and should be treated as a reference integration, not part of the core runtime.
   - For an architecture diagram, these belong on the side as “reference implementations / design artifacts,” not in the request path.

15. **Key boundaries to show**
   - External client → FastAPI routes → auth/logging/queue → backend adapter → model/inference engine.
   - Cross-cutting services: config, error formatting, observability.
   - Non-core but visible surfaces: OpenAI SDK contract tests, mock backend, and example Transformers backend.

Main entry points: [`openai_http/app.py`](https://github.com/loseall/openai-http/blob/main/openai_http/app.py) (App), [`openai_http/backends/base.py`](https://github.com/loseall/openai-http/blob/main/openai_http/backends/base.py) (Backend base), [`openai_http/schemas`](https://github.com/loseall/openai-http/tree/main/openai_http/schemas) (Schemas), [`openai_http/routers/chat.py`](https://github.com/loseall/openai-http/blob/main/openai_http/routers/chat.py) (Chat API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_app["App<br/>ASGI entry<br/>[app.py]"]
  node_server["Server<br/>bootstrap<br/>[_server.py]"]
  node_config["Config<br/>[config.py]"]
  node_logging["Logging<br/>middleware<br/>[_logging.py]"]
  node_auth["Auth<br/>[auth.py]"]
  node_queue["Queue<br/>semaphore throttle<br/>[queue.py]"]
  node_errors["Errors<br/>response mapping<br/>[errors.py]"]
  node_chat["Chat API<br/>router<br/>[chat.py]"]
  node_completions["Completions API<br/>router<br/>[completions.py]"]
  node_embeddings["Embeddings API<br/>router<br/>[embeddings.py]"]
  node_models["Models API<br/>router<br/>[models.py]"]
  node_health["Health<br/>router<br/>[health.py]"]
  node_audio_images["Audio/Image<br/>stub routes<br/>[audio.py]"]
end

subgraph group_backend["Backend"]
  node_backend_base["Backend base<br/>adapter interface<br/>[base.py]"]
  node_mock_backend["Mock backend<br/>backend impl<br/>[mock_backend.py]"]
end

subgraph group_contracts["Contracts"]
  node_schemas["Schemas<br/>Pydantic contracts"]
end

subgraph group_docs["Specs & examples"]
  node_transformers_backend["Transformers backend<br/>example impl"]
  node_specs["Specs<br/>design artifacts"]
end

subgraph group_tests["Tests"]
  node_sdk_tests["SDK tests<br/>compat suite"]
  node_unit_integration["Unit & integration<br/>test suites"]
end

node_server -->|"creates"| node_app
node_app -->|"loads"| node_config
node_app -->|"installs"| node_logging
node_app -->|"guards"| node_auth
node_app -->|"throttles"| node_queue
node_app -->|"normalizes"| node_errors
node_app -->|"routes"| node_chat
node_app -->|"routes"| node_completions
node_app -->|"routes"| node_embeddings
node_app -->|"routes"| node_models
node_app -->|"routes"| node_health
node_app -->|"stubs"| node_audio_images
node_chat -->|"parses/returns"| node_schemas
node_completions -->|"parses/returns"| node_schemas
node_embeddings -->|"parses/returns"| node_schemas
node_models -->|"returns"| node_schemas
node_chat -->|"calls"| node_backend_base
node_completions -->|"calls"| node_backend_base
node_embeddings -->|"calls"| node_backend_base
node_models -->|"calls"| node_backend_base
node_backend_base -->|"implemented by"| node_mock_backend
node_backend_base -.->|"reference impl"| node_transformers_backend
node_backend_base -->|"raises 501"| node_errors
node_specs -.->|"documents"| node_schemas
node_specs -.->|"documents"| node_backend_base
node_sdk_tests -->|"validates"| node_app
node_unit_integration -->|"verifies"| node_backend_base

click node_app "https://github.com/loseall/openai-http/blob/main/openai_http/app.py"
click node_server "https://github.com/loseall/openai-http/blob/main/openai_http/_server.py"
click node_config "https://github.com/loseall/openai-http/blob/main/openai_http/config.py"
click node_logging "https://github.com/loseall/openai-http/blob/main/openai_http/_logging.py"
click node_auth "https://github.com/loseall/openai-http/blob/main/openai_http/auth.py"
click node_queue "https://github.com/loseall/openai-http/blob/main/openai_http/queue.py"
click node_errors "https://github.com/loseall/openai-http/blob/main/openai_http/errors.py"
click node_chat "https://github.com/loseall/openai-http/blob/main/openai_http/routers/chat.py"
click node_completions "https://github.com/loseall/openai-http/blob/main/openai_http/routers/completions.py"
click node_embeddings "https://github.com/loseall/openai-http/blob/main/openai_http/routers/embeddings.py"
click node_models "https://github.com/loseall/openai-http/blob/main/openai_http/routers/models.py"
click node_health "https://github.com/loseall/openai-http/blob/main/openai_http/routers/health.py"
click node_audio_images "https://github.com/loseall/openai-http/blob/main/openai_http/routers/audio.py"
click node_schemas "https://github.com/loseall/openai-http/tree/main/openai_http/schemas"
click node_backend_base "https://github.com/loseall/openai-http/blob/main/openai_http/backends/base.py"
click node_mock_backend "https://github.com/loseall/openai-http/blob/main/openai_http/backends/mock_backend.py"
click node_transformers_backend "https://github.com/loseall/openai-http/blob/main/examples/transformers-backend/transformers_backend.py"
click node_specs "https://github.com/loseall/openai-http/tree/main/specs"
click node_sdk_tests "https://github.com/loseall/openai-http/tree/main/tests/sdk"
click node_unit_integration "https://github.com/loseall/openai-http/tree/main/tests/unit"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_server,node_config,node_logging,node_auth,node_queue,node_errors,node_chat,node_completions,node_embeddings,node_models,node_health,node_audio_images toneBlue
class node_backend_base,node_mock_backend toneAmber
class node_schemas toneMint
class node_transformers_backend,node_specs toneRose
class node_sdk_tests,node_unit_integration toneIndigo
```

## Components

### Runtime

ASGI service, request handling, and cross-cutting concerns

- [App](https://github.com/loseall/openai-http/blob/main/openai_http/app.py): ASGI entry · Builds the FastAPI service and wires routers, middleware, and lifespan · `openai_http/app.py`
- [Server](https://github.com/loseall/openai-http/blob/main/openai_http/_server.py): bootstrap · Process entry and run_server lifecycle control · `openai_http/_server.py`
- [Config](https://github.com/loseall/openai-http/blob/main/openai_http/config.py): runtime config · Loads TOML and environment overrides for deployment settings · `openai_http/config.py`
- [Logging](https://github.com/loseall/openai-http/blob/main/openai_http/_logging.py): middleware · Structured request logging and request ID plumbing · `openai_http/_logging.py`
- [Auth](https://github.com/loseall/openai-http/blob/main/openai_http/auth.py): api key auth · Optional authorization gate for incoming requests · `openai_http/auth.py`
- [Queue](https://github.com/loseall/openai-http/blob/main/openai_http/queue.py): semaphore throttle · Serializes GPU-bound inference and returns backpressure when saturated · `openai_http/queue.py`
- [Errors](https://github.com/loseall/openai-http/blob/main/openai_http/errors.py): response mapping · Normalizes failures into OpenAI-shaped error responses · `openai_http/errors.py`
- [Chat API](https://github.com/loseall/openai-http/blob/main/openai_http/routers/chat.py): router · OpenAI chat completions endpoint with streaming support · `openai_http/routers/chat.py`
- [Completions API](https://github.com/loseall/openai-http/blob/main/openai_http/routers/completions.py): router · Legacy completions endpoint and response translation · `openai_http/routers/completions.py`
- [Embeddings API](https://github.com/loseall/openai-http/blob/main/openai_http/routers/embeddings.py): router · Embeddings endpoint and request/response mapping · `openai_http/routers/embeddings.py`
- [Models API](https://github.com/loseall/openai-http/blob/main/openai_http/routers/models.py): router · Lists and retrieves model metadata · `openai_http/routers/models.py`
- [Health](https://github.com/loseall/openai-http/blob/main/openai_http/routers/health.py): router · Service health endpoint · `openai_http/routers/health.py`
- [Audio/Image](https://github.com/loseall/openai-http/blob/main/openai_http/routers/audio.py): stub routes · Registered compatibility routes that intentionally return 501 · `openai_http/routers/audio.py`

### Backend

Pluggable inference boundary and implementations

- [Backend base](https://github.com/loseall/openai-http/blob/main/openai_http/backends/base.py): adapter interface · Defines the server-side inference contract and optional capabilities · `openai_http/backends/base.py`
- [Mock backend](https://github.com/loseall/openai-http/blob/main/openai_http/backends/mock_backend.py): backend impl · Local compatibility backend for tests and validation · `openai_http/backends/mock_backend.py`

### Contracts

Schemas, API surface, and validation

- [Schemas](https://github.com/loseall/openai-http/tree/main/openai_http/schemas): Pydantic contracts · Shared OpenAI v1 data models for chat, completions, embeddings, and models · `openai_http/schemas`

### Specs & examples

Reference implementations and design artifacts

- [Transformers backend](https://github.com/loseall/openai-http/blob/main/examples/transformers-backend/transformers_backend.py): example impl · Reference Hugging Face backend with model and tool-calling support · `examples/transformers-backend/transformers_backend.py`
- [Specs](https://github.com/loseall/openai-http/tree/main/specs): design artifacts · Feature specs, contracts, plans, and data models · `specs`

### Tests

Compatibility and correctness validation

- [SDK tests](https://github.com/loseall/openai-http/tree/main/tests/sdk): compat suite · Official OpenAI Python client compatibility validation · `tests/sdk`
- [Unit & integration](https://github.com/loseall/openai-http/tree/main/tests/unit): test suites · Backend, router, validation, and library-level correctness checks · `tests/unit`

## Connections

- Server → App: creates · Bootstrap constructs and runs the ASGI app
- App → Config: loads · Runtime settings are sourced at startup
- App → Logging: installs · Middleware adds request IDs and structured logs
- App → Auth: guards · Requests pass through optional API-key authorization
- App → Queue: throttles · Inference requests acquire queue capacity before backend calls
- App → Errors: normalizes · Unhandled failures are mapped to API-shaped responses
- App → Chat API: routes · Chat completions endpoint is mounted on the app
- App → Completions API: routes · Completions endpoint is mounted on the app
- App → Embeddings API: routes · Embeddings endpoint is mounted on the app
- App → Models API: routes · Models endpoint is mounted on the app
- App → Health: routes · Health endpoint is mounted on the app
- App → Audio/Image: stubs · Audio and image routes are registered as 501-compatible stubs
- Chat API → Schemas: parses/returns · OpenAI payloads are converted to and from shared schemas
- Completions API → Schemas: parses/returns · Completions requests and responses use shared schemas
- Embeddings API → Schemas: parses/returns · Embeddings requests and responses use shared schemas
- Models API → Schemas: returns · Model metadata is serialized via shared schema objects
- Chat API → Backend base: calls · Router delegates generation and streaming to backend adapter
- Completions API → Backend base: calls · Completions handler delegates to backend adapter
- Embeddings API → Backend base: calls · Embeddings handler delegates when supported
- Models API → Backend base: calls · Model listing and lookup come from backend capabilities
- Backend base → Mock backend: implemented by · Mock backend provides a default local implementation
- Backend base → Transformers backend: reference impl · Example backend demonstrates a real model-serving integration
- Backend base → Errors: raises 501 · Unsupported capabilities are surfaced as explicit not-implemented responses
- Specs → Schemas: documents · Specs describe the API and data model contracts
- Specs → Backend base: documents · Design artifacts define the extension boundary
- SDK tests → App: validates · OpenAI SDK compatibility is exercised against the service
- Unit & integration → Backend base: verifies · Lower-level tests cover adapter contract and library behavior