loseall/openai-http

1. Top-level shape

- This is a Python 3.12+ service package, not just a library: it exposes an OpenAI v1-compatible HTTP API and can also be embedded as a server via run_server().

- The main runtime is FastAPI/Uvicorn-style ASGI, with the package centered under openai_http/.

2. Core request path

- Requests enter FastAPI, pass through middleware for request IDs and structured logging, then land in route handlers under openai_http/routers/.

- The handlers translate OpenAI-style HTTP payloads into backend calls, then map backend output back into OpenAI v1 response shapes defined in openai_http/schemas/.

3. API surface and routing

- Implemented endpoints are the core OpenAI compatibility set: /v1/models, /v1/chat/completions, /v1/completions, /v1/embeddings, and /health.

- Audio and image routes are already registered, but intentionally return 501 stubs.

- The router layout mirrors the product surface, so an architecture diagram should show separate endpoint groups rather than a single monolithic API layer.

4. Backend abstraction boundary

- The most important architectural seam is openai_http/backends/base.py, which defines BackendBase.

- The server is backend-agnostic: it expects a backend object that can generate completions, stream tokens, list/get models, and optionally implement embeddings and tool/function calls.

- Default behavior for unsupported capabilities is 501, which makes backend support explicit rather than implicit.

5. Pluggable inference implementations

- The repository ships with a mock_backend for local testing and compatibility validation.

- The example backend in examples/transformers-backend/ demonstrates a real model-backed implementation using Hugging Face Transformers, including tool calling support for chat.

- This means the architecture is “HTTP facade + pluggable inference engine,” not a fixed model-serving stack.

6. Streaming and SSE

- Chat/completions support both normal and streaming responses.

- Streaming is implemented as a first-class path, likely using Server-Sent Events semantics from the router layer down through backend generate_stream() yielding chunks.

- In a diagram, streaming should be shown as a separate response mode from standard request/response.

7. Queueing and GPU serialization

- openai_http/queue.py provides request throttling/serialization, described as a semaphore-based queue that prevents concurrent GPU access.

- This is a key operational boundary: route handlers acquire queue capacity before invoking backend inference.

- When the queue depth is exceeded, the service returns 429, so the queue acts both as protection and backpressure.

8. Configuration and runtime control

- openai_http/config.py loads TOML configuration with environment variable overrides using the OPENAI_HTTP__... convention.

- Server host/port, auth, queue depth, and observability settings are all runtime-configurable, so deployment concerns are externalized from code.

- For architecture purposes, config sits beside the server bootstrap layer, not inside request handlers.

9. Authentication and error normalization

- openai_http/auth.py handles API-key style authorization when enabled.

- openai_http/errors.py centralizes OpenAI-shaped error responses, so failures are normalized to the API contract rather than leaking raw framework errors.

- This creates a boundary between internal exceptions and externally visible API semantics.

10. Schemas and contract stability

- openai_http/schemas/ contains the Pydantic models for chat, completions, embeddings, common objects, and model metadata.

- These schemas are the contract layer between routers and backends, and they are what keep the service aligned with the OpenAI v1 format.

- If drawing the architecture, put schemas as the shared “data model” block spanning handlers and backend adapters.

11. Observability

- Logging is structured JSON/text with request IDs, and there is an observability package for metrics integration.

- README indicates OpenTelemetry metrics exposed via a Prometheus endpoint, so there are two notable observability outputs: logs and metrics.

- This is operationally important because it means request tracing and service health are designed into the server, not added externally.

12. Health and lifecycle

- There is a dedicated health route and a server lifecycle path in app.py / _server.py.

- The server likely performs backend setup/teardown during lifespan events, which is important for model loading or resource cleanup in real backends.

- This is the place to show startup/shutdown behavior in a diagram.

13. Testing and compatibility validation

- Testing is stratified into unit, integration, contract, and SDK compatibility tests.

- The SDK suite is especially important: it runs the official openai Python client against the server, validating real-world API compatibility.

- This repo is therefore organized around contract conformance, not only internal correctness.

14. Specs and examples as architecture documentation

- specs/ contains explicit feature specs, API contracts, plans, and data models, which act as design-time documentation for implemented and planned capabilities.

- examples/transformers-backend/ is the canonical real-world extension example and should be treated as a reference integration, not part of the core runtime.

- For an architecture diagram, these belong on the side as “reference implementations / design artifacts,” not in the request path.

15. Key boundaries to show

- External client → FastAPI routes → auth/logging/queue → backend adapter → model/inference engine.

- Cross-cutting services: config, error formatting, observability.

- Non-core but visible surfaces: OpenAI SDK contract tests, mock backend, and example Transformers backend.