How ss-vip/api-gateway works
1. Deployment/runtime boundary
- This is a Cloudflare Workers service, not a traditional server.
- src/worker.js is the runtime entrypoint and wires together the API gateway, admin dashboard, and maintenance endpoints.
- Persistence is split between D1 for durable state and in-memory globals for short-lived caching and rate buffers.
2. Primary API surface
- The main product is an OpenAI-compatible proxy gateway.
- Clients call endpoints like /v1/chat/completions, and the worker forwards requests to upstream providers after routing decisions and policy checks.
- The README indicates the gateway accepts standard OpenAI-style request bodies, including streaming.
3. Request orchestration layer
- src/gateway.js is the central control plane for API traffic.
- It is responsible for authentication/token checks, model-aware channel selection, forwarding requests, and applying response handling rules.
- This is where the “smart routing” behavior lives, rather than in the provider implementation.
4. Channel/provider abstraction
- src/lib/providers/openai.js is the concrete provider adapter for OpenAI-compatible upstreams.
- The architecture implies multiple “channels” or upstream endpoints can be managed, each with independent health, capability, and cooldown state.
- The gateway treats providers as pluggable backends behind a common interface.
5. Routing and failure management
- Channel selection is described as weighted random with dynamic degradation.
- Weights are influenced by model matching, latency, RPM/RPD availability, and error history.
- Error handling includes exponential backoff cooldowns, excluding a channel after repeated failures, and gradual recovery via sliding-window success tracking.
6. Capability learning
- The system remembers per-channel support for features like vision, tools, and streaming.
- Capability detection happens during testing or live calls, and those observations feed back into routing decisions.
- This creates a feedback loop between request execution and future selection quality.
7. State and persistence
- src/lib/db.js is the D1 database access layer.
- src/lib/schema.js centralizes the schema definition; the README says the worker can auto-update schema at startup.
- The architecture intentionally uses SELECT * internally so schema additions do not require broad query rewrites, while the dashboard uses explicit columns for stability.
8. Caching and operational counters
- The gateway keeps channel status cached for 60 seconds to reduce D1 reads.
- RPM/RPD tracking is held in an in-memory rateBuffer and periodically persisted.
- This means operational metrics are split between fast ephemeral state in the worker instance and durable state in D1.
9. Maintenance and health flow
- src/routes/maintenance.js provides the /health endpoint.
- This endpoint is not just a status check: it also performs cleanup and recovery tasks such as expiring cooldowns, flushing rate buffers, and reporting channel statistics.
- The README recommends calling it every 5 minutes via a scheduled job, so it acts like a maintenance tick for the system.
10. Admin/operator interface
- src/dashboard/index.js, src/dashboard/portal.html, and src/dashboard/resources.js implement the /admin management UI.
- The dashboard covers channel CRUD, JSON import/export backups, initial password setup, client token management, and health monitoring.
- This is a separate operator boundary from the API gateway: humans manage configuration here, while clients only interact with the OpenAI-compatible endpoints.
11. Response policy layer
- The gateway also supports content filtering by keyword.
- Filtering can either remove matched content or truncate the response after a match, depending on configuration.
- This happens after upstream execution, so it is part of the outbound response path rather than pre-routing.
12. Overall data flow
- External client request → worker entrypoint → gateway auth/routing → cached/D1 channel lookup → upstream provider call → error/capability/latency observation → optional filtering → client response.
- Separately, operator actions flow through the dashboard into D1, and /health performs periodic maintenance that keeps routing state fresh.
13. Key architectural boundaries to draw
- Client traffic vs admin traffic: API proxy endpoints are distinct from /admin.
- Ephemeral state vs durable state: cache/rate buffers live in Worker memory; authoritative configuration and history live in D1.
- Control plane vs data plane: dashboard, schema, and health tasks manage the system; gateway request forwarding carries user traffic.
Main entry points: gateway.js Gateway · db.js D1 Layer · index.js Admin UI · maintenance.js Health
Components 16
Runtime
Worker execution and request handling
- Worker · Cloudflare Workers entrypoint wiring API, admin, and maintenance surfaces
src/worker.js - Gateway · Central request orchestrator for authentication, routing, forwarding, and response policy
src/gateway.js - OpenAI Adapter · Concrete upstream adapter for OpenAI-compatible providers
src/lib/providers/openai.js - Constants · Shared limits, labels, and runtime constants
src/lib/constants.js - Rate Buffer · In-memory RPM/RPD tracking and short-lived operational counters
- Status Cache · Short-lived cached channel status to reduce D1 reads
- Routing · Weighted channel selection with degradation, cooldowns, and capability awareness
- Capability Learning · Learns per-channel support for vision, tools, and streaming from tests and live traffic
- Filtering · Post-upstream content filtering that removes or truncates matched content
Admin
Operator UI and maintenance endpoints
- Health · Health endpoint that also expires cooldowns, flushes buffers, and reports channel stats
src/routes/maintenance.js - Admin UI · Admin portal entrypoint for channel and token management
src/dashboard/index.js - Portal · Admin dashboard HTML shell
src/dashboard/portal.html - Resources · Admin dashboard client-side resources and interactions
src/dashboard/resources.js
Connections 23
- Worker → Gateway: routes API · Worker hands client API traffic to the gateway
- Worker → Admin UI: serves admin · Worker exposes the operator dashboard under /admin
- Worker → Health: exposes health · Worker mounts the maintenance and health endpoint
- Gateway → Constants: uses limits · Gateway reads shared configuration and operational constants
- Gateway → Status Cache: reads status · Gateway consults cached channel state before hitting D1
- Gateway → D1 Layer: reads/writes · Gateway persists and loads routing and operational state through D1
- Gateway → Routing: selects channel · Gateway applies weighted routing and failure-aware selection
- Routing → OpenAI Adapter: calls upstream · Selected channel is executed through the provider adapter
- OpenAI Adapter → Capability Learning: observes support · Adapter surfaces capability and runtime observations back into learning
- OpenAI Adapter → Rate Buffer: updates counters · Provider execution contributes to in-memory rate tracking
- Rate Buffer → D1 Layer: flushes · Buffered counters are periodically persisted to D1
- D1 Layer → Channel State: stores state · D1 holds channel configuration, health, and history
- Schema → D1 Layer: defines tables · Schema governs the database structure used by the persistence layer
- Health → D1 Layer: repairs state · Health flow performs cleanup and recovery work against durable state
- Health → Rate Buffer: flushes buffers · Maintenance tick persists and clears buffered operational counters
- Health → Status Cache: expires cache · Health flow helps refresh short-lived cached routing state
- Admin UI → Portal: renders shell · Admin app serves the portal HTML
- Admin UI → Resources: loads UI · Admin app uses dashboard resources for interactions
- Admin UI → D1 Layer: manages config · Dashboard creates, updates, exports, and imports operator state in D1
- Admin UI → Channel State: edits channels · Dashboard manages channel records and related operator data
- Filtering → Gateway: post-processes · Gateway applies response filtering after upstream execution
- Gateway → Filtering: applies policy · Outbound response path can remove or truncate matched content
- Capability Learning → Routing: improves choices · Learned capabilities feed future channel selection