How nirb28/dsp_ai_gateway works
Project type and purpose
• This is a microservice-based API Gateway project (“DSP AI Gateway”) that sits behind an APISIX reverse‐proxy layer.
• Purpose: Provide a secured, client‐authenticated REST API (FastAPI/Uvicorn) for calling external LLM providers (OpenAI, Groq), with per‐client configuration, rate‐limiting, endpoint/provider access control, and observability via Prometheus/Grafana.
Main components
a. API Gateway Layer (APISIX)
• Deployed in two modes: containerized (docker-compose under apisix/container) or standalone (apisix/standalone).
• Hosts custom Lua plugins (pipeline-request.lua, file-logger.lua) for request manipulation, logging, and routing to the FastAPI backend.
• Integrates with Prometheus (scrape config) and Grafana (dashboards, provisioning) for metrics.
b. Application Layer (FastAPI service)
• app/main.py: Uvicorn entrypoint
• app/api/endpoints.py: /api/v1/generate, /clients/reload
• app/middleware/debug_middleware.py & auth.py: Authentication and request hooks
• app/clients/configs/*.json: Per‐client configuration store (client IDs, secrets, rate limits, allowed providers/endpoints)
• app/models/llm.py: Client abstractions for calling OpenAI/Groq
• app/core/config.py: Central configuration (env, default provider/model)
c. Monitoring & Observability
• Prometheus: Scrapes APISIX and optionally FastAPI metrics endpoints
d. External Dependencies
• OpenAI and Groq HTTP APIs for LLM calls
• Docker/Docker-Compose for container mode
• Local virtual environment for Python dependencies (FastAPI, pydantic, httpx, uvicorn)
Relationships & Interactions
• Client → APISIX (TLS/HTTP)
• APISIX custom Lua plugin authenticates/passes headers, rate‐limits (if configured in plugin) → forwards to FastAPI
• APISIX → FastAPI service (HTTP)
• FastAPI middleware→ app/clients/auth.py → verifies client_id/secret → loads JSON config → enforces endpoint/provider access and rate limits
• FastAPI → LLM Providers (OpenAI/Groq) via app/models/llm.py (HTTP calls)
• FastAPI → APISIX (response) → back to client
• Prometheus → scrapes APISIX and FastAPI metrics endpoints
• Grafana → reads from Prometheus → dashboard
Architectural patterns & design principles
• API Gateway pattern: APISIX as the single entry point, decoupling client concerns from services
• Plugin architecture: Lua scripts for extensibility in the gateway
• Microservice principle: Service focuses exclusively on AI gateway logic (auth, routing, model calls)
• Configuration-as-code: Client configs in JSON, Grafana provisioning via YAML/JSON files
Key details not to omit
• The custom Lua plugins inside APISIX and their role
• JSON‐based client config store and its dynamic reload endpoint
• Middleware chain inside FastAPI (auth, debug)
• Rate‐limiting and usage metrics capturing
• Two deployment options (container vs. standalone)