# nirb28/dsp_ai_gateway architecture

> Architecture diagram of the GitHub repository nirb28/dsp_ai_gateway, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nirb28/dsp_ai_gateway
- Repository: https://github.com/nirb28/dsp_ai_gateway

## Overview

##### Project type and purpose
- This is a microservice-based API Gateway project (“DSP AI Gateway”) that sits behind an APISIX reverse‐proxy layer.
- Purpose: Provide a secured, client‐authenticated REST API (FastAPI/Uvicorn) for calling external LLM providers (OpenAI, Groq), with per‐client configuration, rate‐limiting, endpoint/provider access control, and observability via Prometheus/Grafana.

##### Main components
a. API Gateway Layer (APISIX)
- Deployed in two modes: containerized (docker-compose under apisix/container) or standalone (apisix/standalone).
- Hosts custom Lua plugins (pipeline-request.lua, file-logger.lua) for request manipulation, logging, and routing to the FastAPI backend.
- Integrates with Prometheus (scrape config) and Grafana (dashboards, provisioning) for metrics.
b. Application Layer (FastAPI service)
- app/main.py: Uvicorn entrypoint
- app/api/endpoints.py: /api/v1/generate, /clients/reload
- app/middleware/debug_middleware.py & auth.py: Authentication and request hooks
- app/clients/configs/*.json: Per‐client configuration store (client IDs, secrets, rate limits, allowed providers/endpoints)
- app/models/llm.py: Client abstractions for calling OpenAI/Groq
- app/core/config.py: Central configuration (env, default provider/model)
c. Monitoring & Observability
- Prometheus: Scrapes APISIX and optionally FastAPI metrics endpoints
d. External Dependencies
- OpenAI and Groq HTTP APIs for LLM calls
- Docker/Docker-Compose for container mode
- Local virtual environment for Python dependencies (FastAPI, pydantic, httpx, uvicorn)

##### Relationships & Interactions
- Client → APISIX (TLS/HTTP)
- APISIX custom Lua plugin authenticates/passes headers, rate‐limits (if configured in plugin) → forwards to FastAPI
- APISIX → FastAPI service (HTTP)
- FastAPI middleware→ app/clients/auth.py → verifies client_id/secret → loads JSON config → enforces endpoint/provider access and rate limits
- FastAPI → LLM Providers (OpenAI/Groq) via app/models/llm.py (HTTP calls)
- FastAPI → APISIX (response) → back to client
- Prometheus → scrapes APISIX and FastAPI metrics endpoints
- Grafana → reads from Prometheus → dashboard

##### Architectural patterns & design principles
- API Gateway pattern: APISIX as the single entry point, decoupling client concerns from services
- Plugin architecture: Lua scripts for extensibility in the gateway
- Microservice principle: Service focuses exclusively on AI gateway logic (auth, routing, model calls)
- Configuration-as-code: Client configs in JSON, Grafana provisioning via YAML/JSON files

##### Key details not to omit
- The custom Lua plugins inside APISIX and their role
- JSON‐based client config store and its dynamic reload endpoint
- Middleware chain inside FastAPI (auth, debug)
- Rate‐limiting and usage metrics capturing
- Two deployment options (container vs. standalone)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Client
    Client["Client"]:::external

    %% Container Mode Group
    subgraph "Container Mode"
        direction TB
        APISIX_CM["APISIX Gateway"]:::gateway
        PR_Lua["pipeline-request.lua"]:::gateway_plugin
        FL_Lua["file-logger.lua"]:::gateway_plugin
        APISIX_DC["docker-compose.yml"]:::gateway_config
        PrometheusCM["Prometheus"]:::monitoring
        GrafanaCM["Grafana"]:::monitoring
        GDash["apisix-grafana-dashboard.json"]:::config
        GProvDash["provisioning dashboards"]:::config
        GProvDS["provisioning datasources"]:::config

        APISIX_CM --> PR_Lua
        APISIX_CM --> FL_Lua
        APISIX_CM --> PrometheusCM
        PrometheusCM --> GrafanaCM
    end

    %% Standalone Mode Group
    subgraph "Standalone Mode"
        direction TB
        APISIX_SM["APISIX Gateway"]:::gateway
        FL_Lua_SM["file-logger.lua"]:::gateway_plugin
        APISIX_DC_SM["docker-compose.yml"]:::gateway_config

        APISIX_SM --> FL_Lua_SM
    end

    %% Backend Service Group
    subgraph "Backend Service"
        direction TB
        SERVICE["FastAPI Service (Uvicorn)"]:::service
        ENDPOINTS["API Endpoints"]:::service
        AUTH["Auth Middleware"]:::service
        DEBUG["Debug Middleware"]:::service
        CONFIG_CORE["Core Config"]:::config
        CONFIG_STORE["Client Config Store"]:::config
        LLM["LLM Client Module"]:::service

        SERVICE --> ENDPOINTS
        SERVICE --> AUTH
        SERVICE --> DEBUG
        SERVICE --> CONFIG_CORE
        SERVICE --> CONFIG_STORE
        SERVICE --> LLM
    end

    %% External LLM Providers
    OpenAI["OpenAI API"]:::external
    Groq["Groq API"]:::external

    %% Connections
    Client --> APISIX_CM
    Client --> APISIX_SM
    APISIX_CM --> SERVICE
    APISIX_SM --> SERVICE
    SERVICE --> OpenAI
    SERVICE --> Groq
    SERVICE --> PrometheusCM
    PrometheusCM --> GrafanaCM

    %% Click Events
    click APISIX_CM "https://github.com/nirb28/dsp_ai_gateway/blob/main/apisix/container/apisix_conf/config.yaml"
    click PR_Lua "https://github.com/nirb28/dsp_ai_gateway/blob/main/apisix/container/dspai_plugins/apisix/plugins/pipeline-request.lua"
    click FL_Lua "https://github.com/nirb28/dsp_ai_gateway/blob/main/apisix/standalone/dspai_plugins/apisix/plugins/file-logger.lua"
    click APISIX_DC "https://github.com/nirb28/dsp_ai_gateway/blob/main/apisix/container/docker-compose.yml"
    click PrometheusCM "https://github.com/nirb28/dsp_ai_gateway/blob/main/apisix/container/prometheus_conf/prometheus.yml"
    click GDash "https://github.com/nirb28/dsp_ai_gateway/blob/main/apisix/container/grafana_conf/dashboards/apisix-grafana-dashboard.json"
    click GProvDash "https://github.com/nirb28/dsp_ai_gateway/blob/main/apisix/container/grafana_conf/provisioning/dashboards/all.yaml"
    click GProvDS "https://github.com/nirb28/dsp_ai_gateway/blob/main/apisix/container/grafana_conf/provisioning/datasources/all.yaml"
    click APISIX_SM "https://github.com/nirb28/dsp_ai_gateway/blob/main/apisix/standalone/conf/apisix.yaml"
    click FL_Lua_SM "https://github.com/nirb28/dsp_ai_gateway/blob/main/apisix/standalone/dspai_plugins/apisix/plugins/file-logger.lua"
    click APISIX_DC_SM "https://github.com/nirb28/dsp_ai_gateway/blob/main/apisix/standalone/docker-compose.yml"
    click SERVICE "https://github.com/nirb28/dsp_ai_gateway/blob/main/app/main.py"
    click ENDPOINTS "https://github.com/nirb28/dsp_ai_gateway/blob/main/app/api/endpoints.py"
    click AUTH "https://github.com/nirb28/dsp_ai_gateway/blob/main/app/clients/auth.py"
    click DEBUG "https://github.com/nirb28/dsp_ai_gateway/blob/main/app/middleware/debug_middleware.py"
    click CONFIG_CORE "https://github.com/nirb28/dsp_ai_gateway/blob/main/app/core/config.py"
    click CONFIG_STORE "https://github.com/nirb28/dsp_ai_gateway/blob/main/app/clients/configs/*.json"
    click LLM "https://github.com/nirb28/dsp_ai_gateway/blob/main/app/models/llm.py"

    %% Styles
    classDef gateway fill:#cfe2f3,stroke:#333,stroke-width:1px
    classDef gateway_plugin fill:#9fc5e8,stroke:#333,stroke-width:1px
    classDef service fill:#d9ead3,stroke:#333,stroke-width:1px
    classDef external fill:#cccccc,stroke:#333,stroke-width:1px
    classDef monitoring fill:#f9cb9c,stroke:#333,stroke-width:1px
    classDef config fill:#ffe599,stroke:#333,stroke-width:1px
    classDef gateway_config fill:#b6d7a8,stroke:#333,stroke-width:1px
```
