# shell-nlp/gpt_server architecture

> Architecture diagram of the GitHub repository shell-nlp/gpt_server, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shell-nlp/gpt_server
- Repository: https://github.com/shell-nlp/gpt_server

## Overview

Based on our analysis of the file tree and README, we can conclude that this project is an open‐source “GPT Server” that emulates OpenAI server capabilities built on top of FastChat. The project is designed to support multiple inference backends (vLLM, LMDeploy, HF, and even Infinity for embedding tasks), and it exposes various API endpoints (including chat, text moderation, TTS, ASR, etc.) while also providing a CLI, a visual UI (via Streamlit), and Docker-based deployment options.

##### Main Components Identification
- API / Serving Layer – Represented by the “serving” directory. This includes modules such as main.py, openai_api_server.py, chat_ui.py, and server_ui.py which handle incoming requests from clients (either via REST APIs, CLI, or via a web UI) and route them appropriately.
- Model Backends – Located in “model_backend”, these modules (e.g., hf_backend.py, vllm_backend.py, lmdeploy_backend.py, sglang_backend.py) encapsulate the interactions with various inference engines and abstract the differences between them.
- Model Handlers – Found in “model_handler” (including the “react” subdirectories for different versions and the prompts and tool_parser functionality). These are responsible for defining how to handle and process model input/output, including special cases like guided decoding and function call (tools) handling.
- Model Workers – In “model_worker”, this layer includes the worker implementations (worker base modules and specific model integrations such as chatglm, deepseek, llama, etc.) that execute the heavy lifting of inference, embedding, and reranking.
- OpenAI API Protocol – The “openai_api_protocol” folder contains modules that adapt the internal calls so they mimic OpenAI’s API spec.
- CLI & Scripts – The CLI (cli.py) and supportive scripts (start.sh, stop.sh, config files in “script”) help with deployment and configuration.
- Infrastructure & Deployment – Top-level Dockerfiles, docker-compose.yml, and GitHub workflows indicate CI/CD, containerization, and deployment-ready components.
- Tests – The “tests” directory contains sample test cases that verify different functionalities (chat, embedding, TTS, ASR, moderation, etc.), ensuring all components work in an integrated fashion.

##### Dependencies, Technologies, and Architectural Patterns
- The project employs a micro-kernel architecture in which discrete modules (backend, handler, worker) allow extensibility and easier integration of new models.
- External dependencies include integration with FastChat, vLLM, LMDeploy, infinity, and recognized libraries for OpenAI API compatibility.
- A clear separation of concerns is visible: API layer (interaction), Model Handlers (input/output processing), Model Workers (execution), and Backends (integration with inference engines).

- Notes on CI/CD processes (represented by GitHub Workflows and Docker Compose) to indicate how builds and deployments are automated.
- If space allows, add a subsection showing supported models from the README (LLMs, VLMs, Embedding/Rerank, ASR) connected to the corresponding backend and worker modules.
- That the system offers extensibility so that new models or backends can be added with minimal code changes (supported by the modular design seen in the file tree).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Interface Layer"
        CLIENT["Client Request"]:::client
    end

    subgraph "API & Serving Layer"
        SERVING["API/Serving"]:::api
        OPENAI["OpenAI API Protocol"]:::adapter
    end

    subgraph "Business Logic Layer"
        HANDLER["Model Handlers"]:::handler
    end

    subgraph "Execution Layer"
        WORKER["Model Workers"]:::worker
        BACKENDS["Model Backends"]:::backend
    end

    subgraph "Deployment & Supporting"
        CLI["CLI & Scripts"]:::script
        INFRA["Infrastructure & Deployment"]:::infra
        TEST["Tests"]:::test
    end

    CLIENT -->|"sends_request"| SERVING
    SERVING -->|"routes_to"| HANDLER
    HANDLER -->|"delegates_to"| WORKER
    WORKER -->|"calls_backend"| BACKENDS
    BACKENDS -->|"returns_response"| WORKER
    WORKER -->|"sends_result"| HANDLER
    HANDLER -->|"formats_response"| SERVING
    SERVING -->|"integrates"| OPENAI
    CLI -->|"triggers"| SERVING
    INFRA -->|"deploys"| SERVING
    TEST -->|"validates"| SERVING
    TEST -->|"validates"| HANDLER
    TEST -->|"validates"| WORKER
    TEST -->|"validates"| BACKENDS

    classDef client fill:#f9c,stroke:#333,stroke-width:2px;
    classDef api fill:#b3e5fc,stroke:#0277bd,stroke-width:2px;
    classDef adapter fill:#fce4ec,stroke:#ad1457,stroke-width:2px;
    classDef handler fill:#fff9c4,stroke:#f9a825,stroke-width:2px;
    classDef worker fill:#c8e6c9,stroke:#388e3c,stroke-width:2px;
    classDef backend fill:#e1bee7,stroke:#8e24aa,stroke-width:2px;
    classDef script fill:#d7ccc8,stroke:#5d4037,stroke-width:2px;
    classDef infra fill:#cfd8dc,stroke:#455a64,stroke-width:2px;
    classDef test fill:#ffe0b2,stroke:#fb8c00,stroke-width:2px;

    click SERVING "https://github.com/shell-nlp/gpt_server/tree/main/gpt_server/serving"
    click OPENAI "https://github.com/shell-nlp/gpt_server/tree/main/gpt_server/openai_api_protocol"
    click HANDLER "https://github.com/shell-nlp/gpt_server/tree/main/gpt_server/model_handler"
    click WORKER "https://github.com/shell-nlp/gpt_server/tree/main/gpt_server/model_worker"
    click BACKENDS "https://github.com/shell-nlp/gpt_server/tree/main/gpt_server/model_backend"
    click CLI "https://github.com/shell-nlp/gpt_server/blob/main/gpt_server/cli.py"
    click INFRA "https://github.com/shell-nlp/gpt_server/tree/main/Dockerfile"
    click TEST "https://github.com/shell-nlp/gpt_server/tree/main/tests"
```
