# guidance-ai/llgtrt architecture

> Architecture diagram of the GitHub repository guidance-ai/llgtrt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/guidance-ai/llgtrt
- Repository: https://github.com/guidance-ai/llgtrt

## Overview

##### Project Type and Purpose
- This project implements a high-performance REST HTTP server that provides an OpenAI-compatible API for text completions and chat endpoints.
- It is built entirely in Rust (with some supporting Python files for various input processors) and integrates NVIDIA’s TensorRT-LLM as the inference backend, along with Microsoft’s llguidance library.
- The focus of the system is on low-overhead, constrained-output generation—using grammars (including JSON schemas and Lark grammars) to direct token sampling via custom CUDA kernels.
- Additional support components include LoRA support for finetuning and multi-model setups.

##### Analyzing the File Structure
- Core Code: The main service is located in the “llgtrt” directory. Within “llgtrt/src” you see various modules (e.g., routes, chat, config, error, tokenizer, logging) which indicate an organized backend with clearly defined API endpoints and supporting routines.
- Docker Directory: The “docker” folder contains scripts for building, pushing, and running the containerized server. There are also several shell scripts (bash.sh, run.sh, build.sh) that help orchestrate the container operations.
- Additional Language/Backend Support: Directories such as “trtllm-c” (C++/CUDA code) and “trtllm_rs” (Rust version of TRTLLM wrapper) indicate supplementary engines or bindings further emphasizing the integration with NVIDIA’s TensorRT-LLM.
- Configuration & Model Parameters: The “model_configs” directory holds model-specific JSON and J2 template files that the server can use. Configuration files are also printed out by the server (llgtrt.json5 and chat_template.j2).
- Supporting Scripts and Docs: The “scripts” directory (with build, test, and regression tools) and “docs” directory provide supporting details for development and integration.

##### Architectural Components
a. API/HTTP Server:
- Core REST HTTP server implemented in Rust (using the “llgtrt/src” folder).
- Modules such as the API routes (completions, chats, health check, openai routing) and the control modules for configuration, logging, error handling, and state management.
b. Inference Engine Integration:
- A component representing the TensorRT-LLM engine. Indicate that it is accessed through shell scripts (the “docker” scripts and build instructions) and via the “trtllm_rs” and “trtllm-c” modules.
- Role of custom CUDA kernels that take the logits from TensorRT and apply the guidance masks computed by the llguidance module.
c. Guidance and Grammar Processing:
- Incorporate the llguidance library as a sub-component that computes token masks based on various constraints (JSON schema, regular expressions, Lark grammar).
- Notice the inclusion of derivative-based regexes and the support for multi-modal inputs.
d. Configuration Management:
- A component for configuration, demonstrating how the server loads configuration information from JSON5 files, chat templates (Jinja2 templates), and model configuration files.
e. The Docker Container/Deployment Layer:
- Role of Docker in containerizing and deploying the server. Show that various scripts (in the “docker” folder) handle build, run, and shell access for in-container operations.
f. External Dependencies:
- External integrations like NVIDIA GPU hardware, LoRA files (with Finetuning support via the LoRA.md instructions), and possible custom input processors in the “llgtrt/py” folder should be modeled.
- Mention the dependency on Linux with nvidia-docker runtime.

##### Relationships & Data Flow
- Interaction between the server’s route handlers and the inference engine: after receiving a request, data is processed, configuration is applied, and then the core inference engine (TensorRT-LLM) is invoked, with guidance masks being applied by the llguidance component.
- Auxiliary flows for starting up the engine (building the engine from checkpoint conversion, moving tokenizer and preprocessor files) as separate initialization steps.

##### Architectural Patterns
- The backend is structured in a modular way with clear separation between API routing, domain logic (chat, completions), and system utilities (state management, error handling).
- The deployment employs containerization, which can be represented by an external “Docker” envelope encompassing the service.
- The integration between CPU-side logic (parsing, configuration, routing) and GPU-side computation (TensorRT engine with custom CUDA kernels) should be clearly delineated.
- The use of grammar-based constraints is a key design element, making it necessary to visually emphasize its path alongside raw logits computation.

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Entities
    Client["Client (API Request)"]:::external
    GPU["NVIDIA GPU (hardware)"]:::external
    LoRA["LoRA Support"]:::external
    PythonSupport["Python Input Processors"]:::external

    %% Docker Container & Deployment
    subgraph "Docker Container & Deployment"
        Docker["Docker Container & Deployment"]:::docker
        Dev["CI/CD & Dev Environment"]:::docker
    end

    %% Backend Service - Rust REST API
    subgraph "Backend Service - Rust REST API"
        API["Rust REST API"]:::api
        Routes["API Routes"]:::api
        Chat["Chat Module"]:::api
        Config["Configuration Module"]:::config
        Error["Error Handling"]:::api
        Logging["Logging"]:::api
        Tokenizer["Tokenizer"]:::api
        State["State Management"]:::api
        Main["Main Application"]:::api
    end

    %% Inference Engine Integration
    subgraph "Inference Engine Integration"
        TRT_Rs["TensorRT-LLM (Rust Binding)"]:::inference
        TRT_C["TensorRT-LLM (C++/CUDA)"]:::inference
        CudaKernels["Custom CUDA Kernels"]:::inference
    end

    %% Guidance and Grammar Processing
    subgraph "Guidance and Grammar Processing"
        Guidance["llguidance Library"]:::guidance
    end

    %% Model & Configuration Files
    subgraph "Model & Configuration Files"
        ConfigFile["Configuration Files (JSON5, Jinja2)"]:::config
        ModelConfigs["Model Configs Directory"]:::config
    end

    %% Connections: Client to API Server
    Client -->|"HTTP_Request"| API

    %% Internal API Module Connections
    API -->|"Routes"| Routes
    API -->|"Chat"| Chat
    API -->|"Config"| Config
    API -->|"Error"| Error
    API -->|"Logging"| Logging
    API -->|"Tokenizer"| Tokenizer
    API -->|"State"| State
    API -->|"Startup"| Main

    %% API invoking Inference Engine
    API -->|"Invoke_Inference"| TRT_Rs
    API -->|"Invoke_Inference"| TRT_C

    %% Inference Engine interactions
    TRT_Rs -->|"Uses"| CudaKernels
    TRT_C -->|"Uses"| CudaKernels
    TRT_Rs -->|"Apply_Guidance"| Guidance
    TRT_C -->|"Apply_Guidance"| Guidance

    %% Configuration management connections
    Config -->|"Load_Settings"| ConfigFile
    Config -->|"Load_ModelParams"| ModelConfigs

    %% Docker enveloping system components
    Docker --> API
    Docker --> TRT_Rs
    Docker --> TRT_C
    Docker --> Guidance
    Docker --> Config

    %% External flows
    PythonSupport -->|"Preprocess_Input"| API
    GPU -->|"CUDA_Processing"| CudaKernels
    Dev --> Docker

    %% Response flow back to Client
    TRT_Rs -->|"Send_Response"| Client
    TRT_C -->|"Send_Response"| Client
    Guidance -->|"Guidance_Info"| API

    %% Click Events for API/HTTP Server Components
    click API "https://github.com/guidance-ai/llgtrt/tree/main/llgtrt/src"
    click Routes "https://github.com/guidance-ai/llgtrt/tree/main/llgtrt/src/routes"
    click Chat "https://github.com/guidance-ai/llgtrt/blob/main/llgtrt/src/chat.rs"
    click Config "https://github.com/guidance-ai/llgtrt/blob/main/llgtrt/src/config.rs"
    click Error "https://github.com/guidance-ai/llgtrt/blob/main/llgtrt/src/error.rs"
    click Logging "https://github.com/guidance-ai/llgtrt/blob/main/llgtrt/src/logging.rs"
    click Tokenizer "https://github.com/guidance-ai/llgtrt/blob/main/llgtrt/src/tokenizer.rs"
    click State "https://github.com/guidance-ai/llgtrt/blob/main/llgtrt/src/state.rs"
    click Main "https://github.com/guidance-ai/llgtrt/blob/main/llgtrt/src/main.rs"

    %% Click Events for Inference Engine Integration
    click TRT_Rs "https://github.com/guidance-ai/llgtrt/tree/main/trtllm_rs"
    click TRT_C "https://github.com/guidance-ai/llgtrt/tree/main/trtllm-c"
    click CudaKernels "https://github.com/guidance-ai/llgtrt/blob/main/trtllm-c/mask_logits.cu"

    %% Click Events for Guidance and Grammar Processing
    click Guidance "https://github.com/guidance-ai/llgtrt/tree/main/llguidance"

    %% Click Events for Configuration Management
    click ConfigFile "https://github.com/guidance-ai/llgtrt/blob/main/llgtrt/src/config_info.json"
    click ModelConfigs "https://github.com/guidance-ai/llgtrt/tree/main/model_configs"

    %% Click Events for Docker Container/Deployment Layer
    click Docker "https://github.com/guidance-ai/llgtrt/tree/main/docker"

    %% Click Events for External Dependencies
    click LoRA "https://github.com/guidance-ai/llgtrt/blob/main/LoRA.md"
    click PythonSupport "https://github.com/guidance-ai/llgtrt/tree/main/llgtrt/py"

    %% Class Definitions for Styling
    classDef api fill:#DDEEFF,stroke:#333,stroke-width:2px;
    classDef inference fill:#FFDDBB,stroke:#333,stroke-width:2px;
    classDef guidance fill:#CCFFCC,stroke:#333,stroke-width:2px;
    classDef config fill:#FFFFCC,stroke:#333,stroke-width:2px;
    classDef docker fill:#FFCCCC,stroke:#333,stroke-width:2px;
    classDef external fill:#E0E0E0,stroke:#333,stroke-width:2px;

    class API,Routes,Chat,Error,Logging,Tokenizer,State,Main api;
    class TRT_Rs,TRT_C,CudaKernels inference;
    class Guidance guidance;
    class Config,ConfigFile,ModelConfigs config;
    class Docker,Dev docker;
    class Client,GPU,LoRA,PythonSupport external;
```
