# huggingface/text-generation-inference architecture

> Architecture diagram of the GitHub repository huggingface/text-generation-inference, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/huggingface/text-generation-inference
- Repository: https://github.com/huggingface/text-generation-inference

## Overview

##### Main Components

a. Client Layer:
- REST API endpoints (/generate_stream, /v1/chat/completions)
- Python Client
- gRPC Client
- Messages API (OpenAI-compatible)

b. Router Layer:
- Request handling and validation
- Load balancing
- Batching system
- Streaming response handling
- Configuration management

c. Server Layer:
- Model loading and management
- Tensor parallelism coordination
- Inference pipeline
- Memory management
- Quantization handlers

d. Backend Components:
- CUDA/ROCm kernels
- Custom optimizations (Flash Attention, Paged Attention)
- Different backend implementations (trtllm, v2, v3)

e. Monitoring & Observability:
- OpenTelemetry integration
- Prometheus metrics
- Distributed tracing

##### Component Relationships

- Client requests flowing through the router to the server
- Batched requests being distributed across GPU workers
- Memory management between shared resources
- Monitoring data flow to observability systems

##### Architectural Patterns to Highlight

- Continuous batching system
- Tensor parallelism for multi-GPU deployment
- Streaming architecture for token generation
- Plugin architecture for different quantization methods
- Modular backend system

##### Implementation Details

- Rust-based core components (router, server)
- Python integration points
- GPU memory management
- Model loading pipeline
- Custom kernel integration points

##### Key Features to Emphasize

- Token streaming capability
- Multiple quantization options
- Custom kernel optimizations
- Support for different model architectures
- Security features (authentication, model access control)

**[Client Layer]**
- ↓
**[Router Layer (Rust)]**
- ↓
**[Server Layer (Python/Rust)]**
- ↓
**[Backend Layer (CUDA/ROCm)]**
- ↓
**[Hardware Layer]**

**[Core Components] | [Optimization Features] | [Monitoring]**

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph CL[Client Layer]
        direction TB
        PC[Python Client]:::client
        GC[gRPC Client]:::client
        RA["REST/Messages API"]:::client
    end

    subgraph RL[Router Layer]
        direction TB
        MR[Main Router]:::router
        RV[Request Validation]:::router
        IH[Inference Handler]:::router
        CF[Configuration]:::router
    end

    subgraph SL[Server Layer]
        direction TB
        CS[Core Server]:::server
        MM[Model Management]:::server
        MemM[Memory Management]:::server
        TP[Tensor Parallelism]:::server
    end

    subgraph BC[Backend Components]
        direction TB
        CK[CUDA Kernels]:::backend
        subgraph BI[Backend Implementations]
            direction TB
            V2[Backend v2]:::backend
            V3[Backend v3]:::backend
            TRTL[TensorRT-LLM]:::backend
        end
    end

    subgraph OF[Optimization Features]
        direction TB
        QZ[Quantization]:::opt
        FA[Flash Attention]:::opt
        PA[Paged Attention]:::opt
    end

    subgraph MO[Monitoring & Observability]
        direction TB
        TR[Tracing]:::monitoring
        MT[Metrics]:::monitoring
        TL[Telemetry]:::monitoring
    end

    %% Connections
    PC --> MR
    GC --> MR
    RA --> MR
    
    MR --> RV
    RV --> IH
    IH --> CS
    CF --> MR
    
    CS --> MM
    CS --> MemM
    CS --> TP
    
    MM --> CK
    MemM --> CK
    TP --> CK
    
    CK --> V2
    CK --> V3
    CK --> TRTL
    
    CS --> QZ
    CS --> FA
    CS --> PA
    
    MR --> TR
    CS --> MT
    IH --> TL

    %% Styles
    classDef client fill:#2374ab
    classDef router fill:#48a9a6
    classDef server fill:#4ad66d
    classDef backend fill:#e4b363
    classDef opt fill:#c1666b
    classDef monitoring fill:#9b4f96

    %% Click events
    click PC "https://github.com/huggingface/text-generation-inference/tree/main/clients/python/text_generation/"
    click GC "https://github.com/huggingface/text-generation-inference/tree/main/backends/client/src/"
    click RA "https://github.com/huggingface/text-generation-inference/blob/main/router/src/server.rs"
    click MR "https://github.com/huggingface/text-generation-inference/tree/main/router/src/"
    click RV "https://github.com/huggingface/text-generation-inference/blob/main/router/src/validation.rs"
    click IH "https://github.com/huggingface/text-generation-inference/tree/main/router/src/infer/"
    click CF "https://github.com/huggingface/text-generation-inference/blob/main/router/src/config.rs"
    click CS "https://github.com/huggingface/text-generation-inference/tree/main/server/text_generation_server/"
    click MM "https://github.com/huggingface/text-generation-inference/tree/main/server/text_generation_server/models/"
    click MemM "https://github.com/huggingface/text-generation-inference/tree/main/server/text_generation_server/layers/"
    click TP "https://github.com/huggingface/text-generation-inference/blob/main/server/text_generation_server/layers/tensor_parallel.py"
    click CK "https://github.com/huggingface/text-generation-inference/tree/main/server/custom_kernels/"
    click V2 "https://github.com/huggingface/text-generation-inference/tree/main/backends/v2/"
    click V3 "https://github.com/huggingface/text-generation-inference/tree/main/backends/v3/"
    click TRTL "https://github.com/huggingface/text-generation-inference/tree/main/backends/trtllm/"
    click QZ "https://github.com/huggingface/text-generation-inference/tree/main/server/text_generation_server/layers/gptq/"
    click FA "https://github.com/huggingface/text-generation-inference/blob/main/server/text_generation_server/layers/attention/flash_attn_triton.py"
    click PA "https://github.com/huggingface/text-generation-inference/blob/main/server/text_generation_server/layers/attention/kv_cache.py"
    click TR "https://github.com/huggingface/text-generation-inference/blob/main/server/text_generation_server/tracing.py"
    click MT "https://github.com/huggingface/text-generation-inference/blob/main/router/src/logging.rs"
    click TL "https://github.com/huggingface/text-generation-inference/blob/main/router/src/usage_stats.rs"
```
