# predibase/lorax architecture

> Architecture diagram of the GitHub repository predibase/lorax, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/predibase/lorax
- Repository: https://github.com/predibase/lorax

## Overview

##### Main Components

- Inference Server (Backend):
- Located in the “server” directory, this is the core of the system. It handles model inference, adapter loading, and dynamic batching.
- Includes modules for adapter management (dynamic LoRA/adapter loading), model layers (various optimizations and CUDA kernels such as flash-attention, paged attention, SGMV, GPT-Q, etc.), caching, CLI operations, and scheduling.

- Routing & API Gateway:
- The “router” component manages incoming requests. Its submodules (client, grpc-metadata, scheduler, etc.) provide routing functionality including load-balancing, batching, and health checks.
- Provides an OpenAI-compatible REST API for multi-turn chat and inference requests.

- Launcher & Deployment Tools:
- The “launcher” directory contains startup scripts (written in Rust/Cargo) for initializing the inference server.
- There are additional deployment scripts (Dockerfile, Dockerfile.dev, container-entrypoint.sh, sagemaker-entrypoint.sh) and helm charts (“charts/lorax”) to support containerized and Kubernetes deployments.

- Clients:
- The “clients/python” directory holds the Python client, which provides a wrapper to interact with the REST API. It also includes test suites to verify client behavior.

- Infrastructure & Testing:
- Integration tests (integration-tests/) and additional tests (server/tests, tests/) exist to simulate end-to-end API, adapter, and inference behaviors.
- CI/CD workflows are defined in “.github/workflows” to execute build, test, and deployment pipelines.

- External Dependencies:
- GPU components (Nvidia CUDA drivers and pre-compiled CUDA kernels in several directories under “server/”) indicate heavy reliance on GPU optimization.
- The system relies on Docker, Kubernetes (with Helm charts), and various Rust libraries (managed via Cargo.toml) to operate.

##### Relationships and Interactions

- Client Layer:
- Python clients and OpenAI API–compatible interfaces interact with the system through REST calls.

- API Gateway/Router:
- Incoming HTTP requests first hit the router, which performs request validation, batching (heterogeneous continuous batching), and routing to the appropriate backend processes.

- Core Inference Engine:
- The server component processes requests by dynamically loading either the base model or the specialized LoRA adapters. It handles scheduling for adapter exchange/ loading and optimizes GPU memory usage.
- Within the server, various modules interact—such as layers for optimized inference (including custom CUDA kernels) and adapter management modules.

- Deployment and Ops:
- Dockerfiles, Kubernetes Helm charts, and CI/CD configurations enable deployment and scaling on GPU-enabled hardware.

- Testing and Continuous Integration:
- Integration and unit tests ensure system reliability and are triggered via designated CI pipelines.

##### Architectural Patterns and Key Principles

- Modular/Microservice-Like Architecture:
- The project is broken into distinct components (server, router, launcher, clients, test suites) that interact over well-defined APIs.

- Dynamic Loading and Scheduling:
- Dynamic adapter loading ensures that multiple fine-tuned LLMs can be served concurrently by loading adapters just-in-time on demand.
- A sophisticated scheduling mechanism asynchronously manages adapter prefetching and GPU/CPU memory balancing.

- Containerization and Orchestration:
- The presence of Dockerfiles, Helm charts, and Kubernetes guides demonstrates a design optimized for containerized deployment.

- High-Performance and Optimized Inference:
- The integration of pre-compiled highly optimized CUDA kernels (flash-attention, paged attention, SGMV) and the use of quantization libraries shows a focus on performance tuning.

##### Final Recommendations for the Principal Software Engineer

- Drill down into the Core Inference Engine to highlight subcomponents such as dynamic adapter loader, scheduling/batching component, and GPU optimization layers.
- Finally, document any external dependencies (hardware drivers, CUDA, Kubernetes) as part of the infrastructure layer to provide context on system requirements.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer"
        PythonClient["Python Client (SDK & Tests)"]
    end

    %% API Gateway / Router
    subgraph "API Gateway / Router"
        APIGateway["Routing & API Gateway (OpenAI Compatible)"]
    end

    %% Core Inference Server
    subgraph "Core Inference Server"
        InferenceServer["Inference Server (Backend)"]
        AdapterManagement["Dynamic Adapter Management"]
        GPUKernels["Optimized Model Layers & GPU Kernels"]
        Scheduling["Scheduling & Batching"]
        InferenceServer --> AdapterManagement
        InferenceServer --> GPUKernels
        InferenceServer --> Scheduling
        %% Breakdown of GPU Kernels modules
        GPUKernels --> GPUKernels_custom["Custom Kernels"]
        GPUKernels --> GPUKernels_exllama["ExLlama Kernels"]
        GPUKernels --> GPUKernels_exllamav2["ExLlamaV2 Kernels"]
        GPUKernels --> GPUKernels_punica["Punica Kernels"]
    end

    %% Deployment & Operations
    subgraph "Deployment & Operations"
        Launcher["Launcher & Startup Scripts"]
        Dockerfile["Dockerfile"]
        DockerfileDev["Dockerfile.dev"]
        ContainerEntrypoint["container-entrypoint.sh"]
        SageMakerEntrypoint["sagemaker-entrypoint.sh"]
        HelmCharts["Helm Charts (Kubernetes/Helm)"]
    end

    %% Testing & CI
    subgraph "Testing & CI"
        IntegrationTests["Integration Tests"]
        ServerTests["Server Unit Tests"]
        Tests["Additional Tests"]
        CICD["CI/CD Workflows"]
    end

    %% Connections between layers
    PythonClient -->|"REST_API_calls"| APIGateway
    APIGateway -->|"Routes_request"| InferenceServer

    %% Deployment tools connecting to server
    Launcher -->|"Deploys"| InferenceServer
    Dockerfile -->|"Deploys"| InferenceServer
    DockerfileDev -->|"Deploys"| InferenceServer
    ContainerEntrypoint -->|"Deploys"| InferenceServer
    SageMakerEntrypoint -->|"Deploys"| InferenceServer
    HelmCharts -->|"Deploys"| InferenceServer
    %% Also deployment can affect the API Gateway
    Dockerfile -->|"Deploys"| APIGateway
    HelmCharts -->|"Deploys"| APIGateway

    %% Testing connections
    IntegrationTests -->|"Tests"| InferenceServer
    ServerTests -->|"Tests"| InferenceServer
    Tests -->|"Tests"| InferenceServer
    IntegrationTests -->|"Tests"| APIGateway
    CICD -->|"Triggers"| IntegrationTests
    CICD -->|"Triggers"| ServerTests
    CICD -->|"Triggers"| Tests

    %% Styles
    classDef client fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef gateway fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef server fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef deployment fill:#FFA07A,stroke:#000,stroke-width:2px;
    classDef testing fill:#F08080,stroke:#000,stroke-width:2px;

    class PythonClient client;
    class APIGateway gateway;
    class InferenceServer,AdapterManagement,GPUKernels,Scheduling,GPUKernels_custom,GPUKernels_exllama,GPUKernels_exllamav2,GPUKernels_punica server;
    class Launcher,Dockerfile,DockerfileDev,ContainerEntrypoint,SageMakerEntrypoint,HelmCharts deployment;
    class IntegrationTests,ServerTests,Tests,CICD testing;

    %% Click Events
    click PythonClient "https://github.com/predibase/lorax/tree/main/clients/python"
    click APIGateway "https://github.com/predibase/lorax/tree/main/router/"
    click InferenceServer "https://github.com/predibase/lorax/tree/main/server/lorax_server"
    click AdapterManagement "https://github.com/predibase/lorax/tree/main/server/lorax_server/adapters"
    click GPUKernels_custom "https://github.com/predibase/lorax/tree/main/server/custom_kernels"
    click GPUKernels_exllama "https://github.com/predibase/lorax/tree/main/server/exllama_kernels"
    click GPUKernels_exllamav2 "https://github.com/predibase/lorax/tree/main/server/exllamav2_kernels"
    click GPUKernels_punica "https://github.com/predibase/lorax/tree/main/server/punica_kernels"
    click Launcher "https://github.com/predibase/lorax/tree/main/launcher/"
    click Dockerfile "https://github.com/predibase/lorax/tree/main/Dockerfile"
    click DockerfileDev "https://github.com/predibase/lorax/blob/main/Dockerfile.dev"
    click ContainerEntrypoint "https://github.com/predibase/lorax/blob/main/container-entrypoint.sh"
    click SageMakerEntrypoint "https://github.com/predibase/lorax/blob/main/sagemaker-entrypoint.sh"
    click HelmCharts "https://github.com/predibase/lorax/tree/main/charts/lorax"
    click IntegrationTests "https://github.com/predibase/lorax/tree/main/integration-tests/"
    click ServerTests "https://github.com/predibase/lorax/tree/main/server/tests/"
    click Tests "https://github.com/predibase/lorax/tree/main/tests/"
    click CICD "https://github.com/predibase/lorax/tree/main/.github/workflows"
```
