# lightning-ai/litserve architecture

> Architecture diagram of the GitHub repository lightning-ai/litserve, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lightning-ai/litserve
- Repository: https://github.com/lightning-ai/litserve

## Overview

##### Project Type & Overall Purpose
- LitServe is an open-source AI inference engine designed to serve various types of AI models (LLMs, vision, audio, etc.) quickly and flexibly.
- It augments FastAPI with AI-specific optimizations (batching, streaming, GPU autoscaling) and provides abstractions (such as LitAPI and LitServer) for building compound systems.
- The README clearly indicates its use for deploying single or multi-model systems and offers both self-hosted and cloud deployment options.

##### Core Components from the File Structure
- Server and API layer:
– The src/litserve/server.py and src/litserve/api.py files form the heart of serving. LitServer wraps a FastAPI instance (with enhancements for performance and scaling) and uses a user-defined LitAPI (with methods like setup, decode_request, predict, encode_response) to handle user logic.
- Request Handling and Data Flow:
– The transport/ directory (with modules such as process_transport.py and zmq_transport.py) provides mechanisms to control data movement—this indicates alternative communication strategies (process-based or ZeroMQ-based) that may be used in different deployment scenarios.
– Middlewares (src/litserve/middlewares.py) allow intercepting or modifying requests/responses.
- Efficient Processing:
– The loops/ directory (with continuous_batching_loop.py, streaming_loops.py, etc.) shows how LitServe manages batching and streaming, optimizing resource usage.
- Auxiliary Components:
– Callbacks (callbacks/ and callbacks/defaults/) are used for handling metrics and custom behaviors.
– Docker support: docker_builder.py provides functionality for containerization.
– The CLI (cli.py) and python_client.py streamline command-line interactions and client testing.
- Testing and CI:
– A robust tests/ directory and numerous GitHub workflows and configuration files (e.g., in .github/workflows) indicate the project’s maturity in terms of continuous integration, testing, and versioning.

##### Key Architectural Patterns & Relationships
- Abstraction via LitAPI: Users define their own API by extending or using provided methods (setup, decode_request, predict, encode_response). This isolates business logic from server management.
- Server Orchestration via LitServer: This component manages FastAPI work and integrates advanced serving features like batching and autoscaling.
- Modular Design: The separation among callbacks, loops, the transport layer, and middlewares shows a modular approach that promotes extensibility and decoupling.
- Communication Patterns: Directional flows are clear—from external clients sending HTTP (or other protocol) requests to the FastAPI server, then into the business logic (via the LitAPI), and out as responses. The transport modules might show further internal messaging (e.g., between different processing loops).
- Containerization & Cloud Deployment: Docker and CLI components denote that LitServe is designed to be easily distributed and managed across environments.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Client and Deployment Subgraph
    subgraph "Client & Deployment"
        A["External Client"]:::client
        subgraph "Deployment Tools"
            M["CLI Tool"]:::deployment
            N["Python Client"]:::deployment
            O["Docker Builder"]:::deployment
        end
        M --> A
        N --> A
        O --> A
    end

    %% Server Components Subgraph
    subgraph "Server Components"
        B["LitServer (FastAPI Server)"]:::server
        C["LitAPI (User API)"]:::server
        D["Middlewares"]:::server
    end

    %% Processing Layer Subgraph
    subgraph "Processing Layer"
        subgraph "Processing Loops"
            E["Continuous Batching Loop"]:::processing
            F["Streaming Loops"]:::processing
            G["Simple Loops"]:::processing
        end
        subgraph "Transport Layer"
            H["Process Transport"]:::transport
            I["ZeroMQ Transport"]:::transport
            J["ZMQ Queue"]:::transport
        end
    end

    %% Auxiliary Subgraph
    subgraph "Auxiliary"
        K["Base Callbacks"]:::auxiliary
        L["Metric Callback"]:::auxiliary
    end

    %% Data Flow Connections
    A -->|"HTTP_Request"| B
    B -->|"routes_to"| C
    C -->|"executes"| D
    D -->|"triggers"| E
    E -->|"sends_data"| H
    H -->|"communicates"| I
    I -->|"queues_via"| J
    E -->|"callback"| K
    F -->|"callback"| K
    G -->|"callback"| K
    K -->|"assembles_response"| B
    D -->|"invokes"| F
    D -->|"invokes"| G

    %% Class Definitions
    classDef client fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef server fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef processing fill:#FFFF99,stroke:#000,stroke-width:1px;
    classDef transport fill:#FFD700,stroke:#000,stroke-width:1px;
    classDef auxiliary fill:#FFB6C1,stroke:#000,stroke-width:1px;
    classDef deployment fill:#D3D3D3,stroke:#000,stroke-width:1px;

    %% Click Events
    click B "https://github.com/lightning-ai/litserve/blob/main/src/litserve/server.py"
    click C "https://github.com/lightning-ai/litserve/blob/main/src/litserve/api.py"
    click D "https://github.com/lightning-ai/litserve/blob/main/src/litserve/middlewares.py"
    click H "https://github.com/lightning-ai/litserve/blob/main/src/litserve/transport/process_transport.py"
    click I "https://github.com/lightning-ai/litserve/blob/main/src/litserve/transport/zmq_transport.py"
    click J "https://github.com/lightning-ai/litserve/blob/main/src/litserve/transport/zmq_queue.py"
    click E "https://github.com/lightning-ai/litserve/blob/main/src/litserve/loops/continuous_batching_loop.py"
    click F "https://github.com/lightning-ai/litserve/blob/main/src/litserve/loops/streaming_loops.py"
    click G "https://github.com/lightning-ai/litserve/blob/main/src/litserve/loops/simple_loops.py"
    click K "https://github.com/lightning-ai/litserve/blob/main/src/litserve/callbacks/base.py"
    click L "https://github.com/lightning-ai/litserve/blob/main/src/litserve/callbacks/defaults/metric_callback.py"
    click M "https://github.com/lightning-ai/litserve/blob/main/src/litserve/cli.py"
    click N "https://github.com/lightning-ai/litserve/blob/main/src/litserve/python_client.py"
    click O "https://github.com/lightning-ai/litserve/blob/main/src/litserve/docker_builder.py"
```
