# rustformers/llm architecture

> Architecture diagram of the GitHub repository rustformers/llm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rustformers/llm
- Repository: https://github.com/rustformers/llm

## Overview

##### Project’s Main Purpose and Type
- This is an ecosystem of Rust libraries and binaries that together offer a framework for running inference on various large language models.
- It comprises a core library (the “llm” crate) that wraps base functionality (via “llm-base”) and model-specific implementations (in the “crates/models” directory), plus several command-line tools for end-user interactions.
- It also integrates with the GGML tensor library (“crates/ggml”) to support efficient machine learning inference on CPU—while also having some support for hardware acceleration (CUDA, Metal, OpenCL) as seen in the accelerator modules.

##### Component Breakdown Based on the File Structure
- Libraries (Crates):
- “crates/ggml”: Contains the core tensor functionality and hardware accelerator integrations (modules for CUDA, Metal, and OpenCL).
- “crates/llm-base”: Provides core functionalities for inference sessions, model loading, quantization, and session management.
- “crates/llm”: The primary public API that wraps “llm-base” and seamlessly integrates with the various supported model crates.
- “crates/models”: Contains sub-crates for different models (bloom, gpt2, gptj, gptneox, llama, mpt, etc.), each encapsulating model-specific logic and possibly pre/post processing.

- CLI and Utilities (Binaries):
- “binaries/llm-cli”: The command-line interface providing interactive and REPL-based access to run inferences.
- “binaries/llm-test”: Likely a suite for testing inference with various configuration files for different models (e.g., bloom.json, gptj.json, etc.).
- “binaries/generate-ggml-bindings”: Utility to generate Rust bindings for the GGML library.
- “binaries/precommit-check”: A tool for precommit validations to ensure code hygiene before commits.

- Additional Supporting Files and Docs:
- “doc”: Provides documentation on contributing, acceleration support, and known-good models.
- “utils”: Contains Dockerfiles, prompts (model specific prompt templates) and possibly deployment scripts.
- CI/CD configurations under “.github/workflows”, as well as build configurations (Cargo.toml, rust toolchain configs) at the root.

- Frontend/Interface for Users:
- Core API/Library Layer:
- The “llm” crate which exposes major functionalities.
- “llm-base” sitting underneath as a support layer for inference sessions, model loading and quantization.
- Model Modules:
- Multiple model-specific crates (bloom, gpt2, gptj, etc.) that plug into the API.
- Infrastructure / Tensor Processing Engine:
- “ggml” crate (and its system-level bindings under “ggml/sys”) that handles the tensor operations, including integrations with hardware accelerators.
- Accelerators (i.e., CUDA, Metal, OpenCL) as subcomponents inside the tensor library.
- Utility Tools and Test Suites:
- The testing binary (“llm-test”) and precommit-check tools, indicating instrumentation for ensuring quality.
- External Dependencies and Bindings:

##### Final Assembly and Detailing
- At the bottom, illustrate the “ggml” crate as the computational engine with sub-modules for hardware acceleration.
- Optionally, position utility tools and documentation around these core components as peripheral but important elements.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface
    U("User"):::user
    CLI("llm-cli"):::user

    %% Core Layers
    API("llm crate"):::core
    BASE("llm-base"):::support

    %% Model-specific Implementations
    Bloom("Bloom"):::model
    GPT2("GPT2"):::model
    GPTJ("GPTJ"):::model
    GPTNeoX("GPTNeoX"):::model
    LLaMA("LLaMA"):::model
    MPT("MPT"):::model

    %% Tensor Processing Engine
    GGML("ggml crate"):::tensor
    CUDA("CUDA Module"):::accelerator
    Metal("Metal Module"):::accelerator
    OpenCL("OpenCL Module"):::accelerator

    %% Utility Tools
    TEST("llm-test"):::utility
    BINDINGS("GGML Bindings Generator"):::utility
    PRECOMMIT("Precommit Check Tool"):::utility

    %% Documentation and Supporting Files
    DOC("Documentation"):::docs
    UTILS("Utility Files"):::docs

    %% Subgraph Groupings for clarity
    subgraph "User Interface Layer"
        U
        CLI
    end

    subgraph "Core Inference Layers"
        API
        BASE
    end

    subgraph "Model Modules"
        Bloom
        GPT2
        GPTJ
        GPTNeoX
        LLaMA
        MPT
    end

    subgraph "Tensor Engine & Accelerators"
        GGML
        CUDA
        Metal
        OpenCL
    end

    subgraph "Utility Tools"
        TEST
        BINDINGS
        PRECOMMIT
    end

    %% Relationships
    U -->|"uses"| CLI
    CLI -->|"triggers"| API
    API -->|"leverages"| BASE
    API -->|"integrates"| Bloom
    API -->|"integrates"| GPT2
    API -->|"integrates"| GPTJ
    API -->|"integrates"| GPTNeoX
    API -->|"integrates"| LLaMA
    API -->|"integrates"| MPT
    BASE -->|"utilizes"| GGML
    GGML -->|"includes"| CUDA
    GGML -->|"includes"| Metal
    GGML -->|"includes"| OpenCL

    %% Utility Tools interact with core components
    TEST -->|"validates"| API
    BINDINGS -->|"generates_bindings_for"| GGML
    PRECOMMIT -->|"checks"| API

    %% Documentation and Utility Files are supportive
    DOC ---|"references"| API
    UTILS ---|"supports"| API

    %% Click Events
    click API "https://github.com/rustformers/llm/blob/main/crates/llm/src/lib.rs"
    click BASE "https://github.com/rustformers/llm/blob/main/crates/llm-base/src/lib.rs"
    click GGML "https://github.com/rustformers/llm/blob/main/crates/ggml/src/lib.rs"
    click Bloom "https://github.com/rustformers/llm/blob/main/crates/models/bloom/src/lib.rs"
    click GPT2 "https://github.com/rustformers/llm/blob/main/crates/models/gpt2/src/lib.rs"
    click GPTJ "https://github.com/rustformers/llm/blob/main/crates/models/gptj/src/lib.rs"
    click GPTNeoX "https://github.com/rustformers/llm/blob/main/crates/models/gptneox/src/lib.rs"
    click LLaMA "https://github.com/rustformers/llm/blob/main/crates/models/llama/src/lib.rs"
    click MPT "https://github.com/rustformers/llm/blob/main/crates/models/mpt/src/lib.rs"
    click CLI "https://github.com/rustformers/llm/blob/main/binaries/llm-cli/src/main.rs"
    click TEST "https://github.com/rustformers/llm/blob/main/binaries/llm-test/src/main.rs"
    click BINDINGS "https://github.com/rustformers/llm/blob/main/binaries/generate-ggml-bindings/src/main.rs"
    click PRECOMMIT "https://github.com/rustformers/llm/blob/main/binaries/precommit-check/src/main.rs"
    click DOC "https://github.com/rustformers/llm/tree/main/doc/"
    click UTILS "https://github.com/rustformers/llm/tree/main/utils/Dockerfile"

    %% Class Definitions for styling and colors
    classDef user fill:#f9c,stroke:#333,stroke-width:2px;
    classDef core fill:#ccf,stroke:#333,stroke-width:2px;
    classDef support fill:#cfc,stroke:#333,stroke-width:2px;
    classDef model fill:#fc9,stroke:#333,stroke-width:2px;
    classDef tensor fill:#9cf,stroke:#333,stroke-width:2px;
    classDef accelerator fill:#ff9,stroke:#333,stroke-width:2px;
    classDef utility fill:#cff,stroke:#333,stroke-width:2px;
    classDef docs fill:#eee,stroke:#333,stroke-width:2px;
```
