# ggml-org/ggml architecture

> Architecture diagram of the GitHub repository ggml-org/ggml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ggml-org/ggml
- Repository: https://github.com/ggml-org/ggml

## Overview

##### Project type and purpose
- ggml is a low-level, cross-platform tensor library for machine learning (not a web or client-server app).
- It offers core tensor operations, automatic differentiation, optimizers, quantization, and multiple hardware backends.

##### File structure
- Top-level folders:
- src/           – core implementation and backend modules
- /       – public headers (ggml.h, ggml-alloc.h, ggml-backend.h, etc.)
- examples/      – sample programs and conversion scripts
- tests/         – unit tests covering individual ops
- cmake/, CMakeLists.txt – build configuration
- scripts/       – utility scripts (model sync, authors generation)
- .github/ci.yml – continuous integration pipeline
- Within src/:
- ggml.c, ggml-*.cpp – core engine and registration logic
- subdirs for each backend:
– ggml-cpu (SIMD, quant, AMX, multi-threading)
– ggml-blas
– ggml-cuda
– ggml-metal
– ggml-opencl
– ggml-sycl
– ggml-vulkan
– ggml-hip, ggml-cann, ggml-musa, ggml-kompute…
- ggml-rpc – optional remote execution interface
- examples/python/ – Python CFFI bindings and utilities

##### Extract insights from the README
- Core features: hardware-agnostic tensor primitives, no external deps, zero runtime allocations, AD and optimizers.
- Build flags enable or disable each hardware backend via CMake (e.g., –DGGML_CUDA=ON).

a. Main components
- Build system (CMake)
- Core tensor engine (ggml core)
- Backend modules (CPU, CUDA, Metal, Vulkan, OpenCL, SYCL, HIP, CANN, MUSA, Kompute)
- Public API (include/*.h)
- Language bindings (Python via CFFI)
- Example applications and CLI tools (gpt-2, mnist, sam, yolo, etc.)
- Tests (unit tests in tests/)
- Scripts and RPC (model conversion, remote execution)
b. Relationships and data flow
- CMake reads CMakeLists.txt → configures & compiles core + selected backends → produces libggml (static or shared) and example binaries.
- Public API exposes functions to client code (examples, Python bindings).
- Backends register implementations with the core engine (ggml-backend-reg.cpp).
- At runtime, user code calls ggml API → core dispatches to active backend → performs tensor operations on CPU or chosen device.
- Python binding layer loads libggml via CFFI → calls into public API.
- Example CLIs load model files (GGUF format) → invoke ggml to run inference/optimizers.
- RPC module can wrap ggml calls over network.
c. Architectural patterns and principles
- Modular plugin-style backends: each hardware target lives in its own folder and is conditionally included.
- Single unified core API → backend abstraction layer via registration and dispatch.
- Zero dynamic allocations: memory management through allocators defined in ggml-alloc.c.
- Build-time configurability: CMake flags select code paths and link only needed backends.
d. Technologies and frameworks
- C/C++ code base
- CMake for build orchestration
- Python CFFI for bindings
- GPU frameworks: CUDA, Metal, Vulkan, OpenCL, SYCL, HIP, CANN, Kompute
- Continuous Integration: GitHub Actions (ci.yml)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Build-time flow
    subgraph "Build System"
        direction TB
        CMakeLists["CMakeLists.txt"]:::tool
        CMakeDir["cmake/"]:::tool
    end

    subgraph "Core Tensor Engine"
        direction TB
        GGML_C["ggml.c"]:::core
        GGML_ALLOC["ggml-alloc.c"]:::core
        GGML_BACKEND_CPP["ggml-backend.cpp"]:::core
        GGML_REG["ggml-backend-reg.cpp"]:::core
        GGML_OPT["ggml-opt.cpp"]:::core
        GGML_QUANTS["ggml-quants.c/.h"]:::core
        GGUF["gguf.cpp"]:::core
        HDR_GGML["ggml.h"]:::core
        HDR_ALLOC["ggml-alloc.h"]:::core
    end

    subgraph "Backend Modules"
        direction TB
        CPU["CPU Backend<br/>(src/ggml-cpu/)"]:::backend
        BLAS["BLAS Backend<br/>(src/ggml-blas/)"]:::backend
        CUDA["CUDA Backend<br/>(src/ggml-cuda/)"]:::backend
        METAL["Metal Backend<br/>(src/ggml-metal/)"]:::backend
        VULKAN["Vulkan Backend<br/>(src/ggml-vulkan/)"]:::backend
        OPENCL["OpenCL Backend<br/>(src/ggml-opencl/)"]:::backend
        SYCL["SYCL Backend<br/>(src/ggml-sycl/)"]:::backend
        CANN["CANN Backend<br/>(src/ggml-cann/)"]:::backend
        KOMPUTE["Kompute Backend<br/>(src/ggml-kompute/)"]:::backend
    end

    subgraph "Public API & Headers"
        direction TB
        PublicAPI["Public API<br/>(include/*.h)"]:::api
    end

    subgraph "Language Bindings"
        direction TB
        PythonBindings["Python CFFI Bindings"]:::api
    end

    subgraph "Client Applications"
        direction TB
        Examples["Example Applications<br/>(examples/)"]:::api
    end

    subgraph "Tools & Tests"
        direction TB
        Scripts["scripts/"]:::tool
        Tests["tests/"]:::tool
        RPC["RPC Module<br/>(src/ggml-rpc/)"]:::tool
    end

    %% Artifacts
    LIB["libggml<br/>(static/shared)"]:::core

    %% Build relationships
    CMakeLists -->|configures & compiles| GGML_C
    CMakeLists --> GGML_ALLOC
    CMakeLists --> GGML_BACKEND_CPP
    CMakeLists --> GGML_REG
    CMakeLists --> GGML_OPT
    CMakeLists --> GGML_QUANTS
    CMakeLists --> GGUF
    CMakeLists --> CPU
    CMakeLists --> BLAS
    CMakeLists --> CUDA
    CMakeLists --> METAL
    CMakeLists --> VULKAN
    CMakeLists --> OPENCL
    CMakeLists --> SYCL
    CMakeLists --> CANN
    CMakeLists --> KOMPUTE
    CMakeLists --> LIB
    CMakeLists -->|uses| CMakeDir

    %% Runtime relationships
    Examples -->|link to| LIB
    PythonBindings -->|loads via CFFI| LIB
    Scripts -->|model conversion| LIB
    RPC -->|wraps calls| LIB
    Tests -->|unit test| LIB

    Examples -->|calls| PublicAPI
    PythonBindings -->|calls| PublicAPI
    RPC -->|calls| PublicAPI

    PublicAPI -->|dispatch to| GGML_C
    PublicAPI --> GGML_ALLOC
    PublicAPI --> GGUF

    GGML_C -->|backend registration| GGML_REG
    GGML_REG --> CPU
    GGML_REG --> BLAS
    GGML_REG --> CUDA
    GGML_REG --> METAL
    GGML_REG --> VULKAN
    GGML_REG --> OPENCL
    GGML_REG --> SYCL
    GGML_REG --> CANN
    GGML_REG --> KOMPUTE

    %% Styles
    classDef core fill:#ADD8E6,stroke:#000
    classDef backend fill:#90EE90,stroke:#000
    classDef api fill:#FFA500,stroke:#000
    classDef tool fill:#D3D3D3,stroke:#000

    %% Click Events
    click CMakeLists "https://github.com/ggml-org/ggml/blob/master/CMakeLists.txt"
    click CMakeDir "https://github.com/ggml-org/ggml/tree/master/cmake/"
    click GGML_C "https://github.com/ggml-org/ggml/blob/master/src/ggml.c"
    click GGML_ALLOC "https://github.com/ggml-org/ggml/blob/master/src/ggml-alloc.c"
    click GGML_BACKEND_CPP "https://github.com/ggml-org/ggml/blob/master/src/ggml-backend.cpp"
    click GGML_REG "https://github.com/ggml-org/ggml/blob/master/src/ggml-backend-reg.cpp"
    click GGML_OPT "https://github.com/ggml-org/ggml/blob/master/src/ggml-opt.cpp"
    click GGML_QUANTS "https://github.com/ggml-org/ggml/blob/master/src/ggml-quants.c"
    click GGUF "https://github.com/ggml-org/ggml/blob/master/src/gguf.cpp"
    click HDR_GGML "https://github.com/ggml-org/ggml/blob/master/include/ggml.h"
    click HDR_ALLOC "https://github.com/ggml-org/ggml/blob/master/include/ggml-alloc.h"
    click CPU "https://github.com/ggml-org/ggml/tree/master/src/ggml-cpu/"
    click BLAS "https://github.com/ggml-org/ggml/tree/master/src/ggml-blas/"
    click CUDA "https://github.com/ggml-org/ggml/tree/master/src/ggml-cuda/"
    click METAL "https://github.com/ggml-org/ggml/tree/master/src/ggml-metal/"
    click VULKAN "https://github.com/ggml-org/ggml/tree/master/src/ggml-vulkan/"
    click OPENCL "https://github.com/ggml-org/ggml/tree/master/src/ggml-opencl/"
    click SYCL "https://github.com/ggml-org/ggml/tree/master/src/ggml-sycl/"
    click CANN "https://github.com/ggml-org/ggml/tree/master/src/ggml-cann/"
    click KOMPUTE "https://github.com/ggml-org/ggml/tree/master/src/ggml-kompute/"
    click PublicAPI "https://github.com/ggml-org/ggml/tree/master/include/"
    click PythonBindings "https://github.com/ggml-org/ggml/tree/master/examples/python/ggml/"
    click Examples "https://github.com/ggml-org/ggml/tree/master/examples/"
    click Scripts "https://github.com/ggml-org/ggml/tree/master/scripts/"
    click Tests "https://github.com/ggml-org/ggml/tree/master/tests/"
    click RPC "https://github.com/ggml-org/ggml/tree/master/src/ggml-rpc/"
```
