How ggml-org/ggml works
Project type and purpose
• ggml is a low-level, cross-platform tensor library for machine learning (not a web or client-server app).
• It offers core tensor operations, automatic differentiation, optimizers, quantization, and multiple hardware backends.
File structure
• Top-level folders:
• src/ – core implementation and backend modules
• / – public headers (ggml.h, ggml-alloc.h, ggml-backend.h, etc.)
• examples/ – sample programs and conversion scripts
• tests/ – unit tests covering individual ops
• cmake/, CMakeLists.txt – build configuration
• scripts/ – utility scripts (model sync, authors generation)
• .github/ci.yml – continuous integration pipeline
• Within src/:
• ggml.c, ggml-*.cpp – core engine and registration logic
• subdirs for each backend:
– ggml-cpu (SIMD, quant, AMX, multi-threading)
– ggml-blas
– ggml-cuda
– ggml-metal
– ggml-opencl
– ggml-sycl
– ggml-vulkan
– ggml-hip, ggml-cann, ggml-musa, ggml-kompute…
• ggml-rpc – optional remote execution interface
• examples/python/ – Python CFFI bindings and utilities
Extract insights from the README
• Core features: hardware-agnostic tensor primitives, no external deps, zero runtime allocations, AD and optimizers.
• Build flags enable or disable each hardware backend via CMake (e.g., –DGGML_CUDA=ON).
a. Main components
• Build system (CMake)
• Core tensor engine (ggml core)
• Backend modules (CPU, CUDA, Metal, Vulkan, OpenCL, SYCL, HIP, CANN, MUSA, Kompute)
• Public API (include/*.h)
• Language bindings (Python via CFFI)
• Example applications and CLI tools (gpt-2, mnist, sam, yolo, etc.)
• Tests (unit tests in tests/)
• Scripts and RPC (model conversion, remote execution)
b. Relationships and data flow
• CMake reads CMakeLists.txt → configures & compiles core + selected backends → produces libggml (static or shared) and example binaries.
• Public API exposes functions to client code (examples, Python bindings).
• Backends register implementations with the core engine (ggml-backend-reg.cpp).
• At runtime, user code calls ggml API → core dispatches to active backend → performs tensor operations on CPU or chosen device.
• Python binding layer loads libggml via CFFI → calls into public API.
• Example CLIs load model files (GGUF format) → invoke ggml to run inference/optimizers.
• RPC module can wrap ggml calls over network.
c. Architectural patterns and principles
• Modular plugin-style backends: each hardware target lives in its own folder and is conditionally included.
• Single unified core API → backend abstraction layer via registration and dispatch.
• Zero dynamic allocations: memory management through allocators defined in ggml-alloc.c.
• Build-time configurability: CMake flags select code paths and link only needed backends.
d. Technologies and frameworks
• C/C++ code base
• CMake for build orchestration
• Python CFFI for bindings
• GPU frameworks: CUDA, Metal, Vulkan, OpenCL, SYCL, HIP, CANN, Kompute
• Continuous Integration: GitHub Actions (ci.yml)