# mozilla-ocho/llamafile architecture

> Architecture diagram of the GitHub repository mozilla-ocho/llamafile, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mozilla-ocho/llamafile
- Repository: https://github.com/mozilla-ocho/llamafile

## Overview

##### Main Components
- Executive Loader & Bootstrapper:
○ Represent the shell script wrapper that performs early runtime tasks (e.g., MZ-based dual image dispatch, extraction of the APE loader on UNIX, and supporting multi-OS bootstrapping).
- Core LLM Engine:
○ Represent the llama.cpp codebase. This should include submodules for core functionality: inference, text processing (tokenization, sampling, vocabulary management), and dispatching to different implementations (SSSE3, AVX, AVX2, etc.) based on CPU capabilities.
- Weights Embedding and ZIP Alignment Module:
○ Show the module or logic (in llamafile/zipalign.c) responsible for embedding model weights (through the ZIP embedding strategy) and mapping them into memory with mmap().
- GPU Support / Dynamic Module Compiler:
○ Include components for GPU support: on-demand compilation (using platform compilers such as nvcc or Xcode tools), the CUDA (ggml-cuda.cu) and Metal (ggml-metal.*) modules, and runtime linking via dlopen().
- Server and API Interface:
- Command-Line Interface (CLI):
○ Indicate the CLI tools (the “main” modules in llama.cpp/main and the various server Binaries) that support interactive mode, guided prompts, and additional command-line options.
- Build and Configuration Subsystem:
○ A build layer including Makefiles, build scripts (in build/), and configuration files that assemble all components together using Cosmopolitan’s toolchain. Also, list external dependencies managed in “third_party” (e.g., double-conversion, mbedtls, sqlite, stb).

##### Relationships and Data Flow
- That at startup, the executable’s bootstrapper loads the embedded LLM engine.
- Loader retrieving the embedded (or external) weights via mmap and handing the pointer to the inference engine.
- For GPU offloading, show a separate branch where the inference engine detects hardware capabilities and (if available) builds/loads GPU modules—this branch communicates with the core inference computation.
- The build system integrates external libraries (from “third_party”) and platform-specific toolchains (Cosmopolitan Libc) and passes the compiled artifacts to the loader.

##### Architectural Patterns and Design Principles
- Modularity and runtime dispatch:
- Cross-platform portability:
○ Illustrate how a single source is compiled twice (for AMD64 and ARM64) and then wrapped in a unifying launch shell script.
- Stress security and sandboxing:
○ Show that the executable applies sandboxing (pledge and SECCOMP) to restrict network/file system access after startup.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Bootstrapping Layer
    subgraph "Bootstrapping Layer"
        BL["Bootstrap Loader"]:::runtime
    end

    %% Inference Engine Layer
    subgraph "Inference Engine Layer"
        WM["Weights/ZIP Alignment"]:::runtime
        IE["LLM Inference Engine"]:::runtime
        WM -->|"maps weights"| IE
        %% GPU Support Subgraph
        subgraph "GPU Support"
            CU["CUDA Module Compiler"]:::gpu
            MT["Metal Module Compiler"]:::gpu
        end
        IE -->|"GPU offload"| CU
        IE -->|"GPU offload"| MT
    end

    %% Interface Layer
    subgraph "Interface Layer"
        subgraph "CLI Interfaces"
            CLI1["CLI (llama.cpp)"]:::interface
            CLI2["CLI (llamafile)"]:::interface
        end
        subgraph "Server & API Interfaces"
            SRV1["HTTP Server/JSON API (llama.cpp)"]:::interface
            SRV2["HTTP Server/JSON API (llamafile)"]:::interface
        end
    end

    %% Build System Layer
    subgraph "Build System"
        BS1["Build Scripts (build)"]:::build
        BS2["Root Makefile"]:::build
        BS3["External Dependencies (third_party)"]:::build
    end

    %% Relationships from Build System to Bootstrapping
    BS1 --> BL
    BS2 --> BL
    BS3 --> BL

    %% Bootstrapping loads the Inference Engine via weights mapping
    BL -->|"loads model weights"| WM

    %% User Interfaces send inputs to the Inference Engine and receive responses
    CLI1 -->|"user input"| IE
    CLI2 -->|"user input"| IE
    IE -->|"result"| CLI1
    IE -->|"result"| CLI2

    %% Bidirectional communication between Server and Inference Engine
    SRV1 <-->|"API calls"| IE
    SRV2 <-->|"API calls"| IE

    %% Styles
    classDef runtime fill:#AED6F1,stroke:#2E86C1,stroke-width:2px;
    classDef gpu fill:#FAD7A0,stroke:#E67E22,stroke-width:2px;
    classDef interface fill:#ABEBC6,stroke:#229954,stroke-width:2px;
    classDef build fill:#D2B4DE,stroke:#8E44AD,stroke-width:2px;

    %% Click Events
    click BL "https://github.com/mozilla-ocho/llamafile/blob/main/llamafile/llamafile.c"
    click IE "https://github.com/mozilla-ocho/llamafile/blob/main/llama.cpp/llama.cpp"
    click WM "https://github.com/mozilla-ocho/llamafile/blob/main/llamafile/zipalign.c"
    click CU "https://github.com/mozilla-ocho/llamafile/blob/main/llama.cpp/ggml-cuda.cu"
    click MT "https://github.com/mozilla-ocho/llamafile/blob/main/llama.cpp/ggml-metal.m"
    click CLI1 "https://github.com/mozilla-ocho/llamafile/blob/main/llama.cpp/main/main.cpp"
    click CLI2 "https://github.com/mozilla-ocho/llamafile/blob/main/llamafile/chatbot_main.cpp"
    click SRV1 "https://github.com/mozilla-ocho/llamafile/tree/main/llama.cpp/server"
    click SRV2 "https://github.com/mozilla-ocho/llamafile/tree/main/llamafile/server"
    click BS1 "https://github.com/mozilla-ocho/llamafile/tree/main/build"
    click BS2 "https://github.com/mozilla-ocho/llamafile/tree/main/Makefile"
    click BS3 "https://github.com/mozilla-ocho/llamafile/tree/main/third_party"
```
