# zjhellofss/kuiperllama architecture

> Architecture diagram of the GitHub repository zjhellofss/kuiperllama, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zjhellofss/kuiperllama
- Repository: https://github.com/zjhellofss/kuiperllama

## Overview

##### Project Type and Purpose
- This is a C++20-based, cross-platform “large model inference framework” (KuiperLLama) with dual backends (CPU & CUDA) for running LLama2/3 and Qwen series models. It includes:
- A core C++ library (`kuiper/`) implementing tensor abstractions, operator kernels (CPU/CUDA), model loaders, and sampling strategies.
- Python-based “export” tools (`tools/`) to convert HuggingFace checkpoints into custom binary.
- Demo applications (`demo/`) and reference inference scripts (`hf_infer/`) showing how to load models, run inference, and compare with HuggingFace outputs.
- A comprehensive test suite (`test/`) with unit tests and CUDA wrappers.
- A CMake-driven build system with optional CPM integration for fetching dependencies (glog, gtest, SentencePiece, Armadillo/OpenBLAS, CUDA).

##### Architectural Overview
- At the highest level, the system comprises three domains:
A. Model Preparation (Python Tools)
B. Core Inference Engine (C++ Library)
C. Clients & Tests (Demo Binaries, Python Inference, Unit Tests)

##### Main Components & Their Interactions
1. Tools Layer (`tools/`)
- Python exporters (`export_llama.py`, `export_qwen2.py`, etc.)
- Input: HuggingFace model files
- Output: Flat binary weight files + tokenizer metadata
- Relationship: Feeds into the C++ loader at runtime.
2. Build System (CMake + CPM)
- Root `CMakeLists.txt` + custom modules (`cmake/*.cmake`)
- Configures CPU/CUDA compilation, third-party libs, feature flags (LLAMA3_SUPPORT, QWEN2_SUPPORT, QWEN3_SUPPORT).
3. Core Library (`kuiper/`)
a. Base Infrastructure (`base/`)
- Memory allocators (CPU/CUDA), logging, Unicode/tokenizer support.
b. Tensor Abstraction (`tensor/`)
- Multi-dimensional arrays, buffer management.
c. Operators (`op/`)
- High-level op interfaces (add, matmul, mha, RMSNorm, etc.)
- Kernel implementations:
- CPU (`op/kernels/cpu/*_kernel.cpp/h`)
- CUDA (`op/kernels/cuda/*_kernel.cu/.cuh`)
d. Model Loading (`model/`)
- Parsers for LLama2/3 & Qwen2/3 formats, raw weight readers.
e. Sampler (`sampler/`)
- Token selection strategies (argmax, top-k, temperature, etc.)
- Exposes a unified inference API used by demo binaries.
4. Demo & Inference Clients
- C++ demos (`demo/main_qwen3.cpp`, etc.) link against the core library to run inference given a `.bin` model and tokenizer.
- Python reference scripts (`hf_infer/`) load the same model via HuggingFace for result comparison.
5. Testing (`test/`)
- Unit tests for allocators, tensors, ops, model loading (CPU & CUDA).
- Ensures operator correctness and performance benchmarks.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Python Tools
    subgraph "Python Tools" 
        Tools["Python Export Tools"]:::tools
    end

    %% Build System
    subgraph "Build System" 
        Build["CMake + CPM"]:::build
    end

    %% Core Inference Library
    subgraph "Core Inference Library"
        direction TB
        subgraph "Base Infrastructure" 
            Base["Base Layer"]:::core_base
        end
        subgraph "Tensor Module"
            Tensor["Tensor Abstraction"]:::core_tensor
        end
        subgraph "Operator Module"
            OpIface["Operator Interface"]:::core_op
            subgraph "CPU Backend"
                CPUK["CPU Kernels"]:::core_cpu
            end
            subgraph "CUDA Backend"
                CUDAK["CUDA Kernels"]:::core_cuda
            end
        end
        subgraph "Model Loader"
            Loader["Model Loader"]:::core_model
        end
        subgraph "Sampler Module"
            Sampler["Sampler"]:::core_sampler
        end
    end

    %% Demo & Reference
    subgraph "Clients & Demos"
        Demo["C++ Demo Apps"]:::client
        HF["Python HF Scripts"]:::client
    end

    %% Testing Suite
    subgraph "Testing Suite"
        Test["Unit & CUDA Tests"]:::test
    end

    %% Third-Party
    subgraph "Third-Party Libraries"
        SP["SentencePiece"]:::third
        AR["Armadillo/OpenBLAS"]:::third
        GL["glog/gtest"]:::third
        CU["CUDA"]:::third
    end

    %% Connections
    Tools -->|"exports model binaries"| Loader
    Build -->|"configures compile & flags"| Base
    Build --> Base
    Build --> Tensor
    Build --> CPUK
    Build --> CUDAK
    Build --> OpIface
    Base --> Tensor
    Tensor --> OpIface
    OpIface --> CPUK
    OpIface -->|-->|CUDAK
    Loader -->|"load weights"| Base
    Loader --> Tensor
    Loader -->|"parse model"| OpIface
    Loader --> Sampler
    Base --> SP
    Tensor --> AR
    CPUK --> GL
    CUDAK --- CU
    Base --> GL
    Demo --> Base
    Demo --> Loader
    Demo --> Sampler
    HF --> Loader
    Test --> Base
    Test --> Tensor
    Test --> OpIface
    Test --> Loader
    Test --> CUDAK

    %% Click Events
    click Tools "https://github.com/zjhellofss/kuiperllama/blob/main/tools/export_llama.py"
    click Build "https://github.com/zjhellofss/kuiperllama/blob/main/CMakeLists.txt"
    click Base "https://github.com/zjhellofss/kuiperllama/tree/main/kuiper/include/base/"
    click Tensor "https://github.com/zjhellofss/kuiperllama/blob/main/kuiper/include/tensor/tensor.h"
    click OpIface "https://github.com/zjhellofss/kuiperllama/blob/main/kuiper/source/op/kernels/kernels_interface.h"
    click CPUK "https://github.com/zjhellofss/kuiperllama/tree/main/kuiper/source/op/kernels/cpu/"
    click CUDAK "https://github.com/zjhellofss/kuiperllama/tree/main/kuiper/source/op/kernels/cuda/"
    click Loader "https://github.com/zjhellofss/kuiperllama/tree/main/kuiper/include/model/"
    click Sampler "https://github.com/zjhellofss/kuiperllama/tree/main/kuiper/include/sampler/"
    click Demo "https://github.com/zjhellofss/kuiperllama/blob/main/demo/main.cpp"
    click HF "https://github.com/zjhellofss/kuiperllama/blob/main/hf_infer/llama3_infer.py"
    click Test "https://github.com/zjhellofss/kuiperllama/blob/main/test/test_main.cpp"

    %% Styles
    classDef tools fill:#eeeeee,stroke:#333,stroke-width:1px
    classDef build fill:#dddddd,stroke:#333,stroke-width:1px
    classDef core_base fill:#f8f0e3,stroke:#333,stroke-width:1px
    classDef core_tensor fill:#f0e8f8,stroke:#333,stroke-width:1px
    classDef core_op fill:#e3f0f8,stroke:#333,stroke-width:1px
    classDef core_cpu fill:#d0f8d0,stroke:#333,stroke-width:1px
    classDef core_cuda fill:#f8d0d0,stroke:#333,stroke-width:1px
    classDef core_model fill:#f8f8d0,stroke:#333,stroke-width:1px
    classDef core_sampler fill:#e8f8e3,stroke:#333,stroke-width:1px
    classDef client fill:#d0e0f8,stroke:#333,stroke-width:1px
    classDef test fill:#d0f0e8,stroke:#333,stroke-width:1px
    classDef third fill:#f8f8b0,stroke:#333,stroke-width:1px
```
