How zjhellofss/kuiperllama works
Project Type and Purpose
• This is a C++20-based, cross-platform “large model inference framework” (KuiperLLama) with dual backends (CPU & CUDA) for running LLama2/3 and Qwen series models. It includes:
• A core C++ library (kuiper/) implementing tensor abstractions, operator kernels (CPU/CUDA), model loaders, and sampling strategies.
• Python-based “export” tools (tools/) to convert HuggingFace checkpoints into custom binary.
• Demo applications (demo/) and reference inference scripts (hf_infer/) showing how to load models, run inference, and compare with HuggingFace outputs.
• A comprehensive test suite (test/) with unit tests and CUDA wrappers.
• A CMake-driven build system with optional CPM integration for fetching dependencies (glog, gtest, SentencePiece, Armadillo/OpenBLAS, CUDA).
Architectural Overview
• At the highest level, the system comprises three domains:
A. Model Preparation (Python Tools)
B. Core Inference Engine (C++ Library)
C. Clients & Tests (Demo Binaries, Python Inference, Unit Tests)
Main Components & Their Interactions
1. Tools Layer (tools/)
• Python exporters (export_llama.py, export_qwen2.py, etc.)
• Input: HuggingFace model files
• Output: Flat binary weight files + tokenizer metadata
• Relationship: Feeds into the C++ loader at runtime.
2. Build System (CMake + CPM)
• Root CMakeLists.txt + custom modules (cmake/*.cmake)
• Configures CPU/CUDA compilation, third-party libs, feature flags (LLAMA3_SUPPORT, QWEN2_SUPPORT, QWEN3_SUPPORT).
3. Core Library (kuiper/)
a. Base Infrastructure (base/)
• Memory allocators (CPU/CUDA), logging, Unicode/tokenizer support.
b. Tensor Abstraction (tensor/)
• Multi-dimensional arrays, buffer management.
c. Operators (op/)
• High-level op interfaces (add, matmul, mha, RMSNorm, etc.)
• Kernel implementations:
• CPU (op/kernels/cpu/*_kernel.cpp/h)
• CUDA (op/kernels/cuda/*_kernel.cu/.cuh)
d. Model Loading (model/)
• Parsers for LLama2/3 & Qwen2/3 formats, raw weight readers.
e. Sampler (sampler/)
• Token selection strategies (argmax, top-k, temperature, etc.)
• Exposes a unified inference API used by demo binaries.
4. Demo & Inference Clients
• C++ demos (demo/main_qwen3.cpp, etc.) link against the core library to run inference given a .bin model and tokenizer.
• Python reference scripts (hf_infer/) load the same model via HuggingFace for result comparison.
5. Testing (test/)
• Unit tests for allocators, tensors, ops, model loading (CPU & CUDA).
• Ensures operator correctness and performance benchmarks.