How zeux/calm works
Main components
• Core Inference Engine: This is implemented in the src folder. It includes components for:
• Tokenization (src/tokenizer.c / tokenizer.h)
• Inference routines (src/infer.c, infer.cu, infer.m, infer.metal) showing support for different GPU backends (CUDA for NVIDIA and Metal for Apple)
• Model handling, tensor manipulation, and sampling (src/model.h, tensors.c/.h, sampler.c/.h, run.c)
• Helper functions (src/helpers.cuh)
• Model Conversion and Data Preparation: Found in the tools folder:
• Python scripts like convert.py, download.py for converting models into the project's expected safetensors format and downloading models.
• Notebooks (e.g., act.ipynb, sol.ipynb) and utility scripts (chart.py, cudaprof.*) for benchmarking and profiling.
• Build and Configuration: The root directory includes Makefile (for building the C/CUDA code), config files for clang-format, git settings, CI workflows (in .github/workflows), and the .gitignore.
• External Dependencies: Although the engine itself is dependency minimal, the conversion and profiling tools rely on Python libraries (listed in tools/requirements.txt) and safetensors/torch packages.
Relationships and interactions
• During development:
• Python tools (conversion, model download, profiling, and optional Jupyter notebooks) prepare and analyze model files.
• The Makefile compiles the C/CUDA source code which contains the core inference engine.
• At runtime:
• The inference binary (compiled into a build/run executable) loads model files (with safetensors metadata), processes input prompts (via tokenization), passes data through the inference engine modules, and applies sampling functions.
• Hardware acceleration:
• Data Flow:
• Input from the user (or from downloaded models) flows into the conversion tools.
• The converted model is loaded into the engine.
• Tokenized input is fed into the inference components, which then produce outputs (tokens) that are sampled and returned.
Architectural patterns and design principles
• Modular Design: Separate modules for tokenization, inference, sampling, and tensor computations.
• Multi-Backend Support: Distinct source files handling CUDA (for NVIDIA GPU acceleration) and Metal (for Apple hardware) emphasizing conditional code paths.
• Minimal Dependencies: The core inference engine is self-contained in C/C/CUDA while Python tools are kept separate.
Technology/framework details
• C/C++ and CUDA for high-performance GPU execution.
• Apple Metal implementation for macOS acceleration.
• Python with safetensors and torch libraries for conversion and model management.
• Makefile for build automation and CI workflows defined in .github for integration/continuous testing.
Specific elements
• Clear labels for each main component:
– Inference Engine (with sub-components: Tokenizer, Inference Module, Sampler, Tensor Operations)
– Model Conversion Tools (convert.py, download.py, notebooks)
– Build and CI/CD Pipeline (Makefile, GitHub workflows)
– From model download/conversion to model loading in the engine.
– From input prompt (CLI or API) through tokenization, inference execution, and back to output token generation.
– Indicate separate paths or branches for CUDA (NVIDIA) and Metal (Apple) acceleration.
– Core engine components (e.g., rectangular blocks)
– Utility and tooling scripts (e.g., rounded rectangles)
• Notes or annotations capturing performance aspects (e.g., handling fp16, fp8, gf4 weight formats and KV cache management).