# zeux/calm architecture

> Architecture diagram of the GitHub repository zeux/calm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zeux/calm
- Repository: https://github.com/zeux/calm

## Overview

##### Main components
- Core Inference Engine: This is implemented in the src folder. It includes components for:
- Tokenization (src/tokenizer.c / tokenizer.h)
- Inference routines (src/infer.c, infer.cu, infer.m, infer.metal) showing support for different GPU backends (CUDA for NVIDIA and Metal for Apple)
- Model handling, tensor manipulation, and sampling (src/model.h, tensors.c/.h, sampler.c/.h, run.c)
- Helper functions (src/helpers.cuh)
- Model Conversion and Data Preparation: Found in the tools folder:
- Python scripts like convert.py, download.py for converting models into the project's expected safetensors format and downloading models.
- Notebooks (e.g., act.ipynb, sol.ipynb) and utility scripts (chart.py, cudaprof.*) for benchmarking and profiling.
- Build and Configuration: The root directory includes Makefile (for building the C/CUDA code), config files for clang-format, git settings, CI workflows (in .github/workflows), and the .gitignore.
- External Dependencies: Although the engine itself is dependency minimal, the conversion and profiling tools rely on Python libraries (listed in tools/requirements.txt) and safetensors/torch packages.

##### Relationships and interactions
- During development:
- Python tools (conversion, model download, profiling, and optional Jupyter notebooks) prepare and analyze model files.
- The Makefile compiles the C/CUDA source code which contains the core inference engine.
- At runtime:
- The inference binary (compiled into a build/run executable) loads model files (with safetensors metadata), processes input prompts (via tokenization), passes data through the inference engine modules, and applies sampling functions.
- Hardware acceleration:
- Data Flow:
- Input from the user (or from downloaded models) flows into the conversion tools.
- The converted model is loaded into the engine.
- Tokenized input is fed into the inference components, which then produce outputs (tokens) that are sampled and returned.

##### Architectural patterns and design principles
- Modular Design: Separate modules for tokenization, inference, sampling, and tensor computations.
- Multi-Backend Support: Distinct source files handling CUDA (for NVIDIA GPU acceleration) and Metal (for Apple hardware) emphasizing conditional code paths.
- Minimal Dependencies: The core inference engine is self-contained in C/C/CUDA while Python tools are kept separate.

##### Technology/framework details
- C/C++ and CUDA for high-performance GPU execution.
- Apple Metal implementation for macOS acceleration.
- Python with safetensors and torch libraries for conversion and model management.
- Makefile for build automation and CI workflows defined in .github for integration/continuous testing.

##### Specific elements
- Clear labels for each main component:
– Inference Engine (with sub-components: Tokenizer, Inference Module, Sampler, Tensor Operations)
– Model Conversion Tools (convert.py, download.py, notebooks)
– Build and CI/CD Pipeline (Makefile, GitHub workflows)
– From model download/conversion to model loading in the engine.
– From input prompt (CLI or API) through tokenization, inference execution, and back to output token generation.
– Indicate separate paths or branches for CUDA (NVIDIA) and Metal (Apple) acceleration.
– Core engine components (e.g., rectangular blocks)
– Utility and tooling scripts (e.g., rounded rectangles)
- Notes or annotations capturing performance aspects (e.g., handling fp16, fp8, gf4 weight formats and KV cache management).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Build and CI/CD Pipeline
    subgraph "Build and CI/CD Pipeline"
        Makefile["Makefile"]:::build
        CI["CI Workflows"]:::build
    end

    %% Model Conversion and Data Preparation Tools
    subgraph "Model Conversion Tools"
        Download["Download Script"]:::tooling
        Convert["Conversion Script"]:::tooling
        Notebooks["Jupyter Notebooks"]:::tooling
        Chart["Chart Utility"]:::tooling
        CUDAProf["CUDA Profiling Tools"]:::tooling
    end

    %% Core Inference Engine
    subgraph "Core Inference Engine"
        EngineCore["Engine Core"]:::core
        Tokenizer["Tokenizer"]:::core
        Inference["Inference Module"]:::core
        HardwareCUDA["CUDA Backend"]:::gpu
        HardwareMetal["Metal Backend"]:::gpu
        TensorOps["Tensor Operations & Model Handling"]:::core
        Sampler["Sampling Module"]:::core
        Helpers["Helper Functions"]:::core
    end

    %% External Entities
    Input["User Input (Prompt)"]:::external
    Output["Output Tokens"]:::external

    %% Data Flow Connections
    Input -->|"tokenize"| Tokenizer
    Tokenizer -->|"prepare"| EngineCore
    EngineCore --> Inference
    Inference -->|"CUDA"| HardwareCUDA
    Inference -->|"Metal"| HardwareMetal
    Inference --> TensorOps
    TensorOps --> Sampler
    Sampler --> Output

    %% Model Conversion Flow
    Download -->|"fetch model"| Convert
    Convert -->|"load model"| TensorOps
    Convert -->|"benchmark"| Notebooks
    Notebooks -->|"visualize"| Chart
    Notebooks -->|"profile"| CUDAProf

    %% Build Pipeline Connection
    CI --> Makefile
    Makefile --> EngineCore

    %% Styles
    classDef core fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef tooling fill:#A9DFBF,stroke:#196F3D,stroke-width:2px;
    classDef build fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef gpu fill:#F5B7B1,stroke:#922B21,stroke-width:2px;
    classDef external fill:#D7BDE2,stroke:#6C3483,stroke-width:2px;

    %% Click Events for Core Inference Engine
    click EngineCore "https://github.com/zeux/calm/tree/main/src"
    click Tokenizer "https://github.com/zeux/calm/blob/main/src/tokenizer.c"
    click Inference "https://github.com/zeux/calm/blob/main/src/infer.c"
    click HardwareCUDA "https://github.com/zeux/calm/blob/main/src/infer.cu"
    click HardwareMetal "https://github.com/zeux/calm/blob/main/src/infer.metal"
    click TensorOps "https://github.com/zeux/calm/blob/main/src/model.h"
    click Sampler "https://github.com/zeux/calm/blob/main/src/sampler.c"
    click Helpers "https://github.com/zeux/calm/blob/main/src/helpers.cuh"

    %% Click Events for Model Conversion Tools
    click Convert "https://github.com/zeux/calm/blob/main/tools/convert.py"
    click Download "https://github.com/zeux/calm/blob/main/tools/download.py"
    click Notebooks "https://github.com/zeux/calm/blob/main/tools/act.ipynb"
    click Chart "https://github.com/zeux/calm/blob/main/tools/chart.py"
    click CUDAProf "https://github.com/zeux/calm/blob/main/tools/cudaprof.cu"

    %% Click Events for Build and CI/CD Pipeline
    click Makefile "https://github.com/zeux/calm/tree/main/Makefile"
    click CI "https://github.com/zeux/calm/blob/main/.github/workflows/build.yml"
```
