# vahe1994/aqlm architecture

> Architecture diagram of the GitHub repository vahe1994/aqlm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vahe1994/aqlm
- Repository: https://github.com/vahe1994/aqlm

## Overview

##### Project and its purpose
- AQLM is an open‐source command-line toolkit and library for quantizing and fine-tuning large language models (LLaMA, Mistral, Mixtral) via additive quantization (AQLM) and PV-Tuning.
- It provides:
- CLI entrypoints (main.py, finetune.py, lmeval.py, convert_to_hf.py, etc.)
- A core Python library (`src/`) implementing quantization, PV-Tuning, beam search, optimizers, data utilities.
- An inference library package (`inference_lib/`) with C++/CUDA, Triton, and Numba kernels.
- Benchmarking scripts (`benchmark/`) and Jupyter notebooks for demos.
- Integrations with Hugging Face Hub, Datasets, WandB, and Transformers.

##### Relationships and Data Flow
- User invokes a CLI script or notebook → script parses arguments → calls core library (`src/aq.py`, `src/finetune.py`, etc.)
- Core library downloads/unpacks model & dataset from HF Hub → performs quantization or PV-Tuning → writes quantized weights to disk
- For inference: core library or notebooks call `inference_lib` → `kernel_selector` picks fastest kernel → loads quantized weights → runs forward pass on GPU/CPU → returns generated tokens
- For benchmarks and evaluations: scripts call either core library or LM Evaluation Harness → aggregate metrics → optionally log to WandB
- Conversion script (`convert_to_hf.py`) reads quantized checkpoint → wraps it into a HF‐compatible `transformers` model → saves to output folder

##### Architectural patterns & design principles
- Modular layering: CLI → Orchestrator → Core quantization library → Inference kernels
- Plugin-style kernel selection: easily add new Triton/CUDA/Numba kernels under `inference_kernels/`
- Separation of concerns: quantization vs. fine-tuning vs. inference vs. evaluation
- Extensibility: can swap in new model families, kernel implementations, or evaluation tasks

##### Essential elements
1. CLI → Core
2. Core ↔ HF Hub (download/upload)
3. Core → Local Disk (save/load quantized model, tokenized data)
4. Core → Inference Library
5. Inference Library → Kernel Selector → [CUDA Kernel, Triton Kernel, Numba Kernel] → GPU/CPU
6. Scripts → WandB (optional)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interfaces (Yellow)
    subgraph "CLI Entrypoints"
        style CLIEntrypoints fill:#FFF2CC,stroke:#E69138,stroke-width:2px
        mainpy["main.py"] 
        finetunepy["finetune.py"]
        lmevalpy["lmeval.py"]
        converttohf["convert_to_hf.py"]
        convertlegacy["convert_legacy_model_format.py"]
    end
    subgraph "Jupyter Notebooks"
        style Notebooks fill:#FFF2CC,stroke:#E69138,stroke-width:2px
        nb1["aq_simple.ipynb"]
        nb2["aqlm_2bit_training.ipynb"]
        nb3["aqlm_cuda_graph.ipynb"]
        nb4["aqlm_vllm.ipynb"]
        nb5["colab_example.ipynb"]
        nb6["streaming_example.ipynb"]
    end

    %% Core Quantization Library (Blue)
    subgraph "Core Quantization Library"
        style CoreLib fill:#D9E1F2,stroke:#2E75B6,stroke-width:2px
        aqengine["aq_engine.py"]
        aqpy["src/aq.py"]
        finetunecore["src/finetune.py"]
        pvopt["src/pv_optimizer.py"]
        beaml2["src/beam_search_l2.py"]
        beamxtx["src/beam_search_xtx.py"]
        kmeans["src/kmeans.py"]
        adam["src/configurable_adam.py"]
        datautils["src/datautils.py"]
        modelutils["src/modelutils.py"]
        pvutils["src/pv_utils.py"]
        memloss["src/memory_efficient_loss.py"]
        utils["src/utils.py"]
    end

    %% Benchmark & Evaluation (Yellow)
    subgraph "Benchmark & Evaluation"
        style Benchmark fill:#FFF2CC,stroke:#E69138,stroke-width:2px
        bench1["benchmark/benchmark_generate_cpu.py"]
        bench2["benchmark/generate_benchmark.py"]
        bench3["benchmark/matmul_benchmark.py"]
        bench4["benchmark/matmul_benchmark_cpu.py"]
    end

    %% Inference Library & Kernel Selector (Green)
    subgraph "Inference Library & Kernel Selector"
        style InferenceLib fill:#D9F2D9,stroke:#27AE60,stroke-width:2px
        infwrap["inference_lib/src/aqlm/inference.py"]
        selector["inference_lib/src/aqlm/inference_kernels/kernel_selector.py"]
        ilutils["inference_lib/src/aqlm/utils.py"]
        
        subgraph "Kernels"
            style Kernels fill:#D9F2D9,stroke:#229954,stroke-width:1px
            cuda_cpp["cuda_kernel.cpp"]
            cuda_cu["cuda_kernel.cu"]
            cuda_py["cuda_kernel.py"]
            triton["triton_kernel.py"]
            numba["numba_kernel.py"]
            dequant["dequantization.py"]
        end
    end

    %% External Integrations & Data (Orange)
    HF["HF Hub & Datasets"]:::external
    Cache["Local Cache / Disk"]:::external
    WandB["W&B Logging"]:::external

    %% Hardware & Runtime (Gray)
    GPU["GPU Compute"]:::hardware
    CPU["CPU Compute"]:::hardware

    %% Connections
    mainpy -->|calls| aqengine
    finetunepy -->|calls| aqengine
    lmevalpy -->|calls| aqengine
    converttohf -->|calls| aqengine
    convertlegacy -->|calls| aqengine

    nb1 -->|imports| aqengine
    nb2 -->|imports| aqengine
    nb3 -->|imports| aqengine
    nb4 -->|imports| aqengine
    nb5 -->|imports| aqengine
    nb6 -->|imports| aqengine

    aqengine -->|uses| datautils
    aqengine -->|uses| modelutils
    aqengine -->|uses| pvopt
    aqengine -->|uses| aqpy
    aqengine -->|uses| utils

    aqengine -->|HTTPS| HF
    HF -->|model/dataset| aqengine
    aqengine -->|save/load| Cache
    aqengine -->|invokes| infwrap

    bench1 -->|benchmarks| aqengine
    bench2 -->|benchmarks| aqengine
    bench3 -->|benchmarks| aqengine
    bench4 -->|benchmarks| aqengine
    bench1 -->|optional| WandB
    bench2 -->|optional| WandB

    infwrap -->|select| selector
    selector -->|CUDA| cuda_cpp
    selector -->|CUDA| cuda_cu
    selector -->|Python CUDA| cuda_py
    selector -->|Triton| triton
    selector -->|Numba| numba
    selector -->|Dequant| dequant

    cuda_cpp -->|runs on| GPU
    cuda_cu -->|runs on| GPU
    cuda_py -->|runs on| GPU
    triton -->|runs on| GPU
    numba -->|runs on| CPU
    dequant -->|runs on| CPU

    mainpy -->|optional logging| WandB
    finetunepy -->|optional logging| WandB
    lmevalpy -->|optional logging| WandB

    %% Click Events
    click mainpy "https://github.com/vahe1994/aqlm/blob/main/main.py"
    click finetunepy "https://github.com/vahe1994/aqlm/blob/main/finetune.py"
    click lmevalpy "https://github.com/vahe1994/aqlm/blob/main/lmeval.py"
    click converttohf "https://github.com/vahe1994/aqlm/blob/main/convert_to_hf.py"
    click convertlegacy "https://github.com/vahe1994/aqlm/blob/main/convert_legacy_model_format.py"

    click nb1 "https://github.com/vahe1994/aqlm/blob/main/notebooks/aq_simple.ipynb"
    click nb2 "https://github.com/vahe1994/aqlm/blob/main/notebooks/aqlm_2bit_training.ipynb"
    click nb3 "https://github.com/vahe1994/aqlm/blob/main/notebooks/aqlm_cuda_graph.ipynb"
    click nb4 "https://github.com/vahe1994/aqlm/blob/main/notebooks/aqlm_vllm.ipynb"
    click nb5 "https://github.com/vahe1994/aqlm/blob/main/notebooks/colab_example.ipynb"
    click nb6 "https://github.com/vahe1994/aqlm/blob/main/notebooks/streaming_example.ipynb"

    click aqengine "https://github.com/vahe1994/aqlm/blob/main/aq_engine.py"
    click aqpy "https://github.com/vahe1994/aqlm/blob/main/src/aq.py"
    click finetunecore "https://github.com/vahe1994/aqlm/blob/main/src/finetune.py"
    click pvopt "https://github.com/vahe1994/aqlm/blob/main/src/pv_optimizer.py"
    click beaml2 "https://github.com/vahe1994/aqlm/blob/main/src/beam_search_l2.py"
    click beamxtx "https://github.com/vahe1994/aqlm/blob/main/src/beam_search_xtx.py"
    click kmeans "https://github.com/vahe1994/aqlm/blob/main/src/kmeans.py"
    click adam "https://github.com/vahe1994/aqlm/blob/main/src/configurable_adam.py"
    click datautils "https://github.com/vahe1994/aqlm/blob/main/src/datautils.py"
    click modelutils "https://github.com/vahe1994/aqlm/blob/main/src/modelutils.py"
    click pvutils "https://github.com/vahe1994/aqlm/blob/main/src/pv_utils.py"
    click memloss "https://github.com/vahe1994/aqlm/blob/main/src/memory_efficient_loss.py"
    click utils "https://github.com/vahe1994/aqlm/blob/main/src/utils.py"

    click bench1 "https://github.com/vahe1994/aqlm/blob/main/benchmark/benchmark_generate_cpu.py"
    click bench2 "https://github.com/vahe1994/aqlm/blob/main/benchmark/generate_benchmark.py"
    click bench3 "https://github.com/vahe1994/aqlm/blob/main/benchmark/matmul_benchmark.py"
    click bench4 "https://github.com/vahe1994/aqlm/blob/main/benchmark/matmul_benchmark_cpu.py"

    click infwrap "https://github.com/vahe1994/aqlm/blob/main/inference_lib/src/aqlm/inference.py"
    click selector "https://github.com/vahe1994/aqlm/blob/main/inference_lib/src/aqlm/inference_kernels/kernel_selector.py"
    click ilutils "https://github.com/vahe1994/aqlm/blob/main/inference_lib/src/aqlm/utils.py"
    click cuda_cpp "https://github.com/vahe1994/aqlm/blob/main/inference_lib/src/aqlm/inference_kernels/cuda_kernel.cpp"
    click cuda_cu "https://github.com/vahe1994/aqlm/blob/main/inference_lib/src/aqlm/inference_kernels/cuda_kernel.cu"
    click cuda_py "https://github.com/vahe1994/aqlm/blob/main/inference_lib/src/aqlm/inference_kernels/cuda_kernel.py"
    click triton "https://github.com/vahe1994/aqlm/blob/main/inference_lib/src/aqlm/inference_kernels/triton_kernel.py"
    click numba "https://github.com/vahe1994/aqlm/blob/main/inference_lib/src/aqlm/inference_kernels/numba_kernel.py"
    click dequant "https://github.com/vahe1994/aqlm/blob/main/inference_lib/src/aqlm/inference_kernels/dequantization.py"

    %% Styles
    classDef external fill:#FDE9D9,stroke:#C00000,stroke-width:1px
    classDef hardware fill:#D9D9D9,stroke:#7F7F7F,stroke-width:1px
```
