# cornell-relaxml/yaqa architecture

> Architecture diagram of the GitHub repository cornell-relaxml/yaqa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cornell-relaxml/yaqa
- Repository: https://github.com/cornell-relaxml/yaqa

## Overview

##### Project Type and Purpose
- This is a research-grade Python library and CLI tool for quantizing large language models (LLaMA) using a Hessian-informed adaptive rounding algorithm (YAQA).
- It builds on the QTIP codebase, implements Hessian collection, quantization algorithms, GPU kernels, and evaluation scripts.

##### File Structure
Top-level directories indicate logical subsystems:
- eval/                ← evaluation scripts (KL, perplexity, zero-shot, interactive)
- hessian_llama/       ← modules to collect layerwise Hessians for LLaMA
- lib/                 ← core Python library
– lib/algo/          ← quantization and finetuning algorithms
– lib/linear/        ← custom linear layers, quantized operators
– lib/codebook/      ← codebook routines (bit-shifting)
– lib/utils/         ← data loaders, math utilities, graph wrappers
- model/               ← high-level LLaMA model wrapper and caching utilities
- qtip-kernels/        ← custom CUDA/C++ kernels and Python bindings
- quantize_llama/      ← end-to-end scripts for quantization workflows
- assets/, README.md, requirements.txt

##### README for Additional Context
- Purpose: reduce KL divergence via Kronecker-factored Hessian approximations.
- Usage flow: collect Hessians → run quantization → evaluate.
- External dependency: HuggingFace model hub for pre-trained models.

##### Core Components & Their Relationships
a. Entry-Point Layer (CLI Scripts)
– quantize_llama/*.py
– eval/*.py
– hessian_llama/README.md points to usage of hessian scripts
↳ Orchestrates workflows and passes arguments/config to lower layers.

b. Model Adapter Layer
– model/llama.py & cache_utils.py
– hessian_llama/get_hess_llama.py & llama_hess.py
↳ Loads pre-trained LLaMA models; manages forward/backward passes to extract Hessians.

c. Core Library Layer (lib/)
– lib/algo (YAQA, LDLQ, finetune modules)
– lib/linear & lib/codebook (quantized operations)
– lib/utils (data handling, math, decomposition, sharding)
↳ Implements the quantization algorithm, data pipelines, utility functions.

d. GPU Kernel Layer (qtip-kernels/)
– inference.cu, qtip_torch.cu, wrapper.cpp
↳ Provides high-performance CUDA kernels; exposed to Python via setup.py bindings.

e. Data & Caching Layer
– model/cache_utils.py
– lib/utils/data_utils.py, gptq_data_utils.py
↳ Manages on-disk caches (Hessians, quantized parameters) and dataset iteration.

f. External Services
– HuggingFace Model Hub (download/upload pre-trained and quantized models)
– Optional: logging/metrics backends if added.

##### Architectural Patterns & Design Principles
- Layered Architecture:
CLI → Model Adapter → Core Algorithm Library → GPU Kernels → Hardware
- Plugin-Style Kernels: CUDA modules are isolated in qtip-kernels and loaded dynamically.
- Modularization by Concern:
– Algorithms vs. linear operators vs. I/O utilities vs. evaluation
- Data Flow:
– Raw model → Hessian collector → YAQA algorithm → quantized weights → evaluation

## Diagram (Mermaid)

```mermaid
graph TD
    %% Entry-Point Layer
    subgraph "Entry-Point Layer"
        direction TB
        QuantizeE2E["finetune_e2e_llama.py"]:::cli
        HFize["hfize_llama.py"]:::cli
        QuantizeFT["quantize_finetune_llama.py"]:::cli
        EvalKL["eval_kl.py"]:::cli
        EvalPPL["eval_ppl.py"]:::cli
        EvalZS["eval_zeroshot.py"]:::cli
        Interactive["interactive_gen.py"]:::cli
    end

    %% Model Adapter Layer
    subgraph "Model Adapter Layer"
        direction TB
        LlamaWrapper["llama.py"]:::adapter
        CacheUtilsMA["cache_utils.py"]:::adapter
        GetHess["get_hess_llama.py"]:::adapter
        LlamaHess["llama_hess.py"]:::adapter
        CLA["custom_linear_A.py"]:::adapter
        CLB["custom_linear_B.py"]:::adapter
        DataUtilsHL["data_utils.py"]:::adapter
    end

    %% Core Library Layer
    subgraph "Core Library Layer"
        direction TB
        FinetuneAlgo["finetune.py"]:::core
        LDLQ["ldlq.py"]:::core
        QuantLin["quantized_linear.py"]:::core
        BitShift["bitshift.py"]:::core
        DataUtilsLib["data_utils.py"]:::data
        GPTQData["gptq_data_utils.py"]:::data
        MathUtils["math_utils.py"]:::data
        GraphWrap["graph_wrapper.py"]:::data
        ShardModel["shard_model.py"]:::data
        FinUtil["finetune.py"]:::data
        KernelCheck["kernel_check.py"]:::data
        KernelDecomp["kernel_decompress.py"]:::data
        MatmulHad["matmul_had.py"]:::data
        MiscPy["misc.py"]:::data
        UnsafeImp["unsafe_import.py"]:::data
    end

    %% GPU Kernel Layer
    subgraph "GPU Kernel Layer"
        direction TB
        InfCU["inference.cu"]:::cuda
        TorchCU["qtip_torch.cu"]:::cuda
        WrapperCPP["wrapper.cpp"]:::cuda
        InfH["inference.h"]:::cuda
        TestCU["test.cu"]:::cuda
        Makefile["Makefile"]:::cuda
        SetupPy["setup.py"]:::cuda
        ProfSh["prof.sh"]:::cuda
        TestDC["test_decompress_matvec.py"]:::cuda
    end

    %% External Services
    subgraph "External Services"
        direction TB
        HFHub["HuggingFace Model Hub"]:::external
    end

    %% Data & Caching Layer
    subgraph "Data & Caching Layer"
        direction TB
        CacheUtils["cache_utils.py"]:::data
        DataUtilsLib
        GPTQDataLib
    end

    %% Relationships
    QuantizeE2E -->|uses| LlamaWrapper
    HFize -->|uses| LlamaWrapper
    QuantizeFT -->|uses| LlamaWrapper
    EvalKL -->|loads| LlamaWrapper
    EvalPPL -->|loads| LlamaWrapper
    EvalZS -->|loads| LlamaWrapper
    Interactive -->|loads| LlamaWrapper

    LlamaWrapper -->|extracts Hessians| GetHess
    GetHess -->|passes Hessians| FinetuneAlgo
    LlamaHess -->|helper| FinetuneAlgo
    FinetuneAlgo -->|quantizes| InfCU
    LDLQ -->|optionally uses| InfCU
    FinetuneAlgo -->|saves weights| CacheUtils
    LlamaWrapper -->|caches models| CacheUtilsMA
    CacheUtilsMA -->|stores| CacheUtils

    LlamaWrapper -->|downloads/uploads| HFHub
    HFHub -->|provides models| LlamaWrapper

    %% Click Events
    click QuantizeE2E "https://github.com/cornell-relaxml/yaqa/blob/main/quantize_llama/finetune_e2e_llama.py"
    click HFize "https://github.com/cornell-relaxml/yaqa/blob/main/quantize_llama/hfize_llama.py"
    click QuantizeFT "https://github.com/cornell-relaxml/yaqa/blob/main/quantize_llama/quantize_finetune_llama.py"
    click EvalKL "https://github.com/cornell-relaxml/yaqa/blob/main/eval/eval_kl.py"
    click EvalPPL "https://github.com/cornell-relaxml/yaqa/blob/main/eval/eval_ppl.py"
    click EvalZS "https://github.com/cornell-relaxml/yaqa/blob/main/eval/eval_zeroshot.py"
    click Interactive "https://github.com/cornell-relaxml/yaqa/blob/main/eval/interactive_gen.py"
    click LlamaWrapper "https://github.com/cornell-relaxml/yaqa/blob/main/model/llama.py"
    click CacheUtilsMA "https://github.com/cornell-relaxml/yaqa/blob/main/model/cache_utils.py"
    click GetHess "https://github.com/cornell-relaxml/yaqa/blob/main/hessian_llama/get_hess_llama.py"
    click LlamaHess "https://github.com/cornell-relaxml/yaqa/blob/main/hessian_llama/llama_hess.py"
    click CLA "https://github.com/cornell-relaxml/yaqa/blob/main/hessian_llama/custom_linear_A.py"
    click CLB "https://github.com/cornell-relaxml/yaqa/blob/main/hessian_llama/custom_linear_B.py"
    click DataUtilsHL "https://github.com/cornell-relaxml/yaqa/blob/main/hessian_llama/data_utils.py"
    click FinetuneAlgo "https://github.com/cornell-relaxml/yaqa/blob/main/lib/algo/finetune.py"
    click LDLQ "https://github.com/cornell-relaxml/yaqa/blob/main/lib/algo/ldlq.py"
    click QuantLin "https://github.com/cornell-relaxml/yaqa/blob/main/lib/linear/quantized_linear.py"
    click BitShift "https://github.com/cornell-relaxml/yaqa/blob/main/lib/codebook/bitshift.py"
    click DataUtilsLib "https://github.com/cornell-relaxml/yaqa/blob/main/lib/utils/data_utils.py"
    click GPTQData "https://github.com/cornell-relaxml/yaqa/blob/main/lib/utils/gptq_data_utils.py"
    click MathUtils "https://github.com/cornell-relaxml/yaqa/blob/main/lib/utils/math_utils.py"
    click GraphWrap "https://github.com/cornell-relaxml/yaqa/blob/main/lib/utils/graph_wrapper.py"
    click ShardModel "https://github.com/cornell-relaxml/yaqa/blob/main/lib/utils/shard_model.py"
    click FinUtil "https://github.com/cornell-relaxml/yaqa/blob/main/lib/utils/finetune.py"
    click KernelCheck "https://github.com/cornell-relaxml/yaqa/blob/main/lib/utils/kernel_check.py"
    click KernelDecomp "https://github.com/cornell-relaxml/yaqa/blob/main/lib/utils/kernel_decompress.py"
    click MatmulHad "https://github.com/cornell-relaxml/yaqa/blob/main/lib/utils/matmul_had.py"
    click MiscPy "https://github.com/cornell-relaxml/yaqa/blob/main/lib/utils/misc.py"
    click UnsafeImp "https://github.com/cornell-relaxml/yaqa/blob/main/lib/utils/unsafe_import.py"
    click InfCU "https://github.com/cornell-relaxml/yaqa/blob/main/qtip-kernels/src/inference.cu"
    click TorchCU "https://github.com/cornell-relaxml/yaqa/blob/main/qtip-kernels/src/qtip_torch.cu"
    click WrapperCPP "https://github.com/cornell-relaxml/yaqa/blob/main/qtip-kernels/src/wrapper.cpp"
    click InfH "https://github.com/cornell-relaxml/yaqa/blob/main/qtip-kernels/src/inference.h"
    click TestCU "https://github.com/cornell-relaxml/yaqa/blob/main/qtip-kernels/src/test.cu"
    click Makefile "https://github.com/cornell-relaxml/yaqa/tree/main/qtip-kernels/src/Makefile"
    click SetupPy "https://github.com/cornell-relaxml/yaqa/blob/main/qtip-kernels/setup.py"
    click ProfSh "https://github.com/cornell-relaxml/yaqa/blob/main/qtip-kernels/prof.sh"
    click TestDC "https://github.com/cornell-relaxml/yaqa/blob/main/qtip-kernels/test_decompress_matvec.py"

    %% Styles
    classDef cli fill:#a2f5a2,stroke:#333,stroke-width:1px
    classDef adapter fill:#a2d2f5,stroke:#333,stroke-width:1px
    classDef core fill:#f5d6a2,stroke:#333,stroke-width:1px
    classDef data fill:#d9a2f5,stroke:#333,stroke-width:1px
    classDef cuda fill:#f5a2a2,stroke:#333,stroke-width:1px
    classDef external fill:#cccccc,stroke:#333,stroke-width:1px
```
