How pilancilab/caldera works
Project Type and Purpose
• CALDERA is a research‐oriented Python library and command‐line toolkit for post‐training compression of large language model (LLM) weights via a low‐rank, low‐precision decomposition.
• It is not a web app or microservice; rather, it is a modular Python package (src/caldera) with companion scripts, shell runners, and Jupyter notebooks to facilitate experiments: compute Hessians, quantize layers, evaluate, and fine‐tune.
File‐Structure Analysis
Top‐Level Components
• src/caldera/ – Core Python package implementing decomposition algorithm, quantized layer classes, and utilities.
– decomposition/alg.py – Core alternating‐minimization algorithm.
– decomposition/weight_compression.py, layer_quantization.py, quantized_layer.py – Classes orchestrating compression and runtime inference of quantized layers.
– utils/ – Enumerations and basic quantizers.
• scripts/ – Standalone Python entry points (CLI scripts) to run zero‐shot, perplexity, GLUE, Winogrande, Wikitext evaluations and fine‐tuning.
• shell_scripts/ – Bash wrappers that supply reasonable defaults to the scripts/ CLIs; coordinate runs, set output directories, point to configuration YAML/JSON.
• quip-sharp/ – Git submodule providing the QuIP# quantizer C++/CUDA library and Python bindings.
• notebooks/ – Jupyter notebooks for interactive testing and throughput evaluation.
• Config files: pyproject.toml, requirements.txt, accelerate_config.yaml, deepspeed_config.json.
a. Core Library (Python Package “caldera”)
• Modules:
• caldera.decomposition.alg
• caldera.decomposition.weight_compression
• caldera.decomposition.layer_quantization
• caldera.quantized_layer
• caldera.utils
• Datastores: Hessian files, quantization parameter JSON outputs, saved quantized model checkpoints.
b. CLI Layer
• Python scripts in scripts/
• Orchestrated by Bash wrappers in shell_scripts/
c. External Dependencies
• PyTorch (model loading, tensor ops)
• CUDA/C++ kernels via QuIP# (quiptools)
• fast-hadamard-transform (optional dependency for Hessian estimation)
• LLM model weights (e.g., LLaMA)
d. Data Flow & Artifacts
1. Compute Hessians
• run_save_hessians.sh → scripts/save_llama_hessians.py → outputs per-layer Hessian files.
2. Decompose & Quantize
• run_quantize_save_caldera.sh → scripts/quantize_save_llama.py
• Calls into caldera.decomposition.* and QuIP# APIs → emits quantized layer artifacts.
3. Evaluation
• run_eval_ppl.sh / run_eval_zeroshot.sh → scripts/eval_ppl.py / scripts/eval_zero_shot.py → reads quantized model → writes metrics.
4. Fine‐tuning
• run_finetune_*.sh → corresponding scripts/finetune_*.py → uses LoRA adaptation on low‐rank factors.
Architectural Patterns & Design Principles
• Modular Python package separating algorithm implementation from CLI interfaces and shell orchestration.
• Data‐driven pipeline: each stage reads/writes files, enabling reproducibility and partial reruns.
• Submodule pattern for external C++/CUDA library (QuIP#) co‐built and version‐controlled.
• Use of dataclasses (dataclasses.py) for clear parameter passing and JSON serialization.
Final Advice
• Every subcomponent in src/caldera and show its interaction with scripts.
• Pipeline nature: decomposer → quantizer → evaluator → finetuner.
• QuIP# build step (setup_quip_sharp.sh) to emphasize the C++/CUDA integration.