codewithdark-git/quantllm

1. What this repo is

QuantLLM is a Python library for loading, optimizing, fine-tuning, quantizing, exporting, and publishing large language models. The architecture is centered on a single high-level model abstraction, TurboModel, with supporting subsystems for hardware/memory inspection, model analysis, export formats, Hugging Face Hub integration, and optional performance kernels.

2. Primary entry points

The public API is exposed from quantllm/__init__.py and is framed around turbo(...) plus TurboModel. The README shows the intended flow: a user calls turbo() with a model ID and config, then uses the returned object for generation, finetuning, export, or push-to-hub operations.

3. Core orchestration layer

quantllm/core/ is the central orchestration boundary. The key modules here are turbo_model.py for the unified façade, smart_config.py for auto-configuration, model_analyzer.py for inspecting model/architecture characteristics, hardware.py for device capability detection, memory.py for VRAM/RAM handling, and training.py / compilation.py for tuning and acceleration decisions.

4. Runtime decision flow

A typical request starts with model metadata and user config, then passes through smart configuration and model analysis to determine architecture compatibility, quantization strategy, attention optimization, and memory settings. This layer is responsible for translating a generic user request into an execution plan that can work across different GPU/CPU capabilities and model families.

5. Quantization and GGUF export path

quantllm/quant/llama_cpp.py is the explicit GGUF/llama.cpp path. This subsystem is the bridge between QuantLLM’s model abstraction and downstream deployment targets like llama.cpp, Ollama, and LM Studio. The README and docs indicate it handles conversion/export concerns rather than being the general model runtime.

6. Export subsystem

quantllm/core/export.py is the cross-format export boundary. The repository advertises multiple export targets: GGUF, ONNX, MLX, and SafeTensors. Architecturally, export sits at the edge of the core runtime: it consumes a loaded/quantized model representation and produces artifacts for external inference ecosystems.

7. Hub integration

quantllm/hub/ handles publishing and model-card generation for Hugging Face Hub. hub_manager.py manages push/pull interactions, while model_card.py generates the metadata and usage-facing documentation. This is a distinct boundary from model execution: it serializes model outputs and metadata into a distributable repository format.

8. Performance extensions

quantllm/core/flash_attention.py and quantllm/kernels/triton/quantized_linear.py represent optional acceleration paths. These are not the primary API, but they materially affect execution by enabling faster attention and fused/quantized linear operations when the environment supports them. The design suggests fallback behavior when these accelerators are unavailable.

9. Training/fine-tuning support

Fine-tuning is treated as part of the same model lifecycle rather than a separate tool. quantllm/core/training.py provides the training path, and the README shows LoRA-style parameters and dataset ingestion. This means the runtime supports both inference and adaptation through a shared model abstraction.

10. Utility and observability layer

quantllm/utils/ contains support code such as memory_tracker.py and progress.py. These are secondary but important for architecture understanding because they provide the user-facing feedback loop around memory usage and progress reporting during long-running quantization, export, or training tasks.

11. Architecture compatibility and fallback behavior

The presence of model_analyzer.py, smart_config.py, and tests like test_architecture_fallback.py and test_resolve_model_type.py shows that model-family detection is a first-class concern. The library is designed to map many architectures into a smaller number of internal execution paths, with fallback handling for unknown or remote-code models.

12. Package boundaries and optional dependencies

The project is a Python package (pyproject.toml, setup.py) with optional extras for GGUF, ONNX, MLX, Triton, and a “full” install. This implies a modular runtime where heavyweight capabilities are opt-in rather than always loaded, which is important when diagramming dependencies and deployment footprint.

13. Documentation and examples as architecture signals

The docs/ tree and examples/ directory mirror the main user journey: quickstart, loading models, generation, finetuning, GGUF export, and Hub publishing. They are not runtime components, but they accurately reflect the intended control flow through the library’s subsystems.

14. How to diagram the system

A good architecture diagram should place TurboModel/turbo() at the center, with arrows to: model analysis/smart config, hardware and memory detection, inference/training, export formats, Hugging Face Hub, and optional acceleration kernels. External boundaries to show are model repositories, local GPU/CPU resources, llama.cpp-compatible deploy targets, and Hugging Face Hub.