# codewithdark-git/quantllm architecture

> Architecture diagram of the GitHub repository codewithdark-git/quantllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/codewithdark-git/quantllm
- Repository: https://github.com/codewithdark-git/quantllm

## Overview

1. **What this repo is**
   QuantLLM is a Python library for loading, optimizing, fine-tuning, quantizing, exporting, and publishing large language models. The architecture is centered on a single high-level model abstraction, `TurboModel`, with supporting subsystems for hardware/memory inspection, model analysis, export formats, Hugging Face Hub integration, and optional performance kernels.

2. **Primary entry points**
   The public API is exposed from `quantllm/__init__.py` and is framed around `turbo(...)` plus `TurboModel`. The README shows the intended flow: a user calls `turbo()` with a model ID and config, then uses the returned object for generation, finetuning, export, or push-to-hub operations.

3. **Core orchestration layer**
   `quantllm/core/` is the central orchestration boundary. The key modules here are `turbo_model.py` for the unified façade, `smart_config.py` for auto-configuration, `model_analyzer.py` for inspecting model/architecture characteristics, `hardware.py` for device capability detection, `memory.py` for VRAM/RAM handling, and `training.py` / `compilation.py` for tuning and acceleration decisions.

4. **Runtime decision flow**
   A typical request starts with model metadata and user config, then passes through smart configuration and model analysis to determine architecture compatibility, quantization strategy, attention optimization, and memory settings. This layer is responsible for translating a generic user request into an execution plan that can work across different GPU/CPU capabilities and model families.

5. **Quantization and GGUF export path**
   `quantllm/quant/llama_cpp.py` is the explicit GGUF/llama.cpp path. This subsystem is the bridge between QuantLLM’s model abstraction and downstream deployment targets like llama.cpp, Ollama, and LM Studio. The README and docs indicate it handles conversion/export concerns rather than being the general model runtime.

6. **Export subsystem**
   `quantllm/core/export.py` is the cross-format export boundary. The repository advertises multiple export targets: GGUF, ONNX, MLX, and SafeTensors. Architecturally, export sits at the edge of the core runtime: it consumes a loaded/quantized model representation and produces artifacts for external inference ecosystems.

7. **Hub integration**
   `quantllm/hub/` handles publishing and model-card generation for Hugging Face Hub. `hub_manager.py` manages push/pull interactions, while `model_card.py` generates the metadata and usage-facing documentation. This is a distinct boundary from model execution: it serializes model outputs and metadata into a distributable repository format.

8. **Performance extensions**
   `quantllm/core/flash_attention.py` and `quantllm/kernels/triton/quantized_linear.py` represent optional acceleration paths. These are not the primary API, but they materially affect execution by enabling faster attention and fused/quantized linear operations when the environment supports them. The design suggests fallback behavior when these accelerators are unavailable.

9. **Training/fine-tuning support**
   Fine-tuning is treated as part of the same model lifecycle rather than a separate tool. `quantllm/core/training.py` provides the training path, and the README shows LoRA-style parameters and dataset ingestion. This means the runtime supports both inference and adaptation through a shared model abstraction.

10. **Utility and observability layer**
   `quantllm/utils/` contains support code such as `memory_tracker.py` and `progress.py`. These are secondary but important for architecture understanding because they provide the user-facing feedback loop around memory usage and progress reporting during long-running quantization, export, or training tasks.

11. **Architecture compatibility and fallback behavior**
   The presence of `model_analyzer.py`, `smart_config.py`, and tests like `test_architecture_fallback.py` and `test_resolve_model_type.py` shows that model-family detection is a first-class concern. The library is designed to map many architectures into a smaller number of internal execution paths, with fallback handling for unknown or remote-code models.

12. **Package boundaries and optional dependencies**
   The project is a Python package (`pyproject.toml`, `setup.py`) with optional extras for GGUF, ONNX, MLX, Triton, and a “full” install. This implies a modular runtime where heavyweight capabilities are opt-in rather than always loaded, which is important when diagramming dependencies and deployment footprint.

13. **Documentation and examples as architecture signals**
   The `docs/` tree and `examples/` directory mirror the main user journey: quickstart, loading models, generation, finetuning, GGUF export, and Hub publishing. They are not runtime components, but they accurately reflect the intended control flow through the library’s subsystems.

14. **How to diagram the system**
   A good architecture diagram should place `TurboModel/turbo()` at the center, with arrows to: model analysis/smart config, hardware and memory detection, inference/training, export formats, Hugging Face Hub, and optional acceleration kernels. External boundaries to show are model repositories, local GPU/CPU resources, llama.cpp-compatible deploy targets, and Hugging Face Hub.

Main entry points: [`quantllm/core/turbo_model.py`](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/turbo_model.py) (TurboModel), [`quantllm/core/smart_config.py`](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/smart_config.py) (Smart config), [`quantllm/core/training.py`](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/training.py) (Training), [`quantllm/core/export.py`](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/export.py) (Export)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_public_api["Public API"]
  node_quantllm_init["quantllm<br/>package api<br/>[__init__.py]"]
end

subgraph group_core["Core Runtime"]
  node_turbo_model["TurboModel<br/>façade<br/>[turbo_model.py]"]
  node_smart_config["Smart config<br/>planner<br/>[smart_config.py]"]
  node_model_analyzer["Model analyzer<br/>inspector<br/>[model_analyzer.py]"]
  node_hardware["Hardware<br/>probe<br/>[hardware.py]"]
  node_memory["Memory<br/>allocator<br/>[memory.py]"]
  node_training["Training<br/>fine-tune path<br/>[training.py]"]
  node_compilation["Compilation<br/>acceleration planner<br/>[compilation.py]"]
end

subgraph group_deployment["Deployment"]
  node_export["Export<br/>artifact boundary<br/>[export.py]"]
  node_llama_cpp["GGUF path<br/>quantization bridge<br/>[llama_cpp.py]"]
  node_hub_manager["Hub manager<br/>publisher<br/>[hub_manager.py]"]
  node_model_card["Model card<br/>metadata generator<br/>[model_card.py]"]
end

subgraph group_extensions["Extensions"]
  node_flash_attention["Flash attention<br/>optional kernel<br/>[flash_attention.py]"]
  node_quantized_linear["Quantized linear<br/>triton kernel"]
end

subgraph group_support["Support"]
  node_memory_tracker["Memory tracker<br/>[memory_tracker.py]"]
  node_progress["Progress<br/>[progress.py]"]
end

node_model_repo[("Model repos<br/>external source")]
node_local_hw[("CPU/GPU<br/>external runtime")]
node_hf_hub[("Hugging Face<br/>external service")]
node_deploy_targets[("Deploy targets<br/>external runtimes")]

node_quantllm_init -->|"exports"| node_turbo_model
node_quantllm_init -->|"API"| node_smart_config
node_turbo_model -->|"plans"| node_smart_config
node_turbo_model -->|"inspects"| node_model_analyzer
node_turbo_model -->|"checks"| node_hardware
node_turbo_model -->|"sizes"| node_memory
node_turbo_model -->|"finetunes"| node_training
node_turbo_model -->|"optimizes"| node_compilation
node_turbo_model -->|"exports"| node_export
node_turbo_model -->|"publishes"| node_hub_manager
node_smart_config -->|"uses"| node_model_analyzer
node_smart_config -->|"uses"| node_hardware
node_smart_config -->|"uses"| node_memory
node_training -->|"observes"| node_memory_tracker
node_training -->|"reports"| node_progress
node_export -->|"routes"| node_llama_cpp
node_export -->|"describes"| node_model_card
node_hub_manager -->|"syncs"| node_hf_hub
node_hub_manager -->|"packages"| node_model_card
node_turbo_model -->|"optionally uses"| node_flash_attention
node_turbo_model -->|"optionally uses"| node_quantized_linear
node_model_analyzer -->|"loads"| node_model_repo
node_hardware -->|"probes"| node_local_hw
node_llama_cpp -->|"produces"| node_deploy_targets

click node_quantllm_init "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/__init__.py"
click node_turbo_model "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/turbo_model.py"
click node_smart_config "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/smart_config.py"
click node_model_analyzer "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/model_analyzer.py"
click node_hardware "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/hardware.py"
click node_memory "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/memory.py"
click node_training "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/training.py"
click node_compilation "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/compilation.py"
click node_export "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/export.py"
click node_llama_cpp "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/quant/llama_cpp.py"
click node_hub_manager "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/hub/hub_manager.py"
click node_model_card "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/hub/model_card.py"
click node_flash_attention "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/flash_attention.py"
click node_quantized_linear "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/kernels/triton/quantized_linear.py"
click node_memory_tracker "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/utils/memory_tracker.py"
click node_progress "https://github.com/codewithdark-git/quantllm/blob/main/quantllm/utils/progress.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_quantllm_init toneBlue
class node_turbo_model,node_smart_config,node_model_analyzer,node_hardware,node_memory,node_training,node_compilation toneAmber
class node_export,node_llama_cpp,node_hub_manager,node_model_card toneMint
class node_flash_attention,node_quantized_linear toneRose
class node_memory_tracker,node_progress toneIndigo
class node_model_repo,node_local_hw,node_hf_hub,node_deploy_targets toneNeutral
```

## Components

### Public API

User-facing entrypoints and package surface

- [quantllm](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/__init__.py): package api · Top-level package export for turbo() and TurboModel · `quantllm/__init__.py`

### Core Runtime

Orchestration, analysis, config, training, and export

- [TurboModel](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/turbo_model.py): façade · Unified model lifecycle abstraction · `quantllm/core/turbo_model.py`
- [Smart config](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/smart_config.py): planner · Derives execution settings from model and user config · `quantllm/core/smart_config.py`
- [Model analyzer](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/model_analyzer.py): inspector · Detects architecture and compatibility details · `quantllm/core/model_analyzer.py`
- [Hardware](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/hardware.py): probe · Detects device capabilities and environment limits · `quantllm/core/hardware.py`
- [Memory](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/memory.py): allocator · Tracks and reasons about RAM and VRAM usage · `quantllm/core/memory.py`
- [Training](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/training.py): fine-tune path · Fine-tuning and adaptation workflow · `quantllm/core/training.py`
- [Compilation](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/compilation.py): acceleration planner · Compilation and acceleration decisions · `quantllm/core/compilation.py`

### Deployment

Conversion and hub publishing boundaries

- [Export](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/export.py): artifact boundary · Cross-format model export interface · `quantllm/core/export.py`
- [GGUF path](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/quant/llama_cpp.py): quantization bridge · GGUF and llama.cpp-oriented conversion/export path · `quantllm/quant/llama_cpp.py`
- [Hub manager](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/hub/hub_manager.py): publisher · Handles Hugging Face Hub push and pull operations · `quantllm/hub/hub_manager.py`
- [Model card](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/hub/model_card.py): metadata generator · Creates hub-facing metadata and documentation · `quantllm/hub/model_card.py`

### Extensions

Optional acceleration kernels and quantization paths

- [Flash attention](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/core/flash_attention.py): optional kernel · Accelerated attention implementation when available · `quantllm/core/flash_attention.py`
- [Quantized linear](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/kernels/triton/quantized_linear.py): triton kernel · Fused quantized linear kernel for Triton backends · `quantllm/kernels/triton/quantized_linear.py`

### Support

Utility helpers and integration surfaces

- [Memory tracker](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/utils/memory_tracker.py): utility · Observability helper for memory usage · `quantllm/utils/memory_tracker.py`
- [Progress](https://github.com/codewithdark-git/quantllm/blob/main/quantllm/utils/progress.py): utility · Long-running task progress reporting · `quantllm/utils/progress.py`

### Other components

- **Model repos**: external source · Remote model sources and checkpoints
- **CPU/GPU**: external runtime · Local compute and memory environment
- **Hugging Face**: external service · Publishing and sharing destination
- **Deploy targets**: external runtimes · Downstream consumers such as llama.cpp-compatible runtimes

## Connections

- quantllm → TurboModel: exports · Package surface exposes the main façade
- quantllm → Smart config: API · Public entrypoints feed configuration into runtime
- TurboModel → Smart config: plans · Builds an execution plan from inputs
- TurboModel → Model analyzer: inspects · Determines model family and compatibility
- TurboModel → Hardware: checks · Adapts to available devices
- TurboModel → Memory: sizes · Chooses memory strategy and placement
- TurboModel → Training: finetunes · Runs model adaptation workflows
- TurboModel → Compilation: optimizes · Selects acceleration and compilation options
- TurboModel → Export: exports · Serializes models to external formats
- TurboModel → Hub manager: publishes · Pushes artifacts and metadata to hub
- Smart config → Model analyzer: uses · Combines model metadata with compatibility rules
- Smart config → Hardware: uses · Adapts config to the environment
- Smart config → Memory: uses · Computes safe allocation and load choices
- Training → Memory tracker: observes · Tracks resource usage during training
- Training → Progress: reports · Surfaces long-running job status
- Export → GGUF path: routes · GGUF export path for llama.cpp ecosystem
- Export → Model card: describes · Bundles export metadata for distribution
- Hub manager → Hugging Face: syncs · Push/pull boundary with Hugging Face Hub
- Hub manager → Model card: packages · Generates repository documentation and metadata
- TurboModel → Flash attention: optionally uses · Can accelerate attention when supported
- TurboModel → Quantized linear: optionally uses · Can use fused quantized linear kernels
- Model analyzer → Model repos: loads · Inspects remote or local model definitions
- Hardware → CPU/GPU: probes · Reads compute and memory capabilities
- GGUF path → Deploy targets: produces · Targets downstream GGUF-compatible runtimes