How wertick01/deep-fold works
deep-fold is an experimental local LLM inference stack built around compressed NF4/VQ weights, with a Go CHR0 compressor and Python GPU runtime. The main workflow is: invoke the CLI, prepare or select a CHR0 model, load packed weights, choose resident or overflow execution, then generate tokens in chat or one-shot mode. The README also documents model pulling/import, hardware diagnostics, and reproducible evaluation. Some runtime links are documented rather than shown in sampled code; unsampled implementation details are not inferred.
Main entry points: main.py CLI commands · generate.py TokenLoop · model.py Model construction · chat.py Chat session
Components 21
User interface
- CLI commands
gpu/cli/main.py - Chat session
gpu/cli/chat.py - Model pull/import
gpu/cli/pull.py - Environment checks
gpu/cli/doctor.py
Compression and format
- CHR compressor
cmd/chr/run.go - CHR0 container
internal/chr0/write.go - NF4 encoding
internal/nf4/nf4.go - Source weights
internal/safetensors/index.go
Model loading
- CHR0 loader
gpu/chr0/loader.py - Model construction
gpu/host/model.py - Packed embeddings
gpu/host/embedding.py - Weight placement
gpu/host/residency.py
Inference runtime
- NF4 kernels
gpu/nf4/__init__.py - TokenLoop
gpu/loop/generate.py - CUDA graph execution
gpu/loop/graph.py - CopyRing streaming
gpu/loop/ring.py - Decode V2
gpu/decodev2/session.py
Evaluation
- Benchmark sessions
gpu/lab/sessions.py - Evaluation runner
gpu/lab/eval.py
Other components
- Local user
- Generated tokens
Connections 23
- Local user → CLI commands: invokes
gpu/cli/main.py - CLI commands → Chat session: routes chat
gpu/cli/main.py - CLI commands → Model pull/import: routes pull
gpu/cli/main.py - CLI commands → Environment checks: routes checks
gpu/cli/main.py - CLI commands → CHR compressor: requests packing
README.md - Source weights → CHR compressor: supplies weights
- CHR compressor → NF4 encoding: encodes NF4
- CHR compressor → CHR0 container: writes CHR0
README.md - CHR0 container → CHR0 loader: loads container
README.md - CHR0 loader → Model construction: provides packed weights
- Model construction → Packed embeddings: loads embedding rows
gpu/host/model.py - Model construction → Weight placement: selects residency
gpu/host/model.py - Model construction → NF4 kernels: attaches packed weights
README.md - Chat session → TokenLoop: generates with
README.md - Chat session → Decode V2: selects resident path
gpu/decodev2/session.py - TokenLoop → CUDA graph execution: executes GEMMs
gpu/loop/generate.py - TokenLoop → CopyRing streaming: streams overflow weights
gpu/loop/generate.py - TokenLoop → NF4 kernels: runs packed GEMMs
gpu/loop/generate.py - Decode V2 → NF4 kernels: runs packed kernels
README.md - TokenLoop → Generated tokens: returns tokens
README.md - Decode V2 → Generated tokens: returns tokens
README.md - Benchmark sessions → Evaluation runner: records sessions
gpu/lab/sessions.py - Evaluation runner → Chat session: uses inference workflow