# wertick01/deep-fold architecture

> Architecture diagram of the GitHub repository wertick01/deep-fold, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wertick01/deep-fold
- Repository: https://github.com/wertick01/deep-fold

## Overview

deep-fold is an experimental local LLM inference stack built around compressed NF4/VQ weights, with a Go CHR0 compressor and Python GPU runtime. The main workflow is: invoke the CLI, prepare or select a CHR0 model, load packed weights, choose resident or overflow execution, then generate tokens in chat or one-shot mode. The README also documents model pulling/import, hardware diagnostics, and reproducible evaluation. Some runtime links are documented rather than shown in sampled code; unsampled implementation details are not inferred.

Main entry points: [`gpu/cli/main.py`](https://github.com/wertick01/deep-fold/blob/main/gpu/cli/main.py) (CLI commands), [`gpu/loop/generate.py`](https://github.com/wertick01/deep-fold/blob/main/gpu/loop/generate.py) (TokenLoop), [`gpu/host/model.py`](https://github.com/wertick01/deep-fold/blob/main/gpu/host/model.py) (Model construction), [`gpu/cli/chat.py`](https://github.com/wertick01/deep-fold/blob/main/gpu/cli/chat.py) (Chat session)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_cli["User interface"]
  node_cli["CLI commands<br/>[main.py]"]
  node_chat["Chat session<br/>[chat.py]"]
  node_pull["Model pull/import<br/>[pull.py]"]
  node_doctor["Environment checks<br/>[doctor.py]"]
end

subgraph group_format["Compression and format"]
  node_compressor["CHR compressor<br/>[run.go]"]
  node_chrformat["CHR0 container<br/>[write.go]"]
  node_nf4codec["NF4 encoding<br/>[nf4.go]"]
  node_safetensors["Source weights<br/>[index.go]"]
end

subgraph group_loading["Model loading"]
  node_loader["CHR0 loader<br/>[loader.py]"]
  node_model["Model construction<br/>[model.py]"]
  node_embedding["Packed embeddings<br/>[embedding.py]"]
  node_placement["Weight placement<br/>[residency.py]"]
end

subgraph group_runtime["Inference runtime"]
  node_nf4kernels["NF4 kernels<br/>[__init__.py]"]
  node_tokenloop["TokenLoop<br/>[generate.py]"]
  node_graph["CUDA graph execution<br/>[graph.py]"]
  node_copyring["CopyRing streaming<br/>[ring.py]"]
  node_decodev2["Decode V2<br/>[session.py]"]
end

subgraph group_research["Evaluation"]
  node_lab["Benchmark sessions<br/>[sessions.py]"]
  node_evaluation["Evaluation runner<br/>[eval.py]"]
end

node_user(("Local user"))
node_response["Generated tokens"]

node_user -->|"invokes"| node_cli
node_cli -->|"routes chat"| node_chat
node_cli -->|"routes pull"| node_pull
node_cli -->|"routes checks"| node_doctor
node_cli -->|"requests packing"| node_compressor
node_safetensors -.->|"supplies weights"| node_compressor
node_compressor -.->|"encodes NF4"| node_nf4codec
node_compressor -->|"writes CHR0"| node_chrformat
node_chrformat -->|"loads container"| node_loader
node_loader -.->|"provides packed weights"| node_model
node_model -->|"loads embedding rows"| node_embedding
node_model -.->|"selects residency"| node_placement
node_model -->|"attaches packed weights"| node_nf4kernels
node_chat -->|"generates with"| node_tokenloop
node_chat -->|"selects resident path"| node_decodev2
node_tokenloop -->|"executes GEMMs"| node_graph
node_tokenloop -->|"streams overflow weights"| node_copyring
node_tokenloop -->|"runs packed GEMMs"| node_nf4kernels
node_decodev2 -->|"runs packed kernels"| node_nf4kernels
node_tokenloop -->|"returns tokens"| node_response
node_decodev2 -->|"returns tokens"| node_response
node_lab -->|"records sessions"| node_evaluation
node_evaluation -.->|"uses inference workflow"| node_chat

click node_cli "https://github.com/wertick01/deep-fold/blob/main/gpu/cli/main.py"
click node_chat "https://github.com/wertick01/deep-fold/blob/main/gpu/cli/chat.py"
click node_pull "https://github.com/wertick01/deep-fold/blob/main/gpu/cli/pull.py"
click node_doctor "https://github.com/wertick01/deep-fold/blob/main/gpu/cli/doctor.py"
click node_compressor "https://github.com/wertick01/deep-fold/blob/main/cmd/chr/run.go"
click node_chrformat "https://github.com/wertick01/deep-fold/blob/main/internal/chr0/write.go"
click node_nf4codec "https://github.com/wertick01/deep-fold/blob/main/internal/nf4/nf4.go"
click node_safetensors "https://github.com/wertick01/deep-fold/blob/main/internal/safetensors/index.go"
click node_loader "https://github.com/wertick01/deep-fold/blob/main/gpu/chr0/loader.py"
click node_model "https://github.com/wertick01/deep-fold/blob/main/gpu/host/model.py"
click node_embedding "https://github.com/wertick01/deep-fold/blob/main/gpu/host/embedding.py"
click node_placement "https://github.com/wertick01/deep-fold/blob/main/gpu/host/residency.py"
click node_nf4kernels "https://github.com/wertick01/deep-fold/blob/main/gpu/nf4/__init__.py"
click node_tokenloop "https://github.com/wertick01/deep-fold/blob/main/gpu/loop/generate.py"
click node_graph "https://github.com/wertick01/deep-fold/blob/main/gpu/loop/graph.py"
click node_copyring "https://github.com/wertick01/deep-fold/blob/main/gpu/loop/ring.py"
click node_decodev2 "https://github.com/wertick01/deep-fold/blob/main/gpu/decodev2/session.py"
click node_lab "https://github.com/wertick01/deep-fold/blob/main/gpu/lab/sessions.py"
click node_evaluation "https://github.com/wertick01/deep-fold/blob/main/gpu/lab/eval.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_chat,node_pull,node_doctor,node_user toneBlue
class node_compressor,node_chrformat,node_nf4codec,node_safetensors toneAmber
class node_loader,node_model,node_embedding,node_placement toneMint
class node_nf4kernels,node_tokenloop,node_graph,node_copyring,node_decodev2 toneRose
class node_lab,node_evaluation,node_response toneIndigo
```

## Components

### User interface

- [CLI commands](https://github.com/wertick01/deep-fold/blob/main/gpu/cli/main.py): `gpu/cli/main.py`
- [Chat session](https://github.com/wertick01/deep-fold/blob/main/gpu/cli/chat.py): `gpu/cli/chat.py`
- [Model pull/import](https://github.com/wertick01/deep-fold/blob/main/gpu/cli/pull.py): `gpu/cli/pull.py`
- [Environment checks](https://github.com/wertick01/deep-fold/blob/main/gpu/cli/doctor.py): `gpu/cli/doctor.py`

### Compression and format

- [CHR compressor](https://github.com/wertick01/deep-fold/blob/main/cmd/chr/run.go): `cmd/chr/run.go`
- [CHR0 container](https://github.com/wertick01/deep-fold/blob/main/internal/chr0/write.go): `internal/chr0/write.go`
- [NF4 encoding](https://github.com/wertick01/deep-fold/blob/main/internal/nf4/nf4.go): `internal/nf4/nf4.go`
- [Source weights](https://github.com/wertick01/deep-fold/blob/main/internal/safetensors/index.go): `internal/safetensors/index.go`

### Model loading

- [CHR0 loader](https://github.com/wertick01/deep-fold/blob/main/gpu/chr0/loader.py): `gpu/chr0/loader.py`
- [Model construction](https://github.com/wertick01/deep-fold/blob/main/gpu/host/model.py): `gpu/host/model.py`
- [Packed embeddings](https://github.com/wertick01/deep-fold/blob/main/gpu/host/embedding.py): `gpu/host/embedding.py`
- [Weight placement](https://github.com/wertick01/deep-fold/blob/main/gpu/host/residency.py): `gpu/host/residency.py`

### Inference runtime

- [NF4 kernels](https://github.com/wertick01/deep-fold/blob/main/gpu/nf4/__init__.py): `gpu/nf4/__init__.py`
- [TokenLoop](https://github.com/wertick01/deep-fold/blob/main/gpu/loop/generate.py): `gpu/loop/generate.py`
- [CUDA graph execution](https://github.com/wertick01/deep-fold/blob/main/gpu/loop/graph.py): `gpu/loop/graph.py`
- [CopyRing streaming](https://github.com/wertick01/deep-fold/blob/main/gpu/loop/ring.py): `gpu/loop/ring.py`
- [Decode V2](https://github.com/wertick01/deep-fold/blob/main/gpu/decodev2/session.py): `gpu/decodev2/session.py`

### Evaluation

- [Benchmark sessions](https://github.com/wertick01/deep-fold/blob/main/gpu/lab/sessions.py): `gpu/lab/sessions.py`
- [Evaluation runner](https://github.com/wertick01/deep-fold/blob/main/gpu/lab/eval.py): `gpu/lab/eval.py`

### Other components

- **Local user**
- **Generated tokens**

## Connections

- Local user → CLI commands: invokes (evidence: `gpu/cli/main.py`)
- CLI commands → Chat session: routes chat (evidence: `gpu/cli/main.py`)
- CLI commands → Model pull/import: routes pull (evidence: `gpu/cli/main.py`)
- CLI commands → Environment checks: routes checks (evidence: `gpu/cli/main.py`)
- CLI commands → CHR compressor: requests packing (evidence: `README.md`)
- Source weights → CHR compressor: supplies weights
- CHR compressor → NF4 encoding: encodes NF4
- CHR compressor → CHR0 container: writes CHR0 (evidence: `README.md`)
- CHR0 container → CHR0 loader: loads container (evidence: `README.md`)
- CHR0 loader → Model construction: provides packed weights
- Model construction → Packed embeddings: loads embedding rows (evidence: `gpu/host/model.py`)
- Model construction → Weight placement: selects residency (evidence: `gpu/host/model.py`)
- Model construction → NF4 kernels: attaches packed weights (evidence: `README.md`)
- Chat session → TokenLoop: generates with (evidence: `README.md`)
- Chat session → Decode V2: selects resident path (evidence: `gpu/decodev2/session.py`)
- TokenLoop → CUDA graph execution: executes GEMMs (evidence: `gpu/loop/generate.py`)
- TokenLoop → CopyRing streaming: streams overflow weights (evidence: `gpu/loop/generate.py`)
- TokenLoop → NF4 kernels: runs packed GEMMs (evidence: `gpu/loop/generate.py`)
- Decode V2 → NF4 kernels: runs packed kernels (evidence: `README.md`)
- TokenLoop → Generated tokens: returns tokens (evidence: `README.md`)
- Decode V2 → Generated tokens: returns tokens (evidence: `README.md`)
- Benchmark sessions → Evaluation runner: records sessions (evidence: `gpu/lab/sessions.py`)
- Evaluation runner → Chat session: uses inference workflow
