# ashhart/tensorfold architecture

> Architecture diagram of the GitHub repository ashhart/tensorfold, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ashhart/tensorfold
- Repository: https://github.com/ashhart/tensorfold

## Overview

TensorFold serves supported text-model families through an OpenAI-compatible API on Apple Silicon (MLX) or NVIDIA GPUs (CUDA). The primary workflow is a client request, prompt preparation and scheduling, model-family inference with optional draft proposals checked for exact decoding, then streamed response tokens. Model acquisition and inspection are separate CLI capabilities. Family-specific implementations use shared engine and kernel layers, while CUDA additionally provides rank coordination. The README documents broader family and backend support than the sampled engine excerpts; links among unsampled family internals are therefore kept at subsystem level.

Main entry points: [`src/tensorfold/server/app.py`](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/server/app.py) (MLX chat app), [`src/tensorfold/cuda/server.py`](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/cuda/server.py) (CUDA API), [`src/tensorfold/cli.py`](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/cli.py) (TensorFold CLI), [`src/tensorfold/engine/lane_engine.py`](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/engine/lane_engine.py) (Shared-round engine)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["CLI and models"]
  node_cli["TensorFold CLI<br/>[cli.py]"]
  node_hub["Model hub<br/>[hub.py]"]
end

subgraph group_api["Request serving"]
  node_http["HTTP interface<br/>[http.py]"]
  node_mlx_app["MLX chat app<br/>[app.py]"]
  node_cuda_api["CUDA API<br/>[server.py]"]
  node_scheduler["Request scheduler<br/>[scheduler.py]"]
  node_request_policy["Request policies<br/>[request_options.py]"]
end

subgraph group_runtime["Decode runtime"]
  node_lane["Shared-round engine<br/>[lane_engine.py]"]
  node_sampling["Exact sampling<br/>[exact_sampling.py]"]
  node_drafting["Draft proposals"]
  node_prefix[("Prefix reuse")]
end

subgraph group_families["Model families"]
  node_family_registry["Family detection<br/>[__init__.py]"]
  node_qwen["Qwen engines<br/>[engine.py]"]
  node_glm["GLM engines<br/>[engine.py]"]
  node_nemotron["Nemotron engines<br/>[engine.py]"]
  node_other_families["Other families"]
end

subgraph group_compute["GPU compute"]
  node_cuda_kernels["CUDA kernels"]
  node_family_kernels["Family kernels"]
  node_cuda_comm["CUDA rank comms<br/>[comm.py]"]
end

node_user(("API client"))
node_mlx(("Apple MLX"))
node_nvidia(("NVIDIA CUDA"))
node_huggingface(("Hugging Face Hub"))

node_user -->|"sends requests"| node_http
node_http -->|"dispatches API"| node_mlx_app
node_http -->|"dispatches API"| node_cuda_api
node_mlx_app -->|"submits jobs"| node_scheduler
node_mlx_app -->|"resolves options"| node_request_policy
node_scheduler -->|"runs rounds"| node_lane
node_lane -->|"verifies samples"| node_sampling
node_lane -->|"requests proposals"| node_drafting
node_lane -->|"retains states"| node_prefix
node_cuda_api -->|"invokes engine"| node_qwen
node_cuda_api -->|"invokes engine"| node_glm
node_cuda_api -->|"invokes engine"| node_nemotron
node_cli -->|"detects family"| node_family_registry
node_cli -->|"pulls checkpoints"| node_hub
node_hub -->|"downloads weights"| node_huggingface
node_cli -->|"starts serving"| node_mlx_app
node_cli -->|"starts serving"| node_cuda_api
node_qwen -->|"invokes kernels"| node_cuda_kernels
node_glm -->|"invokes kernels"| node_cuda_kernels
node_nemotron -->|"invokes kernels"| node_cuda_kernels
node_other_families -.->|"uses kernels"| node_family_kernels
node_mlx_app -->|"runs inference"| node_mlx
node_cuda_kernels -->|"executes on"| node_nvidia
node_glm -.->|"coordinates ranks"| node_cuda_comm
node_qwen -.->|"coordinates ranks"| node_cuda_comm

click node_cli "https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/cli.py"
click node_hub "https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/hub.py"
click node_http "https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/server/http.py"
click node_mlx_app "https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/server/app.py"
click node_cuda_api "https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/cuda/server.py"
click node_scheduler "https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/server/scheduler.py"
click node_request_policy "https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/server/request_options.py"
click node_lane "https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/engine/lane_engine.py"
click node_sampling "https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/engine/exact_sampling.py"
click node_drafting "https://github.com/ashhart/tensorfold/tree/main/src/tensorfold/drafters"
click node_prefix "https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/engine/prefix_snapshots.py"
click node_family_registry "https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/families/__init__.py"
click node_qwen "https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/families/qwen3_5/cuda/engine.py"
click node_glm "https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/families/glm5_next/cuda/engine.py"
click node_nemotron "https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/families/nemotron_h/cuda/engine.py"
click node_other_families "https://github.com/ashhart/tensorfold/tree/main/src/tensorfold/families"
click node_cuda_kernels "https://github.com/ashhart/tensorfold/tree/main/src/tensorfold/cuda/kernels"
click node_family_kernels "https://github.com/ashhart/tensorfold/tree/main/src/tensorfold/kernels"
click node_cuda_comm "https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/cuda/comm.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_hub,node_user toneBlue
class node_http,node_mlx_app,node_cuda_api,node_scheduler,node_request_policy toneAmber
class node_lane,node_sampling,node_drafting,node_prefix toneMint
class node_family_registry,node_qwen,node_glm,node_nemotron,node_other_families toneRose
class node_cuda_kernels,node_family_kernels,node_cuda_comm,node_mlx,node_nvidia,node_huggingface toneIndigo
```

## Components

### CLI and models

- [TensorFold CLI](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/cli.py): `src/tensorfold/cli.py`
- [Model hub](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/hub.py): `src/tensorfold/hub.py`

### Request serving

- [HTTP interface](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/server/http.py): `src/tensorfold/server/http.py`
- [MLX chat app](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/server/app.py): `src/tensorfold/server/app.py`
- [CUDA API](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/cuda/server.py): `src/tensorfold/cuda/server.py`
- [Request scheduler](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/server/scheduler.py): `src/tensorfold/server/scheduler.py`
- [Request policies](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/server/request_options.py): `src/tensorfold/server/request_options.py`

### Decode runtime

- [Shared-round engine](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/engine/lane_engine.py): `src/tensorfold/engine/lane_engine.py`
- [Exact sampling](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/engine/exact_sampling.py): `src/tensorfold/engine/exact_sampling.py`
- [Draft proposals](https://github.com/ashhart/tensorfold/tree/main/src/tensorfold/drafters): `src/tensorfold/drafters`
- [Prefix reuse](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/engine/prefix_snapshots.py): `src/tensorfold/engine/prefix_snapshots.py`

### Model families

- [Family detection](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/families/__init__.py): `src/tensorfold/families/__init__.py`
- [Qwen engines](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/families/qwen3_5/cuda/engine.py): `src/tensorfold/families/qwen3_5/cuda/engine.py`
- [GLM engines](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/families/glm5_next/cuda/engine.py): `src/tensorfold/families/glm5_next/cuda/engine.py`
- [Nemotron engines](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/families/nemotron_h/cuda/engine.py): `src/tensorfold/families/nemotron_h/cuda/engine.py`
- [Other families](https://github.com/ashhart/tensorfold/tree/main/src/tensorfold/families): `src/tensorfold/families`

### GPU compute

- [CUDA kernels](https://github.com/ashhart/tensorfold/tree/main/src/tensorfold/cuda/kernels): `src/tensorfold/cuda/kernels`
- [Family kernels](https://github.com/ashhart/tensorfold/tree/main/src/tensorfold/kernels): `src/tensorfold/kernels`
- [CUDA rank comms](https://github.com/ashhart/tensorfold/blob/main/src/tensorfold/cuda/comm.py): `src/tensorfold/cuda/comm.py`

### Other components

- **API client**
- **Apple MLX**
- **NVIDIA CUDA**
- **Hugging Face Hub**

## Connections

- API client → HTTP interface: sends requests
- HTTP interface → MLX chat app: dispatches API
- HTTP interface → CUDA API: dispatches API
- MLX chat app → Request scheduler: submits jobs
- MLX chat app → Request policies: resolves options
- Request scheduler → Shared-round engine: runs rounds
- Shared-round engine → Exact sampling: verifies samples
- Shared-round engine → Draft proposals: requests proposals
- Shared-round engine → Prefix reuse: retains states
- CUDA API → Qwen engines: invokes engine
- CUDA API → GLM engines: invokes engine
- CUDA API → Nemotron engines: invokes engine
- TensorFold CLI → Family detection: detects family
- TensorFold CLI → Model hub: pulls checkpoints
- Model hub → Hugging Face Hub: downloads weights
- TensorFold CLI → MLX chat app: starts serving
- TensorFold CLI → CUDA API: starts serving
- Qwen engines → CUDA kernels: invokes kernels
- GLM engines → CUDA kernels: invokes kernels
- Nemotron engines → CUDA kernels: invokes kernels
- Other families → Family kernels: uses kernels
- MLX chat app → Apple MLX: runs inference
- CUDA kernels → NVIDIA CUDA: executes on
- GLM engines → CUDA rank comms: coordinates ranks
- Qwen engines → CUDA rank comms: coordinates ranks