How ashhart/tensorfold works
TensorFold serves supported text-model families through an OpenAI-compatible API on Apple Silicon (MLX) or NVIDIA GPUs (CUDA). The primary workflow is a client request, prompt preparation and scheduling, model-family inference with optional draft proposals checked for exact decoding, then streamed response tokens. Model acquisition and inspection are separate CLI capabilities. Family-specific implementations use shared engine and kernel layers, while CUDA additionally provides rank coordination. The README documents broader family and backend support than the sampled engine excerpts; links among unsampled family internals are therefore kept at subsystem level.
Main entry points: app.py MLX chat app · server.py CUDA API · cli.py TensorFold CLI · lane_engine.py Shared-round engine
Components 23
CLI and models
- TensorFold CLI
src/tensorfold/cli.py - Model hub
src/tensorfold/hub.py
Request serving
- HTTP interface
src/tensorfold/server/http.py - MLX chat app
src/tensorfold/server/app.py - CUDA API
src/tensorfold/cuda/server.py - Request scheduler
src/tensorfold/server/scheduler.py - Request policies
src/tensorfold/server/request_options.py
Decode runtime
- Shared-round engine
src/tensorfold/engine/lane_engine.py - Exact sampling
src/tensorfold/engine/exact_sampling.py - Draft proposals
src/tensorfold/drafters - Prefix reuse
src/tensorfold/engine/prefix_snapshots.py
Model families
- Family detection
src/tensorfold/families/__init__.py - Qwen engines
src/tensorfold/families/qwen3_5/cuda/engine.py - GLM engines
src/tensorfold/families/glm5_next/cuda/engine.py - Nemotron engines
src/tensorfold/families/nemotron_h/cuda/engine.py - Other families
src/tensorfold/families
GPU compute
- CUDA kernels
src/tensorfold/cuda/kernels - Family kernels
src/tensorfold/kernels - CUDA rank comms
src/tensorfold/cuda/comm.py
Other components
- API client
- Apple MLX
- NVIDIA CUDA
- Hugging Face Hub
Connections 25
- API client → HTTP interface: sends requests
- HTTP interface → MLX chat app: dispatches API
- HTTP interface → CUDA API: dispatches API
- MLX chat app → Request scheduler: submits jobs
- MLX chat app → Request policies: resolves options
- Request scheduler → Shared-round engine: runs rounds
- Shared-round engine → Exact sampling: verifies samples
- Shared-round engine → Draft proposals: requests proposals
- Shared-round engine → Prefix reuse: retains states
- CUDA API → Qwen engines: invokes engine
- CUDA API → GLM engines: invokes engine
- CUDA API → Nemotron engines: invokes engine
- TensorFold CLI → Family detection: detects family
- TensorFold CLI → Model hub: pulls checkpoints
- Model hub → Hugging Face Hub: downloads weights
- TensorFold CLI → MLX chat app: starts serving
- TensorFold CLI → CUDA API: starts serving
- Qwen engines → CUDA kernels: invokes kernels
- GLM engines → CUDA kernels: invokes kernels
- Nemotron engines → CUDA kernels: invokes kernels
- Other families → Family kernels: uses kernels
- MLX chat app → Apple MLX: runs inference
- CUDA kernels → NVIDIA CUDA: executes on
- GLM engines → CUDA rank comms: coordinates ranks
- Qwen engines → CUDA rank comms: coordinates ranks