How kyuz0/amd-strix-halo-toolboxes works
This repository provides prebuilt llama.cpp toolboxes for accelerated LLM inference on AMD Strix Halo, with Cockpit as the recommended install-and-launch interface. Its main user workflow is selecting a toolbox and running inference; a separate terminal application coordinates distributed runs across RPC hosts. The repository also includes a GGUF memory estimator and a GPU workload watcher. The distributed-runner links reflect shown SSH/RPC behavior; other toolbox and Cockpit relationships are documented in the README. Unsampled implementation details are not inferred.
Main entry points: run_distributed_llama.py Distributed Runner · gguf-vram-estimator.py VRAM Estimator · gpu-workload-watch GPU Workload Watcher · toolboxes llama.cpp Toolboxes
Components 12
Inference Toolboxes
- llama.cpp Toolboxes
toolboxes - Guides and Results
docs
Distributed Inference
- Distributed Runner
scripts/run_distributed_llama.py - Remote Hosts
- RPC Servers
Memory Planning
- VRAM Estimator
toolboxes/gguf-vram-estimator.py
Host Operations
- GPU Workload Watcher
systemd/gpu-workload-watch/gpu-workload-watch
Other components
- User
- AI Toolbox Cockpit
- Container Runtime
- GGUF Model File
- Strix Halo Host
Connections 12
- User → AI Toolbox Cockpit: installs and launches
README.md - AI Toolbox Cockpit → llama.cpp Toolboxes: launches and updates
README.md - User → Container Runtime: runs containers
README.md - Container Runtime → llama.cpp Toolboxes: runs
README.md - User → Distributed Runner: configures and runs
scripts/run_distributed_llama.py - Distributed Runner → Remote Hosts: connects over SSH
scripts/run_distributed_llama.py - Distributed Runner → RPC Servers: starts and checks
scripts/run_distributed_llama.py - User → VRAM Estimator: estimates memory
toolboxes/gguf-vram-estimator.py - VRAM Estimator → GGUF Model File: reads metadata
toolboxes/gguf-vram-estimator.py - User → GPU Workload Watcher: uses workload monitoring
README.md - GPU Workload Watcher → Strix Halo Host: monitors workload
README.md - User → Guides and Results: consults guides
README.md