# kyuz0/amd-strix-halo-toolboxes architecture

> Architecture diagram of the GitHub repository kyuz0/amd-strix-halo-toolboxes, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kyuz0/amd-strix-halo-toolboxes
- Repository: https://github.com/kyuz0/amd-strix-halo-toolboxes

## Overview

This repository provides prebuilt llama.cpp toolboxes for accelerated LLM inference on AMD Strix Halo, with Cockpit as the recommended install-and-launch interface. Its main user workflow is selecting a toolbox and running inference; a separate terminal application coordinates distributed runs across RPC hosts. The repository also includes a GGUF memory estimator and a GPU workload watcher. The distributed-runner links reflect shown SSH/RPC behavior; other toolbox and Cockpit relationships are documented in the README. Unsampled implementation details are not inferred.

Main entry points: [`scripts/run_distributed_llama.py`](https://github.com/kyuz0/amd-strix-halo-toolboxes/blob/main/scripts/run_distributed_llama.py) (Distributed Runner), [`toolboxes/gguf-vram-estimator.py`](https://github.com/kyuz0/amd-strix-halo-toolboxes/blob/main/toolboxes/gguf-vram-estimator.py) (VRAM Estimator), [`systemd/gpu-workload-watch/gpu-workload-watch`](https://github.com/kyuz0/amd-strix-halo-toolboxes/blob/main/systemd/gpu-workload-watch/gpu-workload-watch) (GPU Workload Watcher), [`toolboxes`](https://github.com/kyuz0/amd-strix-halo-toolboxes/tree/main/toolboxes) (llama.cpp Toolboxes)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_toolbox["Inference Toolboxes"]
  node_containers["llama.cpp Toolboxes"]
  node_project_docs["Guides and Results"]
end

subgraph group_distributed["Distributed Inference"]
  node_distributed_runner["Distributed Runner"]
  node_remote_hosts["Remote Hosts"]
  node_rpc_server["RPC Servers"]
end

subgraph group_planning["Memory Planning"]
  node_estimator["VRAM Estimator"]
end

subgraph group_host["Host Operations"]
  node_gpu_watcher["GPU Workload Watcher"]
end

node_user(("User"))
node_cockpit["AI Toolbox Cockpit"]
node_container_runtime["Container Runtime"]
node_model_file["GGUF Model File"]
node_host_hardware["Strix Halo Host"]

node_user -->|"installs and launches"| node_cockpit
node_cockpit -->|"launches and updates"| node_containers
node_user -->|"runs containers"| node_container_runtime
node_container_runtime -->|"runs"| node_containers
node_user -->|"configures and runs"| node_distributed_runner
node_distributed_runner -->|"connects over SSH"| node_remote_hosts
node_distributed_runner -->|"starts and checks"| node_rpc_server
node_user -->|"estimates memory"| node_estimator
node_estimator -->|"reads metadata"| node_model_file
node_user -->|"uses workload monitoring"| node_gpu_watcher
node_gpu_watcher -->|"monitors workload"| node_host_hardware
node_user -->|"consults guides"| node_project_docs

click node_containers "https://github.com/kyuz0/amd-strix-halo-toolboxes/tree/main/toolboxes"
click node_distributed_runner "https://github.com/kyuz0/amd-strix-halo-toolboxes/blob/main/scripts/run_distributed_llama.py"
click node_estimator "https://github.com/kyuz0/amd-strix-halo-toolboxes/blob/main/toolboxes/gguf-vram-estimator.py"
click node_gpu_watcher "https://github.com/kyuz0/amd-strix-halo-toolboxes/blob/main/systemd/gpu-workload-watch/gpu-workload-watch"
click node_project_docs "https://github.com/kyuz0/amd-strix-halo-toolboxes/tree/main/docs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_containers,node_project_docs,node_user toneBlue
class node_distributed_runner,node_remote_hosts,node_rpc_server toneAmber
class node_estimator toneMint
class node_gpu_watcher toneRose
class node_cockpit,node_container_runtime,node_model_file,node_host_hardware toneIndigo
```

## Components

### Inference Toolboxes

- [llama.cpp Toolboxes](https://github.com/kyuz0/amd-strix-halo-toolboxes/tree/main/toolboxes): `toolboxes`
- [Guides and Results](https://github.com/kyuz0/amd-strix-halo-toolboxes/tree/main/docs): `docs`

### Distributed Inference

- [Distributed Runner](https://github.com/kyuz0/amd-strix-halo-toolboxes/blob/main/scripts/run_distributed_llama.py): `scripts/run_distributed_llama.py`
- **Remote Hosts**
- **RPC Servers**

### Memory Planning

- [VRAM Estimator](https://github.com/kyuz0/amd-strix-halo-toolboxes/blob/main/toolboxes/gguf-vram-estimator.py): `toolboxes/gguf-vram-estimator.py`

### Host Operations

- [GPU Workload Watcher](https://github.com/kyuz0/amd-strix-halo-toolboxes/blob/main/systemd/gpu-workload-watch/gpu-workload-watch): `systemd/gpu-workload-watch/gpu-workload-watch`

### Other components

- **User**
- **AI Toolbox Cockpit**
- **Container Runtime**
- **GGUF Model File**
- **Strix Halo Host**

## Connections

- User → AI Toolbox Cockpit: installs and launches (evidence: `README.md`)
- AI Toolbox Cockpit → llama.cpp Toolboxes: launches and updates (evidence: `README.md`)
- User → Container Runtime: runs containers (evidence: `README.md`)
- Container Runtime → llama.cpp Toolboxes: runs (evidence: `README.md`)
- User → Distributed Runner: configures and runs (evidence: `scripts/run_distributed_llama.py`)
- Distributed Runner → Remote Hosts: connects over SSH (evidence: `scripts/run_distributed_llama.py`)
- Distributed Runner → RPC Servers: starts and checks (evidence: `scripts/run_distributed_llama.py`)
- User → VRAM Estimator: estimates memory (evidence: `toolboxes/gguf-vram-estimator.py`)
- VRAM Estimator → GGUF Model File: reads metadata (evidence: `toolboxes/gguf-vram-estimator.py`)
- User → GPU Workload Watcher: uses workload monitoring (evidence: `README.md`)
- GPU Workload Watcher → Strix Halo Host: monitors workload (evidence: `README.md`)
- User → Guides and Results: consults guides (evidence: `README.md`)