How bnice00/k3s-hermes-llama works
This repository describes a single-user K3s inference stack: a Streamlit chat client sends prompts to a Hermes agent, which uses llama.cpp as its inference backend. The README also documents an earlier vLLM/ROCm setup and GPU scheduling/monitoring components; manifests show additional services, but their runtime wiring is not fully established by the sampled source. Paths for those unsampled deployments point to their exact manifest files.
Main entry points: llama.yaml llama.cpp Vulkan · 00-namespace.yaml K3s cluster · agent.py Streamlit chat · hermes-agent.yaml Hermes agent
Components 18
User interaction
- Streamlit chat
streamlit/agent.py
Agent layer
- Hermes agent
manifests/hermes-agent.yaml - Hermes config
hermes-config/hermes-config.yaml
Inference services
- llama.cpp Vulkan
manifests/llama.yaml - Inference endpoint
manifests/llama.yaml - vLLM deployment
vllm-stack/vllm.yaml - vLLM service
vllm-stack/vllm-service.yaml - Whisper service
manifests/whisper.yaml
Cluster platform
- Model storage
manifests/pvc-hermes.yaml - GPU device plugin
manifests/gpu-device-plugin.yaml - GPU device labeller
manifests/gpu-device-labeller.yaml - K3s cluster
manifests/00-namespace.yaml - Obsidian service
manifests/obsidian.yaml - Debian service
manifests/deb-service.yaml - Certificate manager
manifests/certman-values.yaml - Prometheus monitoring
monitoring/prometheus-values.yaml
Other components
- User
- Cluster operator
Connections 9
- User → Streamlit chat: submits prompt
- Streamlit chat → Hermes agent: posts chat request
- Hermes agent → llama.cpp Vulkan: requests inference
- llama.cpp Vulkan → Inference endpoint: serves endpoint
- llama.cpp Vulkan → Model storage: uses model storage
- GPU device plugin → K3s cluster: advertises GPU
- K3s cluster → llama.cpp Vulkan: schedules GPU pod
- GPU device labeller → K3s cluster: labels GPU devices
- Cluster operator → K3s cluster: operates cluster