How mostlygeek/llama-swap works
Purpose and Entry Points
llama-swap is a Go local inference gateway and process orchestrator. It exposes OpenAI-, Anthropic-, llama.cpp-, SDAPI-, audio.cpp-, and ComfyUI-compatible HTTP routes, chooses a configured model, starts/stops its backing inference server, then reverse-proxies the request. Production entry point: llama-swap.go. Configuration begins from YAML and its generated/validated schema: config.example.yaml, config-schema.json.
Configuration Compilation
The configuration subsystem loads platform-aware YAML, merges profiles, expands macros, resolves model aliases/selectors, command templates, request filters, upstreams, and matrix rules into runtime model definitions. This is the principal declarative boundary between user configuration and orchestration behavior. Paths: internal/config/load.go, internal/config/model_config.go, internal/config/matrix.go.
HTTP API and Proxy Boundary
The server owns listener setup, authentication, endpoint grouping, request model extraction/selection, request rewriting, direct upstream access, captures, and API/UI responses. It must preserve streaming behavior while forwarding arbitrary compatible API payloads to selected upstream processes. Paths: internal/server/server.go, internal/server/api.go, internal/server/auth.go.
Routing, Scheduling, and Swap Decisions
The router maps requested model IDs (including aliases/groups/peers) to a loading decision. Basic routing swaps incompatible running models; matrix routing evaluates the configuration DSL to decide which concurrent model set can remain loaded. Scheduler coordination serializes or queues load work and protects capacity/resource decisions. Paths: internal/router/router.go, internal/router/matrix_solver.go, internal/router/scheduler/scheduler.go.
Model Process Lifecycle
The process layer launches configured commands with dynamically assigned ports and environment, tracks readiness/events, forwards process output, and performs graceful termination or platform-specific tree cleanup. Router loading uses this layer to materialize an upstream server before proxying and unloads models due to replacement, manual APIs, or TTL. Paths: internal/router/loading.go, internal/process/process.go, internal/process/process_command.go.
Runtime State, Events, and Observability
In-memory caches/chains coordinate active model state and request flow. Event/log monitoring maintains buffered history plus live streams for proxy and upstream output. Performance monitoring detects hardware and exposes system/GPU Prometheus metrics; activity is persisted through a migration-backed store for UI/API consumption. Paths: internal/event/event.go, internal/logmon/logging.go, internal/perf/prometheus.go, internal/store/store.go.
UI and Deployment
A Svelte/Vite single-page UI is embedded or served by the Go server, providing models, activity, hardware, performance, logs, settings, and multimodal playgrounds over server APIs and event streams. The system runs as a native Go binary across OSes or in Docker/Podman; unified images bundle inference runtimes such as llama-server, stable-diffusion.cpp, and whisper. Paths: ui-svelte/src/main.ts, ui-svelte/src/App.svelte, internal/server/ui.go, docker/unified/Dockerfile.
Main entry points: server.go HTTP server · api.go API & request selection · App.svelte Svelte UI · load.go Config loader
Components 21
Configuration
Compiles declarative model and routing definitions.
- YAML & schema · User-facing configuration contract.
config.example.yaml - Config loader · Loads platform-aware YAML and profiles.
internal/config/load.go - Runtime model definitions · Resolves aliases, templates, filters, and upstreams.
internal/config/model_config.go - Matrix rules · Defines permitted concurrent model sets.
internal/config/matrix.go
Gateway
HTTP compatibility, authentication, and proxy boundary.
- HTTP server · Owns listeners, compatible routes, and responses.
internal/server/server.go - API & request selection · Extracts requested models and rewrites requests.
internal/server/api.go - Authentication
internal/server/auth.go
Orchestration
Model selection, capacity decisions, and process lifecycle.
- Model router · Maps model IDs to loading decisions.
internal/router/router.go - Matrix solver · Evaluates concurrent loading rules.
internal/router/matrix_solver.go - Load scheduler · Serializes load work and protects capacity.
internal/router/scheduler/scheduler.go - Loading lifecycle · Materializes and unloads model upstreams.
internal/router/loading.go - Inference process manager · Launches, monitors, and terminates configured servers.
internal/process/process.go - Managed inference servers · llama.cpp, SDAPI, audio.cpp, and ComfyUI-compatible processes.
internal/process/process_command.go
Runtime State
Events, logs, metrics, and persisted activity.
- Event stream · Coordinates active state and live updates.
internal/event/event.go - Log monitor · Buffers and streams proxy and upstream output.
internal/logmon/logging.go - Prometheus metrics · Exports system and GPU performance metrics.
internal/perf/prometheus.go - Activity store · Migration-backed activity data for APIs and UI.
internal/store/store.go
UI & Deployment
Operator interface and packaged runtime environment.
- Svelte UI · Models, activity, hardware, logs, settings, and playgrounds.
ui-svelte/src/App.svelte - UI serving · Embeds or serves the SPA from the gateway.
internal/server/ui.go - Unified container image · Bundles the gateway with inference runtimes.
docker/unified/Dockerfile
Other components
- Go entry point · Starts the local inference gateway.
llama-swap.go
Connections 26
- Go entry point → Config loader: loads configuration
- Go entry point → HTTP server: starts gateway
- YAML & schema → Config loader: defines input
- Config loader → Runtime model definitions: compiles
- Config loader → Matrix rules: expands rules
- HTTP server → Authentication: applies
- HTTP server → API & request selection: routes requests
- HTTP server → UI serving: serves UI
- UI serving → Svelte UI: delivers SPA
- Svelte UI → API & request selection: uses APIs and streams
- API & request selection → Model router: selects model
- Runtime model definitions → Model router: supplies models
- Model router → Matrix solver: evaluates matrix mode
- Matrix rules → Matrix solver: defines constraints
- Model router → Load scheduler: coordinates load
- Load scheduler → Loading lifecycle: authorizes work
- Loading lifecycle → Inference process manager: starts or stops
- Inference process manager → Managed inference servers: supervises
- API & request selection → Managed inference servers: reverse proxies · Preserves compatible and streaming payloads.
- Loading lifecycle → Event stream: publishes state
- Inference process manager → Log monitor: forwards output
- HTTP server → Prometheus metrics: exposes metrics
- API & request selection → Activity store: records activity
- Svelte UI → Event stream: subscribes live
- Svelte UI → Activity store: views history
- Unified container image → Go entry point: packages runtime