How adt109119/llamacpp-distributed-inference works
Project Type and Purpose
• This is a full-stack desktop application built with Electron for distributed LLM inference.
• The goal is to let multiple machines on a LAN collaborate on llama.cpp model inference, exposing:
• A local HTTP API server (for UI and external API calls)
• Remote RPC nodes (llama.cpp servers)
• A modern GUI (Electron renderer)
• mDNS-based service discovery and manual node management
Main Components
a. Electron Main Process (src/main)
• Orchestrates startup:
• Spawns local rpc-server and llama-server binaries (from bin/)
• Hosts local HTTP API on port 8080 (Node.js/Express)
• Manages mDNS discovery (bonjour-service)
• Reads/writes config via electron-store
• Communicates with renderer via IPC and preload.
b. Preload Script (src/preload)
• Exposes safe IPC channels to the renderer for:
• Fetching node list, logs, status
• Toggling theme, updating settings
c. Renderer (src/renderer)
• UI built with HTML/CSS/JS
• Displays model management, node discovery, logs, controls for starting/stopping services, theme toggle
d. Local HTTP API Server
• Listens on localhost:8080
• Implements OpenAI-compatible endpoints (/v1/chat/completions, /v1/models, /health)
• Forwards inference requests to local llama.cpp process or remote RPC nodes
e. Remote RPC Nodes (llama.cpp rpc-server)
• Each remote machine runs rpc-server on port 50052
• Performs gRPC inference tasks when called by the API server
f. mDNS Service Discovery
• Uses MDNS port 5353 to broadcast/receive cluster node announcements
• Automatically builds a list of available nodes
g. Models Folder
• Contains .gguf model files scanned at startup
• The API server loads model metadata for dropdown selection
h. Binaries Folder (bin/)
• Contains platform-specific executables for llama.cpp servers
Relationships and Data Flows
• Renderer ↔ Preload: IPC for all UI actions (node add/remove, theme change).
• Main Process ↔ Preload: IPC handlers to execute backend operations.
• Main Process → HTTP API Server: Direct instantiation or child process.
• HTTP API Server ↔ RPC Nodes: gRPC calls over port 50052 for inference tasks.
• HTTP API Server ↔ Local llama.cpp: Direct spawning of llama-server for local inference.
• Main Process ↔ mDNS: Broadcast and listen for node announcements every 30s.
• Renderer ← HTTP API (optional): for direct REST API testing (curl examples).
Architectural Patterns & Design Principles
• Electron multi-process architecture (main, preload, renderer)
• IPC boundary to secure renderer
• Modular separation: UI vs. business logic vs. external binaries
• Service discovery pattern (mDNS + manual override)
• Facade pattern: HTTP API facade abstracts local vs. remote inference
• Pluggable binaries: llama.cpp RPC server and inference server loaded per OS
Essential Elements to Capture
• Processes: Electron main, Preload, Renderer
• Child processes: llama-server, rpc-server
• Network protocols: HTTP, gRPC, mDNS
• Filesystem interactions: models folder, binary folder
• Service discovery vs. manual node management
• Configuration storage: electron-store
• Theme toggle and settings persistence