How b4rtaz/distributed-llama works
Main Components
• Root Node: This is responsible for loading the model and weights, conducting the primary inference computation, and coordinating the synchronization state across the network. It runs executables like “dllama inference” and “dllama-api.”
• Worker Nodes: These are additional nodes that each take care of a slice of the neural network. They run the “dllama worker” command and communicate with the root node via designated ports.
• Converter Module: Under the “converter” directory, several Python scripts exist to convert models (and tokenizers) between different formats (e.g., Hugging Face, Llama-specific formats). This component is essential in preparing the models for distributed inference.
• Core C++ Backend: The “src” directory includes all the C++ source files and headers that implement the engine – including files for neural network operations, Vulkan optimizations, and tokenizer processing.
• API & CLI Frontends: There are specific commands for chat, inference, API serving, and benchmarks offered via the provided executables.
• Supporting Documents & Examples: The “docs” and “examples” directories provide additional context and usage examples which can be referenced for deployment scenarios and conversion instructions.
Relationships & Interactions
• The API layer (dllama-api) and CLI interfaces (chat, inference) should be displayed as external interfaces interacting with the root node, demonstrating how external requests (e.g., chats or benchmarks) are handled.
Architectural Patterns & Design Principles
• Distributed/Cluster Architecture: Emphasize the design choice of distributed tensor parallelism, where the model is partitioned among available nodes. Note that the number of nodes must follow a 2^n pattern (with the root node always present).
• Separation of Concerns: Clearly separate the cluster orchestration (root and workers) from model conversion and auxiliary utilities.
• Performance Optimizations: Identify that parts of the system (e.g., high-performance computing code in C++ and Vulkan-based GPU processing) are explicitly optimized for speed.
Relevant Technologies, Frameworks, and Libraries
• C++ (with high-performance computing libraries), including specialized code for AVX2 and Vulkan accelerations.
• Python scripts for orchestration (launch.py) and model conversion.
• External networking over Ethernet for synchronizing the state between nodes.
• Build and deployment tools (Makefile, GitHub Actions workflows, and cross-platform build instructions for Linux, macOS, and Windows).