# b4rtaz/distributed-llama architecture

> Architecture diagram of the GitHub repository b4rtaz/distributed-llama, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/b4rtaz/distributed-llama
- Repository: https://github.com/b4rtaz/distributed-llama

## Overview

##### Main Components
- Root Node: This is responsible for loading the model and weights, conducting the primary inference computation, and coordinating the synchronization state across the network. It runs executables like “dllama inference” and “dllama-api.”
- Worker Nodes: These are additional nodes that each take care of a slice of the neural network. They run the “dllama worker” command and communicate with the root node via designated ports.
- Converter Module: Under the “converter” directory, several Python scripts exist to convert models (and tokenizers) between different formats (e.g., Hugging Face, Llama-specific formats). This component is essential in preparing the models for distributed inference.
- Core C++ Backend: The “src” directory includes all the C++ source files and headers that implement the engine – including files for neural network operations, Vulkan optimizations, and tokenizer processing.
- API & CLI Frontends: There are specific commands for chat, inference, API serving, and benchmarks offered via the provided executables.
- Supporting Documents & Examples: The “docs” and “examples” directories provide additional context and usage examples which can be referenced for deployment scenarios and conversion instructions.

##### Relationships & Interactions
- The API layer (dllama-api) and CLI interfaces (chat, inference) should be displayed as external interfaces interacting with the root node, demonstrating how external requests (e.g., chats or benchmarks) are handled.

##### Architectural Patterns & Design Principles
- Distributed/Cluster Architecture: Emphasize the design choice of distributed tensor parallelism, where the model is partitioned among available nodes. Note that the number of nodes must follow a 2^n pattern (with the root node always present).
- Separation of Concerns: Clearly separate the cluster orchestration (root and workers) from model conversion and auxiliary utilities.
- Performance Optimizations: Identify that parts of the system (e.g., high-performance computing code in C++ and Vulkan-based GPU processing) are explicitly optimized for speed.

##### Relevant Technologies, Frameworks, and Libraries
- C++ (with high-performance computing libraries), including specialized code for AVX2 and Vulkan accelerations.
- Python scripts for orchestration (launch.py) and model conversion.
- External networking over Ethernet for synchronizing the state between nodes.
- Build and deployment tools (Makefile, GitHub Actions workflows, and cross-platform build instructions for Linux, macOS, and Windows).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Components
    CM["Converter Module (Model/Tokenizer Converter)"]:::converter
    LM["Launch / Setup Scripts"]:::launch
    DE["Docs & Examples"]:::docs
    CA["CLI/API Interface"]:::api

    %% Distributed Llama Cluster Subgraph
    subgraph Cluster["Distributed Llama Cluster"]
        %% Root Node Subgraph
        subgraph Root["Root Node (Model Loader & Coordinator)"]
            RN1["Core Execution"]:::cluster
            RN2["Inference_&_API_Server"]:::cluster
        end
        %% Worker Nodes Subgraph
        subgraph Workers["Worker Nodes (Inference Unit)"]
            WN1["Neural_Network_Processing"]:::cluster
        end
    end

    %% Notes
    Note["Note:_Scaling_requires_2^n_nodes"]:::defaultStyle

    %% Arrows and Data Flow
    CM -->|"ModelDataTransfer"| RN1
    LM -->|"ModelConfigInit"| RN1
    CA -->|"ExternalControl"| RN1
    RN1 -->|"InferenceOutput"| CA
    RN2 <-->|"StateSynchronization"| WN1
    Note --- Cluster

    %% Click Events for Converter Module Files
    click CM "https://github.com/b4rtaz/distributed-llama/blob/main/converter/convert-hf.py"
    click CM "https://github.com/b4rtaz/distributed-llama/blob/main/converter/convert-llama.py"
    click CM "https://github.com/b4rtaz/distributed-llama/blob/main/converter/convert-tokenizer-hf.py"
    click CM "https://github.com/b4rtaz/distributed-llama/blob/main/converter/convert-tokenizer-llama2.py"
    click CM "https://github.com/b4rtaz/distributed-llama/blob/main/converter/convert-tokenizer-llama3.py"

    %% Click Event for Launch/Setup Scripts
    click LM "https://github.com/b4rtaz/distributed-llama/blob/main/launch.py"

    %% Click Events for Root Node Components
    click RN1 "https://github.com/b4rtaz/distributed-llama/blob/main/src/dllama.cpp"
    click RN2 "https://github.com/b4rtaz/distributed-llama/blob/main/src/app.cpp"

    %% Click Event for Worker Node Component
    click WN1 "https://github.com/b4rtaz/distributed-llama/blob/main/src/nn/nn-core.cpp"

    %% Click Event for CLI/API Interface
    click CA "https://github.com/b4rtaz/distributed-llama/blob/main/src/dllama-api.cpp"

    %% Click Event for Docs & Examples
    click DE "https://github.com/b4rtaz/distributed-llama/blob/main/docs/HUGGINGFACE.md"

    %% Styles
    classDef cluster fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef converter fill:#A9DFBF,stroke:#117A65,stroke-width:2px;
    classDef launch fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef api fill:#F5B7B1,stroke:#922B21,stroke-width:2px;
    classDef docs fill:#D2B4DE,stroke:#5B2C6F,stroke-width:2px;
    classDef defaultStyle fill:#E8F8F5,stroke:#1F618D,stroke-width:2px;
```
