# youpen-y/rdma-mesh architecture

> Architecture diagram of the GitHub repository youpen-y/rdma-mesh, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/youpen-y/rdma-mesh
- Repository: https://github.com/youpen-y/rdma-mesh

## Overview

##### Project Type & Purpose
- From the README and file tree, note that this is a specialized RDMA communication test program. It’s designed to establish a full connection between hosts in a cluster using multi-threaded RDMA RC (Reliable Connection).
- The primary purpose is to demonstrate establishing RDMA connections as part of a master-slave (or two-host) communication test, with the potential for expansion to larger clusters.

##### File Structure
- Top-Level Files & Directories:
- Configuration and build files (.clang-format, .gitignore, Makefile, compile_commands.json, LICENSE) indicate standard project setup.
- The include directory holds the header files (msg_queue.h, rdma_comm.h, rdma_mesh.h, tools.h), which outline the interface for messaging and RDMA-related functionalities.
- Functional Directories:
- The rdma-mesh directory contains rdma_mesh.c where core functions for client and server threads (run_client() and run_server()) are implemented.
- The rdma-thread directory holds specific files for various thread functions: rdma_client.c, rdma_common.c, rdma_listen.c, and rdma_server.c. These represent the fine-grained operations for connecting, listening, and communication.
- The msg directory with msg_queue.c handles message queue operations.
- The src directory’s main.c acts as the entry point, which configures the host IP array and selects the appropriate mode (master/slave) via command-line parameters.

##### README for More Insights
- The README provides a high-level architectural overview including:
- How client and server threads are set up.
- An explanation that the master sets up one role while the slave sets up another.
- Dependencies on external libraries (librdmacm, libibverbs) which are critical for RDMA networking.
- It also defines how different threads interact – for instance, client threads post send work requests while server threads wait for and serve received messages.

a. Main Components:
- Main Application Layer: Represented by main.c in src; responsible for initialization and configuration, including loading IP arrays and command-line settings.
- RDMA Connection Module: Encapsulated in include/rdma_comm.h and include/rdma_mesh.h, along with their corresponding source files. This module handles the low-level RDMA operations.
- Thread Management:
- rdma-mesh (rdma_mesh.c): Contains the functions, run_client() and run_server(), which represent distinct client and server threads for establishing connections.
- rdma-thread: Includes individual threads such as rdma_client_thread (posting send work requests), rdma_listen_thread (continuously posting receive work requests), and rdma_server_thread (handling incoming messages).
- Message Queue System: Managed by msg/msg_queue.c and include/msg_queue.h, responsible for queuing and managing messages between threads.
- External Library Dependencies: Display dependencies to librdmacm and libibverbs, since these libraries provide the RDMA and IB (InfiniBand) functionalities.

b. Relationships and Interactions:
- That the main application triggers the initialization and starts different threads.
- Demonstrate that once RDMA connections are established, each host uses an array (cm_id_array) for inter-host communication.
- Message queue subsystem as a mediator for handling asynchronous message processing between threads.
- External dependency blocks connected to the RDMA communication module to indicate integration with librdmacm and libibverbs.

c. Major Architectural Patterns & Design Principles:
- Multi-threaded Communication: Clearly outline the separation of client and server responsibilities, emphasizing that different threads manage connection initiation and message processing.
- Modular Design: Indicate how the project cleanly separates concerns: main (initialization), RDMA setup (rdma-mesh, rdma-thread), and message management (msg).

d. Relevant Technologies & Frameworks:
- Mention the use of RDMA over Converged Ethernet (or InfiniBand) as the underlying technology.
- External libraries such as librdmacm and libibverbs along with their interaction with the RDMA modules.
- Build/debug configurations facilitated by Makefile and VSCode settings.

##### Elements
- Clear labels:

- The main entry point (main.c) triggering the network connection setup.
- The segmentation of the system into RDMA connection modules, thread handling components for client/server roles, and the message queuing subsystem.
- The integration with external libraries (librdmacm and libibverbs) needed for RDMA operations.
- Clear directional flows and relationships between modules along with proper annotations and legends.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Application Layer
    subgraph "Application Layer"
        MainApp["Main Application (main.c)"]:::app
    end

    %% RDMA Communication Module
    subgraph "RDMA Communication Module"
        RDMAComm["RDMA Connection Module"]:::rdma
    end

    %% Thread Management
    subgraph "Thread Management"
        ClientThread["Client Thread Operations"]:::thread
        CommonUtils["Common RDMA Utilities"]:::thread
        ListenThread["Listening/Receive Thread Operations"]:::thread
        ServerThread["Server Thread Operations"]:::thread
    end

    %% Message Queue Subsystem
    subgraph "Message Queue System"
        MsgQueue["Message Queue System"]:::msg
    end

    %% External Dependencies
    subgraph "External Dependencies"
        RdmaLib["librdmacm"]:::external
        IbverbsLib["libibverbs"]:::external
    end

    %% Data Flow Relationships
    MainApp -->|"initiate"| RDMAComm
    MainApp -->|"spawn"| ClientThread
    MainApp -->|"spawn"| ServerThread
    MainApp -->|"spawn"| ListenThread
    MainApp -->|"utilize"| CommonUtils

    RDMAComm -->|"supports"| ClientThread
    RDMAComm -->|"supports"| ServerThread
    RDMAComm -->|"supports"| ListenThread
    RDMAComm -->|"supports"| CommonUtils

    ClientThread -->|"send/notify"| MsgQueue
    ServerThread -->|"receive/process"| MsgQueue
    CommonUtils ---|"assist"| ClientThread
    CommonUtils ---|"assist"| ServerThread
    ListenThread ---|"assist"| ServerThread

    RdmaLib -->|"dependency"| RDMAComm
    IbverbsLib -->|"dependency"| RDMAComm

    %% Legend
    subgraph "Legend"
        L1["[Module] Application Layer, RDMA Module, Thread, Message Queue"]
        L2["-->: Data flow / dependency"]
        L3["Subgraphs group related components"]
        L4["External Dependencies are shown in cloud style"]
    end

    %% Click Events for Component Mapping
    click MainApp "https://github.com/youpen-y/rdma-mesh/blob/main/src/main.c"
    click RDMAComm "https://github.com/youpen-y/rdma-mesh/blob/main/include/rdma_comm.h"
    click RDMAComm "https://github.com/youpen-y/rdma-mesh/blob/main/include/rdma_mesh.h"
    click RDMAComm "https://github.com/youpen-y/rdma-mesh/blob/main/rdma-mesh/rdma_mesh.c"
    click ClientThread "https://github.com/youpen-y/rdma-mesh/blob/main/rdma-thread/rdma_client.c"
    click CommonUtils "https://github.com/youpen-y/rdma-mesh/blob/main/rdma-thread/rdma_common.c"
    click ListenThread "https://github.com/youpen-y/rdma-mesh/blob/main/rdma-thread/rdma_listen.c"
    click ServerThread "https://github.com/youpen-y/rdma-mesh/blob/main/rdma-thread/rdma_server.c"
    click MsgQueue "https://github.com/youpen-y/rdma-mesh/blob/main/include/msg_queue.h"
    click MsgQueue "https://github.com/youpen-y/rdma-mesh/blob/main/msg/msg_queue.c"

    %% Styles
    classDef app fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef rdma fill:#AED6F1,stroke:#3498DB,stroke-width:2px;
    classDef thread fill:#A9DFBF,stroke:#28B463,stroke-width:2px;
    classDef msg fill:#F5B7B1,stroke:#E74C3C,stroke-width:2px;
    classDef external fill:#D7BDE2,stroke:#8E44AD,stroke-width:2px;
```
