How youpen-y/rdma-mesh works
Project Type & Purpose
• From the README and file tree, note that this is a specialized RDMA communication test program. It’s designed to establish a full connection between hosts in a cluster using multi-threaded RDMA RC (Reliable Connection).
• The primary purpose is to demonstrate establishing RDMA connections as part of a master-slave (or two-host) communication test, with the potential for expansion to larger clusters.
File Structure
• Top-Level Files & Directories:
• Configuration and build files (.clang-format, .gitignore, Makefile, compile_commands.json, LICENSE) indicate standard project setup.
• The include directory holds the header files (msg_queue.h, rdma_comm.h, rdma_mesh.h, tools.h), which outline the interface for messaging and RDMA-related functionalities.
• Functional Directories:
• The rdma-mesh directory contains rdma_mesh.c where core functions for client and server threads (run_client() and run_server()) are implemented.
• The rdma-thread directory holds specific files for various thread functions: rdma_client.c, rdma_common.c, rdma_listen.c, and rdma_server.c. These represent the fine-grained operations for connecting, listening, and communication.
• The msg directory with msg_queue.c handles message queue operations.
• The src directory’s main.c acts as the entry point, which configures the host IP array and selects the appropriate mode (master/slave) via command-line parameters.
README for More Insights
• The README provides a high-level architectural overview including:
• How client and server threads are set up.
• An explanation that the master sets up one role while the slave sets up another.
• Dependencies on external libraries (librdmacm, libibverbs) which are critical for RDMA networking.
• It also defines how different threads interact – for instance, client threads post send work requests while server threads wait for and serve received messages.
a. Main Components:
• Main Application Layer: Represented by main.c in src; responsible for initialization and configuration, including loading IP arrays and command-line settings.
• RDMA Connection Module: Encapsulated in include/rdma_comm.h and include/rdma_mesh.h, along with their corresponding source files. This module handles the low-level RDMA operations.
• Thread Management:
• rdma-mesh (rdma_mesh.c): Contains the functions, run_client() and run_server(), which represent distinct client and server threads for establishing connections.
• rdma-thread: Includes individual threads such as rdma_client_thread (posting send work requests), rdma_listen_thread (continuously posting receive work requests), and rdma_server_thread (handling incoming messages).
• Message Queue System: Managed by msg/msg_queue.c and include/msg_queue.h, responsible for queuing and managing messages between threads.
• External Library Dependencies: Display dependencies to librdmacm and libibverbs, since these libraries provide the RDMA and IB (InfiniBand) functionalities.
b. Relationships and Interactions:
• That the main application triggers the initialization and starts different threads.
• Demonstrate that once RDMA connections are established, each host uses an array (cm_id_array) for inter-host communication.
• Message queue subsystem as a mediator for handling asynchronous message processing between threads.
• External dependency blocks connected to the RDMA communication module to indicate integration with librdmacm and libibverbs.
c. Major Architectural Patterns & Design Principles:
• Multi-threaded Communication: Clearly outline the separation of client and server responsibilities, emphasizing that different threads manage connection initiation and message processing.
• Modular Design: Indicate how the project cleanly separates concerns: main (initialization), RDMA setup (rdma-mesh, rdma-thread), and message management (msg).
d. Relevant Technologies & Frameworks:
• Mention the use of RDMA over Converged Ethernet (or InfiniBand) as the underlying technology.
• External libraries such as librdmacm and libibverbs along with their interaction with the RDMA modules.
• Build/debug configurations facilitated by Makefile and VSCode settings.
Elements
• Clear labels:
• The main entry point (main.c) triggering the network connection setup.
• The segmentation of the system into RDMA connection modules, thread handling components for client/server roles, and the message queuing subsystem.
• The integration with external libraries (librdmacm and libibverbs) needed for RDMA operations.
• Clear directional flows and relationships between modules along with proper annotations and legends.