Unix World

Application's ( InfiniBand/libibverb based ) Memory Buffer Mapped into the RDMA (IB) Device's Address Space Steps to Map Memory Buffer into RDMA Device's Address Space Memory Allocation : The application first allocates a memory buffer that it intends to use for RDMA operations. This buffer can be used for sending, receiving, or direct memory access operations. Protection Domain Creation : A protection domain is created to define a scope within which resources like memory regions, queue pairs, and completion queues are associated. It acts as a security boundary for RDMA operations. Memory Registration : The application registers the allocated memory buffer with the RDMA hardware. This is done using the Verbs API, which provides functions to interact with RDMA devices. The registration process involves creating a memory region (MR) object, specifying the buffer's virtual address, size, and access permissions (e.g., read, write, atomic). Pinning Memory : During registrati...

 NVMe over Fabric (NVMeOf)  RDMA : NVMe-oF:  Extends the NVMe protocol to enable remote access to NVMe-based storage devices over a network fabric.   RDMA:  A high-performance transport that allows direct memory-to-memory data transfers between computers, bypassing the operating system and processor for lower latency and higher throughput.   NVMe-oF over RDMA:  Combines NVMe-oF's efficient storage access with RDMA's high-speed, low-latency transport, resulting in a powerful storage networking solution for demanding workloads.   How it works: NVMe commands are encapsulated: NVMe commands and data are packaged into "capsules" for transmission over the fabric. RDMA facilitates data transfer: RDMA protocols (like RoCE or InfiniBand) are used to transmit these capsules directly between the host's memory and the storage device's memory. Reduced CPU overhead: This direct memory access bypasses the traditional CPU-intensive data transfer path, minimizing ...

  Remote Direct Memory Access (RDMA) With the use of a technology called remote direct memory access, two networked computers can share data in main memory without depending on each other's operating systems, processors, or cache. Similar to locally based Direct Memory Access (DMA), RDMA increases performance and throughput by releasing resources, which leads to lower latency and quicker data transfer rates across systems that support it. Applications for networking and storage can both benefit from RDMA. RDMA make easy more direct and efficient data movement into and out of a server by implementing a transport protocol in the network interface card (NIC) located on each communicating device. For example, two networked computers can each be configured with a NIC that supports the RDMA over Converged Ethernet (RoCE) protocol, enabling the computers to carry out RoCE-based communications. Note, RDMA is the concept of zero-copy networking, which makes it possible to read data directly...