Optimized Page Fault Handling During RDMA

A Psistakis and N Chrysos and F Chaix and M Asiminakis and M Gianioudis and P Xirouchakis and V Papaefstathiou and M Katevenis, IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, 33, 3990-4005 (2022).

DOI: 10.1109/TPDS.2022.3175666

Remote Direct Memory Access (ROMA) is widely used in High-Performance Computing (HPC) while making inroads in datacenters and accelerators. State-of-the-art RDMA engines typically do not endure page faults, therefore users are forced to pin their buffers, which complicates the programming model, limits the memory utilization, and moves the pressure to the Network Interface Cards (NICs). In this article we introduce a mechanism for handling dynamic page faults during RDMA, named PART, suitable for emerging processors that also integrate the Network Interface. PART leverages the IOMMU already present in modern processors for translations. PARTavoicls the pinning overheads, allows any buffer to be used for communication, and enables overlapping page fault handing with serving subsequent RDMA transfers. We implement and optimize PART for a cluster of ARMv8 cores with tightly-coupled network interfaces. Handing a minor page-fault of a small transfer at the destination takes approximately 38 mu secs, while there is no performance degradation when running three full MPI applications in 16 nodes and 64 cores. Detailed breakdown uncovers the hardware and system software components of this overhead and was used to further optimize the system. A 4MB RDMA transfer performs 1.46x better over pinning.

Return to Publications page