In the rapidly advancing field of networking technologies, Remote Direct Memory Access (RDMA) has emerged as a key player, revolutionizing data transfer processes and enhancing overall network efficiency. Among RDMA technologies, RoCE (RDMA over Converged Ethernet) stands out, with its second version, RoCE v2, delivering significant improvements in performance and versatility. This article delves into RoCE v2, exploring its technology, network cards, and how it compares with InfiniBand.

What is RoCE v2?

RoCE v2 is an advanced RDMA protocol designed for low-latency, high-throughput data transfers over Ethernet networks. Unlike traditional data transfer methods that involve multiple processing layers, RoCE v2 allows direct memory access between systems, minimizing CPU involvement and reducing latency. This makes RoCE v2 particularly beneficial for high-performance computing (HPC) environments, data centers, and cloud computing.

Building upon its predecessor, RoCE v1, RoCE v2 introduces enhancements that address previous limitations and improve performance. It leverages Converged Ethernet infrastructure, enabling both traditional Ethernet and RDMA traffic to coexist on the same network. This convergence simplifies network management and eliminates the need for a separate RDMA fabric, making RoCE v2 a more accessible and cost-effective solution.

RoCE v2 Network Infrastructure

RoCE Network Cards

At the heart of RoCE v2 technology are RoCE network cards, also known as RoCE adapters. These specialized network interface cards (NICs) support RDMA operations and are essential for enabling direct memory access between systems. RoCE network cards are designed to offload RDMA operations from the CPU, resulting in lower latency and improved system performance.

High-Performance Switches

High-performance switches, such as those using Tomahawk3 and the newer Tomahawk4 series chips, play a crucial role in RoCE v2 networks. These chips are widely used in switches and are vital for forwarding data efficiently in commercial networks.

RoCE v2 vs. InfiniBand

Both RoCE v2 and InfiniBand offer high-speed, low-latency communication solutions for data centers and HPC environments. Here are key differences between the two technologies:

• Physical Layer

  • RoCE v2: Utilizes Ethernet infrastructure, allowing for the convergence of storage and regular data traffic on the same network. This integration simplifies setup and reduces costs.
  • InfiniBand: Requires a dedicated fabric, separate from Ethernet, often necessitating specialized cabling and switches.

• Protocol & Network Stack

  • RoCE v2: Uses RDMA over Ethernet, integrating with the traditional TCP/IP stack, making it compatible with standard networking protocols.
  • InfiniBand: Features its own optimized protocol stack and network stack, which may require specialized drivers and configurations.

• Switching

  • RoCE v2: Operates over standard Ethernet switches with Data Center Bridging (DCB) features, supporting lossless Ethernet.
  • InfiniBand: Requires specialized InfiniBand switches designed for low-latency, high-throughput communication.

• Congestion Management

  • RoCE v2: Relies on DCB features of Ethernet switches for congestion management, but lacks built-in congestion control mechanisms.
  • InfiniBand: Includes native support for congestion management with credit-based flow control and adaptive routing.

• Routing

  • RoCE v2: Uses traditional Ethernet routing protocols like RIP or OSPF and operates within standard Ethernet topologies.
  • InfiniBand: Employs specialized routing mechanisms and supports various topologies, including fat-tree and hypercube configurations.

Choosing between RoCE v2 and InfiniBand depends on factors such as existing infrastructure, application requirements, and performance needs. RoCE v2 offers a seamless integration path into existing Ethernet networks, while InfiniBand may be preferred for high-performance computing environments requiring maximum performance and scalability.

UEC’s New Transport Protocol

The Ultra Ethernet Consortium (UEC), founded on July 19th, aims to surpass current Ethernet capabilities. With founding members including AMD, Arista, Broadcom, Cisco, Eviden, HPE, Intel, Meta, and Microsoft, the consortium seeks to develop a modern transport protocol that integrates RDMA for emerging applications. UEC argues that traditional RDMA, with its large traffic blocks, can lead to imbalances and inefficiencies, and advocates for new solutions to meet the demands of modern ML network traffic.

Summary

RoCE v2 stands out as a powerful RDMA solution, offering high-performance, low-latency data communication over Ethernet networks. Its ability to converge with existing Ethernet infrastructure, coupled with advancements from UEC, makes RoCE v2 a versatile and cost-effective choice for a range of applications, from HPC environments to cloud computing.

While comparisons with InfiniBand highlight RoCE v2’s strengths, organizations must consider their specific needs and infrastructure when selecting the most suitable RDMA solution. As technology evolves, RoCE v2 and its innovations are set to play a pivotal role in the future of high-performance networking.