PCI Express Retimers vs. Redrivers: An Eye-Popping Difference (2019)

Retimers and redrivers have enabled longer physical channels in servers and storage systems since Peripheral Component Interface Express (PCIe) 3.0 was first introduced almost 10 years ago. Now that PCIe 4.0 is ramping up and PCIe 5.0 is just around the corner, how do these reach extension tools stack up in the face of new challenges in high-speed connectivity?

A redriver is a mostly analog reach extension device designed to boost the high-frequency portions of a signal to counteract the frequency-dependent attenuation caused by the interconnect: the central processing unit (CPU) package, system board, connectors and so on. A redriver’s data path typically includes a continuous time linear equalizer (CTLE), a wideband gain stage and a linear driver. In addition, redrivers often have input loss-of-signal threshold and output receiver (Rx) detection capability. Figure 1 illustrates a typical redriver block diagram.

Figure 1: Redriver block diagram [1]

A retimer is a mixed signal analog/digital device that is protocol-aware and has the ability to fully recover the data, extract the embedded clock and retransmit a fresh copy of the data using a clean clock. In addition to the CTLE and wideband gain stages also found in a redriver, retimers contain a clock and data recovery (CDR) circuit, a decision feedback equalizer (DFE) and a transmit (Tx) finite impulse response (FIR) driver. Finite state machines (FSMs) and/or a microcontroller typically manage the automatic adaptation of the CTLE, wideband gain, DFE and FIR driver, and implement the PCIe link training and status state machine (LTSSM). Figure 2 illustrates a typical retimer block diagram.

Figure 2: Retimer block diagram [1]

In simple terms, a redriver amplifies a signal, whereas a retimer retransmits a fresh copy of the signal. Figure 3 illustrates this and shows how an attenuated eye opening is boosted by a redriver and completely regenerated by a retimer.

Figure 3: Example of an eye attenuated by a channel (left), the eye after a redriver (middle) and the eye after a retimer (right)

The PCIe 4.0 specification took the unprecedented step of formally defining the terms “retimer,” “redriver” and the superset term “repeater,” all of which are types of extension devices or components whose purpose is to extend the physical length of a link. The definitions are:

• Repeater: An imprecise term for an extension device [2]. (This term causes confusion … please don’t use it!)
• Redriver: A non-protocol-aware software-transparent extension device [2].
• Retimer: A physical layer protocol-aware, software-transparent extension device that forms two separate electrical link segments [2].

Use Cases for Retimers and Redrivers

Reach extension devices are necessary whenever the channel – the electrical path between the root complex (RC) and endpoint (EP) – is longer than the PCIe specification allows. The specification defines the maximum channel length in terms of insertion loss at the Nyquist frequency (an informative specification, but easy to validate) and in terms of a reference receiver’s ability to sufficiently equalize and recover the data assuming a worst-case link partner transmitter (a normative specification, but time-consuming to validate).

Suffice it to say, at PCIe 4.0 speeds, reach extension devices are necessary for:

• Multiconnector topologies.
• Cabled topologies.
• Single-connector add-in card (AIC) topologies with baseboard channels longer than 9.5 inches.

Figure 4 shows an example of a two-connector “riser card” topology, which ordinarily would exceed the PCIe 4.0 loss budget of 28 dB. A redriver or retimer will enable reliable, error-free communication between the RC and EP. But how do you choose which one is the right tool for the job? Well, it helps to know more about the fundamental differences in their capabilities.

Figure 4: Example of redriver (top) and retimer (bottom) used in a two-connector topology

Comparing Retimer and Redriver Capabilities

Not all redrivers and retimers are the same. There are many distinctions between the two, which are universally true for all PCIe reach extension devices. For example:

Retimers actively participate in the PCIe protocol; redrivers do not. The PCIe base specification spells out how and to what extent retimers participate in the protocol during Detect, Recovery, L0 and other LTSSM states. Equalization to the L0 and L1 link states requires value-added functionality from the retimer (handshakes, timeouts, bit manipulation, etc.). Redrivers are unaware of and unparticipating in the protocol. If the link works reliably the first time, that’s great! But if the link experiences marginality of any sort, it becomes exceedingly difficult to pinpoint whether the problem is physically before the redriver or after it, since the redriver’s role in link formation is undefined and unknown to its link partners.

Retimers reset the jitter and insertion loss budgets; redrivers do not. A retimer’s CDR fully recovers the data stream and retransmits it on a clean clock. Starting with a fresh copy of the data enables the extension of the channel to twice the original specification. Without a CDR, the best a redriver can do is attenuate (not reset) the data-dependent jitter (DDJ) caused by intersymbol interference (ISI). A redriver cannot attenuate uncorrelated or random jitter (RJ). In fact, a redriver will always add to RJ due to its own device thermal noise in a root-mean-square (RMS) manner [1].

Retimers have a DFE; redrivers do not. A DFE compensates for reflections in the channel response caused by impedance discontinuities in board vias, connectors and package socket-board interfaces. The nice thing about a DFE is that it is unaffected by crosstalk. The DFE equalizes just as well in the presence of crosstalk, and once the data is sampled by the retimer’s CDR, crosstalk is eliminated for good. Redrivers use a CTLE that boosts both the signal and the noise [1]. Crosstalk is not eliminated or even attenuated through a redriver; in fact, it gets amplified.

Retimers automatically adapt their receive and transmit equalizers to match the characteristics of the channel and the link partner’s needs; redrivers do not. A retimer will examine the signal it receives and adjust the CTLE and DFE to minimize its own bit error rate (BER). Likewise, the retimer’s transmitter will adjust its de-emphasis and pre-shoot equalization to minimize the link partner’s BER according to PCIe equalization protocol. A redriver, conversely, operates with a static equalizer setting. The optimal setting (which can be different for every channel in the system) is often painstakingly selected following an exhaustive search in Input/Output Buffer Information Specification (IBIS) algorithmic modeling interface (AMI) simulations and again in lab testing – a process fondly referred to as “tuning.”

Retimers have built-in features to help diagnose link issues (both electrical and protocol); redrivers do not. Retimers have tools for assessing the electrical performance (internal eye monitors, pattern generators, pattern checkers) and protocol performance (link state history monitors, timeout adjustments). Redrivers cannot offer such diagnostic features because they are neither protocol-aware nor aware of the actual data passing through. Redrivers do not know what state the link is in.

Retimers correct for lane-to-lane skew; redrivers do not. PCIe has a tight requirement on the physical skew between lanes on a board (1.6 ns for PCIe 4.0), typically caused by mismatches in channel routing length [3]. Retimers are required to compensate and reset any lane-to-lane skew, effectively doubling the specification budget. Redrivers cannot compensate for lane-to-lane skew, and what’s worse is that they may degrade the skew depending on how symmetric the redriver package is across all lanes.

Retimers can be placed anywhere between two PCIe-compliant channels; redrivers cannot. By definition, retimers extend the total PCIe channel reach by two times the specification. A redriver’s reach extension, however, depends on where it is placed in the channel – how much loss is before the redriver versus how much is after [1]. The specific placement of a redriver must be carefully determined by IBIS-AMI simulation and experimentation. Too close to the root complex transmitter, and the redriver’s CTLE will enter nonlinear operation and will have limited benefit. Placed too far from the transmitter, the redriver’s device noise may significantly degrade the signal-to-noise ratio (SNR) of the data signal.

It’s not all bad news for redrivers. They do have lower power consumption and lower input-to-output latency compared to retimers. But if the link does not form in the first place or if the BER is too high, none of that matters!

Table 1: Comparison of retimer and redriver capabilities and usage

Outlook for PCIe 4.0 Systems

Looking ahead to the upcoming PCIe 4.0 systems, all signs are pointing to an increased need for reach extension devices – and retimers in particular – due to several trends and challenges:

• CPUs have more PCIe lanes per socket (>100 in some cases [4]) compared to PCIe 3.0. This leads to a greater number of PCIe slots and riser cards, denser routing, and an increased use of multiconnector topologies.
• PCIe is shifting from an I/O bus to a multipurpose system interconnect. This means that more servers will be designed to be modular, allowing an array of compute, storage and networking resources to plug in to an increasing number of PCIe slots. This type of open, “plug anything in and it will work” server architecture requires a reach extension solution that is PCIe compliant with plug-and-play interoperability.
• The disaggregation of resources such as modular servers, storage trays and accelerator trays is pushing endpoints physically away from CPUs, requiring cables or carrier cards to connect everything together. These longer physical topologies will increasingly need reach extension devices.
• Systems are adopting a variety of interconnect styles – M.2, optical/copper link (OCuLink) cables and so on – all of which have unique lane-to-lane skew and crosstalk challenges that must be handled by the reach extension solution.
• PCIe 5.0 is coming right on the heels of PCIe 4.0, and pent-up demand across the industry for higher bandwidth will cause PCIe 4.0 to be short-lived. System designers are looking for a reach extension solution that can easily and quickly scale from 4.0 to 5.0.

In the end, system designers benefit from having multiple options for reach extension solutions. The exact performance requirements, physical constraints and cost targets for a server or storage system will guide the decision-making process, and the industry will benefit from knowing the trade-offs between retimer and redriver devices.

Citations

Samaan, S., Froelich, D., and Johnson, S. (2015). High-Speed Serial Bus Repeater Primer: Re-driver and Re-timer Micro-Architecture, Properties, and Usage [White paper]. https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/serial-bus-white-paper.pdf

PCI Express Base Specification Revision 5.0 Version 0.9, 2019. https://pcisig.com/

PCI Express Card Electromechanical Specification Revision 4.0 Version 0.9, 2019. https://pcisig.com/

Why AMD EPYC Rome 2P Will Have 128-160 PCIe Gen4 Lanes and a Bonus. https://www.servethehome.com/why-amd-epyc-rome-2p-will-have-128-160-pcie-gen4-lanes-and-a-bonus/

3,410

from Hacker News https://ift.tt/3eF2W0l