The chips almost test themselves

With the test features they build into their PCIe switches and 10-Gbps PHY devices, the engineers at PLX Technology can see things their bench instruments can't.

Martin Rowe, Senior Technical Editor -- Test & Measurement World, 4/1/2011 12:00:00 AM

Read more from the April issue.

Sunnyvale, CA—High-speed serial buses such as PCIe (PCI Express) and 10-Gbps Ethernet present significant challenges to designers of computers and communications systems. PCIe data rates of 8 GT/s (gigatransfers per second) and Ethernet data rates of 10 Gbps produce signal degradations in interconnects and PCB (printed-circuit board) traces (Ref. 1), yet system designers have difficulty measuring the degradation, because test equipment can't see inside a device.

Engineers at PLX Technology address this problem by building self-test features into the company's devices. The self-test features were also essential when the engineers developed their first PCIe switches and 10GBaseT PHYs (physical-layer interfaces), because the necessary test equipment hadn't yet reached the market. Initially, test functions inside the devices were available to PLX engineers only, but engineers learned that making these test functions available to customers helped those customers with their designs and helped applications engineers solve problems.

The PLX devices include self-test features such as traffic generators, signal processors, temperature sensors, and sampling circuits. They can perform loopback tests, BER (bit-error-rate) measurements, channel calibration, eye-width measurements, linearity measurements, and packet analysis.

No device can perform all of its own testing, however, so the PLX engineers also need some external test equipment. For example, to test the devices over a full range of PVT (process, voltage, temperature) parameters and to conduct interoperability and EMI (electromagnetic interference) tests, the engineers employ oscilloscopes, power supplies, BER testers, and thermal heads. Serial-bus products also need evaluation boards for physical-layer measurements, data-link measurements, and compliance tests.

PCIe is now in its third generation, called Gen3, which runs at 8 GT/s per lane. Gen3 data rates are up from 5 GT/s per lane for PCIe Gen2 and 2.5 Gbps for Gen1. In September 2010, PLX added 10GBaseT (10-Gbps Ethernet) PHY devices to its line-up through its acquisition of Teranetics. The devices convert XAUI (10 Gigabit Attachment Unit Interface) or XFI (10-Gigabit serial electrical interface) buses on the system side to 10GBaseT Ethernet on the line side. PLX also manufactures PCIe bridges and storage controllers for the consumer market.

Reginald Conley, PLX director of hardware applications, explained that PCIe switches, used in servers and switch products, can have up to 96 lanes grouped into ports of up to 16 lanes. PLX PCIe Gen3 switches with 48 lanes are in production, and 96-lane devices are in development (96-lane Gen2 is already shipping). Typically, 16-lane ports are used by graphics cards. Each lane uses differential signaling, which uses two PCB traces per lane. That requires a test board to have two SMA connectors per lane.

Figure 1 shows a test board for a 96-lane PCIe switch. The 192 SMA connectors give engineers access to each lane so they can make measurements such as BER, return loss, and jitter. The switches have I²C ports that give engineers access to the device for setting parameters and reading registers that contain test results.

Figure 1. A test board for a 96-lane PCIe switch has 192 SMA connectors.

Sampling from inside

The PCIe switch has internal sampling circuits that let engineers see the signal inside the device, which they can't do with an oscilloscope. At PCIe Gen2 and Gen3 data rates, the pins and bond wires inside a device package attenuate signals and add noise and jitter. Furthermore, the switch's internal signal processing takes the incoming signal, which has no eye opening, and opens the eye.

External instruments are still necessary for looking at signals at device pins, but probes can alter signals when connected to device pins or connectors. "An oscilloscope probe on a high-speed signal may clean the signal or make it worse," said Vijay Meduri, VP of engineering for PCIe switches. Conley added "A probe could filter out a high-frequency glitch and cause you to miss it."

Conley explained that the sampling circuit samples a signal's phases and adjusts the sampling based on amplitude. It then puts data about the sampled signal onto a register that PLX's visionPAK PC software can read through the I²C bus. The software uses the combination of phase and amplitude to produce an image, which is an on-chip eye diagram that represents bit errors. PLX design engineers, application engineers, and customers can use visionPAK to control the internal test functions and poll registers for test results.

To produce the eye diagram of BER measurements, visionPAK collects 1's and 0's from every lane on a switch, overlays them, and maps the eye image based on BER. Figure 2 shows the eye diagram that represents BER. Engineers can change the sampling point of the eye and use the software to calculate BER at any point in the eye. The blue area in the eye's center indicates sampling points where no errors occur. As the sampling point (colored bars) moves toward the crossing points, BER increases.

Figure 2. The plot indicates the BER across many PCIe Gen3 lanes, with the blue center indicating a sampling point with no errors. Courtesy of PLX Technology.

 

When PLX began manufacturing PCIe Gen1 switches, test features weren't available through software. Conley noted that engineers had to use jumpers on the test board to activate test features.

A PCIe switch also has loopback capability, which lets engineers use a port to test itself or another port. Figure 3 shows two loopback configurations, which engineers can configure through visionPAK. The blue arrows indicate an internal loop where a lane's transmitted signal loops back to its receiver. In an external loop (red arrows), a signal from another lane—on the same device or another device—passes back to its source. An external loop includes a transmission medium such as PCB traces and connectors. A test of an external loop can reveal problems such as impedance mismatches or CDR (clock-data recovery) errors.

Figure 3. PCIe switches have loopback capabilities for both on-chip and external tests. Courtesy of PLX Technology.

 

As with most serial buses, engineers use the loopbacks as part of a BER test that generates a PRBS (pseudorandom-bit-sequence) pattern. BER tests let engineers build confidence that a device is working, but the tests do nothing to verify that two devices can establish a data link. For that, engineers need to send PCIe packets between two ends of a lane. Internal packet generators let PLX engineers and customers test that a switch can establish a link and send error-free packets. Packets are user-configurable so engineers can force packet errors and dropped packets to see how a link responds.

PCIe has many possible errors, and a device must handle each in a consistent and specified manner. Through the
visionPAK software, a PCIe switch device can inject specific errors into a data stream at random times. The device has registers that log the errors, and engineers can interrogate the logs to find out how the device responded.

Engineers generate high-density traffic using a switch's internal packet generator, which generates layer-2 packets called TLPs (transaction-layer packets) that establish the data link between two ends of a lane. "With the packets," said Conley, "you can test a link from end point to end point."

The data that travels over a serial link has a direct effect on how much power a switch consumes. Because of that, PLX engineers have developed data patterns that force PCIe switches to consume their maximum power. "Customers want to know a device's maximum power consumption," said Meduri. "They need it for their power budgets." He explained that a PCIe switch's core logic consumes about 70% of the total power, with the PHY circuit consuming about 30%. A device's power consumption is at its highest when the maximum number of transistors is switching on and off.

Power sequencing also plays a vital role in testing. A PCIe switch uses three voltages: 0.9 V for core logic, 1.8 V for SerDes (serializer-deserializer) and I/O buffer circuits, and 2.5 V for some I/O buffers. Meduri noted that the switches must function properly regardless of the order in which power comes up, so testing must include all possible power-up combinations.

Even with all of these tests to verify performance and to let engineers characterize the PCIe Gen3 switches, it's interoperability that counts. After all, a PCIe switch is part of a larger system. For interoperability testing, engineers use test boards and PC mother¬boards that can hold several PCIe peripherals such as switches from other manufacturers, network-interface cards, and graphics cards. The PLX interoperability lab for PCIe devices has more than 100 computers so engineers can test with most PCs on the market. Software developed for interoperability testing can detect any PCIe card in the system just as an operating system does. The software can send packets between the switch under test and any peripheral.

The PHY's the thing

With its acquisition of Teranetics, PLX moved into the 10-Gbps PHY market. The company's 10-Gbps PHY device, often used in routers at data centers, is an interface between 10GBaseT Ethernet on the line side and XAUI or XFI on the system side. Dimitry Taich, director of system engineering, said that although most available products use XAUI, more and more new designs use space-efficient XFI.

The PHY device has features that help PLX engineers decide how to test it. For example, imagine that the log file on a device indicates that packet errors occur on a particular port and that the error intervals are equal to multiples of a power-supply switching frequency. This would point toward insufficient isolation between power planes and signal paths and would tell the engineers where to start their tests.

Performance tests begin, of course, with BER. "If you pass a BER test," said Taich, "then the PHY is working properly. If a device doesn't pass, then we have to study the problem."

To run a BER test, the engineers connect a traffic generator from Ixia or Spirent Communications to the device's XAUI receive channel. They then loop the 10GBaseT output from channel 1 to the input of channel 2. Figure 4 shows the setup for what Taich calls a "snake test" that tests all four of a device's ports.

Figure 4. A "snake test" lets engineers measure the BER on a PHY's four line-side and system-side ports.

 
A test for a BER of 10^-12 takes several minutes. That's what the standards require, but customers require a test for a BER of less than 10^-14 or 10^-15, which takes several days. The test uses actual Ethernet packets, not just PRBS patterns. "We drive traffic and look for broken packets," said Taich. "If we test for several days and see zero errors, we're done. If, however, a test fails, we have to go deeper and analyze test results, starting with finding which port or ports are causing the problem."

Once engineers know which port is causing the BER problem, they measure signal parameters such as jitter and driver linearity. They also look for problems related to temperature and power-supply voltage. The PHY devices have on-chip temperature sensors and can log their temperature readings. From the log data, engineers can decide if the device has a temperature problem. They can use the PHY device's internal temperature sensor to monitor device temperature. If required, they can adjust internal processing algorithms to mitigate the problem.

Temperature is just one of the variables that can affect a device's performance. PLX engineers run PVT tests that characterize the 10GBaseT PHYs. Engineers who characterize the PCIe switches and other devices also run these tests.

PVT tests involve subjecting a device to extremes, or corners, of the three variables. Varying power-supply voltage and temperature is easy to do. Changing process variables such as speed requires that manufacturing engineers vary processes when making a device. Typically, design and test engineers will ask manufacturing to vary the process for device speed, but other parameters may also need adjustment.

A PHY can't connect directly to a connector. It must go through a transformer first. The transformer provides isolation, blocks DC voltages, and matches line impedance. PLX engineers test interface devices with transformers from several manufacturers so they can recommend a transformer to customers.

PLX engineers also use evaluation boards to characterize their devices and gain extensive knowledge about how their parts work. Customers, though, design those devices into systems such as switch cards. Problems can occur there because of the interaction between components. At 10-Gbps speeds, even short PCB traces can affect signal quality. PHY devices should be placed as close to connectors as possible. A board may, however, have so many communications ports that designers cannot fit every PHY device near its connector. Taich noted that many multiport designs need two rows of PHY devices. Those devices in the second row will have longer PCB traces between them and their connectors, which can result in signal distortion compared to front-row devices.

To help solve this problem, the PLX engineers developed a test routine that lets the PHY send a test pattern to the system upon power up. The device collects reflections from connectors and analyzes the reflections. In a few tens of microseconds, the device measures the quality of each link and places the results in registers that the host system can read. The device can compensate for signal loss through channel adjustments on its internal DSP (digital-signal processor). Such a test also allows a check for missing components in the data path, thus significantly speeding up systems-level manufacturing testing.

The PHY devices also have the ability to measure port-to-port (or "alien") crosstalk. "Crosstalk is an issue because speed and noise considerations are strict," said Taich. "We developed a routine where one port sends test patterns while another is quiet. Because we know the pattern, we can analyze the noise on the quiet port to see how much energy it receives." The port can measure the noise and compare noise levels to predetermined limits based on IEEE standards.

Taich noted that PHY testing also requires engineers to run performance tests on their devices with cables of different lengths (up to 140 m) and from different suppliers. Measurements such as BER, jitter, and crosstalk take several hours, with a complete test suite taking two days to complete. To reduce test time and automate the changing of cables, PLX engineers developed a "robot" that connects a PHY under test to many cables under software control (Figure 5).

Figure 5. A robot automatically connects an assortment of cables to a PHY under test.

 
Interoperability is key

All communications devices must interoperate with those from other sources, often from competitors, so PLX engineers buy PHYs and SerDes devices from several companies in order to test their devices for interoperability. Based on the test results, the engineers can program internal logic (state machines, timers, signal-processing engines) to match a link partner's expectations and improve interoperability. Unfortunately, a setting that maximizes interoperability with one supplier's part can degrade interoperability with a different supplier's part. "It's a moving target," said Taich.

Because there are many 10GBaseT parts on the market, PLX still sends parts to the University of New Hampshire InterOperability Lab for tests (Ref. 2). "The UNH-IOL has a wide variety of test platforms, and they choose the most difficult products to test against," said Taich. "They also test for interoperability at 1-Gbps and 100-Mbps speeds, which we need to provide backward compatibility with older devices."

A great deal of testing goes into data-communications devices to ensure they comply with industry specifications and satisfy customer demands. By integrating test into the devices themselves, PLX engineers have made it possible to get their devices to market sooner.

REFERENCES

1. Kazmi, Akber, "PCI Express Gen 3 Simplified," EE Times, February, 24, 2009.

2. Rowe, Martin, "Today's testing, tomorrow's engineers," Test & Measurement World, April 2006. p. 28.