Tests ride the PCI Express

Because it's a serial bus, PCIe requires testing at the physical layer (PHY) and at the protocol layers above it (Figure 1). Each layer must comply with many pages of specifications published by the PCI Special Interest Group (PCI SIG, www.pci-sig.com). The PCI SIG also publishes a series of compliance checklists and organizes workshops where you can run interoperability tests. Before attending a workshop, you can perform tests in your lab using the checklists as a guide. (See "Checklists and workshops")

Figure 1.  PCI Express uses two protocol layers on top of a physical layer.

Start at the PHY

Successful PCIe interoperability starts at the PHY, which consists of logical and electrical sub-blocks that manage data encoding and decoding as well as lane configurations and variations. They also cover a myriad of electrical signal specifications. PCIe products must link at the PHY layer before higher layer connections—and data transfer—can occur.

PHY testing begins with the differential signal lanes on the PCIe bus. Transmitted (TX) bits come in two versions: transition bits (Figure 2) and nontransition bits (Figure 3). For each, you need to analyze data streams by using eye diagrams to measure their amplitude and jitter.

You can test the signals on TX lanes for electrical compliance by using an oscilloscope with a differential probe. You can generate a compliance-test pattern from any PCIe device through a device's state machine. To invoke the test pattern, terminate the TX lane with a 50-V resistor or 50-V oscilloscope probe. If the device's receiving (RX) lane has no activity on power up, the state machine will transmit a compliance test pattern.

Figure 2.  Transition bits, which occur when a bit changes state, form a complete eye opening.

Figure 3.  Nontransition bits must comply with eye-diagram specifications such as eye opening and amplitude.

To make eye-diagram measurements, you need to generate 3500 unit intervals (UIs). Use an oscilloscope to make the measurements on the 250 UIs that occur in the center of the 3500-UI pattern. A compliant eye opening will show transition bits with a minimum differential amplitude of 800 mV_p-p. Nontransition bits should have a minimum differential amplitude of 505 mV_p-p. But higher amplitude values aren't necessarily better. Compare the signal's eye opening and amplitude to specified limits.

A perfect eye, if one existed, would have an opening of 1.0 UI. PCIe specifications allow no more then a 30% loss from 1.0 UI (0.7 UI or greater) for TX, and they require 40% (0.4 UI) or larger UIs at a receiver's input. The 30% loss is a jitter budget allocated for board layout and other possible loss contributors between a transmitter and a receiver.

PCI SIG specifications require that you test a receiver's ability to extract a clock that's embedded in transmitted data. Like several other serial buses, PCIe uses 8b/10b encoding to minimize long runs of the same bit, thus making transition bits frequent enough for a receiver to extract the clock. "PHY encoding," explains why 8b/10b character encoding improves a receiver's ability to recover a clock from a data stream.

PHY problems can appear as errors at the protocol layers. For example, problems in the electrical portion of the PHY will often result in invalid characters or disparity errors. You can use a protocol analyzer to track such errors over a specified time interval.

A protocol analyzer can simulate a PCIe device by sending two transaction layer packets (TLPs) to a device under test (DUT). The packets should generate several responses from the DUT across all three layers, and the protocol analyzer can check these responses for errors. You should perform this test before diving into other protocol tests to make sure that the link's physical integrity is intact.

Once you complete the electrical and logical PHY tests, you can move to protocols. Testing progresses up the protocol stack, through the data-link and transaction layers, where you must verify the operability of devices working under control of their drivers and applications.

The PCIe data-link layer ensures the integrity of communications between the two directly communicating devices. The data-link layer serves many functions, including:

• flow control initialization,
• flow control updates,
• ACK/NAK protocol,
• power management control,
• cyclic redundancy checks (CRCs),
• transaction sequencing, and
• replay of failed transactions.

Communications between PCIe devices occur in data-link-layer packets (DLLPs). In addition to providing origin and terminus to various DLLP communications, the data-link layer adds or removes information and checks for proper transaction sequencing and data integrity. When a PCIe transaction passes from the transaction layer to the data-link layer, the data-link layer appends it with a sequence number and a 32-bit link CRC (LCRC).

A PCIe receiver uses the sequence number to verify correct transaction ordering, and it checks the LCRC to verify error-free data. If a receiver detects sequencing or packet-integrity errors, it will instructthe transmitter to resend the failed transaction. If a device receives DLLPs without error and in the proper order, the receiver strips away the sequence number and LCRC fields before it passes the packets to the transaction layer.

As part of a PCI SIG compliance test, you need to test a device's ability to react to malformed packets, including whether the device properly reports errors to system registers through the root complex. See Figure 4 for the compliance test setup. A PCIe bus exerciser acting as a bus endpoint, and under control of an automated compliance test script or used manually, emulates a transmitting device. With the exerciser, you can force noncompliant data-link-layer responses, transaction-layer responses, or no response.

Figure 4.  A typical test setup for a PCI Express device requires setup, stimulus, and response packets. An oscilloscope lets you measure jitter and eye openings.

The PCIe transaction layer encapsulates data-link-layer information and control characters into TLPs that move information from point to point and throughout the PCIe fabric. These transactions echo PCI and PCI-X transactions, with a message transaction added for PCIe communications. Message transactions, used for power management, interrupt signaling, hot-plug support, error reporting, and vendor-specific purposes, eliminate the need for sideband signaling.

The transaction layer performs various error checks, including:

• TLP packet format,
• timeout errors on completion packets,
• flow control operation,
• notification of unsupported requests,
• data corruption (through data poisoning),
• end-to-end CRCs (ECRCs), and
• unexpected completions.

Transactions at this layer come in two variations: posted or nonposted. Memory writes and messages are posted transactions, which don't require transaction-layer completion packets. Memory reads, I/O reads and writes, and configuration reads and writes are nonposted, and thus require a completion packet. Nonposted transactions are modeled similarly to the PCI-X split-transaction protocol (request/completion).

Compliance testing at the transaction layer requires that you simulate a device and verify that the DUT responds properly to errors. A protocol analyzer, for example, must intentionally transmit a packet with a header bit set to indicate data poisoning, with the analyzer programmed to provide a pass/fail indication based on the receiving device's response.

The error-poisoned (EP) bit in a TLP's header indicates corruption of payload data. Devices may send a corrupt payload in various ways. A requester can fetch data from memory that may include a parity error or internal data corruption. Device responses to a poisoned packet might include a returned completion TLP with a status field set to unsupported request (UR), appropriate updates to error registers, or the disposal of the received payload data.

A formatting error associated with the packet will produce a malformed TLP. Examples include mismatches between the payload size and the header's payload length field, byte-enable violations, incorrect type field values, and improper transaction routing. Each possible malformation requires an individual test case to confirm compliance.

The transaction layer also handles proper flow control, which ensures that a device won't send data across the link until a receiver is ready to receive. The transaction layer must ensure that a receiver has enough buffer memory to handle the next transaction. Protocol errors can cause a device to refuse to send transactions or to overrun buffers on the receiving device. You need to test flow control for proper passing of data as part of your compliance tests.

Flow control is important because different types of packets need different priorities. A time-critical application such as video needs a higher priority than data retrieved from a hard drive. PCIe devices use the virtual channel/traffic class (VC/TC) concept to prioritize data, which further complicates compliance testing. Application programs and device drivers make the TC assignments and allocate memory to VC buffers (hardware on switches, endpoints, and root complexes). PCIe switches route traffic based on the VC/TC mapping.

You can test prioritization once a PCIe bus exerciser or application software classifies data to a given TC. Use a protocol analyzer to verify the packet header has the proper 3-bit TC tag. Then, once a PCIe device maps data to the VC configured for that TC, use the analyzer to view the data.

Compare the TC characteristics of this egress data to the source data and verify the expected output. The expected TC packets should match each type of known ingress TC packets and the relative weighting of TC to VC.

A variety of physical and logical factors can produce data-transmission problems. Examples include marginal quality of the transmitted data, corrupt physical media, marginality at the receiving device, or a noncompliant optional reference clock. But with thorough testing of all layers of the PCIe protocol stack and their interactions with drivers and application layer software, you can minimize interoperability problems.