Telecom standards bodies publish test specifications and procedures to help ensure that products from all manufacturers will communicate properly, but they provide no compliance-testing services. It's up to you to verify that your switches, routers, cellular base stations, and other equipment will work when installed into a network.

Unfortunately, many engineers make incorrect assumptions about a design—usually about software—that lead to inadequate testing. To improve the quality of your tests, you need to avoid four common traps.

Relying too heavily on self-testing is perhaps the most common testing error that telecom equipment manufacturers make. A bias can arise when the people who develop a product also test it. Because it is unlikely that members of the product-development team can adequately test every feature, you'll need to employ other engineers or labs to thoroughly test your products.

Self-testing a product is analogous to letting students make up and grade their own exams. Even when completely honest, the students may innocently conclude that wrong answers are correct. Just like students who grade their own exams, engineers can make errors that permit products to pass lab tests but fail field trials. Products based on faulty assumptions in message formats, sequences, or timings may work, but only in the lab. If you misinterpret a specification, you may develop a product that performs as designed but that won't work in real networks. Figure 1 shows how you might mistakenly program a time-out period of 300 ms instead of one that correctly meets a 400-ms specification.

Figure 1 a) If you test with call-connect times that are faster than your time-out, the telecom product will always pass. b) Using larger, variable delays produces a more realistic test.

In the self-test scenario (Figure 1a), the system under test sees a fixed 250-ms delay between sending the "Call Request" and receiving the "Call ACKnowledgement" signal. Because the time-out period is 300 ms, the call goes through and the product passes its test.

When connected to a network, however, a switch, router, or cellular base station will incur longer, variable time delays. Calls that take longer than 300 ms to complete will never connect to their destination, and the product will fail.

Engineers also fall into the self-testing trap when they test new designs using a network built from their own products. Design errors in similar products may cancel each other out. For example, you may decide to build a few extra prototypes and use them to do "back-to-back" testing (Figure 2a). I've seen a wrongly inverted signal in a prototype work perfectly when connected to a second prototype of the same kind. Not surprisingly, the product failed when connected to networks using equipment from other manufacturers.

Figure 2 a) Don't always test your products with other products you make. b) Use competitors' products or use a test system that simulates a network.

When conducting interoperability tests, you should buy or lease market-proven products for use in your test network (Figure 2b), including those from your competitors. If leading-edge products aren't yet available, you may need to simulate a telecom network by building or buying a test system.

In the rush to get products to market, engineers must quickly develop and run tests, so they "make do" with the simple test assumptions. In a common example, telecom engineers often test their products using fixed call durations (Figure 3). In the simple test (Figure 3a), a bank of 500 A subscribers calls a bank of 500 B subscribers. Subscriber A1 calls subscriber B1, subscriber A2 calls subscriber B2, and so on. All calls last 30 s. Obviously, this scenario won't adequately replicate the real world.

Figure 3 a) Don't set all call durations to the same time. b) Use varying call times, random calling, and incomplete calls in your tests.

In a more complex test, subscribers in bank A randomly call bank-B subscribers (Figure 3b), and calls randomly vary in length. This test scenario goes even further than the simple scenario in Figure 3a because it tests the switch with busy-circuit conditions as well as with completed calls. Overall, the procedure in Figure 3b will produce more comprehensive tests than the scenario in Figure 3a. Although the test scenario in Figure 3b may take longer to run than the test scenario in Figure 3a, you'll benefit from knowing that your product is more thoroughly tested.

Testing for typical operating conditions—as in Figures 3a and 3b—is fairly easy. Testing for more stressful conditions, however, is often difficult and doesn't receive enough attention. A product may perform well when receiving the expected control messages but fail miserably when messages arrive out of sequence, at the wrong time, or in the wrong format. Likewise, a product that can easily switch 100 calls per second may fail when trying to switch 150 calls per second.

You need to test your product under as many conditions as possible. One cellular-equipment manufacturer ran into difficulty when performing field trials on base stations and cellular switches using handsets made by several manufacturers. Cell sites would crash and the switch would overload, though mostly at night. But the switch never failed in the lab. Engineers discovered that the base stations became overloaded from traffic generated by one brand of handset only. A bug in that brand of handset caused it to send spurious signals when inserted into its battery charger. At night, when most subscribers charge their handset batteries, these spurious signals overloaded the base stations and brought down the whole network. Anticipating and testing for error conditions requires almost unlimited imagination.

In most product upgrades, new features get most of the attention, sometimes to the detriment of existing features. Engineers often assume that the old features work because they've worked before. Customers may forgive "glitches" in new features, but they expect old features to perform reliably. When adding features to an existing product, make sure the old features still perform properly.

How can seemingly stable features break? When correcting software bugs, you may overlook the impact these changes have on older product features and inadvertently introduce new bugs. In "Reliability Analysis of Large Software Systems: Defect Data Modeling," Levendel showed that for every three bugs that you remove, you introduce one new bug (Ref. 1). Always retest the whole body of software when you modify a program. (See "Software bottlenecks" above.)

Testing early and often can greatly reduce the overall time and expense of bringing a telecom product to market. But the testing must be "smart testing" that avoids the classic testing traps described in this article. You'll achieve faster time-to-market and better quality products.