How do you treat test like a product? (Guest Commentary)

Test engineers must participate during the design process in order to ensure the quality of a design and prevent the product release date from slipping.

Richard McDonell, National Instruments -- Test & Measurement World, 1/14/2010 11:01:49 AM

Editor's note: This article is part 3 in a series on Engineering a Competitive Test Strategy. See part 1, "Do you know your true cost of test?" and part 2, "Standardization—What Does it Really Mean?"

Test engineers and magicians have a lot in common; they are both expected to pull rabbits out of a hat on command. But it doesn't stop there. Test engineers are expected to pull test systems out of a hat while juggling multiple projects at the same time. Unfortunately, many test engineers are nearing the end of their bag of tricks for ensuring test systems are released on time and under budget, especially as the time to release a test system is continually shrinking along with test engineering head count and budgets.

An increasing number of electronic manufacturers have discovered that treating test like a product is essential for engineering a competitive test strategy. This enables them to ensure the optimum quality, budget, release date, and use of test resources.

Typically, the test-development phase of the NPI (new product introduction) process begins when the design process is complete. Test engineers are given the key product specs and a set of test requirements generated by R&D to ensure the product is tested properly. This is most commonly known as the "throw it over the wall" test-development strategy. Despite being an oft-ridiculed approach and the source of many engineering jokes, this test-development strategy is still widely used. Yet, as many companies are discovering, it is becoming increasingly difficult to compete in today's marketplace by using this strategy.

There are many problems with this form of sequential test development. One of the biggest is that the process practically guarantees your product release date will slip, the project will come in above budget, and corners will be cut that will ultimately jeopardize product quality.

The target release date becomes difficult to meet because the development time allocated for the project is often exceeded, which then shortens the time budgeted for test development. As a result, costs increase due to a lack of time to research alternative instrumentation and measurements for performing the tests beyond those specified by R&D, which often uses very expensive, high-precision instruments during design verification and validation. Additionally, there is often little-to-no time for evaluating the potential reuse of test systems from previous projects, and production test times are rarely reduced due to compressed time schedules.

Finally, quality control can be compromised because of the pressure to stay on schedule and under budget. This impact on quality control can manifest itself in the form of tests that are omitted in the absence of specific instruments, test limits that are eased to avoid low first-pass yield rates, and perhaps the most dangerous of all, a test engineer's lack of understanding of the actual design of the product under test which affects his or her ability to ensure the product is being tested thoroughly.

Test offers a competitive advantage

Many leading electronic manufacturers are beginning to address these challenges by viewing test as a competitive advantage, rather than as a necessary evil. In doing so, these companies are treating test more like a product in how they plan their tests and in how they allocate resources and tools to test-system development. This approach to test significantly improves on-time delivery, minimizes budget overruns, and improves product quality through enhanced visibility and negotiation between design and test engineering.

Treating test like a product involves exactly what it implies: giving test-system development its own milestones in the NPI tracking system, assigning the appropriate personnel resources, and investing in the necessary internal and external tools to ensure proper test development. Another subtle, yet crucial, element of treating test like a product involves separating test from its position following the design phase in the sequential NPI flow and allowing test development to begin in parallel to the design flow. In some cases, test development can begin as early as the initial product design kickoff meetings.

From a planning perspective, assigning test-engineering tasks during the design phase of the NPI process helps to ensure proper alignment and planning between the design and test resources. This can save significant amounts of time and money on the back end, as it enables the development team to identify the tests to perform or omit in each phase of the product-release cycle, and it provides feedback into the design process that could make the product easier to test via more accessible test points or test modes in the firmware.

The earlier access to design engineers and product architecture also provides test engineers with insight into the design of the product and how to properly test the device in each phase of the release cycle. This is a key benefit that leads to improved quality control, because test engineers have a different perspective from design engineers, and they understand what components of a design are more likely to pass or fail when going through the manufacturing and test phases of a release.

Treating test like a product means applying more focus on test design and architecture and typically results in an improved use of common architectures and a significantly increased reuse of resources across new product designs. Most companies using this strategy have a common test software framework that all test engineers use while working within the specific product design teams. This eliminates the redundant development of test executives, drivers, and operator interfaces, and it allows test engineers to focus more on the development of test procedures and code. Design and test engineers also benefit from this approach by gaining a common understanding of the test architectures used in both validation and manufacturing test, further helping to reduce time in troubleshooting the differences between test results in each phase.

Finally, one of the biggest benefits of treating test like a product is the negotiated results from having design and test working together in parallel. The objectives of the two teams are naturally different and embody the competing forces within a company that wants to be innovative and fast to market with the latest technologies while also decreasing test system costs. Together, the design and test teams have objectives to deliver their "products" in accordance with these goals, which requires proper negotiation between the groups throughout the entire development process. Cooperating in this manner results in an ideal output for each group in regard to product features, quality, on-time delivery, and cost-control measures.

In today's competitive marketplace, companies must focus on engineering test strategies that do not rely on asking test engineers to produce magical results. By reversing the trend of treating test as a necessary evil and instead treating test as a product, the leading electronic manufacturers are enjoying the benefits of a predictable release cycle: the ability to find issues faster, manage budgets easier, and ultimately allow test engineers to focus on engineering optimal solutions.

Richard McDonell is senior group manager of product marketing at National Instruments. richard.mcdonell@ni.com