Average Rating:

(6)

Rate this:

Test system upgrades can expose problems

Traps can lurk below the surface when you upgrade hardware or software.

By Mike Rutledge, EADS North America Test and Services -- Test & Measurement World, 9/1/2009 2:00:00 AM

In industries such as the defense industry, test systems can remain in service for 20 years or more. The engineers who maintain such systems keep them going by repairing and replacing instruments and components that malfunction or fail. In the past, these engineers could count on identical replacements being available for many years, but now manufacturers frequently redesign or update their products in response to market pressures. The result has been a decline in the life span of each of the key components in test systems.

For instance, CPU life spans are now measured in months, rather than years. Associated operating systems likewise have much shorter life spans. Instrument manufacturers who once operated under a 10- to 15-year product life cycle are now in the 3- to 5-year mode. Thus, the issue of upgrading and managing test assets is becoming an ongoing concern, as opposed to a one-time activity during system development.

Just because parts of your test system need an overhaul does not mean that you can—or should—develop an entirely new system. You may merely need to upgrade some of the instruments or the software. Because upgrading a test system is a business decision, you must determine what upgrades are essential to your business needs and whether the expected improvements justify the investment in new equipment and new programming. And when undertaking an upgrade, be prepared to uncover new problems or to revisit problems for which you found a workaround in the past.

Preserving legacy elements

A test system relies on its application programs, typically in the form of TPS (Test Program Sets), to carry the workload of the platform. A typical TPS development process is shown in the figure. (Go to the figure at the end of this article.) The terminology in the figure reflects US Air Force jargon, so the term ITA (interface test adapter) is used for fixturing, as opposed to ID (interface device) or TUA (test unit adapter). (See a list of acronyms.) Fixturing, here, consists of the interconnectivity between the test resource and the UUT (unit under test).

The TPS development process is the legacy of the test system that you must either preserve or move to a new test asset. To the extent that you can move each box in the figure without change, you preserve investments, reduce cost, and mitigate risk for the upgraded platform. For each box that you must rework, the opposite is true: Past investment becomes sunk cost, expenditure increases, and risks are exposed. Thus, if you move to a totally new platform with completely new software, you are, in essence, performing a new development and not a migration.

The first task in any test system upgrade is to make sure you fully understand and define the problem you need to solve, such as replacing obsolete instruments or parts, expanding capability, improving poor performance, or adapting the system for a new purpose. Once you establish these parameters, you have a strategic direction on which to base decisions.

To define the problem, a systems approach is often useful. The primary issues might be component obsolescence; withered test assets, computer resources, and peripherals; high operational costs; and the need for additional capabilities that new technologies can provide. (For example, you might want to insert new technology to improve information management.) The interfaces to the fixture might be showing signs of wear and tear after years of use, or the UUT itself might have been upgraded with capabilities that are difficult or cost prohibitive to test with the existing test system.

Questions you need to consider include:

• What is the process of validating the resultant new test-system product or products?

• Will I need to complete a comprehensive regression test and fully qualify the new test-system product or products? (Regression testing and qualification can easily cost more than the test assets being replaced, depending on the workload of the test set.)

• Can I demonstrate with a high degree of confidence the traceability from the current fielded baseline to the modified test system with little or no risk?

Once you have defined the problem, you have several possible courses of action:

• Modify the system while keeping the identical capabilities to the current system. A product or item manager might say, “I want what I have, no more, no less. I'm modifying the system only because test equipment is no longer available that's a direct replacement.” Such managers probably are not “test guys.” They want capabilities at the lowest cost.

• Develop a system with a few enhancements, such as better speed or accuracy. Managers who request this option generally have some test savvy, know that bugs exist, or perhaps sense that certain aspects of their product are not perfect. This type of position uses obsolescence as a reason to get problems fixed.

• Develop a completely new system. A manager who asks for this wants today's technology in a system that will last another 20 years. The UUT will limit test system performance because of communications buses.

“Can do better” is a trap

Faster computers or test equipment can expose timing latencies that make newer test systems less compatible with UUTs than a legacy system. Many legacy test systems were designed specifically to test the asset; they did not result from efforts to adapt general-purpose COTS (commercial-off-the-shelf) products to a specific need. These legacy systems were usually highly optimized for that one application, and improving the efficiency of the test system without modifying the design of the UUT can create timing conflicts.

During an upgrade, you will often find timing problems that were masked by the architecture of the legacy system. As processing power removes latency from the system, some settling-time issues with respect to stability or power might emerge. Potential problems might appear in something as simple as a step attenuator in the output of an RF signal, or they might appear in a current-sensing function monitoring a power supply's initial surge during power up. To ease these types of issues, you must fully understand the problem and apply systemic solutions, or the TPS will once again become attached to the particular test set.

Other timing problems may occur when better processors communicate with instrument buses at faster rates than the legacy processor could support. If the test executive does not provide hooks to manage this type of problem, you may need to modify the system drivers so they can support older products. The PAWS Run Time System development software, for example, provides for multiple ways to communicate with instruments and allows bus definitions to account for slow assets.

You might think that because the new test system offers 20, 40, 60, or more times the power of the legacy system, that the throughput should increase significantly. This expectation ignores the fact that the UUT is not 20, 40, 60, or more times more powerful, but remains as it ever was. Thus, while it is not unusual for newer systems to improve throughput, do not expect significant improvement to be the norm without a thorough analysis and quantification of exactly where these improvements will occur.

Hardware issues

The selection of a hardware architecture is one of the first decisions you will make. The increasing power and speed of a variety of buses and controllers provide a multitude of options. There are instrument interfaces based on Ethernet, VXI, PCI, USB, and GPIB, to name only the common ones. Each of these interfaces has advantages and disadvantages (number of vendors, degree of openness, conformance to open specifications, speed, durability, environmental considerations, and so on). Each of these will drive approaches to modularity, sparing and logistics, and maintenance. As you investigate these factors, you will need to address the following issues:

• Old system functionality. Did the designer of the legacy system use instruments in ways that were unorthodox (for example, using a digital multimeter's four-wire resistance-measurement feature as a current source in a way that is undocumented)? Can you duplicate that with new equipment? How?

• Performance envelope. Does the existing system operate at the corners of the test equipment? Can you characterize the entire performance envelope? Can you duplicate the performance envelope with automated software to minimize regression testing?

• System asset improvements. Will the new hardware with better specifications uncover problems masked by the legacy equipment? For example, an improved noise floor might uncover spurs that were not seen before, or the new hardware could present timing issues related to high-speed communication devices, raising bus conflicts and bus-settling issues.

• Interfacing and fixturing. Can you use legacy fixtures, or will you need new fixtures? If you need new fixturing, how can you characterize the performance of the device? Will new instruments detect crosstalk and noise that the older system ignored? Can you develop a standard system interface (ARINC 408 or CTI IEEE 1505) and adapt legacy ITAs to this with an adapter-adapter?

• Grounding, cooling, and mechanical issues. How was grounding implemented in the legacy system? Were floating grounds available to enable the use of the UUT ground as the reference? Can the new instruments provide a floating ground or are they tied to system ground?

• Fundamental clock stability and noise. If you need to phase-lock to a clock signal, can you? Can you use the UUT as a reference for phase measurements?

Software issues

The software architecture issues can be even more critical than the hardware architectural issues, because most commercial software vendors provide support for the current versions and perhaps one to two previous versions of their products. Therefore, for a 15- to 20-year life cycle, the software must either migrate with the technology continuously or be archived and supported with a specialized team at relatively high expense. In any event, you need to develop a strategy for migrating the software technology, or else you need to establish a baseline and hold it.

Software architectures today typically have some layer of abstraction above the primary instrument interface (vendor driver) to aid in the management of obsolescence and limit the dependence of the TPS to the particular instrument. This “middleware” layer can be used for system functions such as calibration factors, cross-instrument communication, and resource management to enable a programmer to adapt COTS resources to particular requirements. Unfortunately, this layer of abstraction does not engender enthusiasm from vendors, because it enables the integrator to treat instruments as commodities. The test industry has attempted to establish layers of abstraction in initiatives including VISA, VXIplug&play, IVI, and synthetic instruments, and it will continue to work on implementing these concepts.

To the degree that the software layers are “open” and not vendor-dependent, the system will be more supportable and susceptible to technology insertion and migration. For example, an IEEE language (such as ATLAS) is not subject to a particular business cycle and will likely provide a longer life than the current fad—although the current fad can often become a standard (see the history of the C language). The tradeoff is usually expediency (graphical or cool languages) versus supportability.

In either event, team discipline and communication is critical. As with hardware, there are issues to consider:

• Rewrite. Should you rewrite test code into a more modern language or try to use existing code? Can you convert existing code?

• Programming assumptions. What programming assumptions were made in the old software code?

• Business or environment assumptions. Elements of legacy code were probably based upon common understandings of software developers at the time they created the legacy code. These assumptions may not be documented, and they may not be obvious today.

• Data. Supporting data and documentation may be tied to the legacy code explicitly, and changing the code can result in significant costs to update and manage the associated data.

• Instrument interchangeability. The ability to interchange instruments combines hardware and software issues.

Role of virtual instrumentation

Virtual, or synthetic, instrumentation has been the topic of much discussion in the test industry. When deciding to replace a test instrument with virtual instrumentation, you need to evaluate how the legacy software took advantage of the characteristics of the legacy instruments, either knowingly or unknowingly. The specifications of the legacy system and its components may not describe the full envelope of performance that was assumed by the legacy applications, and this will create an unforeseen hurdle for migration.

Programmable, modular instruments, like the Talon Instruments T964 digital test resource module from EADS, provide the integrator with flexibility to respond to unforeseen issues as they emerge. The integrator can program such modules to fit into an existing test system rather than having to reprogram the entire system to accept a new instrument; this reduces the risk and cost associated with inserting new equipment and maintaining a legacy system. The T964 has the ability to mimic leakage current, for example, as a way to minimize any changes to the existing legacy TPS base.

As the current inventory of legacy test systems continues to age, more and more equipment will need to be migrated, either to a new platform or to a new computer. A systematic approach should help to identify risk areas. Minimizing the elements of the legacy system that are changing will reduce the cost and risk of replicating the existing performance. Modern tools and virtual-instrument assets are a means to help reach the end of a more supportable and sustainable test capability that preserves past investment to the maximum extent.

Figure.   A test system relies on its application programs, typically in the form of a TPS, to carry the workload of the platform. The terminology shown here reflects US Air Force jargon, so the term ITA (interface test adapter) is used to refer to fixturing. Fixturing, here, consists of the interconnectivity between the test resource and the UUT. See a list of acronyms.

Average Rating:

(6)

Rate this:

Talkback

» Submit Feedback

• 
  

  The introductory part of this article uses the noun "life cycle" as if availability for OEM purchase or post-purchase support is the sole criteria for. Some metrology labs will not provide calibration services for instruments that are identified as no longer supported, but to state that a CPU has a life measured in months or an operating system in years is confusing, when they are both very likely to continue OPERATING without needing replacement or upgrade for years if not decades.

  
  

  
  Shawn Riley - 2009-14-9 17:21:00 EDT

• 
  

  i onced worked as test engineer in production / mfg test for about 8 years. the number of OEM products was about 300, each with their own options. however, most of these products were LOW quantity. it was my job to reverse engineer, redocument test fixtures,test procedures to comply to ISO 9000 requirements.
  
  when the original test fixtures were designed, the interaction between design engineer and production engineer was direct. Test fixture / Test procedure were 'passed over the office partition' once they were done with it in engineering, then pushed onto production test. However, the interaction became much less in the years following. much of the time, the product/test engineer who designed the test fixture/test procedure has since left the company.
  
  lessons learned here:
  
  when you reverse engineer existing test systems that have been passed down, one needs to look at the what the product line is doing currently to justify how far you need to take it. if some particular products are 'going obsolete', then update is probably not worth the effort.
  >was the test system designed and engineered properly?
  >if not, what cheap alternatives are there today to connect to UUT for test?
  >do we need to make the system more robust h/w wise? >what type of s/w upgrades need to occur?
  suggestions:
  attempt to engineer a 'universal test fixture' that has at its heart an FPGA and test controller on desktop pc running say C# .NET control GUI. build test adapters that mate to all the associated UUT where the wiring on the test fixture side of all of them ARE THE SAME.
  
  although i am currently unemployed, between jobs, i design and build semi-automatic test systems for validation of end product. i design the c#.NET control gui, and use various interfaces to the unit under test (microcontroller, CPLD, FPGA , USB, VCP USB) and remote control of test equipment using IVI/SCPI under c#.
  
  i am currently looking for this type of work locally here in the Pacific Northwest in the Puget Sound region.
  
  my two cents -

  
  

  
  Ron Harding - 2009-9-9 13:32:00 EDT
» Submit Feedback

Similar Content from T&MW

• Similar Topics
• More by this Author
• Top Rated

Test-system development: Do everything first

02/01/2005

Embedded Controllers Need Flexible Test Systems

09/01/1999

Tools to tackle costs and complexity (continued)

05/31/2009

Reduce test costs with careful PXI design

04/30/2009

Test system leverages PXI flexibility

03/26/2006

Also by this author

• No articles available

• FLIR offers IR camera for under $3000

• Don't let the economy compromise quality (Guest commentary)

• Danaher speeds up restructuring, acquires life-sciences businesses

• Agilent’s Cover-Extend technology eliminates need for physical test points for in-circuit test

• So many combinations: Testing a switch-matrix board

Sponsored Links

• Resource Center
• Browse Categories
• Browse Companies

Featured Company

• Texas Instruments

  

  Texas Instruments serves the world's most innovative electronics companies, helping them develop new ideas that change the way we live. By providing semiconductor technologies that promote greater power efficiency, enable more features, enhance... more

• Fluke Corporation

  

  Fluke Corporation is the world leader in the manufacture, distribution and service of electronic test tools and software. Since its founding in 1948, Fluke has helped define and grow a unique technology market, providing testing and... more

---

Related Resources

• Learn more about how the Multimedia Test System (MMTS) can solve...
• Best Practices for Designing Automated Test Systems
• How to Test MIMO Devices Via Channel Sounding?