Facing the challenge of test data management

Automotive and aerospace tests generate trillions of bytes of data, which must be securely stored yet easily retrieved by others. Regulators must access the data to review safety test reports, and design engineers need the data to improve their simulation models.

To meet the requirements for storage and access, some companies develop their own systems, while others turn to commercial software designed for test data management (TDM). Recently, I spoke with three men who work for companies that have tackled the problem of test data management:

• Tom Ferraro, DIAdem product manager at National Instruments (Austin TX; http://www.ni.com);
• Tilo Schmalz, director of sales and marketing, data management solutions for Kayser-Threde (Munich, Germany; http://www.kayser-threde.de); and
• Christian Sommer, director of department test data management systems at Werum Software & Systems (Lueneburg, Germany; http://www.werum.com).

I asked them about the problems associated with test data management, about the current state of the art, and what to look for in the future.

What applications generate the most test data?

From top to bottom: Tom Ferraro, Tilo Schmalz, and Christian Sommer.

Schmalz: Applications that make high-resolution measurements or have a high sampling rate generate the most test data. Test applications that capture images also generate lots of test data. In the automotive industry, for example, companies capture high-speed video when performing crash tests. Today's cameras can provide resolutions of 1536x1024 pixels and up to 10,000 frames/s, creating literally gigabytes of data in seconds.

Sommer: Crash tests and long-term observations are most data-intensive. While crash tests produce huge amounts of data in a very short time (spot measurements), long-term observations cause large data stocks over a long time period (environmental observations and remote sensing are two examples).

What's the most data-intensive application with which you or your company has had experience?

Schmalz: Kayser-Threde has its roots in the aerospace industry. In one project, engine manufacturer Rolls Royce required us to handle data volumes of up to 100 Gbytes/hr.

Sommer: Werum has delivered a TDM system for development, integration, and test of the cabin systems of the Airbus A 380. The TDM is a core component within the highly dynamic and adaptable research and test system set up by Airbus Deutschland in Hamburg within the scope of the project CASIV (CAbin System Integration and Verification testing). This system has to deal with up to 10 Mbytes/s for each test bed.

What are some mistakes that companies make when building their own TDM systems or when buying a data-management system?

Ferraro: The biggest problem we find is that companies don't have experience designing this kind of system. This leads to a number of bad choices: The data model they choose is not very flexible or scalable, and they don't get all the parties involved to buy into the project. One comment we frequently hear is, "I have been doing my work the same way for the last 10 years, why should I use this new system?"

Schmalz: One of the most common mistakes that companies make is not thinking in terms of processes and total costs. We see many companies that equate data management and archiving with burning data on CDs or DVDs or storing data on hard disks. The rationale behind this thinking is that the media costs are low and that the manual processes have worked in the past. Our analysis has shown that even though a CD might cost only $0.50, the total cost to create that CD is $20 or more when you include handling costs. Companies also often forget that they need to easily retrieve their data.

Another mistake is that companies don't take into consideration the long-term feasibility of their TDM system. In many cases, test data needs to be accessible for five, 10, or even 30 years. Standard IT systems have a life span of only three years, so it's obvious problems can arise. We have talked to companies who have four-year-old data on older-generation IT systems. They have determined that they cannot afford to migrate this data onto newer systems and are now keeping their fingers crossed that they will never have to access that data again.

Sommer: Our experience shows that some companies work very hard on solutions for a current problem and tend to disregard the process as a whole. They often neglect long-term aspects. Much too often, the transfer of data from the online system to an offline system is ill considered or not considered at all.

We also find that companies do not structure their data storage properly. They often group their test data by test campaign or date, but the engineers who later need to access the data want to retrieve it based on different criteria.

What should companies look for when buying TDM systems?

Ferraro: The features that I tell customers to look for include scalability, cost, and data interfaces. It's also important that it be easy to get data into and out of the system. The system must also handle many data types, including manufacturing test, engineering CAD drawings, and simulation data. Finally, world-class support and ease of administration are also crucial.

Schmalz: Companies have to realize that the solution must include a complete package of hardware, software, and the necessary integration. These components can come from different suppliers.

Companies must also look at the total lifetime costs of a system, in terms of dollars per gigabytes per month. The lifetime cost should also include system support costs and costs for migrating the data to new platforms. There will come a time when the system will have to be upgraded and storage media changed. A solution without a clear vision of how to achieve it will cost dearly in the future.

Sommer: Companies should look for a software package that offers flexibility and open interfaces. In terms of flexibility, the system must offer users the ability to search for similar tests, setups, or specific results. This means that the system must allow users to structure data and the interrelations between data. Flexibility means being open and to be prepared for restructuring and extending the data organization. At best, the users can define their own view to the data.

Open interfaces are important because you do not know what kind of data and what kind of tools you will have to deal with tomorrow. Data structures constantly change, and well-documented open interfaces, based on industrial standards, will allow you to use tools from many different platform providers.

What kinds of things should companies take into account when putting a TDM system into place?

Ferraro: Companies must have a project plan that not only outlines the rollout but also addresses how the system will grow. To do this, figure out how much data you think you're going to collect and then multiply that by 50.

The plan should also cover what will happen if the server goes down or the power fails. Do a complete failure mode analysis. It is important to have backup systems. Imagine the chaos that would ensue if your company has 200 engineers who depend on data from a single server and it is out of commission for two days.

Schmalz: Companies should work with vendors to identify where they are today with their systems and processes and where they want to go. Requirements, processes, and usage scenarios—so called "use cases"—should be identified and agreed to in writing. This document will form the basis of the specifications for the data-management system. The system that is subsequently adapted and delivered needs to be judged against the defined specifications. For sure, this methodology requires more up-front planning and time, but it tends to deliver better results in terms of successful data-management projects.

Sommer: TDM systems are long-lasting systems with steadily growing data-storage needs. That being the case, we recommend using industry-standard hardware and software to allow you to easily migrate to the next generation of storage systems. Also keep in mind that you will need an offline archive; you cannot keep all your test data online.

You also have to involve the end-user in the requirement phase. Do not focus too much on the data-acquisition aspects. Keep in mind that the users have to trust the retrievals and data. Analyze what is really necessary to store permanently. "All" is too much—no one can handle and evaluate all of the test data you'll generate.

What's the future of TDM?

Ferraro: TDM tools will become simpler to use and will have more scalable and flexible centralized storage. They will also gain more powerful interactive analysis tools.

Schmalz: TDM systems will in general become more automated and flexible. We believe, however, that the biggest advances will be in integrating test data among OEMs and Tier-1 and Tier-2 suppliers. Companies will also increasingly want to use test data in the product development process, in production, and in marketing, in order to help them produce higher-quality products.

Sommer: TDM systems will become platforms serving dedicated measurement systems. They will include a knowledge server with intelligent retrieval agents and will analyze test data, extract meta information, and build up information trees. In addition to full text retrieval, they will also allow users to search for particular data.

Contact Dan Romanchik at editor@aatr.net .