Test techniques find firmware faults

Unlike desktop-application software, embedded software has specific functions that operate in a predetermined manner. Because an embedded-systems engineer knows what computations and operations a system must perform, he or she can help ensure that the system always performs properly. Thoroughly testing the embedded-system software is key.

Engineers include many activities under the software-testing heading: performance testing, walk-throughs, timing-checks, and so on. But all these procedures fall into three main groups: demonstration, inspection, or analysis. Knowing what these three techniques can accomplish will help you choose an approach to finding problems in your firmware.

Black-box and white-box testing, two common demonstration techniques, reveal whether software works properly. In black-box testing, you have no information about the inner workings of the code. But you know its intended function, so you can predetermine the expected and correct output for any combination of inputs. To test the code, you apply every combination of input data and compare the actual outputs with the expected outputs. Wherever the code doesn't behave as expected, you can note an error condition. Black-box testing helps identify problems such as missing functions and interface conflicts, as well as misunderstood or "missing" requirements.

Black-box testing has a drawback: The number of unique input conditions can increase exponentially. Testing every combination of 12 binary inputs requires 4096 tests, and 20 binary inputs require over a million tests. In practice, many of these tests provide no useful information about the code undergoing testing. So, engineers may select only the tests they believe will properly exercise the system. Unfortunately, pruning the number of test cases often leaves portions of the software untested. The tests you include should check boundary conditions. (See, "Analyze the boundaries ," below.)

Black-box tests tend to focus on operations that run most of the time, so these tests may not check the code that handles situations such as power-up initialization, power-down processing, and error processing events that occur infrequently. Also, code that puts an embedded system in a safe mode when the system detects a fault also may go untested under normal conditions that experience no faults.

Before you can effectively monitor the outputs of a system during black-box testing, the system must work reasonably well. So, black-box testing occurs late in development—an expensive point at which to discover problems. The earlier in development that you discover a defect, the easier and less expensive it becomes to fix. A rule of thumb holds that the cost to correct an error increases by a factor of 10 at each step, as a product "moves" from analysis, to design, to development, to test, and finally to customers.

Black-box testing has a close relative: white-box testing. This type of testing takes place on one piece of software—generally a function or module—at a time, and it lets you have complete access to the software's operation. White-box testing attempts to find errors by executing every line of code and exercising every condition as you compare outputs with expected values.

During testing, you isolate the code under test and track all the details of its execution. You might monitor register use, stack use, memory allocation, and other characteristics. Your test sequences change the inputs to the test code in as many ways as needed to thoroughly cover all the code's execution paths. By testing every line of code, this method finds missing execution paths, off-by-one errors, and incorrect control flow, as well as logic and processing errors. Like black-box tests, white-box tests also should check boundary conditions.

You must understand the proper operation of a piece of software to test every one of its execution paths. Thus, some companies insist the programmer who developed the code also test it. This requirement shortens the time needed to understand how the code works, but it comes at a price. Programmers typically miss 50% of the errors in their own code. If undiscovered defects in software can prove costly, consider having someone else test it.

Of course, these demonstration techniques show only that an error occurred, so a developer must still debug the code to find what caused the error. In many cases, the programmer must correct and recompile the code before the testing can continue. And after the programmer corrects an error, the tests must be rerun to confirm proper operation of the revised code.

The inspection technique avoids some of the drawbacks associated with demonstration techniques. You might closely inspect code to find, for example, the potential for arithmetic overflow/underflow or to ensure pointers have proper values.

The inspection technique focuses on one piece of software at a time, so it can find many of the same errors that white-box testing will uncover. But inspection offers several benefits. It immediately identifies the source of error, often finding several errors simultaneously, and inspection takes place earlier in code development, so it can cost less than demonstration testing.

The following C++ macro helps show the value of the inspection technique:

#define max (x,y) (x) > (y)

? (x):(y)

The max macro accepts two values, x and y, and returns the value x, if x>y. Otherwise, it returns the value y. Assume a programmer uses the max macro this way:

max(s++, r);

It appears s is only post-incremented once, but if s>r, s actually gets incremented twice and causes an error, because the macro expands to:

(s++) > (r) ? (s++):(r);

Don't assume you know what a macro does. Always inspect the actual operation of macros used within the section of code undergoing inspection. If necessary, expand macros so you can examine their operational details.

Uninitialized local variables also can cause problems you can locate by inspection. The ANSI standard for C does not guarantee that compilers will set local (automatic) variables to zero by default. Nor do all compilers produce a warning when they find an uninitialized local variable, particularly if it lies deep within nested conditions. The code in Listing 1 contains just such a variable. Who knows what value j will contain when the program reaches the else statement? White-box testing might not detect this type of problem, but inspection stands a good chance of finding it.

Of course, inspection techniques aren't a panacea, either. Both the inspection and demonstration techniques may overlook data-synchronization problems. Suppose an electromechanical system responds to an interrupt by calling a speed-limit function that controls the system's speed based on the current gear in use. The system's software stores the current-state data in the snapshot structure shown in Listing 2.

Of course, you want up-to-date information in the snapshot structure at all times. In the first line of the following code, the gear data gets updated. The next line of code updates the speed_limit value to reflect the new gear selected:

snapshot.gear = new_gear;

snapshot.speed_limit =

speed_limit_tbl[new_gear];

If the interrupt that calls the speed-limit function occurs between the update of the gear and the update of the speed_limit data, the function may not properly keep the speed at the expected value, because the speed_limit will still contain an old (not updated) value.

Another synchronization error can occur in step-by-step calculations that produce intermediate results. In the following code, the global variable, voltage, gets changed repeatedly:

voltage = read_port_A();

voltage *= scale;

voltage += offset;

The final update of voltage does not occur until the last step. If an interrupt arrives between these three steps and its service routine uses voltage in a calculation, the service routine may obtain an incorrect result. A programmer could avoid this problem by updating voltage in a single step:

int temp;

temp = read_port_A();

temp *= scale;

temp += offset;

voltage = temp;

But this C code hides a subtle problem. Eight-bit processors store integers as 2 bytes, each of which requires a write cycle. Should an interrupt occur between the two write cycles, its service routine would read 1 byte of new data and 1 byte of old data from voltage. This type of error can elude demonstration testing, because it occurs infrequently and only appears when data changes. Programmers can prevent this sort of problem by disabling interrupts while the software updates a critical global variable:disable_interrupts();

voltage = temp;

enable_interrupts();

The problems caused by the code examples above prove difficult to find using white-box testing because they arise from actions that occur outside of the code under test. Most of the time, the embedded system behaves as expected. Only when certain data values or specific timing conditions exist do the problems appear. These transient errors prove difficult to reproduce—and correct—using testing.You might wonder, "Should I substitute inspection for white-box testing?" I wouldn't recommend it. Even the best code inspectors miss potential problems. And there's no good way to prove an inspection covered every line of code and every important combination of conditions. Both techniques have a place in the quest to uncover firmware defects.

Neither black-box testing, white-box testing, nor inspection techniques can find defects that require a system-wide approach and that require you to consider implementation details. As a result, you may choose to apply the analysis technique. An exhaustive analysis of every program detail generally proves impossible, so analysis usually focuses on problems such as stack space and interrupt latency. By using these two problem areas as examples, you'll better understand how to apply analysis.

In an embedded system, the programmer determines all memory allocations, including the space assigned to the stack. If for any reason a stack exceeds its preset boundaries—a stack overflow—it corrupts adjacent memory, which can cause a catastrophic system failure. For this reason, you must ensure the code allocates enough stack space to account for worst-case conditions.

Many programmers monitor the stack as their program runs. By writing a specific data pattern, say all 1's or all 0's, into the entire stack space at start up, they can see how large the stack grows as their software alters the preset pattern. But this information doesn't prove a stack will always have sufficient space. If a system failure poses a great risk, analyze the code to ensure a stack overflow cannot occur.

For this type of analysis, you need a detailed understanding of the code's structure, and you need an assembly-language listing of your compiled code. Typically, each task in a system operates with its own stack space. Therefore, if you have a multi-tasking application, you must analyze each task individually. Then, you can add the stack requirements to determine an overall stack need.

To start, build a call tree, as shown in Figure 1, from the root function of each task. Include every function and its worst-case stack requirement. Then, find the maximum amount of stack space used by each function and determine which path will use the most stack space. Be sure to add the maximum stack space each interrupt (if any) will use. Because higher-priority interrupts can preempt low-priority interrupts, you must add the stack space used by each interrupt priority. This analysis assumes that only higher-priority interrupts can preempt a current interrupt. If equal-priority interrupts can preempt a current interrupt, you must add the stack space needed by every interrupt at a given level. (I recommend against a design of this sort.) If your embedded system uses a real-time operating system (RTOS), remember to account for the stack space it needs.

Figure 1. Determining a program’s stack requirements requires thorough analysis of assembly-language code to determine paths that use the maximum stack depth. The highlighted paths show maximum stack use for these functions.

Because you gather stack data from assembly-language listings, you must update the program's stack-use information after every recompilation. It makes sense to perform a stack analysis toward the end of the project, when the code won't change much, if at all. Also, you must recheck the entire analysis if you use a different version of a compiler or change optimization settings during software development. If your compiler reports the stack used by each function, your task is simplified—you don't need to determine stack usage from the assembly-language listings. Unfortunately, many popular compilers still don't provide this information.

Other types of analysis include determining the worst-case interrupt latency and examining shared resources to detect any potential for deadlock. You cannot directly measure interrupt latency—the time it takes to recognize and act on an interrupt—but you can calculate it from the timing characteristics of your system. This latency value is critical if you must guarantee the real-time performance of your system under worst-case conditions.

If you use a multitasking system, perform a similar timing analysis on all the tasks. You want to ensure all interrupts or other events get serviced in enough time to properly perform associated tasks. For example, if an interrupt signals, "voltage out of bounds," the computer must react fast enough to bring the voltage back within the proper range before it causes a problem.

Your testing also may include stress or load tests. If your system processes asynchronous interrupts and handles communications with external devices, you should determine the maximum rate at which it can service these tasks. At some point, the system may spend so much time handling these external events that it can do nothing else. Test your system to see when it reaches this saturation point so you can ensure it never gets there during actual use.

You can find a helpful software-testing glossary maintained on the Embry-Riddle Aeronautical University Web site at: faculty.erau.edu/towhid/def.html.