AOI systems simulate human brain

READ OTHER FEBRUARY ARTICLES:  Contents, February 2007

In terms of ease of programming, humans are hard to beat, but we do have downsides. Humans are expensive and are too slow to keep up with the frenetic beat rates of today’s assembly lines. Our eyes are not good enough to match the output of ultra-fine manufacturing processes, and we have difficulty keeping focused on very repetitive tasks. As a result, human inspectors are giving way to automated optical inspection (AOI) systems.

Of course, current AOI systems also have limitations. They are quite brittle to acceptable changes in the appearance of the boards, parts, paste, and solder joints, and the process of programming the systems can be time-consuming and expensive. Some AOI systems need constant modification to accommodate acceptable variations in the appearance of a printed-circuit board (PCB) and its components. Without constant “tweaking,” an inspection program that currently exhibits low false failures and low false accepts may change its performance adversely over time.

What can be done to make AOI systems easier to program and maintain? The answer may lie in the human brain.

Advances in brain science over the past 15 years have been dramatic, partially because inventions such as magnetic resonance imaging (MRI) machines have made it possible to see the brain in action. Researchers have learned several lessons that can be directly applied to improving the intelligence of PCB AOI systems.

Naturally, some people may justifiably be skeptical. For instance, exaggerated claims of being able to create HAL from the Stanley Kubrik movie 2001: A Space Odyssey by the year 2001 led to great disappointment. Also, researchers who wanted to model the neuron and incorporate silicon neurons (popularly known as “neural networks”) in optical inspection machines have managed to achieve only marginal performance.

But with recent advances in brain science, researchers have learned more about the functionality of the human brain and have been able to implement their findings in artificial intelligence systems. There are five principles integral to brain function that can be employed in AOI systems to make them more reliable and resilient to changes as well as easier to program and maintain.

Brains are inherently modular, with sensory information being sent in parallel to multiple areas of the cerebral cortex. These areas analyze the inputs in different ways, and their results are pooled (Ref. 1). This style of functioning is akin to a community in which each member has his or her own domain of expertise (Ref. 2). Each member’s performance might not be sufficient alone, but their collective action confers tremendous advantages to the whole assembly.

In the domain of machine learning, this idea has come to be known as “boosting,” or combining multiple, possibly weak, classifiers to produce one that is surprisingly powerful (Ref. 3). Using this approach when building a machine simplifies the design (since it dispenses with the need to create one monolithic system), and it allows the system to be more adaptive.

In the case of an AOI system, each member of the community would correspond to a software agent that specializes in a particular kind of inspection (Figure 1). For instance, one agent might look for the appearance of the object under test, another might look at its edges or transitions with the board, and a third might look for the board that is supposed to be underneath the part.

Figure 1. Separate software agents in an AOI system can look for the part’s appearance, the part’s structure, and the occlusion of the pads due to the presence of the endcaps. Other agents can look for the absence of the part (or the presence of the board). Each agent provides a decision on whether the part is Present or Absent as well as a Confidence measure. These votes can be combined to make a final decision.

The human brain never stops learning. With each new experience, it updates its world model, including tolerances on allowable variations. This ability allows the brain to anticipate variations in future inputs.

Translating this concept to AOI, a software agent could sample the board on every inspection in order to best understand the visual aspects of the parts and the board under test. The agent could handle changing conditions on an “inspection by inspection” basis rather than relying on learned or programmed conditions that may be no longer valid.

Figure 2. A PCB can be coarsely represented by just a few colors that often change across boards. This figure shows the basic color set for two boards of the same type.

Information that is examined out of context can be ambiguous. Consider an impressionistic painting, in which a paint daub that by itself just looks like a brown smudge is readily interpreted as a face when examined as part of the entire picture. This idea also applies to real-world image analysis. The brain ascribes great significance to context when interpreting images in the presence of noise and other imaging imperfections (Ref. 4).

It is widely known that there is a great deal of variation in how things look on a local level during the inspection of a PCB. Shiny parts have a range of luminances and colors and tend to reflect the image-capture system. PCBs have color changes in the mask across the board and over different substrates. In addition, the boards, parts, and paste change their color or their markings so frequently that it is difficult to tell based on local information whether the object is good or bad.

Figure 3. Without context, the 0201 device on the left looks correctly oriented. When the 0201 is examined with respect to other properly placed components, however, it becomes clear that the part is rotated.

Neurophysiological studies of cells in the visual cortex, the seeing part of the brain, suggest that many neurons encode image information far more coarsely than you might expect. A single neuron might indicate the direction of contrast over large regions (is the left side brighter or darker than the right?) rather than quantitative information like the exact location of an edge, its precise angle, or an exact color.

Why would the brain choose to throw away the fine metric data in favor of coarse relative estimates? It turns out that this is a clever strategy for building in variances. Inequalities are more tolerant to image changes than metric measurements. A representation built from qualitative codes, therefore, turns out to be more adaptable to imaging variations. A face lit in any of a number of different ways yields the same qualitative code, but very different quantitative codes (Ref. 5).

This is a valuable insight for AOI system design. Manufacturers often design their object recognition systems to look at fine details such as the precise transition between an object and its background or the interior transitions between luminance or color regions. These are known as “edges.” Other systems look at the exact colors or luminances of different parts of the image.

Figure 4. One qualitative model of a capacitor encodes regions where the endcaps are brighter than the body, the body is different from the endcaps, and the body is different from the background. This representation is satisfied when the endcaps are bright or dark, the body is brown or pink, and the board color changes. The representation correctly fails the image when the part is absent (right), because the endcap regions are not brighter than the paste, and the body is not different from the background.

Of course, some quantitative information is undeniably useful, especially for process control. A challenge of AOI system design lies in figuring out a good balance between invariant qualitative representations and sensitive quantitative ones.

Within a few months of birth, a human infant is proficient at localizing and even recognizing faces in complex scenes. What underlies such rapid learning?

One proposal suggests that evolution has equipped the brain of a newborn with a coarse face schema, which predisposes the infant to attend to patterns that are more likely to be faces. The resulting biased sampling facilitates the learning of faces (Ref. 6). This is an example of how the brain uses prior knowledge to bootstrap its learning and uses real-world experience to refine its concepts.

Figure 5. Built-in knowledge, including part geometry information, board pad information, and strategies for inspecting each known part type, can give an optical inspection system the ability to inspect parts on a PCB without ever having seen any real-world examples.

Landrex and Imagen have tested these principles on a Landrex Optima 7200 preflow inspection system. Tests done in-house and at customer sites demonstrated that use of the five principles increased the system’s performance over traditional techniques.

To understand how this works, consider an example of a capacitor whose endcaps often appear oxidized and whose body color changes from time to time. Imagine also that the board colors change from assembly to assembly.

A traditional system that has learned one appearance of the part and board may falsely fail a part when its appearance changes, or it may accept a missing part as present when the board color changes. Such a system would have to learn multiple color options and combinations in order to adapt to the changing conditions.

In the 7200, if the part color changes, then from principle 1 (collective action), some models may fail the part or be uncertain about whether it is present. Other models, however, that look for structure or occlusion of the board will be strongly convinced the part is present. This is bolstered by the fact that one of the models uses the qualitative encoding from principle 4 to look for part structure and not part appearance, which is often invariant to color or luminance changes.

Principle 2 (anticipating variations) allows the system to know the board color has changed from the sample board and to distinguish this from a part color or luminance change. Principle 3 allows the system to analyze the part in the context of the board and parts around it, thus, adding more information to the process. Principle 5 (prior knowledge) allows the system to tolerate variations by simply knowing a part is a capacitor; the engineer does not need to develop a complicated program to teach the system all possible variations and combinations.

In terms of false calls and false accepts, the 7200 is able to achieve three orders of magnitude performance difference (from 1000 ppm to 10 ppm) compared to traditional techniques, such as image comparison or edge detection. In addition, the system is able to run over large durations of time with little user intervention or changes to the program while still maintaining its low false call rate. For instance, at one customer site, the 7200 required only 15 discrete modifications over a two-week period of continuous use on roughly 5 million components. Once the modifications were made, there was no re-occurrence of the original issues.

Until recently, the capabilities of computers, cameras, and lighting were quite rudimentary, which constrained the types of images that could be collected and the processing that could be done. With the introduction of faster computers, cheaper memory, higher-resolution cameras with greater color capacity, and white LEDs, automated inspection systems can now “see better,” and they can process information in previously unimaginable ways. In the coming years, as manufacturers apply the synergies between the principles of brain science and computer vision to their algorithms, the industry should produce robust inspection systems that are easier to program and that offer more accurate assessments of PCBs.