MIT teaches computers to interpret user-drawn sketches

MIT student Tom Ouyang and Professor Randall Davis are intending to create a computerized system that can interpret sketches and drawings. Initially, the foundation for this program was to enable to sketching chemical bonds and symbols into a computer, but the leaders of this project see this software as part of a larger project to make interactions with computers as natural as interactions with human beings.Ouyang explains that the only way to translate sketches and drawing data into something that a computer can understand is to use software that requires the user to select an element, such as bonds or chemical symbols, from an on-screen palette, click it, drag it across the screen, drop it into place, and then repeat the process for each successive element. That way, the computer accumulates information about the various elements that are used.

The computer can also combine information about the physical appearance of the particular symbol or bond the user is looking for with information about how it was drawn: the system can recollect the direction in which the stylus was moving when a particular stroke was made. The system then decomposes a symbol into its horizontal elements, its vertical elements, its diagonal elements and the endpoints of the strokes. Then algorithms automatically refine the components to eliminate stray marks and enhance intentional ones. The system comes to its conclusion of which particular item the user wanted through searching a database of similarly decomposed sample symbols, looking for matches.

The researchers have already developed an additional program that translates hand-drawn chemical sketches into a format recognizable by chemical-design software, and while their system recognizes standard symbols for chemical elements-H for hydrogen, C for carbon-it hasn’t yet been trained on the large number of abbreviations that chemists use for more common molecular structures-”like AC for acetyl groups, or ME for methyl groups,” Ouyang explains.