Rick Nelson

Criticizing automated umpires: throw the 'bots out?
September 4, 2008

Should technology replace human referees in sports events? Farhad Manjoo tackles that question in Slate this week in an article titled “Hey, Robot Ref! Are You Blind?” Manjoo’s article is partly inspired by Major League Baseball’s recent introduction of instant replay for certain “boundary calls.” But he focuses extensively on the Hawk-Eye system used in tennis, which “stitches together video footage from several high-speed cameras to produce a 3-D simulation” of a tennis ball’s trajectory. “Tennis,” he writes, “adopted Hawk-Eye after several high-profile matches were marred by bad calls."

But just how accurate is the Hawk-Eye system? Writes Manjoo, “Late in the 2007 Wimbledon final between Rafael Nadal and Roger Federer, Nadal hit a deep ball that Federer let go, thinking it was out. The umpire thought so, too, and TV replays showing the ball from Federer's side seemed to confirm it—the ball looked a good half-inch out.” Nevertheless, Hawk-Eye awarded the point to Nadel. Facing criticism, Hawk-Eye Innovations posted an explanation on its Website—one that I find not totally convincing.

One can acknowledge that an automated system might make the occasional error but concede that robotic umpires will make fewer mistakes than human ones. Robots don’t suffer from fatigue, for example, and they aren’t biased (intentionally or otherwise) toward any one team or player. But apart from fewer obvious blown calls, are robotic umpires good for sports?

That’s a topic that Harry Collins and Robert Evans of Cardiff University address in their paper “You cannot be serious!  Public Understanding of Technology with special reference to `Hawk-Eye,” which appears in the journal Public Understanding of Science. They suggest that viewers don’t understand what they are getting when a system like Hawk-Eye makes a binary in/out call. They write, “The public understanding of measurement errors and confidence intervals could be enhanced if `sports-decision aids,’ such as the Hawk-Eye system, were to present their results in a different way. There is a danger that Hawk-Eye as used could inadvertently cause naïve viewers to overestimate the ability of technological devices to resolve disagreement among humans because measurement errors are not made salient.”

They recommend that results such as Hawk-Eye-generated images be presented with circles of uncertainty, for example, that would indicate with what confidence level the automated system has reached its decision.That sound like a reasonable suggestion to me.

But they also address how robotic umpires might alter the very nature of a game. They describe a situation in cricket (I actually learned something about cricket reading this paper) called “leg before wicket” in which a batsman will be called “out” if his leg blocks a ball that otherwise would have struck the wicket behind him. Human referees, they say, give considerable leeway to the batsman. A robot that’s more likely to call the batsman “out” could drastically shorten the game, with possible financial implications. Collins and Evans also note that because games judged by automated systems could be significantly different from games judged by humans, there could be a gulf between top-level games whose venues are equipped with automated systems and lower level games that continue to be judged by humans.

Their recommendation? “Use automated sports decision aids to reproduce human systematic error while minimizing random error….” That is, program the system to give the benefit of the doubt to the batsman, as a human referee would. Similarly, program the system to call “out” a tennis shot that appears out to human observers—even though the shot may have grazed the line before skidding into its bounce.

What do you think? Are robotic umps a good idea? If so, should they be designed to provide as much accuracy as possible, or should they mirror human umps’ imperfections with respect to systematic errors?

Posted by Rick Nelson on September 4, 2008 | Comments (3)