Winograd Schema Challenge

Our guest this week, Hector Levesque, joins us to discuss alternative ways to measure a machine’s humanity. Hector proposed a new way to test artificial intelligence, called the Winograd Schema Challenge, during the 2011 AAAI Spring Symposium. Designed to be an improved alternative to the Turing Test, it is a multiple-choice test that employs questions of a very specific structure that employ instances of what are called "Winograd Schemas," named after Terry Winograd, a professor of computer science at Stanford University.

For Hector, the Turing test was a thought experiment and that the idea of what Turing was trying to get at was more important than the actual details of the test. Given the possibility that people are going to have disagreements about what defines intelligence, Hector thinks that Turing’s thought experiment was meant to steer this conversation about intelligence toward a different direction. Instead focusing on the question of whether it's possible to get a machine to think or to be intelligent (or conscious), the Turing Test is questioning whether it's possible to get a machine to act like it is intelligent, or thinking, or conscious.

As a follow-up to the Turing Test, Hector introduced the the Winograd Schema Challenge, which we discuss during the episode. What is a Winograd Schema? A Winograd schema is a pair of sentences that differ only in one or two words and that contain a referential ambiguity that is resolved in opposite directions in the two sentences. The schema is designed so that the correct answer is obvious to the human reader, but cannot easily be found using selectional restrictions or statistical techniques over text corpora. So the Winograd Schema challenge would typically involve a small reading comprehension test in which the machine has to identify the antecedent of an ambiguous pronoun in a given statement. For example, “Tom threw his schoolbag down to Ray after he reached the [top/bottom] of the stairs. Who reached the [top/bottom] of the stairs?” Another example would be: “Thomson visited Cooper’s grave in 1765. At that date he had been [dead/travelling] for five years. Who had been [dead/travelling] for five years?”