Improving accuracy in the wisdom of crowds

One day in the early twentieth century, Francis Galton came across a contest at a county fair, in which a crowd of roughly 800 individuals was asked to guess the weight of an ox. While a few participants from the crowd could have had specialized knowledge, such as farmers and butchers, many members of the crowd were non-experts who were just attending the fair. After everyone had given their guesses, Galton was surprised to discover that the mean of the crowd’s guesses was pretty close to the actual weight. The crowd collectively guessed 1,197 pounds, while the actual weight of the ox was 1,198 pounds-- a near-perfect estimate!

In his letter to Nature, Galton noted, “this result is, I think, more creditable to the trustworthiness of a democratic judgment than might have been expected.” A nobleman and scientist, Galton held a condescending view of the average citizen’s intelligence. In fact, he did not believe in the wisdom of the crowd and was convinced that the masses could not be trusted to make important decisions or do much of anything. However, much to his surprise, Galton found the opposite.

Since Francis Galton’s observation, countless examples of the “wisdom of the crowd” effect have been documented scientifically and anecdotally. This phenomenon was further popularized in James Surowiecki’s book “The Wisdom of Crowds,” arguing that the combined opinions of a group of individuals will do better than the judgment of an elite few. But Surowiecki acknowledges that the wisdom of the crowd is not always perfect. He argues that the best collective ideas result when the makeup of the crowd satisfies four conditions, including the diversity of opinion, independence of opinion, decentralization, and finally aggregation.

Surowiecki points to the success of public and corporate markets to demonstrate that a collection of diverse viewpoints with the same motivation can create an accurate aggregate conclusion. He explains that under the right circumstances, “prediction markets” (e.g. the Iowa Electronic Markets), where people buy and sell probabilities as if they were stocks, are an excellent way of turning the knowledge of many people into reasonably accurate predictions. In Surowiecke’s opinion, prediction markets (and other forms of collective thought) tend to be wiser than the individuals who participated in them.

Nonetheless, Surowiecke’s argument for crowd wisdom only seems to work under specific conditions. What happens when a crowd is overwhelming ignorant of the facts germane to deciding the best policy to achieve given objectives? While it may seem only fair to count every vote equally, it might not always lead to the best conclusions. In the wisdom of crowds, people who are less informed about an issue count just as much as the people who are more informed about the issue. Similarly, those who care less about an issue would count just as much as those that care the most about the issue. In situations where specialized knowledge is needed to identify the correct answer, the presence of experts in the crowd might be drowned out by majority ignorance. How can we extract the best answer from a crowd, especially in the case where both the most popular answer is wrong?

One seemingly radical solution is not to weigh each and every vote equally. Instead, confidence ratings can be obtained to improce the accuracy of the crowd. Answers with higher confidence can then be given more weight to improve accuracy. A 2004 study by Drazen Prelec and H. Sebastian Seung showed that it is possible to set up surveys where the incorrect answers are also the most popular answers, which are also typically associated with high confidence. Their method relied on asking questions in pairs: the first queried the individual; the second asked for an estimation of how many other respondents would give the same response. They found that people telling the truth tend to overestimate how common their own answer was.

Given Prelec and Seung’s results, confidence ratings can fail in some situations, such as when questions are deliberately misleading. Consider a case from the study, where a group of people was given the following question: “Philadelphia the capital of Pennsylvania: Yes or No?” Most people gave the incorrect answer (“Yes”). Why was the majority wrong? Prelec et al. reasoned that someone who only knows that Philadelphia is an important city in Pennsylvania could reasonably conclude that Philadelphia is the capital. Individuals who answer “No” might possess a crucial additional knowledge, that the actual capital is Harrisburg. Moreover, those who know that Philadelphia is not the capital of Pennsylvania can imagine that many others will incorrectly think that way.

Prelec and Seung along with John McCoy revisited this problem in a recent study published earlier this year in Nature. They propose a method based on a novel definition of the best answer: it is the one offered by those who would be least surprised by the actual answer when disclosed. They devised a clever yet simple formula, called the “surprisingly popular” solution to the single question crowd wisdom problem. In addition to asking respondents, “Philadelphia is the capital of Pennsylvania, Yes or No?” respondents were also requested to assign a confidence rating and answer “What percentage of people in this survey will answer ‘yes’ to this question?” The key here is to compare the actual responses with the predicted responses.

To illustrate Prelec et al.'s new algorithm, consider the image below.

In the case where researchers asked, "Is Columbia is the capital of South Carolina, Yes or No?" respondents who answered “Yes” believed that the majority would agree with them. Those who answered “No” also predicted that the majority of people would also reply "No.” In other words, both groups believed that others would give the same answer as theirs.

The researchers got a different result when people were asked, "Is Philadelphia is the capital of Pennsylvania, Yes or No?". Philadelphia is the capital of Pennsylvania there was a difference. While those who replied incorrectly "Yes" also predicted that most people would agree with them, the people who gave the correct response "No" also feel that the most others would reply incorrectly.

There was no surprisingly popular answer in the Columbia case and what the majority predicted to be the percentage of people replying “Yes” corresponded with the actual number of responses. However, in the Philadelphia example, the proportion of the group that most people thought would answer incorrectly was significantly less than the actual percentage of individuals that made the incorrect response. Although both the misguided majority and the correct minority predicted that everyone else would give the wrong response, the team found that the minority (but correct) answer was given much more often than predicted. The answers that are selected surprisingly often, compared to what participants on average expected, are more likely to be accurate.

The team tested their method in four different settings. The first experiment focused on 50 questions about US state capitals. The second included 80 True/False general knowledge questions, in which people were asked to include questions most would answer correctly and those the majority would get wrong. For the third scenario, dermatologists were given 80 images of skin lesions and asked to rate their confidence on what they thought was malign or malignant and predict the distribution of other dermatologists' judgments. In their final experiment, the researchers asked a group of art professionals and a group of MIT students who had never taken art courses to evaluate the market value of 90 reproductions of contemporary artworks. Participants were asked to predict the price of each artwork, and then estimate the percentage of people predicting a price more than $30,000.

In all four scenarios, the “surprisingly popular” method out-performed crowd wisdom or confidence-based methods alone, reducing errors by between 21 and 35 percent. They found that people are often as confident in an incorrect answer as the correct one; hence, confidence ratings are not always useful. Rather than weighing people equally like in crowd wisdom, their solution takes into account the knowledge disparity between respondents who know the right answer and those who do not.

The surprisingly popular approach seems to be a compelling approach to the nuances in the wisdom of the crowd. It recognizes that in cases where there are vast differences in specialized knowledge in a group with differing opinions, the more-informed minorities could override the crowd opinion.