August 31, 2011
The Wisdom Of Crowds Turns Into Madness
less independent than they think
In PNAS, an article which is intuitively obvious but terrifying to see played out in science.
The "Wisdom Of Crowds" concept is that the average guesses of a crowd will be closer to the truth than a randomly selected individual guess.
The reason this works is that because the crowd has different individuals with different types of systematic error, e.g. prejudices. With more individuals, the prejudices negate each other.
The Swiss study took 144 college students and asked them a series of questions (population of Switzerland, murder rate, etc). It recorded 5 consecutive guesses, as well as the confidence for the first and last guess.
I.
The first interesting finding is that the crowd is sometimes so incredibly wrong that the mean of their responses is just... really wrong. How many assaults were there in Switzerland in 2006? 10? 100? 1000? 10000? 100000? Those are exponentially different guesses, so an arithmetic mean could be way off, factors of ten off.
In such cases, a geometric mean is much closer to the correct answer. So, point number one, when you are crowdsourcing, choose your mean/distribution appropriately.
The "Wisdom Of Crowds" concept is that the average guesses of a crowd will be closer to the truth than a randomly selected individual guess.
The reason this works is that because the crowd has different individuals with different types of systematic error, e.g. prejudices. With more individuals, the prejudices negate each other.
The Swiss study took 144 college students and asked them a series of questions (population of Switzerland, murder rate, etc). It recorded 5 consecutive guesses, as well as the confidence for the first and last guess.
I.
The first interesting finding is that the crowd is sometimes so incredibly wrong that the mean of their responses is just... really wrong. How many assaults were there in Switzerland in 2006? 10? 100? 1000? 10000? 100000? Those are exponentially different guesses, so an arithmetic mean could be way off, factors of ten off.
In such cases, a geometric mean is much closer to the correct answer. So, point number one, when you are crowdsourcing, choose your mean/distribution appropriately.
II.
The diversity of guesses is quite large-- everyone comes to the question with their own prejudices and errors.
But merely by giving the subjects access to the previous round's guesses-- either the mean of the guesses ("aggregated information") or everyone's individual guess, the diversity disappears and everyone's guesses begin to converge.
The first round the guesses were wildly disparate, but as everyone got to see the other guesses, they converge remarkably.
Why did having the full information (all 12 people's individual guesses) seem to cause less convergence than having the mean of their guesses? It didn't, really; but also because the aggregate is only one number that you converge to; having 12 wildly disparate numbers to converge to is harder. But by the third round, it hardly mattered-- a systematic bias had been introduced into the crowd, which is ironic since it is systematic bias that the Wisdom Of Crowds is supposed to negate. Moo.
III.
People following the herd would be boring but not disastrous, except for the other finding.
Since the guesses converge, since other people are converging with you and you can see that, the confidence in these guesses goes up: a false belief of collective accuracy with no increase in actual accuracy. "It's unanimous!" Yikes.
Also remember, these people weren't being given an expert's guess to converge to, just other (regular) people's. As the authors point out, they didn't even attempt to measure group leader effects, persuasion, talking heads on TV, or twitter.
This is not a trivial problem. It isn't just saying that the beliefs converge; it is saying that since the beliefs converge along with greater confidence in their "truthfulness", it becomes more difficult for any individual to not converge as well-- and feel confident about it.
If you do manage to run from the herd you have to climb a high wall. "Can so many people be so wrong, yet so close together in their guesses? So wrong, yet so confident? Is everyone insane?"
You can imagine the social implications of a highly energized crowd, or electorate, or laity, or polity, or tax base, all converging on a "truth" of which they are supremely confident by virtue of the fact that others believe the same (which is the result of similar convergence on their part.) This is probably supercharged when you have a charismatic figurehead leading convergence, and by "charismatic figurehead" I mean media; no one person came up with this, everyone just knows it's true.
IV.
So much for the paper. Now consider the more general implications.
"Well, I'm going to be an independent thinker and not be affected by the herd and make my own educated guess." No, you won't.
The moment you have the other people's guesses, you cannot shake that information. Your "independent" guess necessarily includes that guess in some way, you can't unlearn it. Either your guess converges towards the herd, or your guess is characterized as against the herd. Either way, the herd affected your thinking in ways you don't realize. You're part of the dialectic and you didn't even want to be. That you don't want to be part of it ensures you are part of it.
The existence of the convergence of ideas, knowing that a convergence exists, either attracts further groupthink, or sets up a second groupthink in opposition to the first. Groupthink certainly reinforces one idea; and it can cause the setting up of a second large idea in opposition, but it makes a third independent idea highly unlikely (unless, again, it forms in opposition to ideas 1 or 2.)
In other words, in cases where social influence is impossible to avoid, the wisdom of crowds becomes the madness of crowds even for those who disagree with the crowd. All it takes is one idiot with a megaphone.
---
How to use your own inherent narcissism to guess more accurately
The special circumstance which causes the wisdom of crowds to fail
The diversity of guesses is quite large-- everyone comes to the question with their own prejudices and errors.
But merely by giving the subjects access to the previous round's guesses-- either the mean of the guesses ("aggregated information") or everyone's individual guess, the diversity disappears and everyone's guesses begin to converge.
The first round the guesses were wildly disparate, but as everyone got to see the other guesses, they converge remarkably.
Why did having the full information (all 12 people's individual guesses) seem to cause less convergence than having the mean of their guesses? It didn't, really; but also because the aggregate is only one number that you converge to; having 12 wildly disparate numbers to converge to is harder. But by the third round, it hardly mattered-- a systematic bias had been introduced into the crowd, which is ironic since it is systematic bias that the Wisdom Of Crowds is supposed to negate. Moo.
III.
People following the herd would be boring but not disastrous, except for the other finding.
Since the guesses converge, since other people are converging with you and you can see that, the confidence in these guesses goes up: a false belief of collective accuracy with no increase in actual accuracy. "It's unanimous!" Yikes.
Also remember, these people weren't being given an expert's guess to converge to, just other (regular) people's. As the authors point out, they didn't even attempt to measure group leader effects, persuasion, talking heads on TV, or twitter.
This is not a trivial problem. It isn't just saying that the beliefs converge; it is saying that since the beliefs converge along with greater confidence in their "truthfulness", it becomes more difficult for any individual to not converge as well-- and feel confident about it.
If you do manage to run from the herd you have to climb a high wall. "Can so many people be so wrong, yet so close together in their guesses? So wrong, yet so confident? Is everyone insane?"
You can imagine the social implications of a highly energized crowd, or electorate, or laity, or polity, or tax base, all converging on a "truth" of which they are supremely confident by virtue of the fact that others believe the same (which is the result of similar convergence on their part.) This is probably supercharged when you have a charismatic figurehead leading convergence, and by "charismatic figurehead" I mean media; no one person came up with this, everyone just knows it's true.
IV.
So much for the paper. Now consider the more general implications.
"Well, I'm going to be an independent thinker and not be affected by the herd and make my own educated guess." No, you won't.
The moment you have the other people's guesses, you cannot shake that information. Your "independent" guess necessarily includes that guess in some way, you can't unlearn it. Either your guess converges towards the herd, or your guess is characterized as against the herd. Either way, the herd affected your thinking in ways you don't realize. You're part of the dialectic and you didn't even want to be. That you don't want to be part of it ensures you are part of it.
The existence of the convergence of ideas, knowing that a convergence exists, either attracts further groupthink, or sets up a second groupthink in opposition to the first. Groupthink certainly reinforces one idea; and it can cause the setting up of a second large idea in opposition, but it makes a third independent idea highly unlikely (unless, again, it forms in opposition to ideas 1 or 2.)
In other words, in cases where social influence is impossible to avoid, the wisdom of crowds becomes the madness of crowds even for those who disagree with the crowd. All it takes is one idiot with a megaphone.
---
How to use your own inherent narcissism to guess more accurately
The special circumstance which causes the wisdom of crowds to fail
Digg | 
del.icio.us | 
StumbleUpon | 
Reddit | 
Yahoo! | 
My favourites
48 Comments