The Special Circumstance Which Causes The Wisdom Of Crowds To Fail
The audience was shown 8 multiple choice questions as a pre-test. Everyone entered their response into their personal keypad, and after each question the presenter showed and explained the correct response.
We were also shown the audience's answers distribution. In half of the questions, the majority chose the right answer. In the other half, the correct answer was tied for most responses. I was a little surprised, thinking that the wisdom of crowds would clearly result in the right answer every time. So it goes.
The presenter then went through his lecture, and then showed the same 8 questions.
Wow. This time the majority never picked the right answer.
I want you to stop and think about this. We had all seen the questions before; we had been told what the right answers were; and we were given a presentation on the material. Not only did "the wisdom of crowds" fail, it did worse than it had initially.
It did worse than worse. The wrong answers weren't randomly distributed. Each time the majority chose the same wrong answer.
What happened? Say what you want about psychiatrists, they're not retarded. So?
II.
Review: the reason the wisdom of crowds works is because each person has their own systematic error (bias) that is usually different than someone else's. It doesn't matter how inaccurate your and his responses are, just that they have different kinds of systematic errors, which thus cancel each other out. By simple analogy: I think it's small, you think it's big, so our average gets us closer to accurate.
And now you may get it: the audience abandoned their individual systematic biases, and took on the presenter's. We stopped being critical, we stopped operating around our own prejudices, and tried to think of what he wanted us to answer. We all adopted his systematic bias. He thinks "it's big," so we all picked "big."
III.
Except: when we willingly abandon our critical thinking to follow a leader, we also lose the ability to infer what the leader means, as opposed to says. This is the unintended consequences of leadership: we collectively misinterpret him the same way. There's no chance for the "wisdom of crowds" to work because we're following someone else; and without the benefit of our own critical thought and prejudices, we misunderstand him as well. "As you can see, this is very big." Wait, whoa-- did he say it's a pig? He's right, it is a pig!
It would be great if people could abandon prejudice in favor of reason; but it's far preferable for them to have their own prejudices (born out of context) than to adopt someone else's (born out of TV). And, at least in a democracy, these prejudices should balance out.
Political parties, leaders, influential books or movies-- when we let them do the thinking for us, when we adopt their position, the balancing out never happens. We do worse for ourselves. (No, this isn't a swipe at the President, I'm making a general point.)
IV .
This would be a sad enough commentary on the limitations of societies if this was the end of the story, but it wasn't. After writing this post, two or three glasses into the rum, I had a realization: none of those questions really had right answers. "What percent of bipolars experience mixed states?" can't be answered without qualification; even using DSM standards on Americans only, the answer varies widely. Not to mention what's bipolar today is often depression yesterday and god knows what tomorrow.
So not only had we followed a leader blindly into worse performance, but reinforcement from the leader when we got questions wrong ("no, no, no, this is the correct answer") means we were blinded to the fact that there weren't really any right answers.
One man had negated the wisdom of crowds, and also failed to teach us anything about nothing.
Biases and heuristics are part of how we think. We can abandon the use of a particular heuristic, but not the use of heuristics. My argument about the relative insignificance of financial bias in research isn't that I think that it is actually insignificant, but that I am more worried about the bias I cannot detect but I know is there.
Neither are heuristics necessarily bad or even wrong. Heuristics don't prevent us from thinking critically, they are actually shortcuts which are supposed to save us from our better judgment.
(from Dave Attell): "if you walk outside right now, and there's a man running naked down the street, cock flapping in the wind, you run with that man."Huh? Why?
"Because there's some scary shit coming the other way."
Digg | 
del.icio.us | 
StumbleUpon | 
Reddit | 
Yahoo! | 
My favourites
March 5, 2009 11:29 PM | Posted by : | Reply
Geez, what did the last psychiatrist do to scare all of the commentators away? I find the idea that "it's far preferable for [people] to have their own prejudices... than to adopt someone else's," due to the balancing-out effect that occurs in the former situation, sort of interesting. I never thought of that.
Score: 4 (4 votes cast)
March 5, 2009 11:52 PM | Posted by : | Reply
When are we going to re-learn to think for ourselves? Funny how kids just do this naturally.
Score: 2 (2 votes cast)
March 6, 2009 8:51 AM | Posted by : | Reply
Hence why critical thinking should never be turned off. It takes conscious effort to consider things without either agreeing with or even being subconsciously snookered by the premise; refuting, say, an ad hominem attack line-by-line against the accusations without noticing that the opponent's argument is essentially fallacious.
I'm also kind of sick of the fetishization (ha, ha) of kids (as critical thinkers). They are probably more susceptible to influence from a strong leader and carry their own retarded viewpoints wrongly deduced from their limited experience. This is probably just some sublimated longing for the abstract concept of "childhood" expressed by putting those still going through it on a pedestal.
Score: 6 (6 votes cast)
March 6, 2009 11:38 AM | Posted by : | Reply
People will almost always do what is expected of them. Everyone wants to fit in. Subscribing to a group or leader not only gives us a ready role in a group of people, it saves us from having to reason and argue issues central to the group. Someone else has done/will do that, and our fellow groupies make this deference feel normal and right. Easy Peasy.
Maybe we need more autistic people.
Score: 2 (2 votes cast)
March 6, 2009 1:25 PM | Posted by : | Reply
Thanks for preserving my IQ points. Just came out of a mind-numbing staff meeting convinced that I had lost at least 10 IQ points. Read your blog and it re-booted my brain. This is an intelligent, well-written entry. I was easily able to imagine a group of education administrators in the same situation.
You also made me think about the HAL 9000 (Heuristically programmed ALgorithmic Computer) from 2110: A Space Odyssey.
From Wikipedia: "Two fundamental goals in computer science are finding algorithms with provably good run times and with provably good or optimal solution quality. A heuristic is an algorithm that abandons one or both of these goals; for example, it usually finds pretty good solutions, but there is no proof the solutions could not get arbitrarily bad; or it usually runs reasonably quickly, but there is no argument that this will always be the case."
HAL to Dave, "I honestly think you ought sit down calmly, take a stress pill and think things over."
The other pertinent reference that comes to mind after reading Charlie's comment is James Clavell's "A Children's Story." Children's critical thinking was gone in 25 minutes becauses they all adopted the systematic bias of a soft-spoken, kind teacher.
I appreciate your excellent writing! Thanks!
Score: 3 (3 votes cast)
March 6, 2009 6:09 PM | Posted by : | Reply
Before anyone spanks me about the typos -- give me a break. I just got out of the hospital (due to the combination of a drug adverse reaction, my VNS just being replaced and the fact that my psychiatrist was out of town), I monitored our state's high stakes testing at an elementary school and it's the Friday before spring break. Yes, I know that HAL was in "2001: A Space Odyssey," and not 2110. Also HAL's comment to Dave should read, "I honestly think you ought to sit down calmly, take a stress pill and think things over."
Now I can go home and start spring break!
Score: 1 (1 votes cast)
March 8, 2009 7:00 PM | Posted by : | Reply
I agree that the idea that children are better at thinking than adults is preposterous. Thinking well is something that needs to be learned, and it takes effort. Children aren't born knowing how to think critically, only to have the skill stolen from them by bad schooling, bad parenting, and the adverse effects of too much TV. That's just stupid.
Score: 1 (1 votes cast)
April 1, 2009 1:40 PM | Posted by : | Reply
The one thing that always strikes me as curious about discussion of the "wisdom of crowds" (and its flip-side implication, the foolishness of individuals) is that the proof of the value of aggregate guesses depends on a measurement.
We would not be able to "prove" how smart groups are without the individual who made the measurement that "proves" groups are smarter than individuals.
So, while it is true that the aggregate of informed guesses will be closer to the measurement than any individual informed guess, the unexamined truth here is that the measurement (made by an individual with a measuring attitude, an ability to measure and to interpret the measurement) is more reliable than both any individual guess and the aggregate guess of the entire group.
What most of these experiments really demonstrate is that rational individuals with the proper tools make far better determinations than entire groups of people who are just guessing. What bias would blind us to that realization, I wonder?
Now, if such an individual insists that his measured opinion is more valuable than that of the informed guessers, indeed more valuable even than the consensus of an entire guesser civilization, will he be condemned by the guessers as a narcissist, or at the very least shunned as an arrogant know-it-all?
Maybe they'd chain him to a rock to have his liver eaten by vultures.
Score: 3 (3 votes cast)
Comments