Why the Wisdom Of Crowds works.  Now why it doesn't.


The audience was shown 8 multiple choice questions as a pre-test.   Everyone entered their response into their personal keypad, and after each question the presenter showed and explained the correct response. 

We were also shown the audience's answers distribution.  In half of the questions, the majority chose the right answer.  In the other half, the correct answer was tied for most responses. I was a little surprised, thinking that the wisdom of crowds would clearly result in the right answer every time.  So it goes.

The presenter then went through his lecture, and then showed the same 8 questions.

Wow.  This time the majority never picked the right answer.

I want you to stop and think about this.  We had all seen the questions before; we had been told what the right answers were; and we were given a presentation on the material.  Not only did "the wisdom of crowds" fail, it did worse than it had initially.

It did worse than worse.  The wrong answers weren't randomly distributed.  Each time the majority chose the same wrong answer.

What happened?   Say what you want about psychiatrists, they're not retarded.  So?

II.


Review: the reason the wisdom of crowds works is because each person has their own  systematic error (bias) that is usually different than someone else's.  It doesn't matter how inaccurate your and his responses are, just that they have different kinds of systematic errors, which thus cancel each other out.  By simple analogy: I think it's small, you think it's big, so our average gets us closer to accurate.

And now you may get it: the audience abandoned their individual systematic biases, and took on the presenter's. We stopped being critical, we stopped operating around our own prejudices, and tried to think of what he wanted us to answer.  We all adopted his systematic bias.  He thinks "it's big," so we all picked "big."

III. 

Except: when we willingly abandon our critical thinking to follow a leader, we also lose the ability to infer what the leader means, as opposed to says.  This is the unintended consequences of leadership: we collectively misinterpret him the same way.  There's no chance for the "wisdom of crowds" to work because we're following someone else; and without the benefit of our own critical thought and prejudices, we misunderstand him as well.  "As you can see, this is very big."  Wait, whoa-- did he say it's a pig?  He's right, it is a pig!

It would be great if people could abandon prejudice in favor of reason; but it's far preferable for them to have their own prejudices (born out of context) than to adopt someone else's (born out of TV).  And, at least in a democracy, these prejudices should balance out.

Political parties, leaders, influential books or movies-- when we let them do the thinking for us, when we adopt their position, the balancing out never happens.  We do worse for ourselves.  (No, this isn't a swipe at the President, I'm making a general point.)

IV .

This would be a sad enough commentary on the limitations of societies if this was the end of the story, but it wasn't.  After writing this post, two or three glasses into the rum, I had a realization: none of those questions really had right answers.  "What percent of bipolars experience mixed states?" can't be answered without qualification; even using DSM standards on Americans only, the answer varies widely.  Not to mention what's bipolar today is often depression yesterday and god knows what tomorrow.

So not only had we followed a leader blindly into worse performance,  but reinforcement from the leader when we got questions wrong ("no, no, no, this is the correct answer") means we were blinded to the fact that there weren't really any right answers.

One man had negated the wisdom of crowds, and also failed to teach us anything about nothing.

Biases and heuristics are part of how we think.  We can abandon the use of a particular heuristic, but not the use of heuristics.  My argument about the relative insignificance of financial bias in research isn't that I think that it is actually insignificant, but that I am more worried about the bias I cannot detect but I know is there.

Neither are heuristics necessarily bad or even wrong.  Heuristics don't prevent us from thinking critically, they are actually shortcuts which are supposed to save us from our better judgment. 

(from Dave Attell):  "if you walk outside right now, and there's a man running naked  down the street, cock flapping in the wind, you run with that man."

Huh?  Why?

"Because there's some scary shit coming the other way."