August 31, 2011

The Wisdom Of Crowds Turns Into Madness

switzerland.jpg
less independent than they think

In PNAS, an article which is intuitively obvious but terrifying to see played out in science.

The "Wisdom Of Crowds" concept is that the average guesses of a crowd will be closer to the truth than a randomly selected individual guess.

The reason this works is that because the crowd has different individuals with different types of systematic error, e.g. prejudices.  With more individuals,  the prejudices negate each other.

The Swiss study took 144 college students and asked them a series of questions (population of Switzerland, murder rate, etc).  It recorded 5 consecutive guesses, as well as the confidence for the first and last guess.

I.

The first interesting finding is that the crowd is sometimes so incredibly wrong that the mean of their responses is just... really wrong.  How many assaults were there in Switzerland in 2006?  10?  100?  1000? 10000? 100000?  Those are exponentially different guesses, so an arithmetic mean could be way off, factors of ten off.

In such cases, a geometric mean is much closer to the correct answer.  So, point number one, when you are crowdsourcing, choose your mean/distribution appropriately. 


pnas lorenz table 1.JPG
II.


The diversity of guesses is quite large-- everyone comes to the question with their own prejudices and errors.

But merely by giving the subjects access to the previous round's guesses-- either the mean of the guesses ("aggregated information") or everyone's individual guess, the diversity disappears and everyone's guesses begin to converge.



pnas collective error.jpg


The first round the guesses were wildly disparate, but as everyone got to see the other guesses, they converge remarkably.

Why did having the full information (all 12 people's individual guesses) seem to cause less convergence than having the mean of their guesses?  It didn't, really; but also because the aggregate is only one number that you converge to; having 12 wildly disparate numbers to converge to is harder.  But by the third round, it hardly mattered-- a systematic bias had been introduced into the crowd, which is ironic since it is systematic bias that the Wisdom Of Crowds is supposed to negate.  Moo.


III.

People following the herd would be boring but not disastrous, except for the other finding.

Since
the guesses converge, since other people are converging with you and you can see that, the confidence in these guesses goes up: a false belief of collective accuracy with no increase in actual accuracy.  "It's unanimous!"  Yikes.

Also remember, these people weren't being given an expert's guess to converge to,  just other (regular) people's.  As the authors point out, they didn't even attempt to measure group leader effects, persuasion, talking heads on TV, or twitter.

This is not a trivial problem.  It isn't just saying that the beliefs converge; it is saying that since the beliefs converge along with greater confidence in their "truthfulness", it becomes more difficult for any individual to not converge as well-- and feel confident about it.

If you do manage to run from the herd you have to climb a high wall.  "Can so many people be so wrong, yet so close together in their guesses?  So wrong, yet so confident?  Is everyone insane?"

You can imagine the social implications of a highly energized crowd, or electorate, or laity, or polity, or tax base, all converging on a "truth" of which they are supremely confident by virtue of the fact that others believe the same (which is the result of similar convergence on their part.)  This is probably supercharged when you have a charismatic figurehead leading convergence, and by "charismatic figurehead" I mean media; no one person came up with this, everyone just knows it's true.

IV.

So much for the paper.  Now consider the more general implications.

"Well, I'm going to be an independent thinker and not be affected by the herd and make my own educated guess."  No, you won't. 

The moment you have the other people's guesses, you cannot shake that information.  Your "independent" guess necessarily includes that guess in some way, you can't unlearn it.  Either your guess converges towards the herd, or your guess is characterized as against the herd.  Either way, the herd affected your thinking in ways you don't realize.  You're part of the dialectic and you didn't even want to be.   That you don't want to be part of it ensures you are part of it.

The existence of the convergence of ideas, knowing that a convergence exists, either attracts further groupthink, or sets up a second groupthink in opposition to the first.  Groupthink certainly reinforces one idea; and it can cause the setting up of a second large idea in opposition, but it makes a third independent idea highly unlikely (unless, again, it forms in opposition to ideas 1 or 2.)

In other words, in cases where social influence is impossible to avoid, the wisdom of crowds becomes the madness of crowds even for those who disagree with the crowd.  All it takes is one idiot with a megaphone.

---

How to use your own inherent narcissism to guess more accurately

The special circumstance which causes the wisdom of crowds to fail









Comments

It's true for just about an... (Below threshold)

August 31, 2011 12:44 PM | Posted by Kevin: | Reply

It's true for just about any piece of extraneous information. If you change the environment and context of just about any question, you get a different answer. E.g. parole board decisions in the morning versus the afternoon. Humans just ain't rational.

Vote up Vote down Report this comment Score: 4 (6 votes cast)
So, when you are in crowds,... (Below threshold)

August 31, 2011 1:21 PM | Posted by André: | Reply

So, when you are in crowds, you don't lose your individuality, but your individuality is changed by the crowd?

Good post, btw.

Vote up Vote down Report this comment Score: 3 (3 votes cast)
TLP: You're part of the di... (Below threshold)

August 31, 2011 2:07 PM | Posted by Gary: | Reply

TLP: You're part of the dialectic and you didn't even want to be. That you don't want to be part of it ensures you are part of it.

Freud: If you arrive late for the appointment, you're passive aggressive, if you arrive early you're anxious, and if you arrive on time, you're obessive.

There's no escaping it. Sit back, relax, and enjoy the ride.

Vote up Vote down Report this comment Score: 18 (18 votes cast)
So is it *ever* possible to... (Below threshold)

August 31, 2011 2:11 PM | Posted by AL: | Reply

So is it *ever* possible to be an "individual"? Or to know that you are not insane?

God, this is depressing ...

Vote up Vote down Report this comment Score: 2 (6 votes cast)
Try playing a game of <a hr... (Below threshold)

August 31, 2011 3:40 PM | Posted by sdenheyer: | Reply

Try playing a game of Paranoid Debating to observe how this works, both within a group and inside your mind.

Vote up Vote down Report this comment Score: 2 (2 votes cast)
Not me man, I think for mys... (Below threshold)

August 31, 2011 3:54 PM | Posted by Lady Gaga Fan: | Reply

Not me man, I think for myself!

Vote up Vote down Report this comment Score: 3 (7 votes cast)
Not sure why I can't just l... (Below threshold)

August 31, 2011 4:16 PM | Posted by Dev Thakur: | Reply

Not sure why I can't just learn from this and try to go beyond it.

"I don't have to go with the group, but I don't have to oppose them either, and I should systematically generate all the possible other answers ...

okay now let me try to figure out which answer makes the most sense ...

and further let me step back and look at my answers in general ... do they tend to line up with one group or another ...

if they do, I can reaxamine more closely to see if I think that's because I'm just following that group, or because those answers actually make the most sense ...

finally I can acknowledge that though I'm still affected by the groupthink, at least I'm not totally dominated by it as I would be if I never considered it ...

Vote up Vote down Report this comment Score: 5 (5 votes cast)
Because no matter how thoro... (Below threshold)

August 31, 2011 6:03 PM | Posted, in reply to Dev Thakur's comment, by ThomasR: | Reply

Because no matter how thorough or analytical and self-aware you may be about your opinions and how they were/are affected by others, you will never know what opinion you might have had if you had never known what others thought.

Vote up Vote down Report this comment Score: 5 (5 votes cast)
Also known as an <a href="h... (Below threshold)

August 31, 2011 6:21 PM | Posted by Dr. Science: | Reply

Also known as an information cascade.

Vote up Vote down Report this comment Score: 1 (1 votes cast)
"it is systematic bias that... (Below threshold)

August 31, 2011 7:32 PM | Posted by Porejide: | Reply

"it is systematic bias that the Wisdom Of Crowds is supposed to negate" - nope. http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds#Failures_of_crowd_intelligence.The WOC is about averaging out *random* errors.

This paper is like an elixir for demonstrating that people (even quite erudite thinkers like yourself, TLP) don't really grok the idea behind the WOC. E.g., see http://neuroself.com/2011/05/29/jonah-lehrer-is-not-a-neuroscientist/.

Vote up Vote down Report this comment Score: 4 (4 votes cast)
How could one possibly infe... (Below threshold)

September 1, 2011 5:08 AM | Posted by Ramachandra: | Reply

How could one possibly infer anything soundly about the truth of a matter merely on the basis of a collection of opinions or guesses or prejudices? Try doing something sensible, e.g., ask a bunch of people to observe something, think about it, and then come to a conclusion. Now compare that with the truth. This will tell tell us something worth taking seriously about the proper exercise of our faculties and the outcomes yielded by a proper exercise of our faculties.

Vote up Vote down Report this comment Score: 3 (3 votes cast)
It is about bias. If you as... (Below threshold)

September 1, 2011 7:28 AM | Posted, in reply to Porejide's comment, by Or: | Reply

It is about bias. If you asked the same person to estimate the value of some similar figure they would likely make an error in the same direction as the first, e.g. if they grossly overestimated the number of rapes they would likely overestimate the number of murders. The idea is that every individual person has a systematic bias within the domain of questions that are being asked, but the biases themselves are treated as a random variable among the population at large; if you randomly select a person, his systematic error becomes a random error in your sample.

That post by Peter Freed is long and rambling, and unfortunately it is terribly unclear in its attempt to reiterate a crucial point stated in the study as follows*: "In the “aggregated information” condition, subjects could reconsider their estimate after having received the average (arithmetic mean) of all 12 estimates of the former round."

But here's what the researchers say about the arithmetic mean, which they report to the subjects after each round of estimates...

"In our case, the arithmetic mean performs poorly, as we have validated by comparing its distance to the truth with the individual distances to the truth. In only 21.3% of the cases is the arithmetic mean closer to the truth than the individual first estimates."

... as opposed to the geometric mean, which they use as a measure the crowd's wisdom after the fact:

"When using logarithms of estimates, the arithmetic mean is closer to the logarithm of the truth than the individuals’ estimates in 77.1% of the cases. This confirms that the geometric mean (i.e., exponential of the mean of the logarithmized data) is an accurate measure of the wisdom of crowds for our data."

Assuming the tendency is to update your estimate in the direction of the aggregated statistic, by reporting the arithmetic mean the researchers ensured that a majority of the subjects would produce a second estimate less accurate than the first, but by reporting the geometric a majority would have given better estimates. It seems that the group was not adjusting its estimates in deference to the wisdom of the crowd, but to the experimenters' authority in choosing a statistic to represent that wisdom. The mistake driving the decrease of their "wisdom-of-crowd indicator" is really appeal to authority as much as it is argumentum ad populum.

From a footnote: "Considering the logarithmic nature of our data, one may argue that the geometric mean would have been a better design choice than the arithmetic mean for the information feedback in the aggregated information condition. However, this measure is hard to understand for most subjects because it necessitates confidence with logarithmic transformations. As the simple average (i.e., arithmetic mean) is known from daily life, this information is more meaningful for subjects. Hence, we decided for the arithmetic mean."

Translation: We assumed these college students were stupid so we gave them data that would prove they were stupid. (The very last footnote also contains more information about why the methods led to the results they got, and in fact implies that they were studying the properties of log-normal distributions more than the properties of human decision-making. It should be the main conclusion of the study.)

Out of everybody involved in this, including the researchers, Freed, and Lehrer, only the lowly undergraduates did an admirable job.

*Also, Freed amazingly fails to notice the main problem with Lehrer's WSJ article: namely that he COMPLETELY fucking lied by saying "each round led to worse guesses." I can't give him the benefit of the doubt here; certainly any Rhodes scholar, or at least the kind of person a Rhodes scholar would hire to read a paper so he doesn't have to, can correctly interpret graph E.

Vote up Vote down Report this comment Score: 5 (5 votes cast)
I'd also crack some jokes a... (Below threshold)

September 1, 2011 7:37 AM | Posted by Or: | Reply

I'd also crack some jokes about how Freed can't stop referencing Malcolm Gladwell in a post whose thesis is that Lehrer is a journalist and not a neuroscientist, but I really need to go to bed.

Vote up Vote down Report this comment Score: 1 (1 votes cast)
Even if you could learn tha... (Below threshold)

September 1, 2011 7:43 AM | Posted by Guy Fox: | Reply

Even if you could learn that this happens and gird yourself to be immune to the effect, it wouldn't necessarily matter because exactly this effect makes much of the world turn, and no individual can swim against that kind of current. How much is a piece of paper, 157mm x 66 mm, 75% cotton/25% linen, worth? Nothing? A pizza? A keg of beer? A Thai teenager to call your own? It can be all of those things, depending on what people collectively agree to be the case.

Sure, the study asked about quasi-natural facts (would be plain 'natural' if Switzerland were a real country), but these mean nothing. They're just trivia. As soon as they start to mean something and start to earn the title 'Truth' (i.e. not 'how many rapes occurred?' but 'how many rapes is so many that we need to start locking women up for their own safety/extending police powers/chopping dangly bits off?"), the answer will be, always has been, and perhaps can only be determined by exactly this mechanism. Yes, it's the crowds doing it. Let's hope that it approximates 'wisdom'.

Vote up Vote down Report this comment Score: 0 (2 votes cast)
"we need to start locking w... (Below threshold)

September 1, 2011 8:24 AM | Posted by Anonymous: | Reply

"we need to start locking women up for their own safety" People always say it this way... if the stats were "unreasonably high" for the percent of man on woman rape.... wouldn't it make more sense to lock all the men up for the safety of the women? I mean if a bunch of tigers escaped the zoo and were rooming the city and only 6% had ever killed anyone.... would we recapture ALL of the lions for the safety of the people or would we tell people to stay in doors in hiding for the rest of their lives?

I'm just saying it's interesting if one segment of the population was statistically likely to prey on another segment of the population--- you would consider it more ethical to put the potential victims through a life spent in hiding rather than the potential predators. It's unthinkable that innocent men have their free life taken away simply because women keep getting raped, but innocent women's freedom should be expendable "for their own good" if women keep getting raped.

Vote up Vote down Report this comment Score: 1 (5 votes cast)
PS- Guy fox- I'm nitpicking... (Below threshold)

September 1, 2011 8:27 AM | Posted by Anonymous: | Reply

PS- Guy fox- I'm nitpicking, your comment was otherwise good. ;)

Vote up Vote down Report this comment Score: -1 (1 votes cast)
And remember: psych meds, a... (Below threshold)

September 1, 2011 8:46 AM | Posted by medsvstherapy: | Reply

And remember: psych meds, and not psychotherapy, should always be first-line treatment for any mental disorder. Care should always begin with a psychaitrist's assessment. If that psychiatrist happens to know a local therapist, the psychiatrist might suggest psychotherapy.

If a psychiatrist is not available, any physician will do.

Could 40,000 pharmaceutical reps be wrong?

Vote up Vote down Report this comment Score: 3 (5 votes cast)
While the irony of quoting ... (Below threshold)

September 1, 2011 11:18 AM | Posted by Larrykoubiak: | Reply

While the irony of quoting Monty Python in a comment about individuality is quite obvious, i just can't resist :

Brian: Please, please, please listen! I've got one or two things to say.
The Crowd: Tell us! Tell us both of them!
Brian: Look, you've got it all wrong! You don't NEED to follow ME, You don't NEED to follow ANYBODY! You've got to think for your selves! You're ALL individuals!
The Crowd: Yes! We're all individuals!
Brian: You're all different!
The Crowd: Yes, we ARE all different!
Man in crowd: I'm not...
The Crowd: Sch!

Vote up Vote down Report this comment Score: 1 (3 votes cast)
So your conclusion is what,... (Below threshold)

September 1, 2011 11:28 AM | Posted by CubaLibre: | Reply

So your conclusion is what, Hegelian? All important ideas are dialectical responses to older ideas? That in itself isn't a world-shattering claim. Seems pretty reasonable actually.

Vote up Vote down Report this comment Score: 0 (4 votes cast)
You are bearing on (I dare ... (Below threshold)

September 4, 2011 4:32 PM | Posted by rontom: | Reply

You are bearing on (I dare say) a false dichotomy: if by being an individual you differ from the thinking of the crowd, your unique thinking directly depends on the general thinking of the crowd, and therefore, your thoughts no matter how uniquely derived, reflects not you at all but the crowd. This undercuts all the discoveries and inventions made in science. Sorry, individuality remains

Vote up Vote down Report this comment Score: 1 (1 votes cast)
Actually I diverge from the... (Below threshold)

September 4, 2011 5:46 PM | Posted by Datura: | Reply

Actually I diverge from the crowd a helluva lot. I'm a high functioning Aspie, and one of the blessings/curses that comes with this is major pattern recognition ability. Lucky me, I recognize behavior patterns.

One of the patterns I recognize very quickly is Bandar-Log... "We all have said it; it must be true."

Soon as I see that little meme eddying about, I can predict the next pattern - "Death to the unbeliever!"

The best response to this is to get the hell away from it. As early as possible in the Bandar-Log stage. Collective insanity indeed...

Wonder if Kipling was Aspie. But more to the point, how the hell has the human race survived this long with the collective mentality of a troop of howler monkeys? I see this pattern on a daily basis, on issues large and small, and the older I get, the more frightening that is.

Vote up Vote down Report this comment Score: -1 (5 votes cast)
I don't think individuality... (Below threshold)

September 6, 2011 1:23 PM | Posted, in reply to rontom's comment, by Anonymous: | Reply

I don't think individuality is even possible for us. Humans are first and foremost a social species, perhaps not as lockstep as a colony of ants, but we aren't exactly cats either. And the whole wisdom of crowds thing is more a social reinforcement phenomena than anything. The whole thing seems to revolve around the idea of converging around a mean number. But it doesn't work if you have no idea of the actual facts.

If I asked about how many apples were sold at piggly wiggly today, we'd converge around a mean, BUT it would only be the truth if someone involved knew something about apples and grocery stores. I think the wisdom of crowds is more illusion than anything real. It doesn't give you information about the world out there, just what people think is out there. Wikipedia is great for general interest things that work in a common sense way, but it doesn't work as well when it's something in science or math that doesn't work in a commonsense way.

Vote up Vote down Report this comment Score: 1 (1 votes cast)
Very happy to see your arti... (Below threshold)

November 14, 2011 2:07 AM | Posted by hermes bags: | Reply

Very happy to see your article, I am very much to like and agree with your point of view. Thank you for sharing. linmei/comment201111

Vote up Vote down Report this comment Score: -4 (4 votes cast)
That you are consequently ... (Below threshold)

March 28, 2012 5:44 AM | Posted by cheap jeans: | Reply

That you are consequently nice! I just adore realising stunning along with creative giveaways this way. They will really jazz up my own morning, many thanks greatly. liulipingcomment201203blog

Vote up Vote down Report this comment Score: -3 (3 votes cast)
Wees zo goed en neem voor u... (Below threshold)

January 31, 2014 10:35 PM | Posted by Kenshaw Jeanne: | Reply

Wees zo goed en neem voor uw eigen goed en profiteren, negeren elke uitleenartikel van deze sites...contact met u kunnen ze helpen you.they zijn kwaad en niet echt!..ze zijn allemaal oplichting! Ik heb ten prooi vallen aan velen zijn er vanwege mijn nieuwsgierigheid en angst te worden vrij van schulden, verloor ik duizenden dollars aan deze persons.well godzijdank nu, het is al een geschiedenis!..Als ik waar u zou ik mijn post verwijderd van die site onmiddellijk, omdat alle kans en de gelegenheid hebben gekregen om deze jongens, zou ze aan het vasthouden, en uit uw hand verdiend money.my zuigen verhaal en ervaring zou vitrine een goede referentie en de voorkeur van een goede bron van legitieme en betrouwbare hulp voor u dat zou u minder dan 48 uur duren, (ongeacht uw krediet score maar moet beschikken over een goed inkomensbron als een bewijs van uw snelle terugverdientijd), in plaats van bloot aan risico's van deze kwade en harteloos mensen hier... e-mail me recht-weg bij "[email protected]" bent u geïnteresseerd..WEES ECHT!!GEEN GIMMICKS!.. Bedankt...JEANNE

Vote up Vote down Report this comment Score: -1 (1 votes cast)
Alone, you REALLY ha