Let me clarify one point about the MMR/Wakefield controversy.  The fact that Wakefield faked his data does not prove there's no link.  Right?  I don't think there's a link, of course, but what do I know?  I'm a pirate.

There's a controversy about a paper published in the Journal of Personality and Social Psychology, a highly reputable academic journal.  The paper is about ESP, which is the controversy.


Either (NYT):

Journal's Paper On ESP Expected to Prompt Outrage

One of psychology's most respected journals has agreed to publish a paper presenting what its author describes as strong evidence for extrasensory perception... The decision may delight believers in so-called paranormal events, but it is already mortifying scientists.

Or (NPR):

Could It Be?  Spooky Experiments That See The Future

One of the most respected, senior and widely published professors of psychology, Daryl Bem of Cornell, has just published an article that suggests that people -- ordinary people -- can be altered by experiences they haven't had yet. Time, he suggests, is leaking. The Future has slipped, unannounced, into the Present. And he thinks he can prove it.

All depends on whether you think "scientists don't know everything, man!" or "scientists are fraudsters, man!"

II.



The experiments are of the type: two groups take a test; one group is then shown the answers, the other group isn't.  The ones who see the answers after the test did better on the test.  Weird, right?

The paper describes nine unusual lab experiments... testing the ability of college students to accurately sense random events, like whether a computer program will flash a photograph on the left or right side of its screen. The studies include more than 1,000 subjects. Some scientists say the report deserves to be published, in the name of open inquiry; others insist that its acceptance only accentuates fundamental flaws in the evaluation and peer review of research in the social sciences.

"It's craziness, pure craziness. I can't believe a major journal is allowing this work in," Ray Hyman, an emeritus professor of psychology at the University Oregon and longtime critic of ESP research, said. "I think it's just an embarrassment for the entire field."

Hyman is right but for the wrong reasons, for self-serving reasons, which makes him wrong.   And the NYT assertion that this "accentuates fundamental flaws in the peer review of research in the social sciences" is also wrong, wrong, wrong, wrong.


There's a subtlety to the experiments that is indeed explicit in the articles but is easily overlooked, so I'll quote from the study:

From the participants' point of view, this procedure appears to test for clairvoyance. That is, they were told that a picture was hidden behind one of the curtains and their challenge was to guess correctly which curtain concealed the picture. In fact, however, neither the picture itself nor its left/right position was determined until after the participant recorded his or her guess, making the procedure a test of detecting a future event, that is, a test of precognition. 

This is the part that's important.  If it was a study of clairvoyance, well, could there be a possible physical explanation?  Perhaps.  But time travel?

Which is why anyone who says this study  "doesn't belong in a scientific journal" is wrong.  It doesn't belong in a psychology journal: this is an experiment about the laws of physics, not the laws of psychology. 

And so to say that  it is a failure of peer review-- like they did with Wakefield--  also misses the point.   Bem's peers are in absolutely no position to review this.  This study is better reviewed by physicists.  Bem himself makes an explicit case for quantum entanglement!  So notwithstanding my own rants about peer review,

"Four reviewers made comments on the manuscript," [said the journal's editor] "and these are very trusted people."

Trusted though they may be, they are not experts in the field being studied. 

All four decided that the paper met the journal's editorial standards, [the editor] added, even though "there was no mechanism by which we could understand the results."

Exactly.  So you should have sent it to the physicists.  You know, the ones who work a building over in the same university that you do.  That was the whole reason for universities, right? 

No, I'm a dummy.  The purpose of universities is to suck up Stafford loan money.  And the purpose of journals is to mark territory, more money in that, like a corporation that spins off a subsidiary.  NO CROSS SCIENTIFIC DISCUSSION ALLOWED IN SCIENCE, EVER, EXCEPT IN SCIENCE, NATURE, AND THE POPULAR PRESS.
 

II.

So I'll be explicit: peer review may have problems, but the entire way we evaluate science is territorial and stuck in the 19th century, which, ironically, was a time when scientists were much less territorial and practiced multiple disciplines.

With a data feed to select articles on "psychiatry," what do I need a psychiatry journal for?  If you wanted to be brain scientists, why do you have separate journals form other brain scientists?

How awesome would it be to have an astrophysics grad student or a PhD economist or a dancer or anyone of the mofos from metafilter to come look at a psychiatric clinical trial and discuss it?  You wouldn't have to pay them, they would think it was fun-- what, you think I'm blogging because of the millions of dollars in donations I get from Denmark and now the pacific northwest (6x in a month-- did you guys find a work around for Cybernanny?)

And no, not just the paper data; why not video the whole process and upload it? I have a phone that shoots HD 720p; good enough for the optical demands of amateur porn, why not good enough for science?

If researchers published their paper along with all of the primary source data on a web page, and let the public wikipedia it up, we might discover that a study was crap but we might also learn something about how studies become crap, the biases or hidden pitfalls, etc.  (No, "available upon request" does not cut it.) 

Instead, we have a near idiotic controversy occurring in self-imposed darkness.   "It's a big butt!"   "No, it's a big leg!"   "No, it's a weird snaky-thing!"   "Well whatever it is, don't turn on the light, let's just keep guessing-- this way we can all get publications out of it."