September 30, 2007

Ten Things Wrong With Medical Journals


I know, right?  Only ten?



This is how references are done now:




















This is madness.

I could make a career out of exposing references that have nothing to do with, or directly contradict, the referenced statement. (Here's one.)  This system of referencing makes it very hard to do this-- and I wonder if that's not the point.

It is, by contrast, very easy to link the exact article referenced-- even the exact page in the article.  Even Jezebel does this.

Online-- assuming it's not a pdf even though this shouldn't matter-- when you click on the superscript it takes you to the references, which has a further link to the actual article (but not the statement.)  Ok-- why the extra step?

Note that the way references are done here is antithetical to science.  Look at the reference pictured above.  What do you see?  What is important?  What you see are the authors and the journal, not the scientific content.  That's what's implied to be important.  We're supposed to accept the science of a statement by the force of the author and journal?  But that speaks to my later point about bias.


Why should government or Pharma sponsored research require me to pay someone else (the Journal) for access to that information?  And why only licensed academics?  So, if I'm a welder in Kentucky, I can't know what's really up with Depakote for bipolar?  I have to read some nut blogger? 

It is a simple process for a Journal to host online all articles as free text.  Or, better, scientists can publish their work on their own site (hosted by a university, etc, if necessary.)

Neither are Journals necessary as repositories of vetted information.  There are numerous ways these scattered articles can be collated, packaged and even summarized for easy use.  Slashdot, Digg, and others are very effective in this regard; and something similar can be done with science.  I know, Digg can be gamed. What, Journal of Clinical Psychiatry can't?


Where's The Raw Data?

Rephrasing from above: why am I not allowed to see the raw data from a government sponsored study?  (And from Pharma-- if they agree to do a study, then they must agree to make all data public.)

You may have heard that there are rumblings about making this data accessible-- but not to everyone; only to those with appropriate access (academics, etc.)  Again-- why?



This is a common lament, but it misses the point: it is artificially slow.  As in on purpose.

Articles that are submitted for peer review should simply be published in a "pending peer review" section.  Other sciences already do this.  To the criticism that doctors may act on unreviewed science, it should be noted that citing "personal communication" (e.g. an email) as a reference is perfectly acceptable.  Is that safer? Oh, and about that peer review:


Peer Review

Even with the most malicious intentions, it would be nearly impossible to create a worse system than peer review.   Peer review does not have the potential for bias; it is specifically designed to retain bias, and to maintain the primacy of subjective opinion over objective findings.  The only people who support peer review are other peer reviewers. If necessary, money should be diverted from pediatric AIDS research to help stop to this oligochracy.  It's that important.

In medicine, "peer review" is the editor of a journal and three other doctors- that the author suggests as reviewers.   While ostensibly the author's identity is unknown to the reviewers, in practice it is simple to determine authorship (type of research; meetings; and even document metadata.)  Oh, and the editor knows who you are.

Most people think peer review is some infallible system for evaluating knowledge.  It's not.  Here's what peer review does not do: it does not try to verify the accuracy of the content.  They do not have access to the raw data.  They don't re-run the statistical calculations to see if they're correct.  They don't look up references to see if they are appropriate/accurate.

So what do they do? They look for study "importance" and "relevance."  You know what that means?  Nothing.  It means four people think the article is important.  Imagine the four members of the current Administration "peer reviewing" news stories for the NY Times.   

On the force of the recommendation of these three reviewers and the editor-- who, by the way, decides whether to even send it to reviewers at all, or reject it outright-- the article gets published or not.  And there is no right to an appeal.

Imagine a movie gets previewed by four people who decide if the movie is important or not, and whether it will play in theatres.  You know what you get?  Notes On A Scandal, that's what.  And riots.

The peer review system also promotes the perpetuation of biases.  Doctors are subtly pressed into writing articles about certain topics-- consider the Depakote madness of 2000-2004; the noradrenergic hypothesis of depression in the 70s (where'd that go?); and how every issue of BMJ has an article on war.   (Except July 2008: that was the month they wrote about whether to boycott Israeli academic institutions.  Ok.)  Academic careers are made, in part, by the number and quality (i.e. journal) of publications, which will be influenced by what they think certain journals would publish. "My research focuses on things my Chairman really likes."  Can't wait to read more about evolutionary psychology.


Lack of Debate

There is no way to have a meaningful debate about an article within the Journal system.  How do you crowdsource a medical study?  As an example: if I find a logical error in an article (e.g. mistaking correlation for causation) I can only point this out by writing a "Letter To The Editor," which, you will be surprised to learn, goes to the editor.

Even if it is published, my Letter will have little attention.  But while anyone smart enough can critique a study, only an academic can write the Letter.

It is unnecessary to point out that the rest of the internet-- including news-- works very differently.

Disclosure of Conflict of Interest

Almost completely invalid for its intended purpose.

If a doctor does a promotional program, a "drug talk", he has to disclose the relationship.  But if a doctor is dating a drug rep-- that relationship he doesn't need to disclose. 

Even more strange is that these are commercial conflicts of interest, only.  If you are a communist, or Priest of Scientology, or a serial pedophile; these are not disclosures, even if your article happens to be "Incidence Of Pedophilia Among Communists."  Neither is being funded by the NIH (any surprise that NIH studies always find that the generic is the best?)  Or being married to the Chair at Harvard. Or having a son on the drug.

Aren't personal beliefs a bias?

To single those out commercial interests as somehow more damaging, more biasing than any others is preposterous.  It's not a false sense of security; it's a deliberate misdirection from all the other things that actually bias science.  And it sidesteps the entire point of scientific articles-- if they are truly scientific, if the articles were truly "peer reviewed"-- it shouldn't matter what your biases are.  I could own Pfizer.  The article on Zoloft should be able to stand on its own.

It's worth observing that the peer reviewers are not asked to disclose any of their commercial interests.



No exposition needed.  Either less words or better words.


Abstracts As Promos

See this blog post, where it starts out, "I know, right: only ten?" and then you have to click to get the full article?  So my promo has really no useful information in it.  You know why?  Because I am a blogger, that's why.

Contrast that with the abstract from an important study on Lamictal for maintenance treatment in bipolar (emphasis mine):

Conclusions  Both lamotrigine and lithium were superior to placebo for the prevention of relapse or recurrence of mood episodes in patients with bipolar I disorder who had recently experienced a manic or hypomanic episode. The results indicate that lamotrigine is an effective, well-tolerated maintenance treatment for bipolar disorder, particularly for prophylaxis of depression.

The Conclusions seem to say Lamictal is good for preventing "mood episodes", mania and depression-- is there any other way to interpret it?  In fact, this study shows it is only good for preventing depression, not mania at all.  Why is it written this way?  Because the authors want to advance the idea that Lamictal is a "mood stabilizer" and not what it actually is: an antidepressant.

You have to understand that most doctors do not read the study, they don't even read the abstract-- they skim the abstract.   For this reason, the abstract has to be an accurate summary of the article, not a promo for an idea. But that's why it is written this way; it's not about the findings, it's about the authors' agenda.

What's stupid about this is that negative findings are as important to a clinician as positive findings.  They are less important, of course, to academics whose careers depend on positive findings, and the drug companies who sponsor them.


Pick up a medical journal-- inside you will see drug ads.   I haven't heard many people complain that this influences the science in the journals, the way authors' "conflicts of interest" is supposed to.  But before you respond, consider that  the ads are only for one product per class.  For example, in the NEJM, there is a two-page, full color ad for Lipitor, but none for any other cholesterol drug.  Oh, my mistake-- there are two, two page ads for Lipitor (running $32,000 per issue).  Same with one inhaled insulin; one antidepressant (Effexor); one sleeper (Rozerem) etc, etc.  If having ties to Pharma influences the outcome of science, what is the effect of having a financial ties to only one Pharma company per class?  (1)

Inflation In Studies

Reducing the value of something by increasing its availability is inflation.  This is magnified when the thing in question didn't have much value to begin with.  Three strategies:

  • The same data set, or the same thesis, is reworked into several different articles for different publications.  This may seem a benign way to pad your CV, but what it does is fool people into thinking something has more support than it actually does.  This is precisely how Depakote became a bipolar colossus.  
  • A topic is investigated multiple times, when even one time was too many.
  • A finding is described as novel or at least interesting, when it had already been published years earlier by a less "important" researcher.

(nod to Glen for this one)



1. I wondered if psychiatry journals, having a more limited range (e.g. no insulin ads) would have broader coverage of companies.  They did, sort of.  Archives, CNS Spectrums, Primary Psychiatry, etc, all had multiple antipsychotic ads (never more than three brands, though) but always only one antidepressant.  Not sure what to make of that.


Dear Doc,Isn’t any... (Below threshold)

October 1, 2007 6:44 PM | Posted by Herb: | Reply

Dear Doc,

Isn’t anything sacred with you? Isn’t anything off bounds with you?

I can’t tell you how much I enjoy your writings, information and even more importantly exposés.

Damn if you don’t upset me. Here I am a support person and care giver making my best efforts to be reasonably informed and educated so as hopefully to make the best possible medical decisions for my charge and you constantly and consistently shoot down almost everything that I would hold dear and rely upon for “truth and accuracy” as well as safety.

And yeah, I’m also in agreement with you about referencing and subscriptions. Damn, how I also hate these journals “nickel and dimeing” folks like me looking and digging for information. It’s a good thing I have some doctor friends and other folks who have access to the full documents but I hate to always be asking for these favors and their time.

Oh well, for that matter I seem to also be in agreement with your 10+ plus items.

So thanks for you thoughts and your particular “bias”. It keeps me thinking, reading and sharing my own brand of “bias”.


Alone's response: thanks for the compliment-- did you DIGG and REDDIT it??? And your being a support person looking for answers speaks to my point- why aren't you allowed access to it, and why can't you trust it? If it's science, shouldn't it be "just the facts?" As for sacred-- you haven't heard me make fun of Missouri, have you?

Vote up Vote down Report this comment Score: 4 (4 votes cast)
I actually saw a medical bl... (Below threshold)

October 1, 2007 7:38 PM | Posted by phill: | Reply

I actually saw a medical blog running on Slashcode once; I think it was neuroscience-specific. I've forgotten where it was,so if anyone knows the site, post it. Slashdot probably isn't the best model for the kind of system you're suggesting though, as it also relies on editors to decide which stories are worthy of posting.

Alone's response: oh, I didn't know that. But I think youget my overall point.

Vote up Vote down Report this comment Score: 0 (0 votes cast)
I couldn't get the full stu... (Below threshold)

October 2, 2007 3:36 AM | Posted by acute mania: | Reply

I couldn't get the full study because I'd have to pay for it, but from what I read of it's not a great antidepressant either. The people on lithium did relapse into depression more than those on lamictal, but they were all switched to lithium after being stable on lamictal. The conclusion of the study completely disregarded those (iirc, half of the original sample) who couldn't get stablized on Lamictal in the first place. The fact that it fared WORSE than lithium even with a responder enriched population in the mania study shows that it's not even worthy of consideration for mania.

Alone's response: Wow-- you got all that from only part of the article? But you are right on the money. P.S. If I send you part of a treasure map...

Vote up Vote down Report this comment Score: 2 (2 votes cast)
Thank you for this post. I... (Below threshold)

October 2, 2007 8:09 AM | Posted by CL Psych: | Reply

Thank you for this post. I posted a somewhat similar rant (maybe a bit more anti-Pharma but similar points) a few months ago, and we need every medical blogger to throw a bloody tantrum on this issue. Immediately. I suspect that medical journals need to be shamed sufficiently in order to change. Perhaps that would require negative media coverage and lots of it. God knows that journals are not going to change just because it's the right thing to do. Please keep up the heat. I'll do likewise as, I'm sure, will others.

Alone's response: yo, brother, DIGG it (and REDDIT it)-- that's the only way to keep up the heat. Oh, and if you think it's bad now, wait till Google gets hold of electronic medical records...

Vote up Vote down Report this comment Score: 0 (0 votes cast)
Nice post. I love reading y... (Below threshold)

October 3, 2007 9:26 AM | Posted by Lee: | Reply

Nice post. I love reading your blog.

I've had so much frustration with not being able to access articles I want to read
this one for example
The title struck my interest because I have Trichotillomania that is primarily pulling body hair and I have wondered if laser hair removal would help. But it sure would suck to do it and have my hair pulling just move to locations where I would like to keep the hair.
Anyways ones like that are frustrating because it doesn't even have a brief summary saying the results. It just wanted me to pay 39 dollars for an article by people I've never heard of that could potentially be useless to me.

A lot of colleges have access to various databases of articles too. But they're all separated with separate usernames and passwords and it can become pretty complicated to figure out what ones I have access to via my school and how to get access to that specific one (some require being in a certain ip range, others a password no one has ever told me these passwords).

Alone's response: Also weird: people in the media have access to the journals. I'm sure they pay, of course, but again, why should CNN have access to something we don't? As for laser hair removal, I am against it (unless you're a model, then I am so for it.). It is a near certainty you will move to other body hair, or to other "techniques"-- chewing your lip, picking your cuticles, etc.

Vote up Vote down Report this comment Score: 0 (0 votes cast)
"nearly impossible to creat... (Below threshold)

October 3, 2007 9:34 AM | Posted by Steve: | Reply

"nearly impossible to create a worse system than peer review"

There is one: law reviews.

Multiple submissions to student-edited, non-peer reviewed journals which rely almost entirely upon author name recognition and informal approval by faculty at the journal's school.

Alone's response: you know, I had this thought in the back of my mind. Let me think about this.

Vote up Vote down Report this comment Score: 0 (0 votes cast)
These are the type of thing... (Below threshold)

October 3, 2007 10:26 AM | Posted by Kevin: | Reply

These are the type of things that should be taught in residency programs. Not just taught, but emphasized...OVER-emphasized. I feel we (as psychiatrists) have to accept the admittedly overwhelming challenge of tackling the neuroscience that the pipeline of new drugs will be based on (purportedly). If we don't, it will become easier and easier for the pharmaceutical companies to wow us with "expert" speakers capable of rattling off names of neurotrophic factors, receptor subtypes, pathways that we're unfamiliar with, and with the aid of slick PowerPoint presentations and a surf & turf, will segue to the conclusion that we need to switch to drug X -- never mind the string of non sequiturs preceding that "conclusion." The moral of the story being, we need to be increasingly cynical of the "evidence" thrusted at as, and that effective cynicism will be to be rooted in a strong neuroscience base in addition to sufficient research/bio-statistics acumen to be able to scrutinize these articles.

Vote up Vote down Report this comment Score: 1 (1 votes cast)
Please, please, please....k... (Below threshold)

October 3, 2007 5:29 PM | Posted by Louis: | Reply

Please, please, please....keep writing articles like this.

"YET ANOTHER 10 things wrong with: Medical Journals/Scientific Studies (bad design)/Medical Practice Today/The Way People (try to) Educate Themselves on Health Matters, Nutrition, etc."

You could make "Yet Another 10 Things Wrong With..." your trademark -- and do a great service for humanity (and its caregivers).

Please do, Sir!

Your greatful fan,


Alone's resposne: PLEASE, please keep DIGGING and REDDITING these articles if you like them. Donations are certainly appreciated, but traffic is just as good...

Vote up Vote down Report this comment Score: 1 (1 votes cast)
I can't help myself, but I ... (Below threshold)

October 4, 2007 12:58 AM | Posted by Raj: | Reply

I can't help myself, but I know a few of you are from Boston or nearby. This Saturday at Northeastern University there is a Neuroscience conference, let's take this fight to the man.

Alone's response: Hey-- make sure you cite me!

Vote up Vote down Report this comment Score: 0 (0 votes cast)
Thank you Alone. T... (Below threshold)

October 6, 2007 6:16 PM | Posted by Anonymous: | Reply

Thank you Alone.

The really pathetic part is most industrial (and probably academic) scientist pay is related to publication in prestigious journals or patent applications filed. As a result, you get a lot of schlocky publications and patent applications. Management hasn't figured out a way to measure the scientific productivity except by quantity, rather than quality.

Alone's response: I know. Worse, many academic centers are de-emphasizing publications, and emphasizing grants-- so fast science chases fast money, in order to keep the career (and consequently, you get about nine million bipolar researchers, all studying seizure drugs-- until they go generic, and suddenly the grant money dries up and everyone's interested in something else. Check out my post, "Farewell Depression.")

Vote up Vote down Report this comment Score: 0 (0 votes cast)
This may be the one Herb is... (Below threshold)

October 8, 2007 4:05 AM | Posted by VelvetElvis: | Reply

This may be the one Herb is thinking of:

Vote up Vote down Report this comment Score: 0 (0 votes cast)
Now can you come up with "1... (Below threshold)

October 9, 2007 3:30 AM | Posted by Randall Sexton: | Reply

Now can you come up with "10 things wrong with a double-blind study" that is published in those journals?

Alone's response: sounds like the guantlet is being thrown down! But hey, man, if you thought the article was good, DIGG and REDDIT it.

Vote up Vote down Report this comment Score: 0 (0 votes cast)
As to "Management hasn't fi... (Below threshold)

October 10, 2007 4:59 PM | Posted by KWL: | Reply

As to "Management hasn't figured out a way to measure the scientific productivity except by quantity, rather than quality.", hasn't Google determined a way? Or perhaps more appropriately, isn't Google practicing a way management can measure scientific productivity? That is, Google's original search algorithm was predicated on references, so that, highly cited pages were given higher rank. In a rewards system that moved beyond the adoption of this practice to a fully realized "cited" model wouldn't that effectively reward those who furthered the science as measured by those who found it useful work? If this were the only way to advance in a career, publish in the means to gain citations from others, the more citations the more important the work, the more quality evident, wouldn't that create a healthier system? Certainly gaming might occur, "cite me and I'll cite you" but wouldn't it wash out in the mean?


Vote up Vote down Report this comment Score: 0 (0 votes cast)
Another thing that is frequ... (Below threshold)

October 30, 2007 7:43 AM | Posted by Glen: | Reply

Another thing that is frequently wrong is that research is portrayed as new and novel when it is not. I have seen many times a 'discovery' labeled as such only to find a reference from 10 years ago. If the editor and the reviewers cannot do simple fact checking what is their purpose?

This didn't even occur to me, you are absolutely right. Actually, this problem is even bigger than that. Many times, the same data is reworked into multiple articles for several different journals; basically, it's inflation: reducing the value of the commodity by increasing its availability. Or the same thesis is investigated several times, and you can't understand why they investigated it int he first place.

Vote up Vote down Report this comment Score: 0 (0 votes cast)
You forgot "hypothetical un... (Below threshold)

March 8, 2008 12:55 PM | Posted by Anonymous: | Reply

You forgot "hypothetical unrealistic and self-aggrandizing two-page semibold-extended-face narrow-column vignettes posing as paper abstracts". This one in particular touts the miraculous redemption of a Seung-Hui-Cho-wannabe schizophrenic teenager.

Vote up Vote down Report this comment Score: 0 (0 votes cast)
Glen:Inflation, re... (Below threshold)

March 8, 2008 1:01 PM | Posted by Anonymous: | Reply


Inflation, recycling and gaming-the-system happen in all science publications. Blame the "quantity beats quality because quality is hard to assess" tenure-track mantra.

Vote up Vote down Report this comment Score: 1 (1 votes cast)