October 6, 2009

The Problem With Science Is Scientists

one of these is a false positive, the other is false

I.  In case you haven't already heard the story:

In a recent fMRI study, a salmon was shown a series of pictures of human faces showing various emotions: can a salmon distinguish them? and what brain regions are involved.  15 pictures, ten seconds each.

I won't bore you with the anatomy.  Because of the small size of the brain, exact brain structures could not be distinguished, but something in the brain did light up.  A statistically significant number of voxels, comprising an area of 81mm3 in the midline of the brain, were active (p<.0001).

So can fish interpret human emotions from a picture?  I have no idea.  I do know, however, that that fish can't do it: it was dead.


Others have discussed the hows/whys of such false positives and what can be done about them.  But there are two other problems not discussed:

These researchers chose a dead fish specifically so they could discuss the issue of false positives and why multiple comparisons correction in MRI studies is important.  Thus, we know these results are false positives because we know that the fish is dead.  Note carefully, however, that both of the things you know are told to you by the researcher; yet you are valuing one as "truth" and the other as "artifact" based on nothing but his word.

The researchers might have been mistaken about the deadness of the fish-- thus nullifying a potentially interesting finding.  Or, they could have lied.

There is no way to check.
  You're rolling your eyes, "why would they lie about that?" or "how would they possibly make a mistake about it being dead?" and you're right, about this they wouldn't.

But what about the old studies?

If they do a study in which an anxious person shows weak activity in the amygdala-- how do you know he was anxious?  How wrong about the anxiety do you have to be to invalidate the weakness of the MRI findings?  Not much.


The danger of the "false positive" discussion is that it is forward looking: from now on, why should tighten our significance thresholds, change the confidence intervals, controls, etc.   But what about all the prior data that finds only "moderate positive correlations" using more liberal significance thresholds, that may be infected by invalid behavioral assessments--

--that because of the passage of time alone-- not better data, but time-- are now knowledge?


The biggest problem with MRI studies is that they're hard for the layman to understand.  Complexity in science protects prejudice.

Understanding the Anxious Mind, in the NYT Magazine, discusses the science of temperament.  Jerome Kagan studied babies, then followed them over the years.  Predictably (i.e. what you'd expect the NYT to say), temperament as a baby predicted temperament as an adult, especially in the extreme cases.

They explore the case of the highly anxious "Baby 19" (defined as being distressed by novelty) who, when she was 15, was a plain looking teenager who liked writing, playing the violin, worrying and fidgeting.  See?  Genetics.

...[Scientists] have put the assumptions about innate temperament on firmer footing, and they have also demonstrated that some of us, like Baby 19, are... born predisposed to be anxious...

...[other scientists] all have reached similar conclusions: that babies differ according to inborn temperament...
You'll observe that those two quotes are about babies-- babies are born a certain way.  No argument from me.  What they do not say is that the inborn temperament is the reason they are also anxious as adults, but that's the conclusion they make every single time.

"Temperament, it turned out, tended to be stable over those five years, at least in children who started out at the extremes."
  Its stability is the evidence that the temperament is biological.   If his haircuts are stable over the years, is that biological?

But more importantly, the kids were raised by parents.  Parents don't parent in an ideal dispassionate manner, they parent in reaction to the kid in front of them.  In other words, kids' temperaments alter the manner in which they are parented, and it's a good bet that the parenting fosters that same temperament.  Not a word on that; it's as if it that couldn't possibly be relevant.

And why is testing a four month old's behavior evidence of an innate quality?  The first four months of parenting don't count?  "Kagan restricted his sample to children who were white, middle class and healthy."  Oh.  So now all white people are the same?


But that's not real science, real science uses MRIs.  If they studied it in an MRI, it must be true.

Teenagers who were in the group at low risk for anxiety showed no increase in activity in the amygdala when they looked at the face, even if they had been told to focus on their own fear. ...In the high-risk kids, even those who were apparently calm in most settings, their amygdalas lighted up more than the others' did.

"Overreactivity in the amygdala" = anxiety.  But we don't really know what the amygdala does, nor how it does it, all we know is where it is.   Saying something occurred in the amygdala is like saying something occurred in Ohio.  "Yes, but we have some sense of what the amygdala does."  And I have a sense of what Ohio does, too, it causes trouble in elections and gets its teens to kill themselves.

But at least whether or anxiety is mediated by the amygdala is worth discussing.  What you can't do is take a structure that may be involved and therefore conclude that anxiety is an innate trait that is generally stable.  Every time I punch someone my shoulder is overactive.  Is my genetically mediated shoulder the cause of my alcoholic rages?

And overactive as compared to what?  What could possible serve as a control?  Seriously, think about this.  Point to a guy you believe could be a control in a study measuring what are here defined as subclinical levels of anxiety.

Not every brain state sparks the same subjective experience; one person might describe a hyperaroused brain in a negative way, as feeling anxious or tense, while another might enjoy the sensation and instead uses a positive word like "alert."

None of those words mean anything.  Brain state?  Hyperaroused?  Alert?  How can anyone know that the "brain state" that two people are describing differently is the same?  These words concepts are so vague that the researcher has to resort to Jungian terms in his descriptions:

The persona can be controlled, but the anima often cannot... Nathan Fox of the University of Maryland says that when the anima erupts in high-risk children, it often takes the form of excessive vigilance and misdirected attention. In the first of his two longitudinal studies...

If you don't even have precise words to describe what you're seeing, how in God's name can you measure it, let alone blame it on the amygdala?


But research has to start somewhere, and my problem isn't with the researchers or their study, nor do I doubt the relevance of genetics.  My problem is that when theory is written up in the NYT, it becomes FACT, it becomes the default understanding.  This understanding becomes part of our cultural filter.  In the same way porn and Cougar Town has assured us that women over 40 can have satisfying extra marital sex with 20 year old bicycle  messengers, we know that behavior is, in large part, genetically determined.

This is how the article ends:

The predictive power of an anxiety-prone temperament, such as it is, essentially works in just one direction: not by predicting what these children will become but by predicting what they will not.  In the longitudinal studies of anxiety, all you can say with confidence is that the high-reactive infants will not grow up to be exuberant, outgoing, bubbly or bold.

Think about this.  Think about what the average person now understands to be true. 

Still, while a Sylvia Plath almost certainly won't grow up to be a Bill Clinton,  she can either grow up to be anxious and suicidal, or simply a poet.


Back to the salmon.  The results were statistically significant, but the fish was dead.  So we laugh.  In 1620, that would have been evidence for the soul, and no one would have laughed.

The problem is the same in both cases.  They are questioning the nature of the data.  They should be questioning the nature of the fish.

part 2 here.


"If you don't even have pre... (Below threshold)

October 6, 2009 6:06 PM | Posted by Mark V Wilson: | Reply

"If you don't even have precise words to describe what you're seeing, how in God's name can you measure it, let alone blame it on the amygdala?"

Not to mention doing so longitudinally.

Vote up Vote down Report this comment Score: 2 (2 votes cast)
Not to mention the fact tha... (Below threshold)

October 6, 2009 8:09 PM | Posted by beelicious: | Reply

Not to mention the fact that half the people are either asleep or nearly asleep in the scanner. This is why I got out of imaging and into services research.

Vote up Vote down Report this comment Score: 1 (1 votes cast)
Let's hear a big AMEN (Dani... (Below threshold)

October 7, 2009 3:18 AM | Posted by David Johnson: | Reply

Let's hear a big AMEN (Daniel, that is) for this post. I'm gonna ask for chips with my next fMRI.

Vote up Vote down Report this comment Score: 0 (0 votes cast)
"My problem is that when th... (Below threshold)

October 7, 2009 7:24 AM | Posted by Andy: | Reply

"My problem is that when theory is written up in the NYT, it becomes FACT"

Same with most press coverage of most empirical studies. The BBC news website is particularly poor, but luckily a lot of the time the reporting of fMRI research is dull, so does no harm.

Cancel that. Just googled "bbc news fmri" and the first hit contained "You could use it to detect people's prejudices, intuition and things that are hidden and influence our behaviour."


We're doomed.

Vote up Vote down Report this comment Score: 3 (3 votes cast)
re 'we don't really know wh... (Below threshold)

October 7, 2009 8:49 AM | Posted by Mike T: | Reply

re 'we don't really know what the amygdala does, nor how it does it'- there have been some very concise and concrete studies in the last few years showing the role of the hippocampus (adjacent to the amydgala) in the role of processing short term memories into long term memories. A casual search for 'amygdala' and 'memory' this morning turned up quite a few research hits too. Note this comment is on 'what the amydgala does', I am still trying to understand the main points of your post. It DOES sound like the people who did the dead fish experiment had some kind of an agenda before they started gathering data. Agendas can get in the way of unbiased research.

Vote up Vote down Report this comment Score: 1 (1 votes cast)
Mike T: Doesn't everyone ha... (Below threshold)

October 7, 2009 10:50 AM | Posted by Matt: | Reply

Mike T: Doesn't everyone have an agenda? I'm not sure I know what unbiased research looks like.

Vote up Vote down Report this comment Score: 1 (1 votes cast)
Sounds like way too ... (Below threshold)

October 7, 2009 11:10 AM | Posted by La BellaDonna: | Reply

Sounds like way too much latitude with their longitude.

Vote up Vote down Report this comment Score: 1 (1 votes cast)
When you find a trout in th... (Below threshold)

October 7, 2009 4:38 PM | Posted by medsvstherapy: | Reply

When you find a trout in the milk, you think, hm ... is this heah a bowl of chowdah?

When you find two trouts in the milk, you think, hm... this fahmah watahed down his milk with watah from the creek.

When you find three trouts in the milk, you think ... wouldn't this farmer make more money just selling fish?

Vote up Vote down Report this comment Score: 2 (2 votes cast)
I'm glad you touched on tha... (Below threshold)

October 7, 2009 5:35 PM | Posted by Mae: | Reply

I'm glad you touched on that Anxious Mind article. I got bored after the third page, due to the arm-waving excitement of those involved. I'm sure someone will suggest we fashion a way to remove the amygdala...

Vote up Vote down Report this comment Score: 0 (0 votes cast)
Don't you know we are all j... (Below threshold)

October 7, 2009 9:24 PM | Posted by Anonymous: | Reply

Don't you know we are all just DNA robots?
When someone becomes "mentally ill" the scientists just have to find the correct chemical cog to get the brain spining properly again.
The foundation of psychiatry is that the brain is physically damaged in some way, if it wasn't damaged, what are you prescribing Doctor? Medicine or drugs?

Vote up Vote down Report this comment Score: 1 (1 votes cast)
I think you mean I threw... (Below threshold)

October 7, 2009 11:42 PM | Posted, in reply to La BellaDonna's comment, by Alone: | Reply

I think you mean I threw out the baby with the bathwater, that I took a small discussion about multiple comparisons corrections and condemned all of science. Not exactly. The focus on the minor problem of fMRI errors distracts us from seeing the major problem of nomenclature and definition. The error is in the "anxious" not in the number of voxels.

Vote up Vote down Report this comment Score: 1 (1 votes cast)
When I originally read that... (Below threshold)

October 8, 2009 2:13 PM | Posted by fraise: | Reply

When I originally read that article, that bit about Sylvia Plath was infuriating. As if her anxiety and suicidal temperament had nothing to do with the way she was raised ("Daddy" just came out of thin air? It had no basis in reality?) or with her unfaithful husband. And "just" a poet? Because if she hadn't killed herself, she wouldn't be remarkable? I realize this next statement is going out on a limb to draw connections, but damn is it ever getting worrisome to see how anti-intellectual US culture is getting. "We all know a [male] president is better than a [female] poet," one can easily read between the lines.

They chose Plath. Not Ernest Hemingway, for instance.

Vote up Vote down Report this comment Score: 4 (4 votes cast)
Interesting points. You con... (Below threshold)

November 3, 2009 9:46 AM | Posted by amygdala: | Reply

Interesting points. You control by using a within subject group, so each person acts as a control. Basic research elements right there. Your argument about not knowing about the parts of the brain is poor. Every research scientist uses a vernacular acceptable in their clique. I'm guessing that unless you've had a background in physics or mechanical engineering that you won't understand half the things they bring up either. Neuroscience is the same. From my perspective the amygdala is boring because it is always brought up and most psychology people with a decent background or neurobiology people will know exactly what it does. If you don't, then take it upon yourself to educate yourself.

Personality traits are thought to be stable and the article you cited is just echoing that common knowledge, not bringing up something new.

I do believe there are some weaknesses with the fMRI research and I believe that you bring up interesting ideas but you should educate yourself first about the things you are talking about to be taken seriously.

Vote up Vote down Report this comment Score: 0 (4 votes cast)
By analogy: if we see a ... (Below threshold)

November 3, 2009 5:02 PM | Posted, in reply to amygdala's comment, by Alone: | Reply

By analogy: if we see a car in a car park, do we know what it does?

When they find more or less activity in the amygdala, you cannot infer from that its significance. It's fairly well established that the amygdala is involved in several different and various behaviors and emotions, e.g. appetite, fear conditioning, explicit memory. No one (including me) disputes the evidence there; the dispute is about the application of the result "activity in the amygdala" to only one of those (e.g. anxiety.) It could mean anything.

The next step is the worst: the conclusion is reversed to use as evidence for other things, e.g. now, whenever we see amydgala activity, it must be anxiety.

"Every research scientist uses a vernacular acceptable in their clique." My point exactly. "Personality traits are thought to be stable... common knowledge. This simply isn't true, and again the extrapolation problem. _The point_ of this study was to see if they are stable. It's not common knowledge, it's popularly assumed.

I'm not sure what you meant about within-subjects design (I know what it is, just not what you're referring to.)

Vote up Vote down Report this comment Score: 5 (5 votes cast)