March 4, 2009
Impulsivity In Kindergarten (Does Not) Predict Future Gambling
Here's how (not) to read a study.
Most people read this (Reuters):
Distracted kindergarteners [sic] become gamblers: study
or perhaps the actual study:
Results A 1-unit increase in kindergarten impulsivity corresponded to a 25% increase in later self-reported child involvement in gambling (SE = .02). This was above and beyond potential child- and family-related confounds, including parental gambling.
and ask critically, "well, I wonder how they define impulsivity?" or "are there other variables that could explain why distractiveness or impulsivity predict gambling?"
That's not where the money is: we can confidently assume psychiatrists have plenty of measures for impulsivity. The real question is whether psychiatrists know anything about gambling.
Typically, people question the causes of pathology; I want to know, what are they labeling as pathological nowadays?
So, I check:
cards; bingo; bought lottery, instant lottery (scratch and win), or sports lottery tickets; played video games for money or video poker (eg, at arcades); and placed bets at sports venues with friends or on games that require skill (eg, billiards, pool, bowling)
Let's all agree that this is precisely not at all what anybody thinks about when they think about gambling. You might say, "well, it still counts!" but it's hard to say you can predict X when everyone thinks X means something different from what you do.
Or another way of saying it: even if the results are accurate, why should we care what predicts playing?
The answer is, we shouldn't-- and neither do the authors.
II.
Psychiatrists, by which I mean residents, who actually spend any time reading journal articles often despair. They hit something like this:
The equation also controls for child (sex and early emotionally distressed behavior) and family (maternal education and family dysfunction) factors. Our results bear on this fully controlled model: CGBi6GR = a1 + β1 IMPiKE + β2 PGi6GR + 1 CHILDiKE + 2 FAMILYiKE + eit, where a and e represent the intercept and stochastic error, respectively, CGB indicates sixth-grade gambling behavior (6GR), IMP indicates kindergarten impulsivity, KE indicates kindergarten entry, and PG indicates parental gambling involvement, for each individual i.
and succumb to self-doubt. "I don't know enough to interpret studies, I guess. How am I going to learn this stuff?" But while wondering if this study was a good one or not, something sneaky happens to them: the premise is absorbed, implicitly. They don't realize they've just been infected by a mind virus. "It's so hard to tell if this study really showed what we all know to be true...."
P values and ANOVA are red herrings. The single most important question to ask when reading any scientific paper, ever, anywhere, is this question only, pay attention: what do the authors want to be true?
Nothing else matters. If you think there's a scientist somewhere publishing a paper that says, "son of a gun, I've devoted my life to this question and it turns out I've been wrong all along" you are beyond a crack pipe. Even if the research comes up negative, the scientist's thoughts inevitably turns to, "that study was flawed." The premise is never questioned.
Don't be fooled: "well, in our lab we looked at the serotonin receptor, and it didn't pan out. So we accepted that we were wrong, and changed focus to the dopamine receptor." Do I need to tell you that that is the exact same thing?
III.
The mistake is thinking the purpose of the study is the outcome of the study. The issue is not problem gambling, otherwise we would be limiting it to people with a gambling problem. The issue is not trying to predict which child will become a gambler. The issue is ADHD:
"Attention problems are a public health issue," she said.That's what the study is about. The gambling is just another excuse to say ADHD needs treatment.
IV.
Once you ask the key question, you'll know exactly where to look for inanities. The Introduction is usually full of them.
Data suggest that in most cases, youthful recreational gambling predates pathological gambling in adulthood. (1,2,8,9)
The authors cite four studies for this premise. Usually, a reference means a primary source that has direct data about the issue. I know without looking that none of those four references are that. I know this not because I know anything about gambling or pediatric studies, but because I know that if they're going to count bingo and bowling as gambling, they're not going to be too careful with their references in the Introduction. "But it's stuff everyone already knows." My point exactly.
NB: I'm not saying they knowingly pick non-supportive references. They absolutely think those references are supportive. They're caught in a self-reinforcing groupthink.
To illustrate my point, I looked up the four references. Only reference 9 had direct data on how many adult gamblers started as kids. The other three references (1,2,8) were about related issues (e.g. psychiatric comorbidities in gambling). They simply asserted the child-adult link in their respective Introductions. They, too, referenced their assertions: references 2 and 8 both used the same other reference to support that claim-- reference 9! Reference 1 and 2 both cited the same author, who wrote in two different venues.
Reference 9 found that 20-30% of adult pathological gamblers began "wagering" before age 15, as defined by 10 types of gambling, including "stockmarket, sports, and games of skill." Not only is this not really the definition the average person uses for gambling, it also isn't the definition used in this kindergarten study.
V.
I want to emphasize that studies are never about discovery, they are about confirmation. Researchers already know what they want to find. This is why I have long argued that financial biases in research are trivial, if for no other reason than they are obvious. Ideological bias, institutional bias, personal bias-- these are far more difficult to detect. There's no disclosure for them. They matter much more. If a researcher's son is on Depakote, how will he interpret a study he does that finds Depakote doesn't work? Would it really matter at all whether Abbott paid him or not?
None of this means that research is corrupt, or that it isn't valuable. But it does mean that you can't accept conclusions as written, and the Discussion section is basically an op-ed.
Nothing wrong with that either, except that everyone thinks it's journalism.
8 Comments