Ten Years Later: Where's the Beef? My letter to AJP RE: Stephen V. Faraone, Ph.D., et al. claim that D4 receptor linked to ADHD using Meta-analysis as proof...

I can be a mean bugger sometimes.  I read some of my old letters and I think, “Am I still that angry?”  I wish I could say no.   In any case, this letter  of mine from 2002 was published by the American Journal of Psychiatry along with  a reply; a reasonably polite one, I must admit, when I compare it to my original in retrospect.  In it, they seemed to suggest that my point wasn’t clear on a couple of occasions. I was happy enough that the American Journal of Psychiatry printed my abrasive and contrary letter, but now that any fool can have his own personal blog, I will see if I can make it clearer.  Here I will print their response and explain my case:

1.We agree that meta-analysis can be misinterpreted if an original positive finding is included. Dr. Pittelli overlooked our Table 3, which showed that the meta-analysis was significant when this study was exclude. 

2. There is no statistical basis for Dr. Pittelli’s assertion that a meta-analysis is biased if it includes a study from the person who performed the meta-analysis. He also overlooked Table 4, which showed that the meta-analysis was significant after omission of our study. Dr. Pittelli claims that research presented at national conferences or solicited from the ADHD Molecular Genetics Network e-mail list is biased; this is incorrect. The main bias of concern to meta-analysis is that negative studies are published, not that conference reports are more positive than published studies. Dr. Pittelli asks why we think it unfortunate that studies of ADHD and the D₄ receptor gene have not consistently confirmed their association. It is unfortunate because the use of small study groups to detect small effects obscures findings and inhibits progress.
3. The claim that a meta-analysis of studies of ADHD and the D₄ receptor gene must include all other genes tested makes no sense. Studies of different genes test different hypotheses. Mixing apples and oranges does not clarify any statistical analysis.
4. Dr. Pittelli incorrectly claims that whether ADHD genes truly exist remains an unproven assumption. The twin literature about ADHD clearly indicates that ADHD is one of the most heritable of psychiatric disorders (1). Moreover, the genetics literature is consistent with a multigenic theory of ADHD (2), despite Dr. Pittelli’s claim that such a theory is circular.We never claimed that our meta-analysis proved the D₄ receptor gene to be a susceptibility gene for ADHD. We concluded that the extant data were strong enough to warrant further studies of the D₄ receptor gene and ADHD.

Alright, I will try to answer all their points by giving a hypothetical meta-analysis below the fold:

Let’s say that there were dozens of studies documenting potential correlations between a person’s favorite snack food and, say, singing talent.  These studies looked at the following snacks:

1. Hostess cupcakes
2. Twinkies
3. Bomb Pops
4. Ding Dongs
5. Oreo Cookies
6. And a few hundred other ones…

Now, we might first wonder why the “few hundred other ones” are going to be ignored.  The reason they are ignored is that previous studies only found the 5 noted above to be correlated to singing talent.  However, here’s where a step is ignored.  If you are inquiring about  hundreds of  different snack foods and only a few of these give you positive results, then these could simply be the few false positives you might statistically expect when you inquire about hundreds of different snack foods.   Now, you might try to decide why these particular snack foods are the only ones that gave positive results.  I suppose, you might conjecture that the curly double line on a Hostess Cupcake has a certain artistic sensibility to it that might make it more popular than some other snack food for someone with musical ability.  Let’s create some hypothetical data for the 5 studies:

Suppose 2 studies for Hostess Cupcakes show a correlation.  3 others show no correlation.

All the rest only really show the one study that originally flagged them and did not show a correlation on the other 4 studies.

Well, is it really fair to now do a meta-anaylysis of these 5 studies?  You already know what the outcome is going to be before you begin if you are an expert in this field of study (particularly if you did one of the studies yourself).  Why is this point important?  Because you are not blinded.  You have already looked at the studies and, while you might not be able to crank out the exact math in your head, you have to have a pretty good idea that doing a meta-analysis on Hostess Cupcakes is going to give you a positive result.  How do we know you already know that?  Because that is the only one you did a meta-analysisi on.  Why didn’t you also do it for Twinkies, Bomb Pops, Ding Dongs and Oreo Cookies? Because you knew those would not come out positively and the point of the exercise is to find a positive result.  All I’m saying is that doing a meta-analysis doesn’t really give you any more information.  You are using it to bolster the last remaining possible correlation from all of those studies.  Get rid of that, and you have a whole house of cards for these genetic studies.  We already know that ALMOST all of the snack food correlations were indeed false positives.  We have to hang our hats on the last remaining that has not been completely debunked to date.  Now, does that mean that if they can’t replicate the studies showing Hostess Cupcakes to be linked to singing ability, they will pack up their bags and give up on the idea?  Of course not.  Because even though the Hostess Cupcake studies have not been replicated years later, they are on to new snack food correlations, such as Snickers Bar, Suzy-Q’s and Blue M&M’s.  Again, I will argue that they are just the latest false positives, but someone will notice enough of a correlation to one of these new snacks to do a meta-analysis again.   

I hope that makes it more clear.  In any case, I win, since almost ten years later and they aren't able to show that the D4 receptor has anything to do with ADHD, right?  Well, you can never win in debunking the shell game of genetic linkage studies for mental illness, because they will always have a new crop of false positives to hang their hat on... 

Let me add, to finish addressing their points, that the fact that something is "heritable" is not the same as saying it is genetic.  I know that the two terms have been intertwined by scientists these days, but something is genetic if you can find a gene for it.  Something is heritable if it tends to run in families.  The fact that it runs in families is interesting, but does not prove that there is an ADHD gene or genes embedded in the DNA of humans.  This is a common fallback argument for those who do genetic linkage studies.  Twin studies do not bolster the case for their bad studies.  Moreover, in the case of something as nebulous as the wastebasket diagnosis of ADHD (for children and adults), the idea that you are going to find a genetic link is wishful thinking.  Just about every kid with a behavioral problem gets this diagnosis at some point. There are a myriad of issues that may be related these problems.   The fact that you can give the kid amphetamines to help him pay attention better doesn't make the diagnosis anymore valid.