55 Chapel Street, Suite 202, Newton, Ma 02458

75 Gilcreast Road, Suite 305, Londonderry, NH 03053

Thank you for visiting. NESCA Notes has moved!

For articles after June 4, 2018 please visit nesca-newton.com/nesca-notes/.

Search This Blog

Wednesday, September 19, 2012

Not So Fast: The Statistical Pitfalls of Scientific Spin

NOTE: Please read both of the articles below.

From Autism Speaks - Science

Genetic Test Shows Promise for Earlier (Autism) Screening and Intervention

By Laurie Tarkan
September 12, 2012

Researchers have developed a gene-based test that shows promise for estimating the risk that a child will develop autism before any symptoms appear. Their study appears in the journal Molecular Psychiatry.

“This test could assist in the early detection of the condition in babies and children and help in the early management of those who become diagnosed,” says lead researcher Stan Skafidas, Ph.D., director of the Centre for Neural Engineering at the University of Melbourne. “It would be particularly relevant for families who have a history of autism or related conditions.”

The authors add that their test represents only the first step in the development of a diagnostic test for the future.

In developing the test, the researchers used genetic material and clinical information from Autism Speaks Autism Genetic Resource Exchange (AGRE) and the Simons Foundation Autism Research Initiative (SFARI) databases.

Research suggests that autism results, in part, from a broad array of genetic differences that increase or decrease the risk that a child will develop the disorder. Scientists have struggled to develop a diagnostic test that captures these genetic differences. For this study, the researchers investigated the prevalence of genetic changes called single-nucleotide polymorphisms (SNPs).

Each SNP represents a single nucleotide change in the DNA sequence that spells out a gene. SNPs occur often. Most prove harmless. However, they can also increase or decrease susceptibility to developmental disorders such as autism spectrum disorders (ASD).

A “snip” (SNP) mutation consists of a one nucleotide change in the DNA strand that makes up a gene. Most such changes have no effect. Some increase risk for a disorder such as autism. Still others decrease risk. Image courtesy of Lauren Solomon, the Broad Institute

“Changes in SNPs contribute to our individual differences,” says senior author Christos Pantelis, M.D., of the Melbourne Neuropsychiatry Centre at the University of Melbourne. “The question that we were trying to address in our study is whether a collection of such SNPs are able to act as biomarkers for autism spectrum disorder.”

The new test measures 146 genetic markers of both risk and protection for ASD. The risk markers increase the test score. The protective markers decrease it. The higher the overall score, the higher the individual’s risk.

The test identified ASD with 86 percent accuracy from the genetic samples of a group of families from Utah. It accurately identified ASD in 84 percent of samples from families in Italy. However, it accurately identified only 56 of those of Han Chinese background.

“The use of genetic testing to identify children who may be at risk for developing an ASD as early as possible is promising,” adds Clara Lajonchere, Ph.D. Dr. Lajonchere oversees the AGRE program as Autism Speaks vice president of clinical programs. “Given the extreme heterogeneity of the disorder, however, further research is needed on larger and more diverse samples,” she says.

“Only then can we determine whether these estimates of autism risk will prove reasonably accurate to broader segments of the population.”

Importantly, Dr. Lajonchere adds, the use of genetics in clinical practice has the potential to lead to more targeted and personalized approaches for the development of medicines and effective therapeutics.

Next the researchers plan to assess their test among babies born into high-risk families that already have an affected child. In this way, they hope to assess whether their test can discern risk among closely related individuals.


From Cracking The Enigma - An Autralian Autism Research Blog

A Genetic Test for Autism?

The problem of Base Rate Bias

By Dr. Jon Brock
September 14, 2012

This week's big autism story was a genetic test able to predict with 70% accuracy [1] whether or not a child had autism. Rather than looking for a specific gene that might differentiate autistic from non-autistic people, Stan Skafidis and colleagues from Melbourne University developed the test by combining information about many different genetic variations. Critically, having developed the test based on one set of genetic data, they then tested the test on genetic data from a completely new set of people.

I don't want to take anything away from the basic science. And it would, of course, be incredibly useful to know early on whether or not a child is likely to develop autism. But the headlines are misleading. The unfortunate truth is that we're still a long way from a genetic test for autism. A screening measure with 70% accuracy would only be slightly better than completely useless.

Here's why.

Say you had 1000 kids and you ran the genetic test to see which ones would become autistic.

If we assume that the rate of autism in the population is around 1% then we'd expect 10 of the 1000 kids to be autistic. Given 70% accuracy, we'd expect 7 to show up on the genetic test as autistic.

The problem is the other 990 who aren't autistic. 70% accuracy means that 30% would be incorrectly diagnosed as autistic. 30% of 990 is 297.

Putting those together, our genetic test thinks that 304 of the 1000 children are autistic. Of those, only 7 really are autistic, the other 297 have been mis-diagnosed.

In other words, if your child took the test and the test came out positive, there would still be a 98% chance that your child was not autistic.

[Edit 15/9]: The researchers describe the test as a test of autism risk rather than a diagnostic test. In those terms, what it tells you is whether you have a 2% risk, as opposed to the usual 1% risk.

This screener's base rate fallacy is a well known problem. Dorothy Bishop covered it when discussing screening for autism based on speech recordings and MRI scans. Ben Goldacre talked about it in relation to screening for terrorists. It's a problem any time you're looking for something that's relatively rare. Even a really good test gets torpedoed by a disasterous rate of false positives.

To be fair, the researchers did suggest that the test would be most useful for parents who already had one autistic child. The latest research suggests that, if you have an autistic child, the chances of a second autistic child may be as high as 19%. Even in that situation, you'd still expect roughly two thirds of the kids picked up by the test to actually be non-autistic.

Update (15/09/12)

On Twitter, Emily asked whether such a test could ever be useful. How high would the detection rate need to be to be useful? Given the rarity of #autism, would any test escape the false positive issue?

As a population screening test, I think the answer would be no. Let's assume that the sensitivity (ability to detect true cases) goes up hand in hand with the specificity (ability to reject non-cases). Then for each level of accuracy, we can plot your autism risk if you test positive for autism.

Even if the test was 90% accurate, you've still only got an 8% risk of autism if you test positive.


[1] "Positive and negative predictive accuracies were 70.8 and 71.8% respectively"


Skafidas E, Testa R, Zantomio D, Chana G, Everall IP, & Pantelis C (2012). Predicting the diagnosis of autism spectrum disorder using gene pathway analysis. Molecular psychiatry PMID: 22965006 Full 

Related post

Can MRI scans be used to diagnose autism?

About Dr. Jon Brock

Jon Brock is a Research Fellow at the ARC Centre for Cognition and its Disorders and the Department of Cognitive Science at Macquarie University in Sydney, Australia. His research focuses on the cognitive and neural mechanisms involved in a range of developmental disorders.

1 comment:

  1. Thanks a lot for this wonderful and admiring information. I hope that you continue to provide us with latest details. Thanks again fengshui