Johns Hopkins Team Wins ‘ADHD-200 Global Competition’ to Develop Imaging-Based Psychiatric Diagnosis Protocol

(NEW YORK - October 12, 2011) The Child Mind Institute (CMI), along with Oregon Health and Science University, today announce the winner of the "ADHD-200 Global Competition." The competition, to develop a predictive tool for ADHD diagnosis based on functional and structural magnetic resonance imaging (MRI) of the brain, is part of a movement towards open science and collaboration fostered by the International Neuroimaging Data-sharing Initiative (INDI). The competition sought to inspire scientists to develop novel approaches to real-world problems. This exciting experiment concluded with 21 submissions from around the world, generated by a mix of disciplines, including statisticians, mathematicians, and computer scientists—some trying their hand at imaging analysis and psychiatric diagnosis for the first time.

The data for the competition was provided by the ADHD-200 Consortium—a self-organized, grassroots initiative, dedicated to accelerating the scientific community's understanding of the neural basis of ADHD through the implementation of discovery-based science. Consortium members from institutions around the world (Brown University, the Kennedy Krieger Institute, the Donders Institute, New York University Medical Center, Peking University, the University of Pittsburgh, and Washington University in St. Louis) made de-identified, HIPAA-compliant imaging datasets from almost 800 children with and without ADHD available to the scientific community via INDI.

The winner is the team from Johns Hopkins University, including Brian Caffo, PhD, Ciprian Crainiceanu, PhD, Ani Eloyan, PhD, Fang Han, MS, Han Liu, PhD, John Muschelli, ScM, Mary Beth Nebel, PhD, and Tuo Zhao, MSc. The Hopkins team scored the most points of the eligible competitors, with 119. One point was awarded for the correct diagnosis—typically developing, ADHD primarily inattentive type, or ADHD combined type. A half point was awarded for a diagnosis of ADHD with incorrect subtype. The team is invited to attend the National Institute on Drug Abuse (NIDA) symposium preceding the Society for Neuroscience Annual Meeting in Washington, DC, this November, where they will be publicly recognized for their accomplishment and given the opportunity to present their work.

The Hopkins team shined with regard to the "specificity" of their method, or its ability to identify typically developing children (TDC) without falsely classifying them as ADHD-positive. The Hopkins team correctly classified 94% of TDC, and showed that a diagnostic imaging methodology may be developed that presents very low risk of false positive, a fantastic result. Highly specific diagnostic tests ensure that the mental health care system runs smoothly, saving money and time as they steer professionals towards the correct diagnosis—even if that is no diagnosis at all.

The Hopkins team was not as effective in increasing "sensitivity," or the effectiveness of the test in identifying true positive ADHD diagnoses. They caught just 21% of cases. (The method was very successful at discerning between the subtypes of ADHD, with 89.47% accuracy.) Other teams demonstrated, however, that there is ample room to improve sensitivity scores. NICA and UNC, while not winning with points, both scored quite high on a joint measure of specificity and sensitivity called the J-statistic, suggesting that tests may be developed in the future which satisfy needs in both of these crucial diagnostic areas.

Depending on the classification approach, the prediction accuracy of chance (i.e., randomly assigning diagnoses) would be 33% or 38.75%—all teams except one demonstrated greater overall accuracy than by chance. Scores ranged between 37.44% and 60.51%. The average was 54.09% for datasets included in the training set, 40.24% for datasets not included in the training set (the overall average was 49.52%).

These results are all the more heartening because of the unique design of the competition. After the teams developed their models with the training sets, they put them to work on imaging information from sites that were not included in the training set. From the beginning the teams worked to build predictive models based on imaging data from MRI scanners they had never seen before. And neither the training nor the validation sets contained data that was gathered in any coordinated fashion. With large-scale, coordinated recruitment of data, the results would almost certainly have improved.

"This is real-life science," said Michael Milham, MD, PhD, ADHD-200 Consortium coordinator, INDI founder, and director of the Center for the Developing Brain at the Child Mind Institute. "The results of the ADHD - 200 show where this research is going and the promise of imaging-based diagnostic tools for psychiatric illness—and the hurdles between here and there." But the future is bright. "The open science model that we're bringing to child psychiatry research is what has helped us tackle childhood cancer, and to begin to get a handle on HIV/AIDS. Massive coordination is going to accelerate the pace of discovery and help a lot of kids."

Damien Fair, PA-C, PhD, competition co-coordinator and assistant professor in the behavioral neuroscience department at Oregon Health and Science University, added: "The results of this competition highlight the potential for investigators to identify biomarkers based on neuroimaging for developmental neuropsychiatric disorders. This prospect has been quite elusive over the years, but will be extremely beneficial for the field if this work can be improved upon."

This was a real world test and its results will guide this field as it grows and develops, which it must before these tools are ready for the clinical spotlight. To illustrate: the highest score to come out of the competition was from the Alberta team, which disregarded imaging entirely to construct their predictive model. The intent of the competition was to explore the diagnostic potential of imaging data, but the Alberta team's result did show that imaging has not yet reached maturity and issued a healthy reminder that we not lose sight of the limitations at the same time as we celebrate the advances. As our imaging collection protocols mature and standards are adopted across sites, the diagnostic power of MRI will only improve.

The ADHD-200 Global Competition illustrates, in addition to the dedication and skill of the participants, three major lessons about imaging research and the future of brain science:

• Open data sharing yields results. The contestants were faced with more data than many of them had ever seen before, but in a short amount of time were able to analyze the data and make working models. This would be literally impossible if this information were not free and freely available. But these samples, though relatively large for the field, are still much smaller than they need to be to make these techniques applicable in clinical settings.
• Collaboration is key. The winning group from Hopkins was made up of statisticians, giving a glimpse into what the future could hold if we dramatically open our disciplines to cross-pollination. This group also made use of a preprocessed version of the ADHD-200 dataset provided by the Neuro Bureau, illustrating how scientists with different skill sets can augment and bolster the achievements of others.
• The future depends on bold and cautious effort. Large, coordinated recruitment of datasets and diagnostic studies are necessary to improve the accuracy and repeatability of these diagnostic imaging methods. And it is precisely because of the fact that we need such bold action to achieve these future goals that we must be cautious about perceptions of this technology now. There are some who claim imaging is ready for "prime time;" but this will never happen if the community does not continue to soberly evaluate this promising area of research.

About the Child Mind Institute

The Child Mind Institute is dedicated to transforming mental health care for children everywhere. Founded by Harold S. Koplewicz, MD, and Brooke Garber Neidich, the Child Mind Institute is committed to finding more effective treatments for childhood psychiatric and learning disorders, building the science of healthy brain development, and empowering children and their families with help, hope, and answers. To learn more, visit childmind.org. 

For more information about the ADHD-200 Global Competition, please visit http://fcon_1000.projects.nitrc.org/indi/adhd200/results.html.