Adjusting research statistical methods could transform mental health care for young people

A new publication by two University of Cincinnati researchers contends that adjusting how researchers approach their statistical analysis has the potential to change the lives of children and adolescents struggling with mental health issues across the world.

Jeffrey Mills, Ph.D., and Jeffrey Strawn, MD, have been collaborating on interdisciplinary research and data analysis for years. Their latest paper, “Myths of Randomized Controlled Trial Analysis in Pediatric Psychopharmacology,” was selected as an Editor’s Pick for the Journal of Child and Adolescent Psychopharmacology.

Improvements in biostatistics have deepened researchers’ understanding of some of the limitations of analytical methods, especially when it comes to nuances in pediatrics.

The “myths” referred to in the paper’s title are common misconceptions within the field of child and adolescent medicine, specifically in regard to treatments for mental health studied through randomized controlled trials.

“As a physician-scientist, you hope your work is both rigorous and clinically useful. Being selected as an Editor’s Pick means that we’d accomplished this,” said Strawn, professor in the Department of Psychiatry and Behavioral Neuroscience in UC’s College of Medicine.

Strawn, who is also a UC Health adolescent psychiatrist, collaborates with Mills to make sure data analysis in child and adolescent psychiatric clinical trials is clear and accurate.

“The myths in the paper are problems within statistical analysis, things researchers should keep in mind and errors repeatedly being made,” said Mills, professor of economics in UC’s Carl H. Lindner College of Business. “We’re trying to point those out for people so it’s top of mind, not only when they read the journals, but when they are producing their own clinical analyses.”

Strengthening research through cross-collaboration

Strawn emphasized the importance of interdisciplinary research, or team science, for improving medicine. “Imagine you’re a parent whose child is struggling with anxiety or depression. You’re told a medication doesn’t work because the trial showed ‘no significant effect.’

“The work Mills and I have done together over the years shows that those ‘no effect’ results might not mean the treatment failed,” he said. “Instead, the results might mean the analysis failed to see an effect.”

“In psychiatry, and especially child and adolescent psychopharmacology, trials are hard. The sample sizes are smaller. The statistical noise is greater. And the stakes for young patients are high. We need sound trial design, advanced statistical methods and variance in perspectives,” Strawn said. “It’s no longer enough to ask: ‘Did the drug work?’ We have to ask for whom, when and why.”

Mills said that it is incredibly difficult to squeeze in data analysis while also working as a medical provider, which is one of his motivations for getting involved in adolescent psychiatry as an econometrician.

His expertise in collaboration with Strawn has them questioning the typical approaches to biostatistics, such as 5% being the threshold for statistical significance. Mills argues that 5% is an arbitrary level and that the threshold of significance varies depending on context.

“In clinical trials, you have all that expense, time and effort of all these people involved in your research, and the results are important in terms of helping people. So you should do the best you can with the statistical analysis,” he said.

Strawn said that in research on children and adolescents, there are so many variables that are easy to overlook, such as age, economic status and even specific symptoms. “By improving how we interpret trials, we’re not just tweaking statistics, we are developing ways to get more accurate answers, more personalized treatments and better outcomes,” he added.

The myths of randomized controlled trial analysis

A patient’s environment, including their school, socioeconomic status and potential other conditions are all factors in a treatment’s effectiveness. As it is impractical to control all of these extraneous variables, randomization works better for large samples where these additional factors are less influential on results.

The 5% level of significance is a convention, but not a law. The broader context and possibilities for error are important to consider in pediatrics, where sample sizes are typically small with high levels of variability.

Time and budget constraints often lead researchers to gather a large amount of data with multiple varying factors instead of a smaller, more “reliable” set. But a small, well-designed trial can prove to be much more reliable, even if it takes more time to complete.

With a better understanding of biostatistics, flaws in the “mixed-model for repeated measures” (MMRM) analysis method becomes exposed. In child psychopharmacology, there are complex datasets that may involve nonlinear relationships that could benefit from newer, dynamic methods of analysis.

It is common to adjust the statistically significant level for a dataset depending on the number of tests conducted. Correcting for multiple comparison bias can be overly conservative, potentially resulting in missed effects in the data.

Prediction is one of the most crucial steps of the scientific method of evaluating a hypothesis. By only testing if data from a researcher’s own sample supports a given hypothesis, it is easy to miss whether that hypothesis can predict treatment results on patients that are not part of the current study.

Refusing to revisit data as new theories emerge can be counterproductive to scientific advancement. In fact, more data and new and improved analyses can enhance the reliability of clinical findings even further.

The duo hopes that their work will continue to forge the path ahead for transdisciplinary research and collaboration. “There’s a lot of cross-fertilization that has happened in terms of what I’ve learned from working in econometrics and what Dr. Strawn knows working in psychiatric medicine,” Mills said.

“Team science is critical in modern clinical research,” Strawn said. “Mills and I speak different scientific languages—psychiatry and economics—but we have the same goal. That is to say, we are both committed to getting better treatments to patients faster and improving the treatments that we already have.”