Cancer scientists may be too confident in their ability to repeat the experiments in mice and get similar results the second time, according to a new study that offers new evidence of a crisis called reproducibility in medical research.
For the study, researchers asked 196 scientists to predict whether six mouse experiments published in prominent medical journals could be replicated – that is, they again achieve the same results with the same effect size, as reducing tumors in it Quantity, and the same level of statistical significance, which means that the results are not simply due to chance.
None of the six researchers in the experiments asked that the scientists review acquired the same statistical significance or effect size when they were repeated by researchers in the reproductive project: Cancer Biology, a collaboration between the science exchange and the center For the open science that is independently testing the reliability of experiments published in prominent medical journals (osf.io/e81xl/).
But, on average, the scientists who participated in the survey predicted a 75 percent probability of replicating statistical significance and a 50 percent probability of obtaining the same effect size.
“This is the first study of its kind, but it deserves more research to understand how scientists interpret key reports,” said study author Jonathan Kimmelman of McGill University in Montreal.
“I think there’s probably good reason to think that some of the problems we have in science are not because people are sloppy on the bench, but because there is room for improvement in the way they interpret the results,” Kimmelman said. email.
The work follows numerous reports exploring the crisis of reproducibility of biomedicine. In the past 10 or 15 years, there has been growing concern that some of the techniques and practices used in biomedical research lead to inaccurate assessments of the clinical promise of a drug, Kimmelman’s team writes in PLoS Biology.
The results, however, raise the possibility that training could help many scientists overcome certain cognitive biases that affect their interpretation of scientific reports, the researchers propose.
The study team asked established cancer scientists and apprentices in elite education programs to evaluate the six mouse experiments, and found that more experienced and more influential scientists tended to be more accurate.
“What is surprising here is that researchers are not very accurate, they are actually less accurate than chance, to predict whether a study will be repeated,” said Dr. Benjamin Neel, director of the Perlmutter Cancer Center at New York University.
Participants in the study reported their own level of experience in the field, however, they were not on average the most influential scientists according to the number of publications they had and the frequency with which other researchers cited their work, Neel, who Did not participate in the study, said by e-mail.
“Probably the biggest reason the studies do not hold up is because the sample size is too small. For example, if only 5 to 10 mice are used, a study of 50 animals might not produce the same result,” Said Neel.
“I think all preclinical results must be validated by an independent laboratory before they are used as a basis for clinical trials,” he added.
For patients, even experiments that fail in mice or can not be reproduced can help scientists determine if human research is possible, a necessary step in discovering new treatments, said Dr. Anthony Olszanski, director of the research program. Phase 1 therapeutic development at Fox Chase Cancer Center in Philadelphia.
“Today, it is very difficult to predict whether promising results in preclinical studies will be seen in humans,” Olszanski, who did not participate in the current study, said via e-mail. “However, most of the effective anticancer agents used today became drugs based, in part, on the findings of early investigators.”
