Kevin Drum, a blogger I like, linked to this PLOS One paper from a number of years ago showing which trials sponsored by the National Heart Lung and Blood Institute published beneficial results between 1974 and 2014. “Beneficial” means the treatment group did better than the comparison group to an extent that chance could be excluded as an explanation. Fewer treatments were found to be beneficial after 2000 which was when study investigators were required to register a study’s primary outcome in advance of the completion of a the trial at ClinicalTrials.gov. Mr. Drum concludes:
Before 2000, researchers cheated outrageously. They tortured their data relentlessly until they found something—anything—that could be spun as a positive result, even if it had nothing to do with what they were looking for in the first place. After that behavior was banned, they stopped finding positive results. Once they had to explain beforehand what primary outcome they were looking for, practically every study came up null. The drugs turned out to be useless.
It’s hard to tell if he’s being sarcastic, but his conclusion is misinformed and dangerous regardless. Mr. Drum is wrong to imply that a null trial is unsuccessful. In many cases, trials are testing wide spread medical practices that had never been evaluated, and when they are finally evaluated using rigorous methods they are revealed to be ineffective. Many post-menopausal women were already on estrogen treatment when the WHI found this was ineffective and potentially harmful. The WHI is a success in that it prevented further harm even though it doesn’t get a star on the chart. The only unsuccessful trial is one that fails to provide a clear cut result.
Mr. Drum’s fantasy of scientists combing data and some how “torturing” them is odd, and I would be shocked if the results presented before 2000 had “nothing to do with what they were looking for in the first place.” I’ve been an observer and a participating researcher in clinical trials for about 20 years. I don’t think data analysis practices change much from before to after 2000. The key data analysis principles have been in place at least since I started graduate school (1983). What has changed is how strict major journals are with the trial conclusions they are willing to print; they began to focus rigidly and exclusively on prespecified primary endpoints. This is good and bad. It’s good in that there is transparency, and it does cut-down on false-positive conclusions – the treatment appears better but it isn’t. But, it can also lead to more false-negative conclusions – the treatment is better but it appears not to be. Researchers often use multiple measures to make sure their conclusions are robust. A paper by Kitzman et al presents clinical trial data related to the potential benefits of exercise and weight reduction to manage a kind of heart failure. Quality of life (QOL) was a primary endpoint of the study. The prespecified measure was the Minnesota Living with Heart Failure Questionnaire score. The team also assessed QOL using the Kansas City Cardiomyopathy Questionnaire and another very widely used questionnaire, the SF-36. The weight loss group improved on the Minnesota Questionnaire by 6 units but chance could not be ruled out as an explanation. Weight loss also improved QOL based on the two other questionnaires and chance could be ruled out in both cases. The Journal did not allow the authors conclude that weight loss improved QOL despite the fact that all three questionnaires showed improvement, and two convincingly so. So, who is helped here? Are overweight/obese heart failure patients to conclude that they might feel better if they lost some weight? According to the journal editors NO, but any reasonable interpretation of the totality of the findings would conclude that they might very well feel better if they followed this course of action.
The attribution of the pre – post differences seen in the graph to the venality of scientists is dangerous given the crises facing the human race the management of which depend on scientific analysis. Rick Santorum was just quoted as saying that climate change is mostly a thing because climate scientists want to become rich– hopeful lies while the ice-caps melt. Certainly scientists have egos and many enjoy the attention that discovery brings, but Mr. Santorum and Mr. Drum should be ashamed to undermine those who seek to learn and share the truth about the natural world to benefit us all.
.
The Aging Report 