Rigor Mortis

""Rigor Mortis" explores how perverse incentives and a "broken" scientific culture are fueling the reproducibility crisis in modern biomedical research. Published in April, Harris's jeremiad was a perfect fit for my 2017 reading theme on <a href="http://books.max-nova.com/2017-focus/">"The Integrity of Western Science"</a> and an excellent companion to Goldacre's <a href="http://books.max-nova.com/bad-science/">"Bad Science"</a>.<br/><br/>Harris argues that competition for funding, politicized peer-review, and pressure to be the first to publish have enabled sloppiness and dubious practices like "p-hacking" to flourish. He investigates how these incentives have created a lack of transparency in research and how they have damaged the self-correcting nature of scientific research. As one of his interviewees points out, "it’s unfortunately in nobody’s interest to call attention to errors or misconduct".<br/><br/>Diving into the technical details, Harris points out how cell line contamination, non-transferability of animal models to human systems, and statistical blundering (the batch effect!) often render vast swathes of biomedical research unreliable. He also explains how much bad science is driven by conflating "exploratory" and "confirmatory" research. You can't run an "exploratory" experiment, mine the results for correlations, and then publish it as a "confirmatory" result. Or rather, you can - and apparently it is done all the time - but it is bad science. This was a "big idea" for me and is one that I suspect the general public (and even most scientists) are not aware of.<br/><br/>One of the major questions I'm trying to answer in my year of reading about "The Integrity of Western Science" is "How much confidence do we need to have in the 'science' before we start making policy decisions based on it?" As another one of Harris's interviewees notes, "it takes years for a field to self-correct" - this complicates the situation even further. How long do we need to wait before we can have a high level of confidence that a particular finding is real? Harris offers no answers here - he just hammers home how difficult of a question this actually is.<br/><br/>As he discusses the inability to impose standards of rigor and transparency on research, Harris briefly touches on how most biomedical science is funded (The National Institutes of Health) and includes the odd and unexplained quote, “The NIH is terrified of offending institutions.” This strikes me as completely absurd. The NIH is funding the research. Don't they know the golden rule? "He who has the gold, rules." Why shouldn't the NIH mandate some easy wins like an experimental registry and a best-practices checklist for each publication? The institutions (read: universities) can either play ball or... find a more lenient funding source. Would love to read more about this.<br/><br/>As far as concrete ideas for cleaning up science, Harris doesn't offer many new ideas. There were three that caught my eye though. The first is that researchers should report their level of confidence in the results of every study they publish. The second is the idea of a one-time scientific "Jubilee" where researchers can retract any of their old research with no penalty. It's tough to see how these first two ideas could work culturally, but they're intriguing nonetheless. The final idea probably has the best chance of working because it has worked before. Harris discusses the 1975 Asilomar conference where leading genetics scientists developed guidelines to ensure the safety of recombinant DNA research. This seems to be an instance of relatively successful self-regulation of the scientific community, and it's nice to think there might be some hope of the community internally cleaning up its act. I'm not holding my breath though.<br/> <br/>One final side-effect of reading this book was being introduced to the institutions leading the effort to clean up science:<br/><br/>* Center for Open Science - https://cos.io/<br/>* Science Exchange - The Reproducibility Project: Cancer Biology -<br/> http://validation.scienceexchange.com/#/cancer-biology<br/>* Retraction Watch - http://retractionwatch.com/<br/>* Meta-Research Innovation Center at Stanford - https://metrics.stanford.edu/<br/><br/>Overall, Harris's book is a great summary of current issues in biomedical research and it comes at a time when the general public is finally becoming aware of some of these problems.<br/><br/>Full review and highlights at <a href="http://books.max-nova.com/rigor-mortis/">http://books.max-nova.com/rigor-mortis/</a>"