Celebrating Peer Review Week at BMC: What’s next for Research Integrity and the Reproducibility Crisis?

John Ioannidis’ 2005 article Why Most Published Research Findings Are False was an uncomfortable wake-up call for many in the scientific community. The paper has led many to take a more critical eye to published research, leading to a wave of high-profile retractions and several major studies warning of a reproducibility crisis in science. So great is this concern that it has moved out of the academic sphere with governments and regulators taking an interest, such as the 2021 call from the UK House of Commons Science and Technology Committee for evidence on reproducibility and research integrity across research institutions.

Alongside the reproducibility crisis the meaning of the term ‘Research Integrity’ has evolved. Originally used to define a governing ethos for best practice in scientific research and the practices it encompasses, Research Integrity has become a research field in its own right spanning sociology, philosophy, statistics, science education and meta-research. BMC Research Notes recently closed its collection on Reproducibility and Research Integrity, launched in collaboration with the UK Reproducibility Network. The submissions reflect how this field has matured, shifting focus from identifying problems – to offering solutions.

Education

A popular solution to improving research integrity is to educate aspiring researchers on the importance of robust, reproducible scientific practice. Andreas Meid describes a pilot lecture series created to equip their students with this understanding and gives an honest account of what worked and what did not. Dr Meid’s lectures placed focus on developing statistical skills, an approach supported by Penny Reynolds. In her commentary, she lays out her case for better statistical education for all investigators as a solution to the pervasive reproducibility issues found in translating pre-clinical research stating “Properly designed and analyzed experiments are a matter of ethics as much as procedure…reducing the numbers of animals wasted in non-informative experiments and increasing overall scientific quality and value of published research.”

“Properly designed and analyzed experiments are a matter of ethics as much as procedure…reducing the numbers of animals wasted in non-informative experiments and increasing overall scientific quality and value of published research.”

Technical education alone is not enough, argue Ilinca Ciubotariu and Gundula Bosch. They highlight the importance of science-communication to ratify research quality and promote trust in researchers. In their commentary, they showcase the framework of responsible science communication taught to students at Johns Hopkins Bloomberg School of Public Health. They hope fostering this ethos will promote more transparent and accountable science in the future. Daniel Pizzolato and Kris Dierickx also consider ways to stimulate responsible research practices across the hierarchies of academic institutions. They advocate turning traditional ideas of pedagogy on their head through reverse mentoring, where a junior academic shares their scientific insights with a senior colleague. Such approaches are well established across many other industries; it seems high time for academia to catch up.

Institutions

Across the collection, many call for change at the institutional level. Olivia Kowalczyk describes how institute heads can cultivate an environment of reproducible, open research by changing funding and hiring criteria to align with such principles. Sweeping changes in how we value research are called for by Stephen Bradley. He argues that funders should stop using citations and impact factors to assess research and instead make judgements based on quality, reproducibility, and societal value. Scientific publishers must also play their part in the pursuit of better science. Patrick Diaba-Nuhoho and Michael Amponsah-Offeh call for journals to promote the publication of unexpected results and null findings to help dispel the distinctly unscientific taboo of publishing null and negative data*.

Peer Review

Peer review also comes under critique. Robert Schultz ponders the potential of automated screening tools to aid manuscript assessment, hoping it might help to screen a growing influx of new submissions more quickly, allowing reviewers more time to consider the papers they receive. Alexandru Marcoci advocates a total overall to the traditional peer review system. He proposes it be treated like an expert elicitation process, applying methods from mathematics, psychology, and decision theory to mitigate biases and enhance the transparency, accuracy, and defensibility of the resulting judgment. In this way, he argues, peer review will become an intrinsically more transparent and reliable process. An ambitious initiative but perhaps it is time to think big.

Collaboration

I contacted two contributors to the collection with the question, ‘If money were no object – what one initiative would about the greatest improvement in research integrity? One wanted the methods section to be mandated as open access across all journals, believing that transparent methods can lead to better science. The other wished to begin a longitudinal study of academic supervisors and their students to identify the best means to promote reproducible science across generations of researchers. Clearly, potential solutions to the reproducibility crisis are manifold, transcending fields and institutions. In a sentiment echoed across the collection, Andrew Stewart of the UK Reproducibility Network argues that the only way to improve research integrity is to drive systematic change across the key scientific stakeholders: academic institutions, research organizations, funders, publishers, and the government. In agreement, Natasha Drude emphasizes that the research community should “view initiatives that promote reproducibility not as a one-size-fits-all undertaking, but rather as an opportunity to unite stakeholders and customize drivers of cultural change.”

Overall the view of the collection seems cautiously optimistic. Now that we have established the issues of reproducibility and research integrity, perhaps we have a chance to change science for the better. An attitude well summarized in the title of Guest Editor Marcus Munafò’s article –

“The reproducibility debate is an opportunity, not a crisis”.

 

My sincere thanks to my predecessor Dr. Tamara Hughes for instigating this collection and to Professor. Marcus Munafò for putting it together.

*I feel I must note that BMC Research Notes welcomes submissions presenting null and negative results.

 

View the latest posts on the BMC Series blog homepage

Comments