A case for open peer review for clinical trials

2

A few weeks ago I played the part of an expert witness in the Medical Journalist Association’s mock trial, Trials on Trial. The charge was: “Is the current system of publishing clinical trials fit for purpose?” The jury’s verdict was a resounding ‘no’. You can read more about the event in Jane Feinmann’s write up on the BMJ Group Blogs.

As an expert witness on a panel tasked with giving evidence on different elements of publishing clinical trials, I was asked “how does open peer review work and does it address all the flaws of the current system of publishing clinical trials?”.

Blog - open peer review for clinical trials - quoteI gave my evidence on why I believe that open peer review is the best system of peer review for publishing clinical trials. I explain my arguments below.

It’s vital that that articles reporting results of clinical trials have been appropriately peer reviewed as they form the basis of doctors’ clinical decision making. Traditionally, this process has taken place anonymously (‘closed peer review’) so only the editor knows the reviewers’ identities.

image credit: John Martinez Pavliga flickr
image credit: John Martinez Pavliga (flickr)

What is open peer review?
Some journals – including the BMC-series medical journals, Trials, Journal of Negative Results in BioMedicine and BMJ Open – operate a system of  open peer review.

It’s a peer review model that involves two levels of openness. The first level is that authors know who reviewed their article, and the second is that readers know too – if the article is published, a pre-publication history is posted online alongside the article. This contains all versions of the manuscript, the reviewers’ signed reports and the authors’ responses to reviewers’ and editors’ comments.

If you want to know more about the history of open peer review, this blog from F1000 Research gives a nice description and timeline.

blog - open peer review black boxWhat are the problems with anonymous peer review of clinical trials?
There are some drawbacks with closed peer review for clinical trials. Peer review is akin to a ‘black box’ – no-one outside of the journal knows how decisions were made. There’s less accountability for the decision and also room for poor practice and misconduct – editors, for example, may choose the wrong peer reviewers, or reviewers may review manuscripts on which they have an undeclared conflict of interest and potentially stall, or even prevent, publication.

How does open peer review deal with these issues?
It may not be a panacea, but open peer review goes some way to solving these problems. Most importantly it increases transparency – something that is key to the whole process of publishing clinical trials.

blog - open peer review transparent boxIncreased transparency increases accountability:

–  Peer reviewers are more accountable for their reports as they’re attaching their name to them.

–  Editors are also more accountable – their decisions are no longer made behind closed doors. Anyone can see whether peer review was appropriate.

It also brings transparency to the thorny issue of competing interests – rather than simply relying on the editor’s opinion of whether something counts as a competing interest, reviewers’ statements of competing interests in the pre-publication history allows the reader to make their own judgments.

Transparency in open peer review allows doctors to make more informed decisions – they can look at whether research was reviewed appropriately and therefore to what extent they believe the results.

Does it work?
There’s little empirical evidence on open peer review, but from what little there is available, we do know that telling reviewers that their names will be revealed to authors (van Rooyen et al. 1999) and that their reports may be posted online (van Rooyen et al. 2010) does not decrease the quality of their reports.

In our recent research, we compared the quality of peer reviewers’ reports on an open peer review journal (BMC Infectious Diseases) and a closed peer review journal (BMC Microbiology), and found an increase in the quality of reviewers reports on open peer review. This improved quality was  specifically seen in the areas of comments on the methods and study design, supplying evidence to support comments, and constructiveness.

We also know that open peer review is feasible. As argued by Fiona Godlee, experience from her time at both BioMed Central and the BMJ shows us that the model works.

Summing up
Open peer review increases transparency by bringing the peer review process into the public domain to help doctors make informed decisions. It not only provides information on how a decision was made on a specific article, but it also provides data for much needed research into peer review allowing us to better understand the process by which we evaluate clinical trials on which we base clinical decisions.

Despite the jury’s verdict that the current system of publishing clinical trials is not fit for purpose, I would argue that open peer review does go some way to solving some of the issues.

View the latest posts on the BioMed Central blog homepage

2 Comments

By commenting, you’re agreeing to follow our community guidelines.

Your email address will not be published. Required fields are marked *

gaudetjj

Thank you for the excellent piece! Although I beg to differ that there is no empirical evidence on open peer review… unfortunately journal peer review is consistency used as a pre-constructed object of study, the common experience of peer review as process. This is one of the reasons why research on peer review is so contradictory. My doctoral research is on journal peer review, in this short preprint I outline how journal peer review can be constructed as a scientific object of study “Investigating journal peer review as scientific object of study” http://www.ruor.uottawa.ca/en/handle/10393/31161. Scholarship in law has much to contribute in the conversation and in the research…

Reply
gaudetjj

In this preprint, I cite some of the clinical trials performed with open peer review (including those at BMJ) and other open review changes at journals – with their inconclusive results… mostly always being compared with closed review at pre-publication journal peer review. Gaudet, J. 2014. All that glitters is not gold: The shaping of contemporary journal peer review at scientific and medical journals. uO Research. Pp. 1-23. http://hdl.handle.net/10393/31238.

The “improved quality” at open review that you observed is no doubt due to the increased potential for rational decision-making that transparency brings – this is well-documented in legal studies where judges are very seldom ‘anonymous’. The few instances of ‘faceless’ courts there are worldwide provide an excellent example of how anonymity in the role of those who judge can be exploited…

You might also be interested in research on the purported resistance to new scientific ideas by scientists at journal peer review: Gaudet, J. 2014. How pre-publication journal peer review (re)produces ignorance at scientific and medical journals: a case study. uO Research. Pp. 1-67. http://hdl.handle.net/10393/31198

Reply