Peer review frequently comes under fire for failing to detect important deficiencies or recognize outstanding work, and frustration with its flaws has led to calls to abandon it altogether. However, a lot of these opinions are based on surveys and personal experiences, and there are not many studies about the actual process itself.

BioMed Central’s Research Integrity Group in collaboration with Frank Dudbridge, an academic Associate Editor for BMC Bioinformatics, analysed the quality of reports and recommendations made by reviewers under open and single-blind peer review. Under ‘open’ peer review, all parties know the players involved, reviewer reports are signed and if the manuscript is accepted, they accompany publication. Under single-blind ‘closed’ peer review the reviewers are anonymous.

We also compared reports from reviewers originally suggested by the authors to those found by the Editor independently. A preliminary version of our findings was presented as a poster at the 7^th Peer Review Congress in Chicago. The full research paper was published in BMJ Open.

Does the peer review model matter?

We compared two journals that are part of the BMC series, which share the same policies and processes, and have similar rejection rates, size and threshold. BMC Infectious Diseases operates open peer review while BMC Microbiology operates single-blind peer review. We used an established Review Quality Instrument to rate 200 reviewer reports in each journal and we found that the overall quality of the reports was 5% higher for BMC Infectious Diseases (open peer review), compared to BMC Microbiology (single-blind peer review). Interestingly this finding is also consistent with findings from the British Journal of Psychiatry.

Reviewer reports for BMC Infectious Diseases (open peer review) provided more detail on the strengths and weaknesses of the methods, and the reviewers’ comments were more constructive and better substantiated compared to reports for BMC Microbiology. This is reassuring for supporters of open peer review, however our findings do not undermine other journals using closed peer review.

In order to attempt to control for differences between fields which may have impacted on our findings we also wanted to compare open versus closed peer review on a single journal. We therefore rated 200 reviewer reports for the Journal of Inflammation from the time when it operated open peer review and 200 reviewer reports submitted after the journal implemented the single blind model in 2010.

We did not find a difference in reviewer report quality in this case, which agrees with the findings of a previous randomized controlled trial reported in The BMJ. However, the change in peer review model was accompanied by a change in editorship and coincided with the journal receiving its first impact factor. This may have impacted on how the peer reviewers were selected and how they perceived the journal and made recommendations, but not related to the peer review model per se.

Does it matter who suggested the reviewer?

The manuscripts we analysed in BMC Infectious Diseases and BMC Microbiology had two reviewers each, one suggested by the authors and one by another party (the editor, another reviewer or the BioMed Central’s PubMed search tool). We found that the quality of reports written by author-suggested reviewers was similar to the reports coming from the other reviewers although they were more likely to recommend acceptance. We obtained the same result for the Journal of Inflammation. This echoes the research on BMC-series journals from 2006 and research done in other subject areas (for example see here and here).

Is open peer review better than single blind?

Our results have shown that open peer review is of at least as high quality as single blind. Open peer review has both pros and cons, but overall it seems to be gaining in popularity, as shown by a growing number of open peer review journals at BioMed Central, and the success of F1000 Research or PeerJ where about 40% of peer reviewers choose to sign their reports. Open peer review is deemed ethically superior by some and as Richard Smith observed, science is moving away from anonymity.

To answer the above questions, further research into peer review is needed. As there are not many venues for this kind of study, BioMed Central has recently launched Research Integrity and Peer Review covering all aspects of peer review, standards of reporting and research and publication ethics. The four co-Editors-in-Chief are Stephanie Harriman, Maria Kowalczuk, Iveta Simera and Liz Wager.