Results-free review is a new model of peer review aiming to reduce publication bias, preventing impressive ends justifying poor means. BMC Psychology launched results-free review in December 2016; today the first article to have undergone the results-free peer review process, by Zorana Zupan and colleagues from the University of Cambridge (UK), was published. The authors also discuss their research in this blog.

The authors of the article and Koula Asimakopoulou, the Associate Editor who handled the peer review of the article, talk to us about their thoughts on results-free review.

How do you think results-free review could help move towards a more robust publication process?

Authors: Results-free review is one of several measures that could improve the scientific process towards the goal of research that is more transparent, reproducible and efficient as described in the Manifesto for Reproducible Science. Results-free peer review shifts the focus of the decision to publish from the results of the research towards the importance of the research question and the adequacy of the methods used to answer it. Given that the stages of the scientific process are inter-connected, introducing measures at the evaluation stage may also positively impact other stages. For example, researchers may focus more on ensuring robust designs for their studies if these are the key point of evaluation, as in results-free peer review. Further, if findings are not central to the evaluation, this may dis-incentivize researchers from data dredging (also known as p-hacking) to produce statistically significant results to increase the chances of publication.

Associate Editor: It is well known that studies reporting null results are less likely to receive favorable attention in terms of publication than studies reporting significant effects. The difficulty with this approach is the resultant publication bias that can take place and which, in some cases, can mask our understanding of science, to include cases where results showing that eg. Drug therapy is no better than placebo never appear. For this reason, a results-free review is likely to help a more objective peer review process to take place.

Do you have any previous experiences of publication bias?

Authors: We have experience of journals rejecting manuscripts describing studies that aimed to replicate and extend previously published studies because they are not seen as sufficiently novel.

Associate Editor: Yes – especially with systematic reviews where there are not enough studies showing an effect to justify their publication.

Well aware of the bias against null results we opted for a review process we judged could guard against this

As far back as 1977, Michael J. Mahoney found that reviewers judged the methods and quality of null studies more critically than those of positive studies. More recently, John P. A. Ioannidis identified a substantial time lag in the publication of trials reporting null results, with null results taking several months longer from the time of submission to eventual publication than positive results. Could results-free review offer a solution to current barriers of publishing null results?

Authors: Results-free review does indeed have the potential to overcome the publication bias against null results. This is a testable hypothesis.

Associate Editor: Of course.

Our experience of the review process was positive. It seemed like a more scientific approach to peer review with the paper judged on the question asked and the methods used to address the question.

As an author, what factors influenced your decision to opt into results-free review?

Authors: Our study provided no evidence to support the hypotheses tested. Well aware of the bias against null results we opted for a review process we judged could guard against this.

Did taking part in results-free review impact the preparation or reporting of your manuscript at all?

Authors: We prepared the manuscript in the same manner as for all our publications then submitted the sections sequentially.

What was your experience of having peer review conducted in this way? And how did it compare to standard peer review?

Authors: Our experience of the review process was positive. It seemed like a more scientific approach to peer review with the paper judged on the question asked and the methods used to address the question.

Associate Editor: It helped me focus on the methods of the study. The difficulty that I did have was that it felt like I was missing a big part of the paper and in many respects a paper can have a great method section but totally incomprehensible / poorly analyzed results, which worried me at first. So it felt like I was reviewing half a paper rather than a complete piece of work which will take some getting used to!

Do you think that the open peer review model further improves the transparency of results-free review?

Authors: Open peer reviews could help build confidence in the results-free process. Moreover, there is some evidence that open peer reviews are of higher quality than those prepared anonymously, although the potential risks of open-identity peer review should also be taken into consideration.

Associate Editor: Most certainly. I am sure we can all think of peer reviewers who have been unkind at times, probably inadvertently. I find that an open peer review system encourages reviewers to think carefully about their recommendations.

From the readers’ point of view, do you think that results-free review could change the appraisal of a study’s quality?

Authors: Results-free review does indeed have the potential to alter how readers approach papers, highlighting the importance of the research question and the methods used to address this as the context in which results are to be understood. If it results in more studies being published with null findings this would provide an evidence base that is more representative of a field than the one that currently exists.

Associate Editor: I think it places greater emphasis on ensuring that when you read a Method section and an Introduction, these two tie well together and are logically related. So in that sense, yes, a results-free review may well make it harder for papers to get through a first review, unless the sections that are available to the reviewer are particularly well written.

From your perspective, what are the key pros and cons of results-free review?

Author: There are many advantages of results-free review. As well as removing the bias towards publishing statistically significant findings, this process could improve the quality of systematic reviews if it fosters a less biased pool of published studies, and help in generating unbiased hypotheses. Overall this would improve the quality, reliability and credibility of scientific findings.

In terms of cons, the division of the peer review process into two stages may require additional work for editors and reviewers, with the potential to lengthen the peer review process, although this was not our experience. Without formal evaluation, it has the potential to appear as though it is addressing important sources of bias in the evaluation of research findings but may not actually do so and thereby inadvertently lead to false reassurance about its contribution to reproducible science.

Associate Editor: The pro is obviously a move towards minimizing publication bias. The cons have to do with the paper feeling incomplete when you first look at it!

Can you see a future for results-free review in becoming the ‘norm’ for peer review?

Author: Yes, we can see this becoming the norm. However, high-impact journals must lead in adopting these procedures.

Associate Editor: Yes

Results-free review offers one potential solution to tackling publication bias. However, what other key steps do you think need to be taken to address the problem?

Author: Other key areas to tackling publication bias are identified in the Manifesto for Reproducible Science. These include: pre-registering studies, data sharing and incentivizing good practice.

Associate Editor: Dedicated outlets for studies reporting null results may be helpful.

Your contribution to the future of peer review
The number of new publishing initiatives is on the rise and it is important that these are rigorously evaluated to ensure they are having the desired outcomes. The evaluation of results-free review, will be conducted by our randomized controlled trial, in order to determine how (an if) it influences publication bias and the editorial decision-making process. If deemed effective, it is our hope that results-free review will become the ‘norm’ for peer review.

In a field which has been plagued by the under-reporting of both replications and null findings, BMC Psychology has always had a policy of publishing all sound science, thus aiming to reduce the impact of publication bias. We have never made editorial decisions on the basis of the perceived interest of a study or its likely impact. We hope that results-free review is another feather in our bow in the effort to improve the reproducibility of our published research even further.

BioMed Central is committed to combating publication bias and promoting reproducibility. In a complementary approach to results-free review, BioMed Central is now also offering authors the chance to submit Registered Reports, another format shifting the emphasis from research results, to the scientific questions being asked and the strength of the study design. BMC Biology is the first BioMed Central journal to embrace the format and is now welcoming submissions. You can learn more about the initiative by reading this blog from the journal’s Chief Editor.

Do you want to help shape the future of peer review? Simply submit to BMC Psychology and click to opt-in to results-free review.

Need more information? Read our Editorial or contact us at bmcpsychol@biomedcentral.com.