ESOF brings together scientists, policy makers and business people to debate how research and innovation impact science, policy and society. Science communication and peer review is part of this discussion and the “Peer review – the nuts and bolts” session attracted a large number of participants, most of whom had some experience in conducting peer review.
The Panelists
The session was organized and moderated by Julia Wilson, director of operations at Sense about Science. The four panellists came from varied backgrounds but were all involved in peer review in different ways.
Irene Hames, editorial and publishing consultant and former director at COPE, painted a bigger picture of the changing peer review landscape and new and emerging peer review models and innovations.
Franciso Lacerda, professor of phonetics at Stockholm University in Sweden spoke from the perspective of a reviewer.
Bahar Mehmani, Publishing Innovation Manager at Elsevier with a research background in theoretical physics presented case studies of pilots of models of peer review and reviewer recognition and mentoring programs.
I described the editor’s role in peer review and how my perspective changed on transition from being a researcher in genomics to becoming an editor for a range of biology journals.
Themes and Thoughts
The audience was very engaged and challenged the panel with some very interesting questions. The emerging themes were trust, bias, transparency, recognition and quality. Below are some thoughts from the panel and the audience regarding these themes.
We agreed that trust needs to be at the heart of the interactions between authors, editors and reviewers. We are all part of the scientific communities around journals and can influence the standards in our respective fields.
We agreed that trust needs to be at the heart of the interactions between authors, editors and reviewers. We are all part of the different scientific communities around journals and thus we are able to influence the standards in our respective fields. Reviewers carry a burden of responsibility in light of the trust that editors and eventually the reading public have in their ability to assess the quality of the science and the reliability of the findings presented by the authors. Early career researchers need to learn to recognize trustworthy journals and organizations and they need to be careful when approached by third party services that offer help in writing and submitting a manuscript but which are not always legitimate or reliable.
One of the challenges faced by everyone involved in publishing is bias inherent in the decisions we make. Authors may perceive reviewer recommendations, editorial decisions and journal policies as biased towards or against specific institutions, research findings or even entire fields of research. It was suggested that perhaps the fairest system would be double-blind peer review, where the authors do not know the identity of the reviewers, and the reviewers do not know who the authors are and who they are affiliated with. However, this system is not easy to implement in practice as surveys have shown that reviewers are often able to guess who the authors are.
Transparency was put forward as a way of building trust and overcoming bias.
Transparency was put forward as a way of building trust and overcoming bias. Open peer review, publishing reviewer reports and increased transparency and accountability of the peer review process help to open up the ‘black box’ and also provide great learning tools for new reviewers. The process of peer review needs careful management in order to work properly, and any innovations should be tested and changes to the process should be based on evidence. I spoke about the open access journal Research Integrity and Peer Review which aims to stimulate and bring together research into this area.
While tremendous effort is put into peer review, sadly here is a noticeable lack of recognition of that effort, especially from reviewers’ institutions and research councils. Publishers and organizations put forward a number of projects aimed at recognizing, rewarding and mentoring reviewers, for example Elsevier or Publons. However, while it is easy to count the number of reviews submitted, it is much harder to assess their quality. An unbiased and objective assessment of quality of both research output and the process of peer review remains a challenge.
A revolution in peer review?
In conclusion, it was a thought provoking session. Given the ESOF motto of ‘Science as Revolution’, someone asked if we should be expecting a revolution in peer review. Should we do away with journals and substitute them with something better? It seems the feeling from the panel was that in light of much innovation and change that is already happening, we may not need to completely overturn the current system. Instead, we should support existing initiatives aimed at improving it and make sure that journals change with the times.
Comments