Reviewing the review

Few resources are available to support less experienced peer reviewers of literature reviews. A recently published commentary in Research Integrity and Peer Review will help to fill this gap.

Peer review can be considered the skeleton upon which scientific research is built. Researchers engage in peer review constantly- we review, we are reviewed, and over time, we oversee the peer review activities of others. Given the central role that peer review plays in improving research integrity, not to mention in our professional lives, you’d imagine that peer review would receive a lot of research attention. Well, yes, and no.

Peer review has been described as being like common sense, multi-factorial, of many parts. Attempts to unpick peer review to find out what makes it work best (or at least better), have proved frustrating. Somewhat like studying clouds, which appear so solid from a distance, peer review seems to dissolve under close scrutiny.

One feature that has emerged is that younger peer reviewer age is reproducibly associated with higher-quality manuscript reviews. This is indeed good news. The peer review system badly needs more reviewers, as publication rates continue to rise, some say unsustainably.

Younger peer reviewers, with relatively fewer responsibilities to juggle, would be really welcome.

To compound these rising numbers, manuscripts are also becoming larger and increasingly multi-disciplinary. Investigator workloads are also rising, many would say just as unsustainably. Peer review often has to be done “as time allows”, alongside administration, teaching, and of course producing manuscripts and grant applications that ironically require peer review.

So yes, younger peer reviewers, with relatively fewer responsibilities to juggle, would be really welcome. And there’s plenty of resources out there to train these peer reviewers, right? Well again, yes and no.

There are great resources for reviewers of manuscripts describing original data and indeed much of this advice is broadly applicable. But what about specific advice for other publications, such as literature reviews? Surprisingly, such peer review guides are hard to find.

This was brought to my attention by a former student, when she was asked to review a literature review for the first time. She’d reviewed original data manuscripts, but never a literature review, and she hadn’t been able to find any form of guidance. After firing off an email containing some rudimentary advice, I decided that this was something that needed investigation.

The increasingly unwieldy volume of original articles means that literature reviews are more important than ever. Just as investigators are overloaded with work, they are overloaded with information. Increasingly, many report that they just can’t keep up.

Enter the trusted literature review. These are read by investigators already in the field, to check that they haven’t missed anything, and to enjoy someone else’s overview and perspective. However, literature reviews can also be used by those entering a field. Some of these entrants may be intending to stay long-term.

Literature reviews need to be reliable, particularly for those readers who won’t be able to pick up errors of fact.

Others may be looking for a one-stop shop of information when, for example, triaging experimental results. These readers, and I count myself firmly amongst them, may use a single literature review to decide whether to keep reading, or move on.

Like most investigators, I’ve written original data manuscripts, and literature reviews. My experience of peer review of these manuscript categories has, on the whole, been different. Sometimes, the peer review of original manuscripts can feel like a fight to the death, gloves off. However, I’ve no doubt that my publications have emerged the better for this experience, however painful it may have felt at the time.

The peer review of most of my literature reviews has been a genteel affair by comparison. Peer reviewers can seem less confident of their opinions, and I’m less sure that the manuscript has been pulled apart to expose any flaws.

These reviewers also sometimes asked odd questions- for clarifications and explanations that I thought were too obvious to include. But here, I was forgetting my own use of literature reviews. I’d forgotten the many times that I’d read a literature review, without having previously read a single line on that topic.

So literature reviews need to be reliable, particularly for those readers who won’t be able to pick up errors of fact, and may unwittingly propagate these errors further. Literature reviews also need to sort the wheat from the chaff, while remaining fair and free from bias.

They need to show where the field is heading, what’s missing, and where it needs to go. And while doing all of this, they need to remain accessible. So it makes sense that the peer review process needs to be specifically tailored to fulfill these ambitions.

My own how-to guide is unlikely to represent the last word on this topic. There will be points that I’ve missed, or perhaps overstated. Furthermore, my article deals only with narrative literature reviews, and not systematic reviews.

There is undoubtedly scope to develop similar advice for other review sub-types. Nonetheless, if this article leads to greater focus on the peer review of different manuscript types, and encourages more people to review literature reviews, my efforts (and those of my peer reviewers- thank-you!!) will have been worth it.

Share this post

View the latest posts on the Research in progress blog homepage