Writing in BMC Medicine last week, one of our Medical Editors, Jigisha Patel, made a case for training and specialization in peer review. With kind permission from The BMJ, here we republish a piece by Jane Feinmann which takes a look at the article’s recommendations.

This article first appeared on The BMJ blogs.

Blind faith that the publication of medical research in peer reviewed journals elevates a study to the status of “the evidence,” and therefore “the truth,” may be on the wane among those in the know. But for the public, and a vast number of doctors, this “naïve and misplaced” credulousness persists.

According to Dr Jigisha Patel, medical editor of BioMed Central, this idea must be challenged. Writing in an opinion piece published this week in the open access journal BMC Medicine, Patel calls for innovation in medical publishing that addresses the quality of peer review, rather than simply the process. To this end, Patel proposes mandatory training for peer reviewers, with a “kitemark” identifying research papers that have been peer reviewed by people with the necessary skills.

What is in effect the launch of a new initiative, which parallels the demand for greater transparency in clinical trials, has been commended by Professor Doug Altman, director of the Centre for Statistics in Medicine at Oxford University. In his open peer review, published alongside the online paper, Professor Altman says the issues raised “are of major importance to the integrity and value of the medical research literature.” The problems identified by Patel are well known, although “not amenable to easy resolution,” he says.

According to Patel, these problems can be summed up as: “the aims of peer review are poorly defined, with no evidence that it works, and no established way to provide training.” There is evidence, she says that “reviewers fail to detect deliberately introduced errors and do not detect deficiencies in reporting methods, sometimes even suggesting inappropriate revisions.” Yet current initiatives focus on the need to reduce delays for authors and the burden for reviewers, while ignoring the fundamental weaknesses of a system on which evidence based medicine relies.

What’s more, adds Altman, “editors generally explain inadequately to reviewers what they are expecting them to do [so that] . . . peer review is quite poor on some of the major issues.” Indeed, as a peer reviewer, he chides Patel for understating the scale of the problem. In an early draft, she quotes only one study as evidence of peer review’s all too frequent poor quality. Yet Altman points out: “In fact, we know from dozens of reviews of peer reviewed publications that many published articles contain methodological errors and also that reporting is poor . . . [to the point that] key information is frequently not provided.”

So what’s the way forward? Patel points to CONSORT (Consolidated Standards of Reporting Trials), updated in 2010, as providing a model for establishing a single, universal framework, which should be taken on board by any journal publishing randomised controlled trials.

But peer reviewers need more than CONSORT, Professor Altman told The BMJ. “This is a guideline for reporting what was done, but [it] does not give guidance on how to judge whether what was done was sensible. And we can see very clear reporting of very bad, biased methods,” he explains. Which leads us back to Patel’s essential point: reviewers need to learn “to spot fundamental flaws,” and be tested on this skill on a regular basis.

A wide number of bodies—including medical schools, regulatory and accreditation organisations, funding bodies, publishers, journal editors, and lay people—would need to be involved, although “initially a positive response from journal editors could get the process going,” she told The BMJ. The endpoint would be peer reviewed papers with ‘‘a searchable quality assurance symbol . . . so that readers [including lay people] know whether a study was reviewed by at least one appropriately trained and accredited expert.”

Professor Altman wonders whether, on its own, such a scheme “would have much impact,” as well as whether the suggested quality assurance scheme is practical. “Partly this is because there is no truth about what is a reliable paper, or indeed what is a high quality review.”

But he does see value in further discussion of the evidence that peer reviewers “miss a lot” and are “generally rather poor at detecting errors and omissions.” And he is in favour of postgraduate mandatory training for peer reviewers, which he suggests could be provided by research organisations. “All parties need to take steps,” he says. “We should try to make progress and this paper offers one way forward.”

Jane Feinmann is a freelance medical journalist and copywriter based in London. She writes about patient safety issues including pharmacovigilance, mental health, ageing, and women’s issues.