Are current scholarly metrics leading us away from research transparency?

Is it time for a transparency metric? Scholarly metrics are thriving right now, both in number and what (and how) they measure, there is a real interest in them that is literally buzzing.

6

Scholarly metrics are abuzz!

In one manic week in April 2015, all of the following took place:

  • The Committee on Publication Ethics held their annual European Seminar focused on How Metrics Shape Publication Behaviour (summarized in a blog here)
  • The Royal Society rang in 350 years of Philosophical Transactions with meetings focused partially on the impact of metrics on research assessment
  • A Nature publication offered up the Leiden Manifesto – ten guiding principles for best practice in metrics-based research assessment.

The misapplication of metrics

A metric is defined as a ‘standard of measurement’. One popular metric, the journal impact factor, has become a widely used mechanism for comparing journals across fields. It is not uncommon for authors to select a journal for publication consideration based on its Journal Citation Report impact factor or ranking within the field.

The appropriateness of this approach is questionable, given that it does not account for relevance and readership of the journal, among other things. Nonetheless, the seed for such thinking is planted quite early on in a researcher’s career.

Research coming out of the Evaluation Practices in Context group at Leiden University found that life science post-docs tend to value grant money, number of publications and journal impact factors as ‘capital’, influencing their decisions on what research questions to ask, where to work, and even social interactions, and not just simply where to publish their research.

Over time, new metrics have emerged quantifying article-level impact such as views, downloads and Altmetrics, and researcher-level impact, such as the H-index. With so many ways to measure research output, some stakeholders (institutions, funders) are using metrics as measures of merit for research and researchers – although few will admit to it.

Not only do we have to contend with problems in the way that metrics are calculated and manipulated but, critically, their misapplication as a measure of quality is troubling. The implication is that researchers or research articles with more impressive metrics (and hence more popular) are perceived to be more impactful and tacitly misconstrued as better quality and worthy of reward. Having not done a systematic review on the topic, I cannot say for certain, but this is surely not a sound hypothesis on which research evaluation should be based.

Are metrics really leading us away from research transparency?

Despite the plethora of metrics, and certainly as a result of the misdirected importance placed on them as an assessment of quality, transparency – probably the most important of all measurements of quality – is actually not covered by a metric.

In my view, this is highly problematic. I have spent the last several years whetting my scientific chops by applying my epidemiological skills to evaluating the completeness, bias, and transparency of reporting of biomedical research, a primary focus of the EQUATOR Network with whom I work.

For the uninitiated, transparency in reporting is all about understandable, complete, available/accessible, reproducible, and usable biomedical research. Indeed, transparency may just be the X factor of evidence-based medicine – the single largest variable on which the reliability and trustworthiness of the entire evidence base is hinged.

Like metrics, research transparency is very much in vogue; but the impetus is different. As a whole, biomedicine is failing on all transparency accounts. A fair proportion of research is either discontinued or never made public, particularly because it is littered with reporting biases or not reported in a usable/useful manner. Essential components (such as individual patient results) are inaccessible.

Achieving transparency in research should be straightforward, but it is not.

Achieving transparency in research should be straightforward, but it is not. Many have speculated who is at fault and have set priorities for what needs to be done to improve this situation. Essentially a systemic culture change is needed within the scientific enterprise and this seems to have (finally) caught the attention of funders, regulators, publishers, and lay media, all at the same moment. Improvements – or promises to that effect – seem to be looming.

As a young researcher, I am troubled that the system in which we are trained, and later expected to work in, is weighted so heavily towards bean counting and away from transparency.

Two years ago I chose a small piece of the transparency pie to explore further in a PhD in epidemiology. Beyond my good fortune to have mentors that share my research interests on the topic, more broadly speaking there seems to be little focus on educating young scientists on best practices in research transparency, ensuring that at a minimum, the research they, or we, produce will be a usable and reproducible contribution.

There is, however, omnipresent encouragement to publish at an early stage as training for our academic careers ahead. This imbalance between encouraging, and even rewarding, numerous publications in high impact journals over transparent research practices in academia is an unintentional brainwashing of epic proportions and with disastrous consequences (i.e. impairing truly evidence-based medicine).

What can be done?

Transparent and ‘high impact’ research can, and does, co-exist, but there does seem to be a trade off between the two. The incentives and motivations towards ensuring transparency in academia are far fewer than those aimed at many publications in high impact journals.

The current range of impact–popularity metrics reveal nothing about the inner workings of a research article and its potential reporting biases. What if the two came together, such that articles could be measured in terms of their transparency in a way that is truly meaningful and could not be readily manipulated or misapplied?

A new transparency metric?

Scientists like measuring things. Wouldn’t it be neat if we could motivate them to produce transparent research by measuring their publications on the basis of transparency, completeness and reproducibility, rather than (or in addition to) the number and type of ‘beans’ they accrue?

This is not a second call for the previously suggested transparency index, aimed at grading the transparency of journal policies. Instead, what if there was an article-level metric that would provide a snapshot of the transparency of the research contained within?

Consider a metric derived from the following features in a publication:

  • Inclusion of a signed declaration of transparency by authors
  • Compliance with an appropriate reporting guideline (checked and signed off by the publishing journal)
  • Link to a pre-study registration record
  • Link to a date-stamped public protocol
  • Link to fully accessible individual patient results

It may seem far off, but an emerging tool in the publishing industry may be starting up the transparency engines. PRE-val is a tool aimed at bringing transparency to the journal peer review process by making it apparent to those reading an article that it has been peer reviewed. It also provides a snapshot of whether the journal endorses various publication ethics standards or emerging practices.

PRE-val
Peer Review Evaluation
PRE-val.org

Perhaps the PRE-val model can be replicated, or even extended, to include the above items, and a badge bearing the article’s transparency metric could accompany each publication. Ultimately, such a badge may enable readers to see: what was done, whether it deviated from intended plan (and formulate a judgment about the impact of any changes), what was found, and whether ethical reporting practices were in place.

These are just the ramblings of a young researcher. Perhaps one day we will get there, and the beans we are counting will be so transparent that there’ll be no need to count them anymore.

View the latest posts on the Research in progress blog homepage

6 Comments

Adam Etkin

Larissa,

Thank you for mentioning our work at PRE. Your idea is quite in line with what we’re doing with PRE-val as well as other initiatives we have planned. Technically, implementing your idea would (I think) actually be quite easy. As you say, there would need to be a culture change among academe for a transparency index to be widely accepted. I do think there is movement in that direction.

Coincidentally, I just spoke at the CSE meeting on the topic of how to best evaluate impact using both metrics and other tools. I wish you were there as your ideas here are outstanding!

Larissa Shamseer

Hi Gustav,
The Open Science Foundation badges are fascinating and indeed go into far more detail than I have proposed in my metric at this time. The elements of adequate/complete reporting of articles is also a must! Thanks for sharing!

Karen Goodman, Professor of Epidemiology, University of Alberta

Larissa,

I thoroughly enjoyed reading this beautifully written piece about your current work – and I’m so impressed that you have taken up this crucial topic as your scholarly focus.

I can see the appeal of introducing a transparency metric, though collapsing a set of criteria into one score always has the disadvantage of leaving unclear which criteria are met. Whether or not a single transparency metric is advocated, I think it would be of potentially greater value to promote adherence to the list of transparency components as a transparency standard. A minor suggestion pertaining to your list of transparency criteria would be to broaden “fully accessible individual patient results” to research that is not conducted in a clinical setting (that is, where subjects are not patients), or even to studies not based on persons as the unit of observation (for example, where the unit of observation is communities), and make clear that this pertains to anonymized information – for example, “fully accessible anonymized data pertaining to individual units of observation”, assuming I understand this criterion correctly.

Please keep us posted as your work progresses.

Larissa Shamseer

Thanks Karen,
How wonderful to hear from you here! Indeed the proposal and components for such a metric requires further scrutiny. Your suggestion is a sensible one. Having reflected further, i would think that something of an altmetric-type score (i.e. different weighting for the different elements of transparency) might also be a good consideration.
Thanks for taking the time!

Comments are closed.