Randomized controlled trials are praised as one of the highest forms of evidence in healthcare. However, to be valuable, all research must have valid methods, and be reproducible and useable.
The usability of research is affected by a multitude of factors, stemming from the research not being reported in sufficient detail. Clinical trials particularly have come under heavy fire recently, with a few very high-profiles cases highlighting some of these issues.
The Tamiflu case, where the full evidence suggested no additional benefit over paracetamol, highlighted major impacts from publication bias and selective outcome and analyses reporting, whereas the case of selective serotonin reuptake inhibitors (SSRIs) increasing the risk of suicide emphasized the disastrous effect inadequate reporting of harms can have on patients.
Media hype stemming from such stories has created the caricature of a malevolent trialist, gleefully suppressing data for their own publication record.
But, in a world dominated by regulation, funders and grant applications, has anyone actually asked the trialists?
Excuse me Trialist, what are your thoughts?
Rebecca Smyth and colleagues set out to do just that. They identified 286 trialists, who had either published a trial covered in Cochrane systematic reviews used in the ORBIT project or were randomly sampled from PubMed between 2007 and 2008, interviewing the 59 that replied.
These interviews highlighted issues for trialists relating to five major stages of the clinical trial process:
- Framing the research question,
- Defining key outcomes,
- The role of the study protocol,
- Conducting the research,
- Getting the research published.
Many of the reported challenges have been highlighted before, but the importance of qualitative studies to gain insight from those on the front-line of research, as well as provide depth and detail to the evidence-base, cannot be underestimated.
There are some findings that will always be disheartening. Thirty-three trialists commented that, independent of outcome, they had never chosen to withhold from publishing their results; however, sixteen of these (48%) said they had been involved in a trial that was never published, 3 of which were industry-funded:
“… that was an industry sponsored trial… Not only was it a negative result but the study was stopped early because of safety. And that has not been published.”
…Not only was it a negative result but the study was stopped early because of safety. And that has not been published…
Smyth and colleagues, Trials 2015
So, what can we do?
Existing initiatives concentrate on combating issues at the final stage of a clinical trial, i.e. publishing the results. But Smyth and colleagues identified issues for trialists that occur in many of the stages leading up to, and ultimately contributing to, this final stage.
Reporting guidelines, such as those from the EQUATOR Network, including CONSORT, provide guidance for the minimum set of reporting items to ensure that research is transparent and reproducible. However, if a trialist does not see these or isn’t referred to them until their article is rejected, it’s too late. Trialists need to be aware of these guidelines through the entire trial process for them to have their intended impact.
Therefore, maybe we need to focus our efforts, not just on the final stage of a clinical trial, but equally on all the other stages that will impact this final stage.
Some projects are already underway with the goal of supporting better trial design and reporting practices, including the National Institute of Health Research Clinical Research Network and Trial Forge, but do they go far enough?
In particular, difficulty in defining key outcomes was highlighted by those interviewed. When it comes to defining key outcomes, the COMET Initiative aims to develop standardized sets of outcomes for specific conditions, an agreed minimum of what should be measured during clinical trials.
These should be incorporated into the study protocols, which should be a trial’s ‘how-to guide’ and form the core of their eventual report paper. The deliberation and detail that goes into creating a protocol means that adherence should assist all stages of the clinical trial process. However, some trialists mentioned that after completion, the protocol was shelved and forgotten:
“…You write the protocol you spend months doing it, and you have got a really good grasp of it and then as the trial progresses the protocol fades…”
Another recurring theme was that of recruitment and retention, something that will feel all-to-familiar to many trialists out there. Many of those interviewed commented on difficulties achieving their target sample sizes:
“… it was supposed to be a huge trial, looking to recruit 1800 men, and we closed after two years with 35 randomized, so clearly you are not going to get anything useful out of that…”
While this is a well-known bugbear, perhaps it is time to acknowledge that some of these trials should never have gone ahead. Does this show we are doing too many trials? Instead, maybe we should be conducting more pilot studies with predefined feasibility outcomes for factors such as recruitment, to give a clear go/no-go for definitive trials.
Whatever the future may hold for the clinical trial process, we need to ensure that the voice of all parties involved are heard.
30/01/15: *Update to text*: paragraph 9 has been updated. The original paragraph read, “There are some findings that will always be disheartening. For example, 48% of trialists interviewed stated they had been involved in a trial that was never published:”