How do we know if surgery works?
At first sight, this seems a bit of a daft question. If you’re bleeding to death, or your leg has taken on a right angle bend where there shouldn’t be one, it stands to reason that stopping the bleeding or straightening out the bones is bound to be helpful.
No-one seriously suggests you need a scientific trial to prove this. But there are many more conditions where surgery is nowadays one of a number of possible treatment options (some cancers, for instance, or appendicitis).
Some conditions may get better by themselves or with physiotherapy for example just as well as with an operation.
And which operation is better?
When we start asking questions about whether surgery is better than another treatment, or which operation is better, things get complicated.
When we start asking questions about whether surgery is better than another treatment, or which operation is better, things get complicated. Reliable scientific methods for treatment comparisons have only been available for the last 70 years or so.
When it’s not a no-brainer like the examples we started with, human variation often makes comparisons unreliable. If we were all identical widgets, treatments would have identical effects, but we aren’t, and if the patient groups we compare are different, bias can creep in, ending up with a wrong answer.
The conventional (and excellent) solution to this is the randomized controlled trial (RCT), where chance decides which treatment patients get. But experience shows that it is much more difficult to conduct an RCT for a surgical treatment than for a drug treatment.
Historically RCTs in surgery have been lacking. Surgeons got a lot of stick for this initially – some of it deserved. We are a self-selected bunch of – well let’s just say very decisive people: it’s a job requirement. We are also not good at admitting uncertainty – and that’s essential if you are going to put your patient into an RCT.
However it’s not really all about us. When a group called the Balliol Collaboration started to look seriously at why RCTs were difficult in surgery, they soon realized that it was because surgical innovations develop in quite a different way from new medicines, and there are several stages an innovation needs to go through before it is ready for an RCT.
The ‘tinkering stage’ of surgery
The first few times a surgeon does a new surgical technique there is almost always a bit of refinement and modification of the technique in response to its results. Until this tinkering stage has finished an RCT is impractical, because the new operation to be evaluated is not yet stable and defined.
Once other surgeons get in on the act there is always a good deal of controversy about which patients to operate on: there is also an inevitable operator learning curve for new surgeons taking up the procedure.
Getting agreement from the surgeons is very difficult, and even if you did, the learning curve would bias things against the new treatment.
Again, an RCT is not sensible at this stage. Getting agreement from the surgeons is very difficult, and even if you did, the learning curve would bias things against the new treatment.
In 2009 the Balliol group published the IDEAL Framework and Recommendations, which described surgical innovation in 5 stages (Idea, Development, Exploration, Assessment and Long-term study, hence IDEAL) and suggested appropriate study methods and reporting formats for research at each of the stages. The proposals included two types of prospective uncontrolled cohort studies which could be used to prepare the ground for an RCT.
Happy birthday to The IDEAL Collaboration
The IDEAL Collaboration is now five years old, and we are having a celebration – the IDEAL International Conference at St Catherine’s College, Oxford on 7 April 2016. Evaluating innovation in surgery and therapeutic technology: the IDEAL approach. @IDEALCollab #IDEAL2016
The Conference will look at progress in using IDEAL in surgical research, and update the Framework to keep up with the latest ideas on improving trials. It is becoming clear that IDEAL works well for other complex treatments as well as surgery, and this will be discussed too.
One of the most topical is its possible use to help regulators decide whether new invasive medical devices like stents and heart valves are safe and effective and ready for market. At present the rules on what evidence is needed to declare a device safe are very lax.
An IDEAL-based system could cut costs and time in the evaluation process, whilst providing better evidence than is generally available at present. Another use for IDEAL may be in helping those who need to make difficult decisions about buying complex expensive treatments for the NHS, such as hyperbaric oxygen or proton beam therapy.
Again, using the stages of IDEAL could provide an integrated evaluation pathway which allowed the NHS to select the use of treatments as the evidence strengthened. The Conference will be a great opportunity to discuss and share new ideas on the science of evaluating surgery and other complex treatments, and we hope we’ll see you there. Please go to our website to register for the event, or submit an abstract.
For further details of the conference please contact Allison Hirst, IDEAL Project Manager: allison.hirst@nds.ox.ac.uk
Comments