When I started my PhD project it just made perfect sense to me. I would implement a prediction model that could give physicians their patients’ predicted risks for nausea and vomiting after surgery. The physicians would then adjust their preventive strategies according to those risks, creating a personalized plan for each patient. That is exactly what the physicians did, but it did not help their patients.
I tried to analyze why this model had not been successful. I performed additional analyses of the data and interviewed the participating physicians, but in hindsight it comes down to this: I had underestimated the complexity of implementing the prediction model.
At the start of my implementation project I considered implementation to be easy, but I was wrong. Although implementation is often addressed as the ‘rest’, successful implementation is hard to accomplish. It often requires a back-and-forth approach, rather than being an isolated process.
Making a difference is hard
Although my focus is on clinical medicine, it will probably hold for other fields: making a difference is challenging. Of course, I could elaborate about all the problems I faced during my implementation efforts, however I believe that there is a much more convincing argument: there is hardly any evidence that prediction models make a difference in real life.
Within clinical medicine there is substantial proof that healthcare workers change their behavior when using prediction models – or other forms of decision support. Nonetheless, there is little evidence that such change in behavior leads to better results. Even in this age of big data and artificial intelligence, reports on prediction models having a true impact on patients’ lives are incidental.
All that remains is implementation
There is another reason why reports on successful implementation of prediction models are lacking – their implementation is rarely studied. Perhaps we simply assume that the models will work, I know I did.
Abiding by that assumption makes sense to me. Tinkering with algorithms is much more alluring than implementing them. You get to play around with data and produce results, but more importantly, tinkering with the algorithm keeps the dream alive that your model will one day change the world. Implementation often crushes such dreams, leaving not much of an incentive to study implementation.
A back and forth process
Funding agencies for scientific research try to motivate researchers to study implementation. Most agencies require a plan for knowledge utilization and implementation within the grant proposal. Although a noble objective, it sends the message that implementation is a separate phase, independent from the solution that has been developed.
A prediction model is a good example. A prediction model in itself is not ready for use; it needs a format in which the predictions are presented, including choices on when, to whom and how to present the predictions. Moreover, knowledge gained during the implementation of the prediction model may require changes in the model.
What to do – funding agencies
The implementation strategy and the prediction model are all part of the same solution to a problem. That is what funding agencies should focus on: implementation is part of the solution and not simply the ‘rest’ that follows the development of a model.
Rather than separating implementation as a separate phase, funding agencies should encourage scientists to use a more agile approach – such as a theory-driven adaptive intervention study. In such designs, implementation is part of the proposal rather than a separate chapter. Funding agencies would then truly stimulate scientists to focus on finding a solution, rather than dreaming of making a difference.
What to do – scientists
For my next implementation project, I would start by the assumption that I will fail. Failure until proven otherwise ensures that I will not blame the physicians for not using the risks or following the recommendations.
In addition, it opens my mind for ideas to arm myself against failure. Not only for ideas that will improve implementation, but also for ideas that – when implementation fails – will help me understand why.
Not making a difference is not in vain as long as you are able to learn from it. Implementation failure was the best thing that happened to me during my PhD project. I hope to have many more of such failures during the rest of my scientific career.
Comments