Sometimes the important concepts underlying established research methods, even those we take for granted, can turn out to be unclear on closer inspection.
Take for example the concept of intention-to-treat (ITT), the widely promoted approach to analyzing randomized controlled trial (RCT) data. Alshurafa and colleagues identified that there are actually multiple definitions for ITT, something that has interfered with appraising the risk of bias of published RCTs. For this reason they set out to resolve the methodological confusion by conducting a methods overview on the topic.
In addition to explicitly highlighting important ambiguities in common conceptualizations of ITT and inconsistencies in how methods authors have defined it, they were able to recommend resolutions to these problems, paving the way for a clearer shared understanding.
Methods overviews—also called reviews of methods—differ from traditional systematic literature syntheses since, instead of synthesizing empirical findings from published study reports, they address methods-relevant content in the literature. Such reviews can address either the methodological or empirical literature, depending on the purpose.
While the ITT review above addressed only the methodological literature (including methods manuals and methods-focused journal articles), overviews addressing the methods-relevant sections of published empirical study reports are also common. This latter category is often used, for example, to establish the needs to be addressed in methodological reporting guidelines.
Regardless which category of literature is reviewed, methods overviews commonly serve to identify key areas where methodological clarity is lacking and needs improvement.
Up to now, this distinct form of literature synthesis has been under-addressed in the literature despite clear implications for unique review procedures. Specifically, there has been a lack of guidance on how to adapt commonly accepted systematic review methods to the specific realities of methods overviews.
With the publication of our recent methodology article in Systematic Reviews, however, we hope that methods overviews may now start receiving more attention. Using sampling in qualitative research as a worked example of a methods overview, my colleagues and I provide tentative initial guidance for conducting methods overviews in the form of seven principles and strategies.
It is our hope that this article will not only help anyone undertaking a methods overview, but that it will help catalyze further discussion about what a rigorous systematic approach to this type of review should look like.
More definitive guidance for this review type will depend on further work to integrate a diversity of reviewers’ perspectives and experiences from methods overviews covering a broad range of qualitative and quantitative methods topics.
To that end, we hope others see the value of undertaking methods overviews. For one, reviews of this type are excellent opportunity to develop methodological expertise and capacity, particularly when it is possible to engage methodologically oriented students in the review process.
More importantly, high quality critical evaluations of the literature regarding problematic methods topics generally promote clearer, shared understandings. And that can accelerate advances in rigorous research methods.
In addition to more methods overviews, we also need to start pooling our wisdom from conducting this type of review to establish the standards for doing them rigorously. If you have participated in a methods overview yourself, it would be great to have you connect with us with your experience and perspective. This important review type can only be developed as more reviewers contribute to the discussion.
Comments