Survey response rates: all about the money?

Surveys and questionnaires sent to patients, clinicians and the general public provide valuable data to researchers, but they often suffer from poor response rates which reduces their usefulness. BMC Medical Research Methodology has published a selection of articles over the past year, investigating how factors such as financial incentives can be used to improve response rates for both the general public and specifically for healthcare professionals. The research has also looked at how this can be balanced with the cost of the study to provide cost-effective solutions.

The importance of response rates

Whether it’s a large-scale cross-sectional study or a follow up to a clinical trial, it is important to maximize the number of people that respond. Lower than anticipated response rates will lead to a reduced sample size, and this can in turn affect the quality of the statistical analyses that can be performed. In the worst case, this could result in an inability to answer the proposed research question with the data obtained. Poor response rates may also lead to potential biases in the data – especially in follow-up studies for example, and this can potentially invalidate the results.

Financial incentives: a powerful motivator for everyone?

It is widely accepted and well-tested that including a financial incentive is an effective way to increase response rates in the general population,  but few studies had confirmed whether this was also the case for General Practitioners (GP’s).

A study published in BMC Medical Research Methodology was able to show that, surprisingly, the inclusion of a prize draw incentive did not significantly improve response rates. The authors argue that this is because the motivators for the specific population of GP’s are different and complex, and it is thought that this may be due to the large number of invitations to complete surveys combined with a lack of time due to a high workload. Clinicians have a wealth of experience and are an invaluable source of information, providing a unique perspective on the healthcare system. It is therefore important to investigate novel methods to ensure that they are motivated to respond to surveys and pass their experiences on to researchers.

https://pixabay.comCost-effective solutions – 3 different approaches

Method 1 – money up front

Whilst it is true that the use of a prize draw incentive did not significantly increase the response rate in GPs, other research has shown that larger incentives can lead to increased responses. The main issue with this strategy is that many researchers are unable to fund these larger incentives.

Another study in BMC Medical Research Methodology tried to address this issue by altering the way in which the reward is administered.

Traditionally the incentive will be paid on completion of the survey, but in this study the researchers gave an upfront incentive of $50, with no requirement to complete the questionnaire. The key difference to this method is that the $50 was provided in the form of a redeemable gift card that would only cost the researchers if it was actually used.

The authors observed impressive results, showing that only 1 in 100 participants used the gift card without responding, while 36% of those who did respond chose not use the gift card. The results showed that compared to the conditional incentive, $2,150 was saved. This method therefore enabled the researchers to recruit those who needed an incentive whilst capitalizing on those who participate for more altruistic reasons.

Method 2 – incentive in reminders

Research recently published in BMC Medical Research Methodology has looked at how the timing of a financial incentive affects the response rate to a follow-up questionnaire in a clinical trial. They found that by only offering the incentive in a reminder, they were able to reduce the average cost of vouchers per participant from $10.04 to $3.96, a mean difference of $6.08. Further to this, the authors showed that there was no significant difference to the overall return rate.

Method 3 – delayed payments

Another article has looked at the effect of the timing of incentive payments in a cohort study. In cohort studies, the incentive payment can be made either up-front or on completion of the follow-up. Similar to method 2, the authors were able to show that there was no significant difference in the response rates between the methods, but money was saved by delaying payments. The cost per successful case was reduced to $13.85 from $16.64 – a saving of $2.79 per participant. When applied to large cohorts, this method would result in a significant reduction in cost, making the survey significantly more cost-effective.

Is this fair to the participants?

While it is clear that impressive results have been achieved by varying the way in which the incentive is presented, strategies like this undoubtedly come with some ethical issues.

Consideration has to be given to the question of whether it is fair to reward only some of the participants for performing the same task.Perhaps, of the methods discussed here, the use of reclaimable gift cards is the more ethical solution because all participants are presented with the same opportunity.

Although more discussion is needed regarding the ethical issues of some of these methods, it is clear that progress is being made in the optimization of response rates whilst balancing cost, and BMC Medical Research Methodology will continue to publish high quality research into this area.

View the latest posts on the BMC Series blog homepage