The Effect of an Incentive on Response Rates and Timing in an
Economic Survey
Rob Burnside, Glenys Bishop and Tenniel Guiver
Australian Bureau of Statistics Methodology Division
Locked Bag 10
BELCONNEN ACT 2616
Canberra, Australia
rob.burnside@abs.gov.au
1. Background
Systematic methods for improving response rates in surveys has been an area of interest for
researchers for some time following the articulation of the Total Design Method (Dillman 1978)
which included applying aspects of social influence and social exchange theory to encourage
responses. One part of the literature focused on the idea of offering a tangible but token incentive
to respondents at the time of the request for data. Subsequent research and experience has shown
that appropriate incentives do improve respondent cooperation in terms of obtaining better response
rates (eg Singer and Wilmot 1997; Yammarino, Skinner and Childers 1991).
The focus of most work has been on using monetary or token material incentives for surveys
of individuals, so little is known about the problem of how to motivate businesses and the
individuals in them to respond. The situation of National Statistical Organisations (NSOs)
conducting surveys of businesses is also different to that of many other survey researchers.
Differences can include larger sample sizes, greater frequency of repeating surveys, the mandatory
nature of such surveys, legislative provisions that limit the types of incentives that can be offered
(eg no paying for data), and community perceptions about appropriate of activity funded from
taxation. Official surveys of businesses require very reliable data for major economic indicators
and so expend considerable effort on getting high response rates to minimise non-response bias.
Obtaining high response rates is a costly and labour-intensive process involving follow-up
techniques that impose a further burden and irritation on respondents, so any methods to achieve
even small increases in response rates offers significant cost, data quality and timeliness gains.
One incentive that NSOs may be able to offer is access to their own published statistics. This
conforms to principles that incentives should be token but also worthwhile, show regard for the
respondent’s effort in proving data and illustrates how their data is used in major economic
statistics, decision based on which may affect them. Our results show that the incentive had a
positive effect on response by the due date.
2. Design
An experiment that was designed that examined the effect on response rate of offering a URL
and password for a secure download of the publication to which their data contributed. The aim
was to make statistically valid judgements about the effect on timeliness of response by offering the
incentive of free access to a publication. Phase I offered an Internet address link (URL) to half of
the respondents for the previous quarterly publication along with the data request (form) for the
current quarter collection. In the following quarter Phase II split the groups offering the most
recent publication to half of those who had had the initial offer and, for the first time, to half those
who did not receive the initial offer. Different form types were despatched to financial sector units
(F90) and non-financial sector businesses (F80). These groups potentially had different response
patterns so the design was blocked by form type (F90 or F80) and four treatments (1, 2, 3, 4) were
randomly allocated inside each block. The initial allocation cell sizes (91 and 198) for treatments
were eroded by a number of factors. The small number of new selections were excluded because it
seemed possible that they may respond differently to the incentive. Losses during enumeration
resulted from operational factors such as cancellations, changes to businesses and other contact
issues, reducing final live block counts to around 70 and 130. During both phases hits on the
password protected download pages were logged. The Web pages offering the incentive were
accessed around 35 times in Phase I and around 65 times in Phase II.
3. Analysis
Three operationally meaningful time points in each quarter were chosen for exploratory
analysis - due date, middle of follow-up and close-off. A logistic regression model was fitted with
a logit link function based on frequency counts of trials (total forms in any of the experimental
cells) and events (form received / not received) at the analysis dates for each cell. For each analysis
time point four models were fitted (Table 3).
Table 1: Percentage of Treatment Group/block responding in Second Quarter at key analysis
dates
Group for Incentive
Block
% response: due % response: mid % response:
Offer:
date
follow-up
close-off
1 in both quarters
F80
36.8
76.5
93.4
F90
52.1
85.5
97.1
2 in first quarter only
F80
33.8
77.4
92.5
F90
32.9
71.2
89.0
3 in second quarter only
F80
32.8
77.1
94.6
F90
36.0
80.0
92.0
4 in neither quarter
F80
37.1
76.5
93.2
F90
34.8
81.2
98.5
4. Results
The incentive had a positive effect on response rates by the due date in the second quarter.
The odds of response in quarter 2 where the incentive was offered in quarter 1 were increased by a
factor of 1.40 compared to when it was not offered (odds ratio 1.40 with Chi-sq= 2.76 and p-value
= 0.0964). The odds of response in quarter 2 where the incentive was offered in quarter 2 were
increased by a factor of 1.43 times compared to when it was not offered (odds ratio 1.43 with
Chi-sq= 3.12 and p-value = 0.0774). Interaction between the four combinations of offer of
incentive over the quarters also had an additional significant effect (Chi-sq= 2.53 and p-value =
0.1117).
REFERENCES:
Dillman, Don A (1978) Mail and Telephone surveys: The Total Design Method, Wiley (New
York)
Singer E. and Wilmot A,.(1997). Incentive payments on Social Surveys: a Literature Review.
Proceedings of the section on survey research methods AAPOR
Yammarino, F. J., Skinner, S.J., Childers, T.L. (1991). Understanding mail Survey Response
Behaviour A Meta-analysis. Public Opinion Quarterly Vol 55
Résumé
Les incitations d'exposition de théorie et d'expérience améliorent la coopération de répondant en
termes d'obtenir de meilleurs taux de réponse dans les aperçus. Le but de cet essai était de faire
statistiquement des jugements valides au sujet de l'effet sur l'opportunité de la réponse dans une
enquête conjoncturelle en offrant l'incitation du libre accès à une publication.