The Effect of an Incentive on Response Rates and Timing in an Economic Survey

The Effect of an Incentive on Response Rates and Timing in an Economic Survey Rob Burnside, Glenys Bishop and Tenniel Guiver Australian Bureau of Statistics Methodology Division Locked Bag 10 BELCONNEN ACT 2616 Canberra, Australia rob.burnside@abs.gov.au 1. Background Systematic methods for improving response rates in surveys has been an area of interest for researchers for some time following the articulation of the Total Design Method (Dillman 1978) which included applying aspects of social influence and social exchange theory to encourage responses. One part of the literature focused on the idea of offering a tangible but token incentive to respondents at the time of the request for data. Subsequent research and experience has shown that appropriate incentives do improve respondent cooperation in terms of obtaining better response rates (eg Singer and Wilmot 1997; Yammarino, Skinner and Childers 1991). The focus of most work has been on using monetary or token material incentives for surveys of individuals, so little is known about the problem of how to motivate businesses and the individuals in them to respond. The situation of National Statistical Organisations (NSOs) conducting surveys of businesses is also different to that of many other survey researchers. Differences can include larger sample sizes, greater frequency of repeating surveys, the mandatory nature of such surveys, legislative provisions that limit the types of incentives that can be offered (eg no paying for data), and community perceptions about appropriate of activity funded from taxation. Official surveys of businesses require very reliable data for major economic indicators and so expend considerable effort on getting high response rates to minimise non-response bias. Obtaining high response rates is a costly and labour-intensive process involving follow-up techniques that impose a further burden and irritation on respondents, so any methods to achieve even small increases in response rates offers significant cost, data quality and timeliness gains. One incentive that NSOs may be able to offer is access to their own published statistics. This conforms to principles that incentives should be token but also worthwhile, show regard for the respondent’s effort in proving data and illustrates how their data is used in major economic statistics, decision based on which may affect them. Our results show that the incentive had a positive effect on response by the due date. 2. Design An experiment that was designed that examined the effect on response rate of offering a URL and password for a secure download of the publication to which their data contributed. The aim was to make statistically valid judgements about the effect on timeliness of response by offering the incentive of free access to a publication. Phase I offered an Internet address link (URL) to half of the respondents for the previous quarterly publication along with the data request (form) for the current quarter collection. In the following quarter Phase II split the groups offering the most recent publication to half of those who had had the initial offer and, for the first time, to half those who did not receive the initial offer. Different form types were despatched to financial sector units (F90) and non-financial sector businesses (F80). These groups potentially had different response patterns so the design was blocked by form type (F90 or F80) and four treatments (1, 2, 3, 4) were randomly allocated inside each block. The initial allocation cell sizes (91 and 198) for treatments were eroded by a number of factors. The small number of new selections were excluded because it seemed possible that they may respond differently to the incentive. Losses during enumeration resulted from operational factors such as cancellations, changes to businesses and other contact issues, reducing final live block counts to around 70 and 130. During both phases hits on the password protected download pages were logged. The Web pages offering the incentive were accessed around 35 times in Phase I and around 65 times in Phase II. 3. Analysis Three operationally meaningful time points in each quarter were chosen for exploratory analysis - due date, middle of follow-up and close-off. A logistic regression model was fitted with a logit link function based on frequency counts of trials (total forms in any of the experimental cells) and events (form received / not received) at the analysis dates for each cell. For each analysis time point four models were fitted (Table 3). Table 1: Percentage of Treatment Group/block responding in Second Quarter at key analysis dates Group for Incentive Block % response: due % response: mid % response: Offer: date follow-up close-off 1 in both quarters F80 36.8 76.5 93.4 F90 52.1 85.5 97.1 2 in first quarter only F80 33.8 77.4 92.5 F90 32.9 71.2 89.0 3 in second quarter only F80 32.8 77.1 94.6 F90 36.0 80.0 92.0 4 in neither quarter F80 37.1 76.5 93.2 F90 34.8 81.2 98.5 4. Results The incentive had a positive effect on response rates by the due date in the second quarter. The odds of response in quarter 2 where the incentive was offered in quarter 1 were increased by a factor of 1.40 compared to when it was not offered (odds ratio 1.40 with Chi-sq= 2.76 and p-value = 0.0964). The odds of response in quarter 2 where the incentive was offered in quarter 2 were increased by a factor of 1.43 times compared to when it was not offered (odds ratio 1.43 with Chi-sq= 3.12 and p-value = 0.0774). Interaction between the four combinations of offer of incentive over the quarters also had an additional significant effect (Chi-sq= 2.53 and p-value = 0.1117). REFERENCES: Dillman, Don A (1978) Mail and Telephone surveys: The Total Design Method, Wiley (New York) Singer E. and Wilmot A,.(1997). Incentive payments on Social Surveys: a Literature Review. Proceedings of the section on survey research methods AAPOR Yammarino, F. J., Skinner, S.J., Childers, T.L. (1991). Understanding mail Survey Response Behaviour A Meta-analysis. Public Opinion Quarterly Vol 55 Résumé Les incitations d'exposition de théorie et d'expérience améliorent la coopération de répondant en termes d'obtenir de meilleurs taux de réponse dans les aperçus. Le but de cet essai était de faire statistiquement des jugements valides au sujet de l'effet sur l'opportunité de la réponse dans une enquête conjoncturelle en offrant l'incitation du libre accès à une publication.