Effect of Small Monetary Incentive and Demographic Characteristics on Response Rate of Self-Administered Questionnaire Mailed to Rural Women

Objectives: The National Children’s Study (NCS) is a longitudinal observational study that will examine the effects of genetics and environment on the health and development of children in the United States. The NCS is in a Vanguard, or pilot phase, so it is important to determine the feasibility, acceptability and cost of different data collection methods. The purpose of this study was to determine whether demographic characteristics differed in self-administered questionnaire (SAQ) completion rates, and to examine response rates when a $2 incentive was included with the mailed questionnaire. Methods: Subjects for the study (~500) were being followed by a local call center and were mailed SAQs. Women who did not complete their last telephone event were eligible. Women were randomized to receive a $2 bill incentive with their SAQ (approximately 250 women in each group). Results: Approximately 450 women comprised the final sample. In the incentive group, 35% of SAQs were returned, which was significantly higher than the 27% returned in the non-incentive group (p<0.001). No significant differences in completion rates based on demographic characteristics were seen, except in divorced women. The final cost per completed SAQ was $12.51 in the incentive group compared to $7.57 for the non-incentive group. Conclusions: Adding a $2 bill as an incentive significantly increased response rates of hard-to-reach rural women completing mailed questionnaires about pregnancy information. Future research should focus on different incentive amounts and types and how they influence response rates. Introduction The National Children’s Study (NCS) is a longitudinal observational study that will examine the effects of genetics and environment on the health and development of children in the United States. Participants are followed from before birth through 21 years of age. During the initial vanguard phase of the study, eligible households were enumerated to determine if women Effect of Small Monetary Incentive and Demographic Characteristics on Response Rate Vol. 8, Issue 1 (2013) 2 between the ages of 18 and 49 years resided in them. These women then completed a pregnancy screener to determine pregnancy status and probability of becoming pregnant. Those women who were not pregnant at the time of screening, but were age-eligible, were followed-up periodically through a local Call Center to collect information on pregnancy status and update contact information. Approximately 4,000 rural women were followed at the South Dakota State University Study Center (SDSU SC) Call Center. A paper-based version of the Call Center questionnaire was developed for women who were difficult to contact by phone. This one-page self-administered questionnaire (SAQ) collected the same information as the telephone interview, but could be completed at the participant’s leisure. A business-reply paid envelope was included for returning the SAQ. Anecdotal evidence from other Study Centers suggested mailing $5 with the SAQ increased response rates. Published research also has shown that including an incentive with a questionnaire increases response rates. The NCS is in a Vanguard phase, so it is important to determine the feasibility, acceptability and cost of different methods of data collection. Specifically, it is important to examine whether women who are hard to reach over the phone are easier to reach via postal mail. Poor response rates for a certain mode of data collection may lead to a biased sample and ultimately limit generalizability of study findings. Achieving high response rates can be time-consuming and expensive; therefore it is valuable to know how influential a small monetary incentive can be. The purpose of this study was to determine whether there were differences in SAQ completion rates by demographic characteristics (age, race/ethnicity, marital status), and to compare response rates when a $2 incentive was included with the mailed questionnaire. We hypothesized that those receiving the incentive would have significantly higher response rates. Methods Subjects Subjects for the incentive study (520) were a subset of nearly 2,300 rural women being followed by the SDSU SC Call Center. This includes women from Brookings County, South Dakota and Pipestone, Lincoln and Yellow Medicine Counties in Minnesota. Women who did not complete their last telephone event were eligible to receive the mailed SAQ. Approval for the study was obtained from the South Dakota State University Institutional Review Board and from the NCS Program Office. Procedures Mailed SAQs were prepared according to protocol. This involved printing a questionnaire with the local SDSU SC logo and contact information and attaching a label with the woman’s ID. A cover letter addressed to the woman was also generated that contained information about the questionnaire as well as thank you message. Additionally, a brochure about the NCS and a magnet indicating household eligibility with the SDSU SC toll-free number also was included. Women were randomized to either receive a $2 bill incentive with their SAQ (approximately 250 women in each group) or no incentive at all. The Online Journal of Rural Research and Policy Vol. 8, Issue 1 (2013) 3 Data Analysis Data were analyzed using SAS Version 9.2 software (SAS Institute, Inc., Cary, North Carolina, USA). A chi-square test was calculated to determine if including an incentive significantly increased the completion rate of the SAQ. Differences in completion rates were assessed among different age groups, marital statuses, ethnicities and races. PROC FREQ procedures were used to determine returned percentages of the different categories that were analyzed. Results Table 1 gives an overview of the status of all SAQs. Table 1. Final status and description of SAQs, by Incentive or No Incentive Final Status Description Incentive (n=269) No Incentive (n=251) Complete* The SAQ was completed and returned within 75 days. 30% 23% Moved, Unknown Address** Return information indicated a move, but no forwarding address. 1% 1% Moved, Out of Study** Return information indicated a move to an ineligible address, so the woman was removed from the study. 13% 12% Telephone Research Center Event** Call Center information obtained before SAQ was returned. <1% <1% Other Did not return SAQ. 55% 64% *Includes a small number that were initially returned by the Postal Service and then re-mailed ** Not included in final analysis, ineligible Overall, 30% of completed SAQs were received from the incentive group and 23% were received from the non-incentive group while 14% from the incentive group and 13% from the non-incentive group were returned by the postal service. Completed SAQs included those that, based on return information received from the postal service, were re-mailed (11% in the incentive group and 6% in the non-incentive group) and subsequently returned completed (3% and 1%, respectively). In some cases, the Call Center was able to contact the woman and complete the questionnaire over the phone (<1% in both groups). In other cases, a participant had moved, making her no longer eligible for the study (13% in the incentive group and 12% in the non-incentive group). Only 1% of the women moved and did not provide a forwarding address. More than half (55%) of the women in the incentive group and 64% in the non-incentive group did not return the SAQ and did not fall into any of the previous categories. Only those in the ‘Complete’ and ‘Other’ groups were considered for final analysis, which resulted in 229 in the incentive group and 218 in the non-incentive group (Table 2). Effect of Small Monetary Incentive and Demographic Characteristics on Response Rate Vol. 8, Issue 1 (2013) 4 Table 2. Description of SAQs included in final analysis, by Incentive or No Incentive Final Status Description Incentive (n=229) No Incentive (n=218) Complete* The SAQ was completed and returned within 75 days. 35% 27% Other Did not return SAQ. 65% 73% *Includes a small number that were initially returned by the Postal Service and then re-mailed In the incentive group, 35% of SAQs were returned which was significantly higher than the 27% returned in the non-incentive group (p<0.001). The cost to initially mail the SAQ was $1.59 plus the incentive, if applicable (including printing, labeling, postage and envelopes). Thus it cost approximately $966 to mail the SAQ's with an incentive and $399 to mail the SAQs without an incentive (Table 3). Table 3. Summary of costs for mailed SAQs and returned envelopes Original Cost Returned Incentives Returned Envelopes Re-mailed SAQs Completed & Returned Final Costs Cost $1.59 + incentive $2 each $0.44 each $1.59 + incentive $0.44 Incentive $965.71 (269) -$78.00 (39) $17.16 (39) $104.11 (29) $3.96 (9) $1,013 No Incentive $399.09 (251) $0.00 (32) $14.08 (32) $23.85 (15) $1.32 (3) $439 If an SAQ status wasn’t ‘Complete’ or ‘Other’, the envelope had been returned, at a charge of $0.44 for return service. Additionally, with returned mail, incentives were also returned. Those in the ‘Re-mailed’ group incurred charges both for returning of the first envelope and mailing of the second. The final cost per completed SAQ was $12.51 in the incentive group compared to $7.57 for the non-incentive group. Based on these figures and including only those women ultimately found to be eligible for the study, it cost 2.3 times more to include the $2 incentive and the response rate increased by 1.3 times if the incentive was included. Demographic information was collected on all women who were sent SAQs. No significant differences in completion rates based on demographic characteristics were seen, except that divorced women in the incentive group returned significantly more than divorced women in the non-incentive group (p=0.001). Discussion The purpose of this study was to examine response rates and associated costs when a $2 incentive was included with a mailed questionnaire and to determine whether differences in The Online Journal of Rural Research and Policy Vol. 8, Issue 1 (2013) 5 demographic characteristics between responders and non-responders were present. In general, the mailed SAQ is used when telephone contacts have been unsuccessful. It was hypothesized that although these women are hard to reach, adding a monetary incentive might increase response rates. Previous research has suggested ways to increase response rates of mailed questionnaires such as following up with a postcard or changing questionnaire color or hand-addressing envelopes and sending them via certified mail. Completion rates for mailed surveys have been shown to be increased when telephone contact was made prior to the mailing and when stamped return envelopes were used instead of business reply. In addition to cosmetic and logistical alterations, research has shown that providing study participants with incentives generally increases response rates, especially when the incentive is non-contingent or is some form of direct payment. It is not clear what magnitude of monetary incentive is most effective at eliciting a response, and if that response rate differs by demographic characteristics of the study participants. Various studies have examined how incentives impact clinicians responding to surveys, including lotteries versus fixed incentives and contingent versus non-contingent cash and checks, with immediate cash payments being most successful. Hawley and colleagues examined how differing values of non-contingent incentives affected response rates among mental health clinicians. The clinicians received a magnet, $1, $2 or $5 bill or no incentive. The highest response rate was from those who received the $5 bill. Response rates increased between each category: no incentive (39.8%), magnet (41.8%), $1 bill (51.0%), $2 bill (60.6%), and $5 bill (64.6%). Analyses determined that type of clinician did not affect response rates. These results are similar to the current study, which indicated that including a monetary incentive significantly increased response rates. The response rates in this study are lower, however, which could be due to the hard-to-reach nature of these participants. Additional research could assess whether a great monetary incentive elicits a significantly greater response rate in women between the ages of 18 and 49 years. We chose to use a $2 bill because of its uniqueness and to see if a relatively small dollar amount could elicit significant differences in response rates. Similarly, Kanaan and colleagues explored survey response rates when including an immediate incentive, a promised incentive or no incentive. Those who received the incentive immediately responded at a significantly higher rate (61%) than those who were promised the incentive upon completion (48%). Response rates were not different between the no incentive group (43%) and the promised incentive group (48%). The authors cited guilt as a possible reason for the effectiveness of the immediate incentive; however it may not be applicable to all groups of people. Future research could explore the effectiveness of immediate versus promised incentives in the NCS study population. Although previous research has shown that college students who received a $5 or $2 incentive with a questionnaire were more likely to respond (41% and 36%, respectively) than those who receive no incentive (23%), we did not find any significant differences in response rates among age groups or within the college-aged individuals. Effect of Small Monetary Incentive and Demographic Characteristics on Response Rate Vol. 8, Issue 1 (2013) 6 Additional forms of incentives could be explored as well. Liu and Geidenberger found African American women of childbearing age had increased response rates when given a gift card as opposed to a calling card. Whiteman and colleagues found that middle-aged women who received either a scratch-off lottery ticket and a postcard, or a $1 bill had higher response rates for a mailed questionnaire than those who received no incentive. In this case it seems that a potential monetary incentive acts just as well as a monetary incentive in eliciting higher response rates. Conclusions Adding a $2 bill as an incentive significantly increased response rates of hard-to-reach rural women completing mailed questionnaires about pregnancy information. Completion rates did not vary by demographic factors, however divorced women in the incentive group returned significantly more than divorced women in the non-incentive group. Although response rates were increased overall, cost must be taken into consideration and the benefits of mailing a monetary incentive must outweigh the costs. Future research should focus on different incentive amounts and types and how they influence response rates in young to middle-aged women, as well as the optimal method for data collection in this population. The Online Journal of Rural Research and Policy Vol. 8, Issue 1 (2013) 7 Acknowledgements This analysis was conducted as part of the National Children’s Study, supported by the Eunice Kennedy Shriver National Institute of Child Health and Human Development, and funded, through its appropriation, by the Office of the Director of the National Institutes of Health. Supported in part by NICHD Contract Numbers HHSN275200603416C and HHSN275201100004C. The views expressed in this article are those of the authors and do not necessarily represent the position of the National Children’s Study, the National Institutes of Health, or the U.S. Department of Health and Human Services. Effect of Small Monetary Incentive and Demographic Characteristics on Response Rate Vol. 8, Issue 1 (2013) 8 End Notes: McCormack, Lacey A. “Effect of Small Monetary Incentive and Demographic Characteristics on Response Rate of Self-Administered Questionnaire Mailed to Rural Women.” Online Journal of Rural Research & Policy (8.1, 2013). 1. Hirschfeld, S., et al., National Children's Study: update in 2010. Mt Sinai J Med, 2011. 78(1): p. 119-25. [back] 2. Hawley, K.M., J.R. Cook, and A. Jensen-Doss, Do noncontingent incentives increase survey response rates among mental health providers? A randomized trial comparison. Adm Policy Ment Health, 2009. 36(5): p. 343-8. [back] 3. Fox, R.J., M.R. Crask, and J. Kim, Mail survey response rate. Public Opinion Quarterly, 1988. 52(4): p. 467-491. [back] 4. Rimm, E.B., et al., Effectiveness of various mailing strategies among nonrespondents in a prospective cohort study. Am J Epidemiol, 1990. 131(6): p. 1068-71. [back] 5. Linsky, A.S., Stimulating responses to mailed questionnaires: a review. Public Opinion Quarterly, 1975. 39(1): p. 82-101. [back] 6. Urban, N., G.L. Anderson, and A. Tseng, Effects on response rates and costs of stamps vs business reply in a mail survey of physicians. J Clin Epidemiol, 1993. 46(5): p. 455-9. [back] 7. Brealey, S.D., et al., Improving response rates using a monetary incentive for patient completion of questionnaires: an observational study. BMC Med Res Methodol, 2007. 7: p. 12. [back] 8. Halpern, S.D., et al., Lottery-based versus fixed incentives to increase clinicians' response to surveys. Health Serv Res, 2011. 46(5): p. 1663-74. [back] 9. James, K.M., et al., Getting physicians to respond: the impact of incentive type and timing on physician survey response rates. Health Serv Res, 2011. 46(1 Pt 1): p. 232-42. [back] 10. Hawley, K.M., J.R. Cook, and A. Jensen-Doss, Do noncontingent incentives increase survey response rates among mental health providers? A randomized trial comparison. Adm Policy Ment Health, 2009. 36(5): p. 343-8. [back] 11. Kanaan, R.A., S.C. Wessely, and D. Armstrong, Differential effects of pre and post-payment on neurologists' response rates to a postal survey. BMC Neurol, 2010. 10: p. 100. [back] 12. Szelenyi, K., A.N. Bryant, and J.A. Lindholm, What Money Can Buy: Examining the Effects of Prepaid Monetary Incentives on Survey Response Rates among College Students. Educational Research and Evaluation, 2005. 11(4): p. 385-404. [back] 13. Liu, S.T. and C. Geidenberger, Comparing incentives to increase response rates among African Americans in the Ohio pregnancy risk assessment monitoring system. Matern Child Health J, 2011. 15(4): p. 527-33. [back] 14. Whiteman, M.K., et al., A randomized trial of incentives to improve response rates to a mailed women's health questionnaire. J Womens Health (Larchmt), 2003. 12(8): p. 821-8. [back] The Online Journal of Rural Research and Policy Vol. 8, Issue 1 (2013) 9 Author Information Lacey A. McCormack (back to top) Lacey A. McCormack, MPH, RD, LN is Program Director with the EA Martin Program at South Dakota State University and a Registered Dietitian with a Master of Public Health in Public Health Nutrition from the University of Minnesota (2009). She is in the final year of her doctoral program. During graduate school she worked on projects and published articles in the area of food and beverage advertising and youth and weight-related teasing in youth. Her professional experience includes the management and execution of the National Children’s Study at the SDSU Vanguard Center, which involved management of electronic-, telephoneand paper-based data collection. Her current responsibilities center on developing project and proposal ideas and applying for funding along with analyzing and interpreting data for publication. Her research interests include: determinants of body weight and composition in rural children, adolescents and young adults; eating behaviors and food environments; and program evaluation. The Online Journal of Rural Research and Policy © 2013 New Prairie Press ISSN 1936-0487


Introduction
The National Children's Study (NCS) is a longitudinal observational study that will examine the effects of genetics and environment on the health and development of children in the United States. Participants are followed from before birth through 21 years of age. 1 During the initial vanguard phase of the study, eligible households were enumerated to determine if women between the ages of 18 and 49 years resided in them. These women then completed a pregnancy screener to determine pregnancy status and probability of becoming pregnant. Those women who were not pregnant at the time of screening, but were age-eligible, were followed-up periodically through a local Call Center to collect information on pregnancy status and update contact information. Approximately 4,000 rural women were followed at the South Dakota State University Study Center (SDSU SC) Call Center.
A paper-based version of the Call Center questionnaire was developed for women who were difficult to contact by phone. This one-page self-administered questionnaire (SAQ) collected the same information as the telephone interview, but could be completed at the participant's leisure. A business-reply paid envelope was included for returning the SAQ. Anecdotal evidence from other Study Centers suggested mailing $5 with the SAQ increased response rates. Published research also has shown that including an incentive with a questionnaire increases response rates. 2 The NCS is in a Vanguard phase, so it is important to determine the feasibility, acceptability and cost of different methods of data collection. Specifically, it is important to examine whether women who are hard to reach over the phone are easier to reach via postal mail. Poor response rates for a certain mode of data collection may lead to a biased sample and ultimately limit generalizability of study findings. Achieving high response rates can be time-consuming and expensive; therefore it is valuable to know how influential a small monetary incentive can be.
The purpose of this study was to determine whether there were differences in SAQ completion rates by demographic characteristics (age, race/ethnicity, marital status), and to compare response rates when a $2 incentive was included with the mailed questionnaire. We hypothesized that those receiving the incentive would have significantly higher response rates.

Subjects
Subjects for the incentive study (520) were a subset of nearly 2,300 rural women being followed by the SDSU SC Call Center. This includes women from Brookings County, South Dakota and Pipestone, Lincoln and Yellow Medicine Counties in Minnesota. Women who did not complete their last telephone event were eligible to receive the mailed SAQ. Approval for the study was obtained from the South Dakota State University Institutional Review Board and from the NCS Program Office.

Procedures
Mailed SAQs were prepared according to protocol. This involved printing a questionnaire with the local SDSU SC logo and contact information and attaching a label with the woman's ID. A cover letter addressed to the woman was also generated that contained information about the questionnaire as well as thank you message. Additionally, a brochure about the NCS and a magnet indicating household eligibility with the SDSU SC toll-free number also was included. Women were randomized to either receive a $2 bill incentive with their SAQ (approximately 250 women in each group) or no incentive at all.

Data Analysis
Data were analyzed using SAS Version 9.2 software (SAS Institute, Inc., Cary, North Carolina, USA). A chi-square test was calculated to determine if including an incentive significantly increased the completion rate of the SAQ. Differences in completion rates were assessed among different age groups, marital statuses, ethnicities and races. PROC FREQ procedures were used to determine returned percentages of the different categories that were analyzed. Table 1 gives an overview of the status of all SAQs. Overall, 30% of completed SAQs were received from the incentive group and 23% were received from the non-incentive group while 14% from the incentive group and 13% from the non-incentive group were returned by the postal service. Completed SAQs included those that, based on return information received from the postal service, were re-mailed (11% in the incentive group and 6% in the non-incentive group) and subsequently returned completed (3% and 1%, respectively). In some cases, the Call Center was able to contact the woman and complete the questionnaire over the phone (<1% in both groups). In other cases, a participant had moved, making her no longer eligible for the study (13% in the incentive group and 12% in the non-incentive group). Only 1% of the women moved and did not provide a forwarding address. More than half (55%) of the women in the incentive group and 64% in the non-incentive group did not return the SAQ and did not fall into any of the previous categories.

Results
Only those in the 'Complete' and 'Other' groups were considered for final analysis, which resulted in 229 in the incentive group and 218 in the non-incentive group (Table 2). In the incentive group, 35% of SAQs were returned which was significantly higher than the 27% returned in the non-incentive group (p<0.001).
The cost to initially mail the SAQ was $1.59 plus the incentive, if applicable (including printing, labeling, postage and envelopes). Thus it cost approximately $966 to mail the SAQ's with an incentive and $399 to mail the SAQs without an incentive (Table 3). If an SAQ status wasn't 'Complete' or 'Other', the envelope had been returned, at a charge of $0.44 for return service. Additionally, with returned mail, incentives were also returned. Those in the 'Re-mailed' group incurred charges both for returning of the first envelope and mailing of the second. The final cost per completed SAQ was $12.51 in the incentive group compared to $7.57 for the non-incentive group. Based on these figures and including only those women ultimately found to be eligible for the study, it cost 2.3 times more to include the $2 incentive and the response rate increased by 1.3 times if the incentive was included.
Demographic information was collected on all women who were sent SAQs. No significant differences in completion rates based on demographic characteristics were seen, except that divorced women in the incentive group returned significantly more than divorced women in the non-incentive group (p=0.001).

Discussion
The purpose of this study was to examine response rates and associated costs when a $2 incentive was included with a mailed questionnaire and to determine whether differences in demographic characteristics between responders and non-responders were present. In general, the mailed SAQ is used when telephone contacts have been unsuccessful. It was hypothesized that although these women are hard to reach, adding a monetary incentive might increase response rates.
Previous research has suggested ways to increase response rates of mailed questionnaires such as following up with a postcard or changing questionnaire color 3 or hand-addressing envelopes and sending them via certified mail. 4 Completion rates for mailed surveys have been shown to be increased when telephone contact was made prior to the mailing 5 and when stamped return envelopes were used instead of business reply. 6 In addition to cosmetic and logistical alterations, research has shown that providing study participants with incentives generally increases response rates, especially when the incentive is non-contingent or is some form of direct payment. 7 It is not clear what magnitude of monetary incentive is most effective at eliciting a response, and if that response rate differs by demographic characteristics of the study participants.
Various studies have examined how incentives impact clinicians responding to surveys, including lotteries versus fixed incentives 8 and contingent versus non-contingent cash and checks 9 , with immediate cash payments being most successful. Hawley and colleagues examined how differing values of non-contingent incentives affected response rates among mental health clinicians. 10 The clinicians received a magnet, $1, $2 or $5 bill or no incentive. The highest response rate was from those who received the $5 bill. Response rates increased between each category: no incentive (39.8%), magnet (41.8%), $1 bill (51.0%), $2 bill (60.6%), and $5 bill (64.6%). Analyses determined that type of clinician did not affect response rates. These results are similar to the current study, which indicated that including a monetary incentive significantly increased response rates. The response rates in this study are lower, however, which could be due to the hard-to-reach nature of these participants. Additional research could assess whether a great monetary incentive elicits a significantly greater response rate in women between the ages of 18 and 49 years. We chose to use a $2 bill because of its uniqueness and to see if a relatively small dollar amount could elicit significant differences in response rates.
Similarly, Kanaan and colleagues explored survey response rates when including an immediate incentive, a promised incentive or no incentive. 11 Those who received the incentive immediately responded at a significantly higher rate (61%) than those who were promised the incentive upon completion (48%). Response rates were not different between the no incentive group (43%) and the promised incentive group (48%). The authors cited guilt as a possible reason for the effectiveness of the immediate incentive; however it may not be applicable to all groups of people. Future research could explore the effectiveness of immediate versus promised incentives in the NCS study population.
Although previous research has shown that college students who received a $5 or $2 incentive with a questionnaire were more likely to respond (41% and 36%, respectively) than those who receive no incentive (23%) 12 , we did not find any significant differences in response rates among age groups or within the college-aged individuals.
Additional forms of incentives could be explored as well. Liu and Geidenberger found African American women of childbearing age had increased response rates when given a gift card as opposed to a calling card. 13 Whiteman and colleagues found that middle-aged women who received either a scratch-off lottery ticket and a postcard, or a $1 bill had higher response rates for a mailed questionnaire than those who received no incentive. 14 In this case it seems that a potential monetary incentive acts just as well as a monetary incentive in eliciting higher response rates.

Conclusions
Adding a $2 bill as an incentive significantly increased response rates of hard-to-reach rural women completing mailed questionnaires about pregnancy information. Completion rates did not vary by demographic factors, however divorced women in the incentive group returned significantly more than divorced women in the non-incentive group. Although response rates were increased overall, cost must be taken into consideration and the benefits of mailing a monetary incentive must outweigh the costs. Future research should focus on different incentive amounts and types and how they influence response rates in young to middle-aged women, as well as the optimal method for data collection in this population.