attach9

ATTACHMENT 9

NIAAA’S REPORT TO OMB ON THE OVERALL EFFECTIVENESS OF THE NESARC INCENTIVE SCHEME, RESULTS OF REFUSAL CONVERSION AND FIELD REPRESENTATIVE OBSERVATIONAL QUESTIONS

BACKGROUND

On February 24, 2004, the Office of Management and Budget (OMB) approved Data Collection Number 0925-0484, entitled “Wave 2 National Epidemiologic Survey on Alcohol and Related Conditions (NESARC).” The Wave 2 NESARC was the second wave of a longitudinal survey, conducted in 2004-2005, its first wave of data having been collected in 2001-2002. The NESARC is a representative survey of the noninstitutionalized civilian U.S. population designed by the National Institute on Alcohol Abuse and Alcoholism (NIAAA), National Institutes of Health (NIH), with fieldwork conducted by the Census Bureau.

Because noninterview rates have been on the rise since about 1990 for federally sponsored surveys, the Wave 2 NESARC OMB packet included a detailed and comprehensive justification for incentives. The OMB approved the Wave 2 NESARC data collection with the use of a $40/$40 pre- and post-incentive scheme for all respondents initially interviewed in the Wave 1 NESARC. The $40 prepayment was unconditional and the $40 postpayment was conditional, after completion of the interview. As a condition of that approval, OMB required NIAAA, NIH, to submit a report on:

1. The overall effectiveness of the incentive scheme;

2. Results of refusal conversion; and

3. Results of Field Representative observation questions on the incentive payments.

This report represents NIAAA’s response to the above condition of the OMB approval for the Wave 2 NESARC.

Overall Effectiveness of the Incentive Scheme

In the Wave 1 NESARC, 43,093 respondents were interviewed with a response rate of 81.0%. After respondents classified as out of universe (Census Bureau Nonresponse Type Bs and Cs—3,134 deceased, institutionalized, mentally and/or physically incapacitated, returned to the native country in which the respondent held citizenship) were eliminated, the response rate in Wave 2 was 86.7%, with refusals totaling 5,306 and completed interviews totaling 34,653 (34,653 / 34,653 + 5306 = 86.7). The cumulative response rate at Wave 2 was 70.2% (81.0% x 86.7%). In addition, there were no breakoffs in the Wave 2 NESARC, item nonresponse was under 4.0%, and data quality was excellent as determined by weekly monitoring of key data element prevalences throughout the fieldwork and preliminary comparisons of Wave 1 and Wave 2 data elements for consistency.

The overall effectiveness of the Wave 2 incentive was excellent for the following reasons:

1. The excellent Wave 2 response rate (86.7%), a rate that exceeded the Wave 1 rate of 81.0%, as well as excellent data quality; and

2. The cumulative response rate (70.2%), higher than that achieved in surveys similar to the NESARC in both content and length; in fact, those similar surveys (e.g., the 2001-2002 National Comorbidity Survey – Replication, sponsored by the National Institute of Mental Health, had a Wave 1 response rate of 72.3%).

As previously mentioned, the $40 prepayment was unconditional and did not require the respondent to participate in the interview, while the $40 postpayment was available to those who additionally completed the interview. The extant literature on the effects of prepayment and postpayment was split, as was whether each of these types of payments should be unconditional or conditional on completion of the interview. As the Wave 2 incentive procedure has shown, the combination of unconditional prepayment and conditional postpayment was extremely successful in encouraging respondents to participate in the Wave 2 NESARC. Our experiences with this incentive program, which is consistent with the extant literature, have convinced us that such an incentive design would have increased the NESARC response rate in Wave 1, when no incentive was paid.

The mechanisms involved in making the incentive scheme successful can never be completely known, but we will continue to analyze the Wave 2 data in order to understand them. Initially, however, we hypothesize that the success of the scheme may have been due, in part, to several of the following factors:

1. Psychologically, the majority of people do not like to accept something for nothing (i.e., the $40 prepayment); and

2. The total $80 incentive is significant to most people, apparently regardless of income; they view it as a vehicle to provide something special to a family member, loved one, or significant other (see analyses of incentive questions in the next section of this report).

Results of Refusal Conversion

Our review of the extant literature on incentives in survey research showed that differential incentives of any kind, especially those used to reward initial nonresponders, were felt to be unethical, with the majority of people believing that differential incentive payments for nonrespondents represents an unfair practice. As a consequence, we made the decision not to offer initial nonresponders any additional incentives.

Toward the end of the fieldwork, each Census Bureau Regional Office (RO) was allowed to send the $40 unconditional prepayment, that was only offered by the Field Representative making physical contact with the respondent, to 25 initial nonresponders in the mail (for a total of 300 initial nonresponders, 25 for each RO). As expected based on our review of the literature, only 18% of the 300 initial nonresponders were converted using this procedure.

Results of Field Representative Observational Questions

A. Percentage of respondents accepting incentive payments.

Nearly all (99.24%, n=34,392) Wave 2 NESARC respondents accepted both the $40 prepayment and the $40 postpayment. Another 0.34% (n=117) were classified as accepting only the $40 prepayment. In these cases, the Field Representative had problems verifying the debit card number for the $40 postpayment, and that payment had to be sent by the RO to the affected respondents by mail.

Of the 144 (0.42%) respondents who did not accept the $40 prepayment and $40 postpayment, 63.7% stated that they didn’t need the money, 18.6% stated that they couldn’t accept money from the government and/or from the Census Bureau, 10.2% stated that they didn’t understand how ATMs worked, and 3.8% asserted that it was their civic duty to participate. The percentages of the remaining respondents reporting that they felt the incentive payments were a scam (1.3%), could not accept the money for religious reasons (1.2%), or were prohibited to do so by virtue of their status as federal employees (1.2%) were quite low.

B. Did the Field Representatives think the incentive payment influenced the respondents’ decision to participate in the interview?

A lot: 47.43%

A little: 36.31%

Not at all: 15.5%

Don’t know: 0.76%

C. Did the Field Representatives think that they would have been successful in convincing the respondents to participate if they had not been able to offer the incentive payments?

Definitely yes: 9.90%

Probably yes: 52.20%

Probably not: 27.40%

Definitely not: 9.21%

Don’t know: 1.29%

D. Did the Field Representatives think that the incentive payments allowed them to work their cases more efficiently, that is, make fewer visits to the households or spend less time gaining cooperation than they would have?

Yes: 80.61%

No: 18.82%

Don’t know: 0.56%

E. Did the respondents make any comments that suggested they would have participated in the survey without the incentive?

Yes: 15.56%

No: 84.29%

Don’t know: 0.15%

F. Did the respondents make any comments that suggested they felt it was inappropriate to offer money in exchange for a person’s participation in the survey (i.e., referring to the $40 postpayment)?

Yes: 1.47%

No: 98.50%

Don’t know: 0.04%

G. Did the respondents make any comments about what they planned to do with the incentive payment?

Yes: 21.35%

No: 78.62%

Don’t know: 0.04%

Of the 7,398 respondents who provided comments concerning what they would do with their incentive payments, there were 8,120 responses to this question (i.e., some respondents gave more than one response). In these cases, each response was counted as a separate answer.

The results of our classification of respondents’ answers to this question are presented in Table 1. About 28.4% of the respondents reported that they would use the incentive payments for shopping in general (15.4%) or specifically for Christmas shopping (13.0%). Responses accounting for between 7.9% and 10.8% of respondents’ answers were: (1) paying bills; (2) buying gifts (for kids/anniversaries/birthdays); (3) buying groceries/food; (4) taking someone out to dinner; and (5) going out to dinner. The remaining 26.5% of the responses were all of lower prevalence (0.6%-3.7%) and included paying for essential items (e.g., home/car repairs, medications, utility bills, school books/supplies, baby clothes/food, rent, taxes, pet food/toys) and vacations, charity, and savings. A very small percentage of respondents reported using the incentive payments to buy cigarettes and/or alcohol (1.5%) and in one-half of these cases, the interviewer added a note to indicate that the respondent was kidding or not really serious in that response.

H. Did the respondents make any comments that indicated how they felt about the amount of the incentive?

Yes: 14.01%

No: 85.93%

Don’t know: 0.06%

I. Did the respondents’ comments suggest that they thought the amount of the incentive was too high, about right, or too low?

Too high: 10.58%

About right: 79.21%

Too low: 8.06%

Don’t know: 2.15%

J. Did the respondents show any reluctance to participate in the interview before you mentioned the incentive payment?

Yes: 35.12%

No: 63.95%

Don’t know: 0.93%

Summary of Incentive Scheme

Based on the results of prior longitudinal surveys on alcohol, drugs, and mental health, we anticipated a response rate for the Wave 2 NESARC as low as 75.0%. We achieved a rate of 86.7%. According to the responses to the Field Representative questions, 99.24% of the respondents accepted the $80 incentive, and Field Representatives felt that: (1) 36.61% of the respondents would not have participated in the survey if the incentive had not been offered; (2) the incentive payment resulted in a gain in the efficiency with which they were able to work their caseloads (which, of course, reduced fieldwork costs) in 80.61% of the cases; (3) very few respondents indicated that they would have participated in the survey without the incentive (15.56%); and (4) 35.12% of the respondents indicated reluctance to participate in the survey without the incentive payment. Taken together, these Field Representative results strongly suggest that our anticipated response rate of 75.0% without an incentive payment may have been too high and underscore the success of the incentive payments in the Wave 2 NESARC. Further, a very, low percentage of respondents made comments suggesting that it was inappropriate to offer money in exchange for a person’s participation in the survey (1.47%), whereas 87.27% of the respondents indicated that the amount of the entire incentive payment was about right (79.21%) or too low (8.06%).

Most importantly, the overwhelming evidence from the extant scientific literature on incentives strongly suggests that incentives are effective among respondents to both initial and subsequent waves of surveys of all types, regardless of burden. The results of the Wave 2 NESARC incentive scheme bear out this finding, both in terms of increases in response rate and in terms of data quality. Further, taken together, the results of the NESARC incentive scheme support the use of an unconditional prepayment and a conditional postpayment, the prohibition against using differential incentives to reward nonresponders, and the amount of incentive ($80 in 2004-2005) that seems acceptable to respondents.

Table 1

Respondents’ Responses to “What They Planned to Do with the Incentive Payment”

Response	N	%
Shopping (in general)	1,250	15.4%
Christmas shopping	1,056	13.0%
Groceries/food	878	10.8%
Pay bills	763	9.4%
Take someone out to dinner	723	8.9%
Go out to dinner	658	8.1%
Buy gifts (for kids, anniversaries, birthdays)	641	7.9%
Put in savings account	300	3.7%
Put toward home repairs	300	3.7%
Give to charity	219	2.7%
Buy medications	171	2.1%
Car repairs	170	2.1%
Gas for car	170	2.1%
Put toward vacation	154	1.9%
Buy baby clothes/food	138	1.7%
Buy cigarettes/alcohol	121	1.5%
Pay utility bill(s)	106	1.3%
Buy school books/supplies	106	1.3%
Pay rent	82	1.0%
Pay taxes	65	0.8%
Buy pet food/toys	49	0.6%
TOTAL	8,120	100.0%

10