O. Use of Tokens of Appreciation PRAO

Attachment O. Use of Tokens of Appreciation

OMB No. 0584-[NEW]

Job Search as a Supplemental Nutrition Assistance Program (SNAP) Employment and Training (E&T) Component

January 14, 2021

Project Officer: Danielle Deemer

Office of Policy Support

Food and Nutrition Service

U.S. Department of Agriculture

1320 Braddock Place

Alexandria, VA 22314

703.305.2952

danielle.deemer@usda.gov

Overview of Use of Tokens of Appreciation for the Study

Former and current SNAP E&T participants who complete an interview for this study will receive a token of appreciation in the form of a generic $30 gift card. The tokens of appreciation for interview respondents are an essential component of the multiple methods that will be used to ensure the necessary sample sizes are met, improve data quality, and reduce nonresponse bias. A primary objective of this study (as noted in appendix B: Research Objectives and Questions by Data Source) is to conduct in-depth participant interviews to understand client perceptions of job search outcomes. Because individuals may come to job search with different backgrounds, skills, and expectations, and as a result may have a range of experiences with and outcomes from job search, a sufficiently large sample of completed interviews is required to collect information on the range of participant experiences with job search in a given State.

1. Factors Supporting the Use of Tokens of Appreciation

In this section, we provide more detail on the decision to provide a token of appreciation and the amount of the token of appreciation.

1. Improve data quality

A token of appreciation is essential to obtain the sample sizes needed to understand the full range of various participants’ experiences with job search. Response rates for research have declined in recent years (Brick & Williams, 2013; Curtin, Presser, & Singer, 2005). Offering tokens of appreciation is an effective method for improving response rates without compromising the quality of the data (Singer & Ye, 2013). The knowledge that respondents will receive a monetary incentive to complete a survey is expected to increase their likelihood of spending the time to respond. In a recent study, Mercer and colleagues (2015) conducted a meta-analysis of the dose-response association between tokens of appreciation and response and found a positive relationship between tokens of appreciation with a greater monetary value and response rates for household telephone surveys offering tokens of appreciation. Respondent incentives are particularly effective at increasing response rates for populations similar to SNAP participants, which include people with low income and lower educational levels.

2. Improve coverage of hard-to-reach or specialized respondents

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-xxxx. This is a reference document and no burden is associated with this document; it will not be sent to respondents. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to the following address: U.S. Department of Agriculture, Food and Nutrition Services, Office of Policy Support, 1320 Braddock Place, Alexandria, VA 22314, ATTN: PRA (0584-xxxx*). Do not return the completed form to this address.

The target populations for the interviews are socially disadvantaged groups, namely low-income SNAP participants and former SNAP participants, who are considered hard to reach (Bonevski et al., 2014). Moreover, households in the study are specialized respondents because they are limited in number and difficult to recruit, and their lack of participation jeopardizes the outcomes evaluation. These groups may also have work and childcare schedules that make it difficult to reach SNAP participants. Tokens of appreciation may encourage greater participation among these groups.

3. Reduce respondent burden

The tokens of appreciation will reduce respondent burden because they can help offset the costs associated with participation, including childcare that may be needed while respondents complete the interview, cell phone and data usage costs associated with calls and texts needed to set up appointments as well as the telephone interviews themselves, and potential lost wages (assuming a minimum wage of $7.25 per hour). SNAP agencies have indicated that their low-income populations often use “pay-as-you-go” cell phones, which incur costs at a standard rate of $0.10 per minute. Others face limits on data usage and may incur fees for exceeding those limits. Finally, some low-income SNAP participants without a telephone may have to travel to a location with telephone service. Tokens of appreciation will help cover these costs of participating.

4. Reduce nonresponse bias and improve sample representativeness

Low response rates are problematic because they can lead to increased nonresponse bias (Groves et al., 2006). Tokens of appreciation reduce efforts to recruit low-income study participants and lower overall survey costs and time to achieve completion rates without affecting data quality (Dillman, 2000; Singer, 2006). In a meta-analysis, Mercer and colleagues (2015) estimated that surveys that promised $10 generated response rates 5 percentage points greater than surveys that offered no token of appreciation. Frederickson and colleagues (2005) found a $10 contingent token of appreciation increased responses by 20 percentage points among Medicaid recipients, a similar population to those being recruited for this study. (Children’s income eligibility for Medicaid is at least 133 percent of the Federal Poverty Level (FPL) but is higher in many States, and SNAP income eligibility is set at 130 percent of the FPL for the household size.)

Providing interview participants with a monetary token of appreciation reduces nonresponse bias and ensures a full range of job search experiences are reflected in the data, especially in populations defined as being in poverty; monetary tokens of appreciation are particularly effective relative to other types (Singer, 2002; James, 1997; Groves et al., 2009; Singer & Ye, 2013). Tokens of appreciation can improve sample representativeness and reduce nonresponse bias (Groves et al., 2000; Messer & Dillman, 2011) by encouraging those less interested in research to participate, including low-income respondents (Groves et al., 2006; Singer & Kulka, 2002). Several studies have shown offering tokens of appreciation may improve representation for low-education, low-income, and ethnic minority subgroups. Response rates among minorities and those with low education are generally lower for all types of surveys, especially those conducted by mail. For example, in a survey for the National Cancer Institute, the sample was divided into two strata: (1) one with a greater percentage of minority individuals, and (2) one with a smaller percentage. The response rate to the strata with a greater proportion of minority individuals was 12 percentage points lower than the rate for the strata with a smaller proportion (23 percent versus 35 percent; Westat, 2017). Similarly, when testing within household selection procedures, Olson and colleagues (2014) found all procedures led to underrepresenting non-Whites, Hispanics, individuals with less education, and individuals in the lowest income groups. The population of interest in the current study is comparable to the populations discussed earlier in this section. The current study will be recruiting participants who currently participate or previously participated in SNAP E&T.

2. Past Experience Using Tokens of Appreciation in Telephone Interviews With Similar Low-Income Populations

The study team is proposing that respondents who complete the 30-minute telephone interview (see appendix M: Participant Protocol) receive a $30 token of appreciation, an amount based on tokens of appreciation that were approved for similar data collections in other recent studies with similar burden among low-income populations. For example, the WIC Cost Containment Study (Office of Management and Budget [OMB] Control Number 0584-0627; expiration date September 30, 2020) provided a $30 token of appreciation to Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) participants to complete a 30-minute telephone survey and a $30 token of appreciation to former WIC participants to complete a 20-minute telephone survey. For the Evaluation of the Pilot Project for Canned, Frozen, or Dried Fruits and Vegetables in the Fresh Fruit and Vegetable Program for FNS (OMB Control Number 0584-0598; expiration date September 30, 2017), $50 tokens of appreciation were offered to community members, including low-income parents, who completed 1-hour telephone interviews. The Evaluation of the Summer Food Service Program Participant Characteristics used $25 prepaid gift cards to increase the participation of parents/caregivers of participants and eligible nonparticipants in 30-minute telephone interviews (OMB Control Number 0584-0595; expiration date April 30, 2016).

The National Survey of WIC Participants II (NSWP-II) obtained complete data for approximately 81 percent of participants for the in-person interviews; respondents who completed the interview received a $20 token of appreciation. However, such tokens of appreciation were not provided for NSWP-II interviews conducted over the telephone; those surveys generated an overall response rate of only 51.3 percent (Huang et al., 2012a). A nonresponse bias analysis conducted for NSWP-II showed nonresponse did lead to some potential bias in the telephone survey results (Huang et al., 2012b). Therefore, the study’s research team advised caution when interpreting the results from the telephone survey. Respondents for the USDA Healthy Incentive Pilot (HIP), which surveyed SNAP participants in Massachusetts, were provided monetary tokens of appreciation (Bartlett, Klerman, Olsho, et al., 2014). Respondents were provided a $20 token of appreciation for the baseline telephone interview, which achieved a 63-percent response rate; $30 for the round 2 interview (3–6 months after baseline), which achieved an 83-percent response rate; and $40 for the round 3 interview (11–13 months after baseline), which achieved an 81-percent response rate (OMB Control Number 0584-0584; expiration date August 31, 2014).

The tokens of appreciation for the current study will be offered equally to all potential interview participants. The tokens of appreciation will not be targeted to specific subgroups, nor will they be used to convert refusals. If tokens of appreciation were to be offered only to the most disadvantaged households or those incurring the highest participation costs, the differing motivations to participate used would limit and bias the responses. Based on the evidence discussed earlier in this section, the contractor plans to offer all respondents to the interview a $30 token of appreciation, which will be mailed to the respondents after they complete the interview. As discussed in the pretest memorandum (see appendix P), pretest respondents were sent a $30 gift card for participating in the pretest. Furthermore, pretest respondents are not part of the study and cannot receive the token of appreciation again.

In summary, the planned tokens of appreciation for the current and former SNAP E&T participant interviews are designed to promote cooperation and improve data quality by minimizing nonresponse bias and improving sample representativeness. If all the other strategies to achieve high response rates are used without also offering the planned tokens of appreciation, the nonresponse bias will be higher, resulting in poor data quality. The likelihood of detecting the full range of job search experiences and client perceptions of job search outcomes will be significantly compromised.

References

Bartlett, S., Klerman, J., Olsho, L., et al. (2014). Evaluation of the Healthy Incentives Pilot (HIP): Final report. Retrieved from https://fns-prod.azureedge.net/sites/default/files/ops/HIP-Final.pdf

Bonevski, B., Randell, M., Paul, C., Chapman, K., Twyman, L., Bryant, J., Brozek, I., & C. Hughes. (2014). Reaching the hard-to-reach: A systematic review of strategies for improving health and medical research with socially disadvantaged groups. BMC Medical Research Methodology, (14)42.

Brick, J. M., & Williams, D. (2013). Explaining rising nonresponse rates in cross-sectional surveys. The ANNALS of the American Academy of Political and Social Science, 645(1), 36–59.

Curtin, R., Presser, S., & Singer, E. (2005). Changes in telephone survey nonresponse over the past quarter century. Public Opinion Quarterly, 69(1), 87–98.

Dillman, D. (2000). Mail and internet surveys: The tailored design method (2nd ed.). New York, NY: John Wiley & Sons.

Fredrickson, D. D., Jones, T. I, Molgaard, C. A., Carman, C. G., Schukman, J., Dismuke, S. E., & Ablah, E. (2005). Optimal design features for surveying low-income populations. Journal of Health Care for the Poor and Underserved, 16, 677–690.

Groves, R., Fowler, F., Couper, M., Lepkowski, J., Singer, E., & Tourangeau, R. (2009). Survey methodology (2nd ed., pp 205–206). New York: Wiley.

Groves, R., Singer, E., & Corning, A. (2000). Leverage-saliency theory of survey participation: description and an illustration. Public Opinion Quarterly. 64(3), 299–308.

Groves, R. M., Couper, M. P., Presser, S., Singer, E., Tourangeau, R., Acosta, G., & Nelson, L. (2006) Experiments in producing nonresponse bias. Public Opinion Quarterly, 70(5), 720–736.

Huang, G., et al. (2012a). National Survey of WIC Participants II: Improper payments report. Retrieved from http://www.fns.usda.gov/sites/default/files/NSWP-II_Vol3.pdf

Huang, G., et al. (2012b). National Survey of WIC Participants II: Technical report. Retrieved from https://fns-prod.azureedge.net/sites/default/files/NSWP-II_Vol4.pdf

James, T. L. (1997). Results of wave 1 incentive experiment in the 1996 survey of income and program participation. Proceedings of the Survey Research Methods Section (pp 834–839). Washington, DC: American Statistical Association.

Mercer, A., Caporaso, A., Cantor, D., & Townsend, R. (2015). How much gets you how much? Monetary incentives and response rates in household surveys. Public Opinion Quarterly, 79, 105–129.

Messer, B., & Dillman, D. (2011). Surveying the general public over the internet using address-based sampling and mail contact procedures. Public Opinion Quarterly, 75, 429–457.

Olson, K., Stange, M., and J. Smyth. (2014). Assessing within-household selection methods in household mail surveys. Public Opinion Quarterly, 78, 656–678.

Singer, E. (2002). The use of incentives to reduce non response in households surveys. In R. Groves, D., Dillman, J. Eltinge, & R. Little (Eds.), Survey non response (pp. 163–177). New York: Wiley.

Singer, E. (2006). Introduction: Nonresponse bias in household surveys. Public Opinion Quarterly, 70(5), 637–645.

Singer, E., & Ye, C. (2013). The use and effectives of incentives in surveys. Annals of the American Academy of Political and Social Science, 645(1), 112–141.

Singer, E., & Kulka, R. A. (2002). Paying respondents for survey participation. In M. Ver Ploeg, R. A. Moffitt, & C. F. Citro, Committee on National Statistics, Division of Behavioral and Social Sciences and Education (Eds.), Studies of welfare populations: Data collection and research issues. Panel on data and methods for measuring the effects of changes in social welfare programs (pp. 105–128). Washington, DC: National Academy Press.

Westat. (2017). Health Information National Trends Survey 5 (HINTS 5), Cycle 1 methodology report. Retrieved from https://hints.cancer.gov/docs/methodologyreports/HINTS5_Cycle_1_Methodology_Rpt.pdf