Memo for Survey of SNAP and Work Generic ICR

MEMORANDUM

Date: February 28, 2019

To: Steph Tatham, OMB Desk Officer

Through: Rachelle Ragland-Greene Rg

Food and Nutrition Service, Information Collection Officer, Planning & Regulatory Affairs

Ruth Brown, Desk Officer, United States Department of Agriculture, Office of Chief Information Collection Officer

From: Michael Burke, PhD

Social Science Analyst, Food and Nutrition Service

Special Nutrition Evaluation Branch, Office of Policy Support

Re: Generic OMB Clearance No. 0584-0606 – Pretesting Instruments for the Survey of SNAP and Work

The U.S. Department of Agriculture’s (USDA) Food and Nutrition Service (FNS) is requesting approval for pretesting of data collection instruments for the referenced study under approved Generic OMB Clearance No. 0584-0606. The following information is provided for review:

1. Title of the Project: Pretesting Instruments for the Survey of SNAP and Work

2. Control Number: 0584-0606, Expires 03/31/2019

3. Public Affected by this Project: Individuals/households; 50 Supplemental Nutrition Assistance Program (SNAP) participants not currently receiving disability benefits and between the ages of 18 and 69

4. Number of Respondents and Research Activity:

Respondents	Research Activity	# of Respondents
SNAP participants ages 18- 69 who are not currently receiving disability benefits	Conduct cognitive interviewing to pretest data collection instruments	250 (sample size) 50 will go on to participate fully

5. Time Needed per Response:

Respondents	Research Activity	Time (hours)*
SNAP participants ages 18- 69 who are not currently receiving disability benefits	Craigslist recruitment announcement	0.0334
	Pretest recruitment flyer	0.0334
	Filtered databased recruitment email	0.0334
	Email to confirm pretest interview appointment	0.0334
	Pretest consent	0.0833
	CATI or web survey	0.4500
	Semi-structured guide	0.4333
	Pretest incentive receipt	0.0334

* Average time per response.

6. Total Burden Hours on Public: 48 burden hours and 650 responses

See the enclosed Excel burden table for the detailed burden estimate by affected public (Appendix A-1, Detailed burden table).

7. Project Purpose, Methodology, and Formative Research Design:

Background

SNAP is the largest of the nutrition assistance programs administered by FNS of USDA. It is considered the cornerstone of nutrition assistance in the US and the primary policy mechanism to provide a nutrition safety net and reduce food insecurity among low-income Americans by increasing access to healthy foods. Assistance is provided in the form of electronic benefit transfer (EBT) cards which can be used to purchase food in authorized food retailers. In Fiscal Year (FY) 2017, SNAP served 42.2 million people with an average monthly benefit of $125.80 per person.

About one-third of SNAP households have earnings from employment. Few details are known, however, about current job characteristics and work histories of adults participating in SNAP. In order to better understand current and past workforce participation characteristics among nondisabled adult SNAP participants, FNS contracted with Westat to conduct the Survey of SNAP and Work. Through a national survey of SNAP participants aged 18 to 69 who are not receiving disability benefits, information on employment status, length of workforce detachment, types of job held, education and training, and social, physical, and environmental barriers to work will be collected. Both national and state-level estimates will be calculated. Once we have concluded the pretesting data analysis, we plan to seek OMB clearance and launch a full data collection in 2020.

In order to assess construct validity of the survey questions and to improve data quality, the research team seeks to pretest the survey documents and applicable instruments:

1. CATI Survey of SNAP and Work—English

2. CATI Survey of SNAP and Work—Spanish

3. Web Survey of SNAP and Work—English

4. Web Survey of SNAP and Work—Spanish

Recruitment

The Survey will be pretested in two locations: the Washington, D.C. area and the Oakland, CA area (locations of the contractors on the research team). Two of the three recruitment activities described below (i.e., Craigslist and flyer) will be conducted in both locations.

SNAP recipients will be recruited simultaneously through three channels. First, an advertisement will be placed on the website Craigslist (for the Washington, D.C. area and for the Oakland, CA area)¹ (see Appendix A-2, Craigslist recruitment announcement). Second, a flyer advertising the study will be placed in the Manna Food Center (for Oakland), the Capital Area Food Bank in D.C., and at Montgomery County Health and Human Services in Maryland (for Washington, D.C.) (see Appendix A-3, Pretest recruitment flyer). Both the flyer and the Craigslist posting will direct interested people to call the research team. Third, an email to potential participants in the Washington, D.C. area will be sent using individuals contained in a database of potential participants supplied by a vendor to Westat that specializes in recruitment for research studies (see Appendix A-4, Filtered database email). The database includes contact information for volunteers who previously participated, or expressed interest, in participating in testing data collection instruments. Potential participants who call the research team will be screened for eligibility on the phone (see Appendix A-5, Pretest recruitment incoming call script). Eligible participants will be scheduled for in-person pretesting at Westat in Rockville, MD or at SPRA in Oakland, CA. The research team aims to conduct pretesting with 20 English-speaking respondents and 10 Spanish-speaking respondents.

Purpose and Procedure

The objectives of the data collection activity covered by this generic clearance are to (1) assess respondents’ understanding of the survey questions, (2) gauge feasibility of response (e.g., respondent ability to recall information and willingness to provide potentially sensitive information), (3) gauge the sequence of the survey topics and question order within topics, and (4) assess reporting accuracy (based on respondents’ descriptions of how they answered the questions). The survey includes questions on current employment status of the respondent and adult members of the respondent’s household; educational attainment; professional certification or licensure; jobs held within the last twelve months; job characteristics; barriers to employment; and household demographics (see Appendix A-6, Survey of SNAP and Work). Within one week of OMB approval, the research team will begin recruitment. As discussed above, interested respondents will be screened upon contacting the research team. If they are eligible, interviews will be scheduled. Once the interviews are scheduled, the research team will send an email to respondents confirming the details of the pretesting (including date, time, and location) (see Appendix A-7, Email to confirm pretest interview appointment).

Methodology/Research Design

Approach. In order to understand individuals’ thought processes, the research team will conduct qualitative cognitive interviews to pretest the English and Spanish versions of the CATI and web versions of the survey instruments. For all formats, the pretest is expected to take about 60 minutes, with about half of the time devoted to completing sections of the survey and the other half for interviewing. The Spanish pretest interviews will be conducted using bilingual interviewers who are experienced conducting cognitive interviews.

A semi-structured guide (Appendix A-8, Semi-structured guide) will be used to conduct cognitive interviewing of the survey. Interviews will be conducted by a team of trained and experienced cognitive interviewers. All interviews will be audio recorded and respondents will be notified before being recorded. After the respondent has completed a section of the survey or a series of questions interviewers will ask respondents a series of probes to examine issues related to comprehension and ability to recall of information. Respondents will be asked open ended probes (e.g., how did you come up with your answer; How did you decide how many hours you worked?) so we understand respondents in their own words.

Recruitment. Current SNAP recipients will be asked to participate in cognitive testing of the survey. Interested eligible participants will be invited to Westat in Rockville, MD or Social Policy Research Associates in Oakland, CA.

Informed Consent

Interviewees will be asked for their written consent before participating in any interview and for audio recording of the interview (see Appendix A-9, Pretesting in-person consent). Individuals refusing consent to both elements will be excused and thanked for their time.

The interviewers will also answer any questions and review consent information. Interviewers will read the public burden disclosure statement to telephone respondents before conducting the interview. All participants will be asked to confirm consent prior to starting the cognitive interview.

Incentives

Given the importance of recruiting a representative range of respondents and the difficulty typically experienced recruiting low-income respondents, especially SNAP able-bodied without dependents (ABAWDs), we will provide prospective respondents $40 for completing the 60-minute cognitive interview. The incentive will offset any participation cost respondents may incur such as child care as a result of participation in this study and can offset any travel cost incurred by traveling (e.g., parking, bus fare, gasoline) to the testing facility.

While the use of cash incentives is currently not the preferred method by OMB for increasing response rates, FNS believes that for this study population of low-income households, cash incentives are the most appropriate way to ensure high survey response rates and to provide reliable and generalizable results. Cash is better than checks as a medium for incentives for low-income populations (Groves, et al., 2009) because many low-income households are unbanked. For instance, over a quarter of households with incomes of $15,000 or less are unbanked (Federal Deposit Insurance Corporation, (2014). If these households are to monetize their checks, they often must use check-cashing services that often require customers to pay substantial service charges in order to cash a check (Swagler, Burton and Lewis, 1995; Fox and Woodall, 2006). Cash incentives are also preferable to pre-paid gift cards as cash has shown to provide a higher response rate than prepaid vouchers (Van Veen, Göritz, and Sattler, 2004) or gift cards (Birnholtz, Horn, and Finholt, 2006).

Data Analysis

The cognitive interviews will be audio-recorded to allow interviewers to conduct a more thorough analysis than could be achieved simply by listening and taking notes. Our general process for analyzing cognitive interviewing data involves the five steps of synthesis and reduction from interview to conclusions described by Miller, et al. (2014) and shown in
Figure 1. For each interview conducted, our cognitive interviewers will summarize their findings into an Excel reporting template which will allow for both case as well as themed-based analysis (Collins, 2015 See Figure 2 below). Interviewers will listen to the recordings in order to complete the templates. Respondents will be assigned unique codes that can be linked with the sampling criteria for a more informed interpretation of the findings. A senior cognitive interviewer/survey methodologist will summarize the results across all the interviews, including those conducted in Spanish. Results and recommendations will be presented following each round of interviews. Both English and Spanish versions of the survey will be revised based on the results of the pretesting.

Figure 1. Process for the analysis of cognitive interviews

Analytic step	Data reduction product
1. Conducting	Cognitive interview text
2. Summarizing	Detailed summaries
3. Comparing across respondents	Thematic schema
4. Comparing across groups	Advanced schema
5. Concluding	Conclusions
Source: Figure 4-1 (p. 37) Products of data reduction for analytic steps. Cognitive Interviewing Methodology, First Edition. Edited by Kristen Miller, Stephanie Willson, Valerie Chepp, and José-Luis Padilla. ©2014 John Wiley & Sons, Inc. Published 2014 by John Wiley & Sons, Inc.

Figure 2. Example of an Interviewer Summary Template.

Summary Template

Survey of SNAP and Work

Date:

1. Participant ID:

2. General comments on interview:

3. Cognitive testing, For each survey section item probed on:

3.1. Survey answer:

3.2. Observations/ think aloud findings:

3.3. Findings from general probes:

3.4. Findings from comprehension probes:

3.5. Findings from retrieval probes:

3.6. Findings on judgment probes:

4. Observed usability issues (if applicable)

5. Cross cultural differences (if applicable)

6. Other findings/ comments:

Adapted from: Collins, D (2015).

8. Confidentiality

Participation in the cognitive testing study is voluntary. FNS published a system of record notice (SORN) titled FNS-8 USDA/FNS Studies and Reports in the Federal Register on April 25, 1991, volume 56, pages 19078-19080, that discusses the terms of protections that will be provided to respondents. Personally identifiable information (PII), including names and contact information (telephone number and/or email address), will be collected for recruiting and scheduling proposes only. Participants will be informed that their PII will be kept private. These data will be securely stored in password-protected files to which only project staff will have access, and will be destroyed after the project is finished. Names provided by participants on consent and incentive receipt forms (Appendix A-11, Pretest incentive receipt) will be stored in locked cabinets, separate from the data. Participant PII will never be associated with the data collected during the interview. PII for individuals not selected for interview will be destroyed immediately.

9. Federal Costs

The total cost to the Federal government is $142,558.40, which includes the total cost for Contractor and Federal staff costs for the abbreviated supporting statement, developing the study instruments, and collection of the pretesting information. The Contractor cost is estimated at $140,638. This is based on an estimate of 790 hours, with a salary range of $87 – $356/hour, and includes overhead costs. This information collection also assumes a total of 40 hours of Federal employee time, for GS-13, Step 2 at $48.01 per hour, for a total of $1,920.40 on an annual basis. Federal employee pay rates are based on the General Schedule of Office of Personnel Management (OPM) for 2018.

10. Study Instruments

Appendix A-1. Detailed burden table

Appendix A-2a. Craigslist recruitment announcement, English, CA

Appendix A-2b. Craigslist recruitment announcement, Spanish, CA

Appendix A-2c. Craigslist recruitment announcement, English, DC area

Appendix A-3a. Pretest recruitment flyer, English, CA

Appendix A-3b. Pretest recruitment flyer, English, DC

Appendix A-3c. Pretest recruitment flyer, Spanish, CA

Appendix A-4a. Filtered database recruitment email, English

Appendix A-5a. Pretest incoming call script eligibility pre-screener guide, English

Appendix A-5b. Pretest incoming call script eligibility pre-screener guide, Spanish

Appendix A-6a. CATI Survey of SNAP and Work—English

Appendix A-6b. CATI Survey of SNAP and Work—Spanish

Appendix A-6c. Web Survey of SNAP and Work—English

Appendix A-6d. Web Survey of SNAP and Work—Spanish

Appendix A-6e. Screen Shots Web Survey of SNAP and Work—English

Appendix A-6f. Screen Shots Web Survey of SNAP and Work—Spanish

Appendix A-7a. Email to confirm pretest interview appointment, English, CA

Appendix A-7b. Email to confirm pretest interview appointment, Spanish, CA

Appendix A-7c. Email to confirm pretest interview appointment, English, DC area

Appendix A-8a. Semi-structured guide—CATI, English

Appendix A-8b. Semi-structured guide—CATI, Spanish

Appendix A-9a. Pretest consent form, English

Appendix A-9b. Pretest consent form, Spanish

Appendix A-10a: Pretest incentive receipt, English

Appendix A-10b: Pretest incentive receipt, Spanish

Appendix A-11: Statement of Confidentiality and Nondisclosure

Appendix A-12: Westat IRB Approval Letter

References

Birnholtz, J.,  Horn, D., Finholt T., and Bae, S. (2006). The effects of cash, electronic, and paper gift certificates as respondent incentives for a web-based survey of technologically sophisticated respondents. Social Science Computer Review.

Collins, D. (2015). Cognitive interviewing practice. London, UK: Sage.

Federal Deposit Insurance Corporation (2014). 2013 FDIC National Survey of Unbanked and Underbanked Households. Available at: https://www.economicinclusion.gov/surveys/2013household/documents/2013_FDIC_Unbanked_HH_Survey_Report.pdf.

Fox J A, Woodall P (2006). Cashed Out: Consumers Pay Steep Premium to “Bank” at Check Cashing Outlets. Available at: http://consumerfed.org/pdfs/CFA_2006_Check_Cashing_Study111506.pdf.

Groves, R., Fowler, F., Couper, M., Lepkowski, J., Singer, E., and Tourangeau, R. 2009. Survey Methodology, Second Edition. Hoboken, NJ: John Wiley & Sons, Inc.

Harkness, J. A., Van de Vijver, F. J., & Mohler, P. P. (2003). Cross-cultural survey methods (Vol. 325). Hoboken, NJ: Wiley-Interscience.

Miller, K., S. Willson, V. Chepp, and J.M. Ryan, (2014). Chapter 4: Analysis. In Miller, K. S. Willson, V. Chepp, and J.-L. Padilla (Eds.), Cognitive Interviewing Methodology, First Edition. Hoboken, NJ: John Wiley & Sons, Inc.

Swagler, R., Burton, J and Koonce Lewis, J, (1995). The Alternative Financial Sector: An Overview.  Advancing the Consumer Interest. Vol. 7, No. 2 (Fall 1995), pp. 7-12.

Van Veen, F., Göritz, A., Sattler, S., (2004). Response effects of prenotification, prepaid cash, prepaid vouchers, and postpaid vouchers: an experimental comparison. Social Science Computer Review. Available at: http://ssc.sagepub.com/content/early/2015/05/05/0894439315585074.abstract.

1 Websites may include: https://washingtondc.craigslist.org/ and https://sfbay.craigslist.org/eby/.