Part B_08_15_2014_OMB

U.S. Department of Agriculture

Food and Nutrition Service

Examination of Cash Nutrition Assistance

Program Benefits in Puerto Rico

AG-3198-C-14-0007

Request for Clearance

Supporting Statement and

Data Collection Instruments

Part B: Collection of Information Employing Statistical Methods

August 14, 2014

Project Officer: Robert Dalrymple

Office of Policy Support

3101 Park Center Drive, 10^th Floor

Alexandria, VA 22302

(T) 703.305.2125 / (F) 703.305.2576

Bob.Dalrymple@fns.usda.gov

TABLE OF CONTENTS

part B. Collection of Information Employing Statistical Methods 2

B.1. Respondent Universe and Sampling Methods 2

B.2. Procedures for the Collection of Information 2

B.3. Methods to Maximize Response Rates and the Issue of Nonresponse 2

B.4. Tests of Procedures 2

B.5. Consultants 2

part B. Collection of Information Employing Statistical Methods

B.1. Respondent Universe and Sampling Methods

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The purpose of this section is to document any statistical sampling procedures used for the “Examination of Cash Nutrition Assistance Program Benefits in Puerto Rico.” The only sampling component of the study is a telephone survey of participants in Puerto Rico’s Nutrition Assistance Program (NAP). The selection method for the other study components are described below.

Focus group participants will be identified and recruited by community-based organizations (CBOs) using a convenience-based selection process. These partner organizations will be responsible for gathering a group of approximately nine individuals who meet the screening criteria and who are able to attend the group discussion at the specific time and location. Retailers for the semi-structured interviews will be selected purposively from a list of active retailers provided in July 2014 by the Administration for Socioeconomic Development of the Family (ADSEF), which is the agency that administers NAP in Puerto Rico. CBOs and group residential facilities will be recruited using a snowball selection technique and from a list obtained through searches on the web and supplemented by contacts with local organizations.

The survey sampling frame of NAP participants will be based on a list of current NAP participants provided by ADSEF in July 2014. The survey design will consist of a stratified systematic random sample of 1,000 NAP participants in Puerto Rico. To allow for comparison, the sample will be stratified by urban/rural and an equal number of NAP participants will be selected from each stratum. Substrata based on key subgroups (such as household composition) may be created at the time of sampling based on the subgroups’ representation in the NAP population.

1. Target Population

The target population for the telephone survey includes all NAP participants currently receiving NAP benefits (as of June 2014) in Puerto Rico. The target population for the focus groups will be NAP participants, and the population for the semi-structured interviews will be retailer managers, and CBO and group residential facility directors.

2. Survey Eligibility

All individuals in the survey target populations are eligible for the study, so no screening will be needed beyond verification that the individual received NAP benefits in June 2014 and conducts shopping for the household (at least sometimes).

3. Sampling Frame

The sampling frame for the telephone survey will be built from a list of NAP participants obtained from ADSEF’s database of current NAP participants receiving benefits as of June 2014. The ADSEF NAP office provided their database of current NAP participants in July 2014.

4. Statistical Methodology for Stratification and Sample Selection

We plan to select a stratified systematic random sample of 1,000 NAP participants in Puerto Rico. Prior to sample selection, we will first stratify the data based on urban and rural location. We will then perform systematic sampling within urban and rural strata. This method involves numbering the NAP participants in the population from 1 to N (= total records in population). To select a sample of participants, we take a participant at random from the first K participants and every kth participant thereafter until the designated sample size of participants is achieved in the stratum. In this way, each participant in the sampling frame will be given a known, nonzero probability of selection so that weighted inferences can be made about the entire population of participants. This will ensure a random sample within urban and rural strata.

5. Response Rates

Our goal for the telephone survey is to achieve an overall response rate of 75 percent. We feel that this is a likely response rate for this survey for numerous reasons. First, we plan to use a proven data collection methodology (telephone survey with locating and follow-up of nonrespondents). Second, to reduce the respondent burden, we have kept the questionnaire length to a minimum (15 minutes). Third, respondents will be offered a $10 gift card as a token of appreciation for participation in the survey. Fourth, the survey addresses a subject matter that is likely to be important and relevant to respondents. Finally, the survey will be conducted by an experienced survey research firm in Puerto Rico—Ipsos—with local employees who speak with the local dialect, cadence, and idioms familiar to the target population.

6. Reliability of Estimates

Estimates of percentages (such as the percentage of NAP participants who use their cash benefits in stores that do not accept their EBT transaction cards (Family Cards)) will have 95-percent, two-tailed confidence intervals of between 1.2 and 3.6 percentage points, and the rural (and urban) estimates will have 95-percent, two-tailed confidence intervals of between 1.7 and 5.1 percentage points. The confidence interval becomes larger for subgroup estimates. For example, with a sample size of 100 in a particular subgroup, the survey will be able to detect differences in percentages of larger than 8.8 percent, as compared to the sample as a whole. Follow-up computer-assisted telephone interviewing (CATI) methods will be used to ensure that the response rate goal of 75 percent is achieved.

7. Estimation Procedures

The primary purpose of the NAP participant survey is to assess NAP participants’ access to certified retailers, identify items purchased with the cash portion of their benefit, and explore potential impacts of eliminating the cash portion of the benefit with a corresponding increase in the non-cash portion of the benefit. We will calculate the final survey response rate and adjust the initial sample weights for nonresponse based on relevant variables available from the NAP database from which the sample frame was drawn. Following data collection, sample weights for NAP participants will be prepared and adjusted to account for 1) the initial probability of selection, 2) unit nonresponse, and 3) multiple selection opportunities (if needed). The product of these three weights will result in final weights suitable for use in analysis of responses. This weighting scheme inflates the respondents' data to represent the entire universe of NAP participants. Next, Insight will populate table shells, analyze the survey data, and provide written survey results (using charts/graphs and cross tabulations, as appropriate). Subgroup data will be presented as sample sizes allow.

B.2. Procedures for the Collection of Information

Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection

• Estimation procedure

• Degree of accuracy needed for the purpose described in the justification

• Unusual problems requiring specialized sampling procedures

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden

1. Data Collection

The proposed survey is designed as a telephone survey using CATI with nonresponse telephone follow-up to obtain information on the NAP participants’ use of the cash portion of the NAP benefit and potential adverse effects of discontinuing the cash portion with a corresponding increase in the non-cash benefit. We considered various methods of data collection, including mail, and recommend utilizing CATI for this target population for several reasons. CATI is an efficient way to reach a substantial number of respondents where the sampling frame is sufficiently large and the contact information is adequate to provide a reasonably high response rate. Phone numbers are likely to be accurate given that ADSEF’s NAP maintains current telephone contact information for their participants. Still, we acknowledge that cell phone numbers, which may be the primary telephone of many of these individuals, are less stable and more likely to change than landlines. When we find that telephone contact information is inaccurate, we plan to utilize standard locating procedures to identify a current phone number for the sample member. The use of CATI offers several advantages that can shorten the data collection period. For example, call attempts can be scheduled to maximize the chances of reaching the intended respondent, and interviewers can often obtain immediate locating information when the contact information on file is incorrect.

We believe that, overall, CATI would yield a higher response rate for this study than other modes of data collection, with an anticipated response rate of 75 percent for the final results. In addition, CATI will improve the quality of the data by ensuring that the most knowledgeable respondent is interviewed for the survey. The data collection methodology is as follows:

• An advance letter with frequently asked questions and answers about surveys will be mailed to potential respondents prior to the first telephone contact.

• The CATI instrument will be developed, tested, and programmed to assign interim and final status codes to track refusal, ineligible, and unlocatable cases.

• A survey management system will be programmed to track completed cases, partially completed cases, call history, and locating history.

• A training program will be developed and interviewers will be thoroughly trained on all aspects of the study.

• Tracing efforts using commercial locating databases will be implemented to obtain updated phone numbers for nonrespondents.

• Response rates will be monitored and analyzed by completed cases by time of day and days of the week to optimize calling times.

• Refusal conversion calls will be made by specialists trained in refusal conversion.

2. Follow-Up Methods

We propose a multipronged strategy for ensuring high response rates, including 1) obtaining the most current contact information from NAP administrative records; and 2) use of respondent-locating techniques¹ as needed. The interview script will assure sample members that their NAP benefits will not be affected by their responses and their privacy will be maintained. We will also design the scripts so they are not too lengthy. Our interviewers are trained in refusal conversion techniques and will utilize a wide range of methods to minimize nonresponse and maximize the complete data available for analysis. Procedures to maximize the response rate include the following:

• Pre-survey notification letters will be mailed to each sampled respondent approximately 1 week prior to outbound calling to inform him/her of the forthcoming call and the available token of appreciation for participation. The letter will be a single-page document printed in black and white. Note that the letter will be translated into Spanish.

• Seven-attempt protocol on different days/at different times of day. If needed, we may choose to add additional callbacks; however, the incremental increase in response rates diminishes beyond seven calls. Messages will be left for recipients to call a toll-free number to complete the survey. After seven attempts to reach a number, a replacement number will be used if available.

• Call rotation and flexibility. The CATI system can schedule calls to rotate among various times throughout the day and evening. The system allows respondents to call in to complete a survey or continue a survey over multiple sittings. Interviewers can also schedule appointments so that respondents can participate at a time convenient to them.

• Refusal conversion. Ipsos will work with Insight to implement refusal conversion appropriate to the needs of the project. The level of conversion will be communicated to interviewers as part of the training.

• Cross-sectional design. The survey is cross-sectional, so no future contacts are planned after the interview is completed by phone.

B.3. Methods to Maximize Response Rates and the Issue of Nonresponse

Describe methods to maximize response rates and to deal with issues of nonresponse. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

The methods described above have been proven in methodological research to yield response rates of 75 percent when the survey is of reasonable length and sample members consider the topic important. The following strategies will be used to help achieve this response rate, unless otherwise noted:

• Personalized pre-notification letters

• Strategically scheduled follow-up attempts

• Survey sponsorship by a recognized Federal agency

• A brief introduction that underscores the importance of the survey topic to sample members

• A short 15-minute survey

• Interviewer training that addresses potential obstacles in reaching or communicating with NAP participants and offers strategies for overcoming these obstacles

• A toll-free number for respondents with questions

• Locating efforts using commercial locating databases and directory assistance in an effort to obtain updated phone numbers for unreachable sample members

The pre-notification letter will briefly explain the purpose of the study and the reasons why sample members should volunteer their time. The letter will also include the estimated completion time of the survey, and assurances of privacy. Stating the sponsorship of the survey helps to engage sample members by providing immediate assurance that the survey is legitimate and not an attempt to sell them something. The likelihood of acceptance is greatly increased when sample members are told early why the survey is being conducted and why their responses are important.

B.4. Tests of Procedures

Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The instruments were pre-tested with less than 10 respondents which resulted in clarifications in instructions, changes in question wording and question order, and changes in response options. In addition, the pre-tests clarified the overall flow of the questionnaires and confirmed estimates of questionnaire length and burden calculations. Final edits to the instruments were implemented and submitted following FNS review of suggested changes.

B.5. Consultants

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Anne Peterson, Principal of Insight Policy Research (703-504-9483), provided consultation on the statistical aspects of the design. A review by the USDA National Agricultural Statistics Service (NASS) was also conducted (Attachment C of Part A). No other individuals outside the evaluation project were consulted on statistical aspects of the design. FNS has contracted with Insight Policy Research to conduct this study. Insight has subcontracted with Ipsos Marketing (787-753-8370) to collect the telephone survey data. Table B5.1 identifies the individuals at these organizations who will be responsible for collecting and analyzing the data. The Project Officer for the contract providing funding for the evaluation, Bob Dalrymple, will be responsible for receiving and approving all contract deliverables. His contact information is also included in Table B5.1 below.

Table B5.1

Individuals Responsible for Statistical Aspects and Data Collection and Analysis

Name	Title (Project Role)	Organizational Affiliation and Address	Phone Number
Carole Trippe	Associate Director (Project Director)	Insight Policy Research 1901 N. Moore Street, Suite 204 Arlington, VA 22209	(703) 504-9486
Alex Suchman	Senior Researcher (Project Manager)	Insight Policy Research 1901 N. Moore Street, Suite 204 Arlington, VA 22209
Meg Tucker	Senior Researcher (Task Lead for survey data collection and analysis)	Insight Policy Research 1901 N. Moore Street, Suite 204 Arlington, VA 22209	(703) 504-9485
Anne Peterson	Principal (Quality Assurance)	Insight Policy Research 1901 N. Moore Street, Suite 204 Arlington, VA 22209	(703) 504-9483
Esteban Rivera	Field Liaison (On-site liaison for Telephone survey and Focus Group)	Ipsos Marketing 463 Fernando Calder San Juan, Puerto Rico, 00918	(787) 753-8370
Bob Dalrymple	Senior Analyst for the SNAP Research and Analysis Division (Project Officer)	USDA Food and Nutrition Service, Office of Policy Support 3101 Park Center Dr. Alexandria, VA 22302	(703) 305-2122
Jeff Bailey	Chief, Summary, Estimation and Disclosure Branch Methodology Division, USDA/NASS NASS Reviewer	USDA, National Agricultural Statistics Service, Statistical Methods Branch 1400 Independence Ave., S.W. Washington, DC 20250	202-720-4008

1 We will use locating databases such as LexisNexis and residential telephone listings to locate sample members based on names, addresses, current or former telephone numbers, and/or other identification numbers.