FMIPS OMB Supporting Statement B 08032012_103112_REV

The respondent universe will include key staff at selected organizations that agree to participate. The total number of respondents is 225 individuals at 115 organizations. This includes up to 3 individuals working for each of the 15 Type I and 40 Type II organizations, and 1 individual at each of the 45 Type III organizations. It is estimated that 10 percent of each type of organizations invited to participate, a total of 15 organizations, will refuse (2.5 Type I, 8 Type II, and 4.5 Type III organizations). If a selected organization refuses to participate, another similar organization will be selected as a replacement. We have developed these numbers based on the best possible information currently available. Little is known about the interrelations and operations of incentive organizations; thus, there is minimal empirical grounding from which to build estimated numbers for a sampling plan.

B.1.2. Sampling Methods

Sampling Frame

The sampling procedure is summarized in Table B1. The unit of analysis for the Study of Organizations Providing or Administering SNAP Incentives at Farmer’s Market (Farmers Market Incentive Provider Study, FMIPS) will be the cluster of organizations serving a particular geographic area. Development of a frame of organizations to be included in the FMIPS will be drawn from the Nutrition Assistance in Farmers Markets: Understanding Current Operations (FM Ops). The FM Ops Survey includes a stratified random sample of all FMs with four strata: (1) SNAP authorized FMs and direct marketing (DM) farmers that redeemed benefits during the last 12 months; (2) SNAP authorized FMs and DM farmers that did not redeem benefits during the last 12 months; (3) FMs and DM farmers that were SNAP-authorized sometime during the last 5 years, but not authorized during the last 12 months; and (4) FMs that have never been authorized to redeem SNAP benefits. Stratum 1 will be the starting point for selecting the sample for the FMIPS.

Table B1. Steps for Building the Sampling Frame

Step	Activity
1.	Using FM Ops data, build a list of FMs offering SNAP financial incentives. [If Question D1 equals “offered financial incentives’, then output to list 1, which will include record ID, D3 (# orgs providing support to incentives); D4 Names of top 3 orgs providing support); D5 (what do you do to see if incentives are successful. Additional data from table in Section E will be extracted as well)]
2.	Convert list by FM to list by organization
3.	Sort by organization name and record ID;
4.	Clean list and remove duplicates
5.	Output a new list in which the unit of analysis is the organization to include the name of the organization
6.	Conduct internet search to complete and finalize the list with information for sampling (e.g. geographic region, type; communities served, contact information.) Work closely with FMC and FNS to refine the list.

The sampling frame will be formed by responses to specific questions on the FM Ops survey regarding the names of organizations that provide or administer SNAP incentives for use at their markets. Many FMs will provide more than one organization in their response. Thus, the size of the FMIPS sampling frame is currently somewhat ambiguous.

One of the goals of this study is to obtain the data needed to build a typology of organizations involved with the funding and administration of SNAP incentives in FMs. The technical working group (TWG) of experts that was formed for this study was consulted, along with an additional expert, Ms. Stay Miller of the Farmers Market Coalition, on the number and types of organizations that participate in SNAP incentive programs. The consensus was that these characteristics would vary significantly within and between communities, as well as in different parts of the country. Given this variation, it has proven difficult to develop an a priori typology of organizations for sampling because not enough is known about the organizations and the interrelationships of stakeholders involved in SNAP incentive administration and operations. Summaries of the meeting and conference call held with the TWG are included in Attachments 4a and 4b.

Selection of Organizations

As described in Supporting Statement A, three types of organizations will be interviewed. There will be two stages of sampling. Stage one will begin by sampling 15 Type I (large incentive granting) organizations that serve multiple FMs based on the responses received in FM Ops (Table B1). One of the unique traits of these organizations will be that they provide incentive grants to other organizations. The second stage will be to build the remainder of the sample by directly identifying candidate organizations in interviews with the Executive Director and the person in charge of the incentive program in the sampled Type I organizations. As part of the interview, Type I participants will be asked to identify organizations with which they work as part of the incentive program. The Executive Director of these smaller organizations will then be contacted and asked to be interviewed about their organization and their relationship to the larger organization. In this way, the participation of these organizations in SNAP incentive programs and the division of support among the organizations will be an empirical question that will be explored as part of the study rather than an a priori site selection criterion.

We will select one or more backups for each stratum of Type I organizations to be used as replacements as necessary. Once 15 Type I organizations agree to participate, we will identify and purposively sample one or more Type II and Type III organizations working with them. About 100 organizations in total will be selected with the aim of conducting approximately 210 telephone interviews with key informants at supporting organizations. These numbers were developed based on estimates provided by the Farmers Market Coalition, a key stakeholder in the farmers’ market community and of which many organizations of the type we will interview have memberships. Thus, we have developed these numbers based on the best possible information currently available. Little is known about the interrelations and operations of incentive organizations, so there is minimal empirical grounding from which to build estimated numbers for a sampling plan.

Selecting Individuals for Interview

It is important that the people at selected organizations most useful to interview are identified. As described previously, collaboration with the Farmers Market Coalition, FNS, and the project working group will assist in identifying the Executive Director of each organization, specifically those at Type I organizations. During the scheduled interview, the director will be asked to provide one or two other people in their organization who are knowledgeable and involved with the SNAP incentive program. These people will be interviewed and also asked to identify the Type II and III organizations that they interface with and who should be contacted at these organizations for interviews. At least two persons in each Type I and Type II organization will be interviewed. We anticipate that the Type III organizations will have fewer resources and employees devoted to farmers market incentive programs. This may be because their primary mission is something other than incentive programs and/or that they have smaller operating budgets. Because of this, only one individual in each selected Type III organization will be interviewed. Table B2 summarizes the total number of interviews to be conducted and how these fit into the scheme of the organizational universe.

Table B2. Sample Design for Interviews with Supporting Organizations

Organization type	Number of organizations in the frame	Sampled organizations	Maximum number of interviewees in each organization.	Total number of interviews
Type I	30	15	3	45
Type II	153	40	3	120
Type III	Unknown	45	1	45
Total		100		210

B.1.3. Response Rates, Sample Size, and Non-Response Analysis

Procedures that will be applied for selecting organizations and individuals for the study have been explained in details in section B.1.2.

B.2. Describe the Procedures for the Collection of Information including:

Statistical methodology for stratification and sample selection,
Estimation procedure,
Degree of accuracy needed for the purpose described in the justification,
Unusual problems requiring specialized sampling procedures,
and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Data for this study will be conducted by means of scheduled telephone interviews with selected individuals. A Westat staff member, who is an expert in the area of qualitative research, will conduct these interviews and analyzing the data qualitatively. Sampling frame and sample selection procedures have been explained in section B.1.2.

B.2.1. Statistical Methodology for Stratification and Sample Selection

As previously mentioned this is a qualitative study and is not intended to be nationally representative. Statistical procedures have not been applied to calculate study sample.

B.2.2. Estimation Procedures

The descriptive analysis of ALERT data will include simple statistics for each of the characteristics of FM SNAP incentive programs. These will include means, medians, standard deviations and/or confidence intervals, frequency distributions and cross-tabulations. Once all data is entered, we will begin the analysis by generating reports for important questions across all data sources within each organization. We can then produce narrative summaries for each question across all organizations by type, e.g., we can summarize the perspectives of directors of all umbrella organizations across communities. A computerized set of matrices will be used to develop narratives addressing specific research questions, such as the advantages and disadvantages of alternative strategies or factors that facilitate or impede the ability of organizations to convert incentives into increased sales of fruits and vegetables.

There are two advantages to this kind of analysis of qualitative data. First, it grounds conclusions in evidence that can be traced and assessed by policymakers, program managers, and others with a need to know. Anyone can pull up the analysis database and track the logic underlying the conclusions. The database is also a resource for further research. FNS will be able to replicate analyses or develop new ones as questions emerge. Preservation of all steps in the analysis is an important part of protecting validity, reliability, and scientific integrity of any qualitative data collection.

The descriptive analysis of ALERT data will include simple descriptive statistics for each of the characteristics of FM SNAP incentive programs. These will include means, medians, standard deviations and/or confidence intervals, frequency distributions and cross-tabulations.

B.2.3. Degree of Precision Needed for the Purpose Described in the Justification

Not applicable.

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

No specialized sampling procedures are involved.

B.2.5. Any use of Periodic (less frequent than annual) Data Collection Cycles to Reduce Burden

The study design requires a one-time data collection from respondents. All data collection activities will occur within a 3 month period.

B.3. Describe Methods to Maximize Response Rates and to Deal with Issues of Non-Response.

The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

Much of the data will be collected using telephone interviews. Specific procedures to maximize response rates include:

A cover letter from USDA/FNS.
Follow up calls to respondents to schedule interviews.
Interviews will be conducted by trained personnel who are experts in qualitative research area.
Implement standardized training interviewers. The interviewer training will focus on basic skills of telephone interviewing and probing.

Non-response is anticipated at the organization level. If a selected organization refuses to participate, the plan is to substitute a similar type of organization in its place instead. Once a selected organization’s leader agrees to participate, all selected staff members are expected to respond as well and a complete data matrix is expected to be obtained. It is anticipated that not all organizations that participate in general will be willing to provide self-evaluation data, even if the organization has such data.

B.4. Describe any Test of Procedures or Methods to be Undertaken.

Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

A pretest of the interview instruments was conducted by telephone with a panel of key informants who are knowledgeable about SNAP incentive programs and not currently working with eligible organizations. Pretest participants were individuals who have worked for eligible organizations in the past. A total of 7 individuals participated in cognitive testing to ensure testing is conducted in line with Office of Management and Budget (OMB) provisions. None of the data captured as part of the pretest will be part of the study data.

The purpose of the pretest was to assess whether the questions included in the interview protocols are understood by the group with whom they will be used. It also ensured that the way the question is worded is acceptable and not offensive, particularly for potentially sensitive topics such as the existence of misuse within the incentive programs. The initial pretest panel participants (n=3) received an electronic copy of the instrument and were asked to comment on clarity and comprehension. Based on comments and feedback received from these pretest participants, the instruments were revised and tested on the remaining members of the pretest panel (n=4). All instruments were then finalized for submission and approval by OMB. See Attachments 3a and 3b for templates of letters that were used to invite individuals to participate in the pretest panel.

B.5. Provide the Name and Telephone Number of Individuals Consulted on Statistical Aspects of the Design and the Name of the Agency, Unit, Contractor(s), Grantee(s), or Other Person(s) Who Will Actually Collect and/or Analyze the Information for the Agency.

Table B3. Individuals Consulted on Statistical Aspects of the Design

Name	Organization	Phone	Email
Alyson Abrami	NY City Health Department, Healthy Bucks Program (New York)	(347) 396-4221	aabrami@health.nyc.gov
Rachel Chadderdon	Fair Food Network (Michigan)	(734) 213-3999	RChadderdon@fairfoodnetwork.org
Anna Curtin	Portland Farmers’ Market (Oregon)	(503) 241-0032	anna@portlandfarmersmarket.org
Jezra Thompson	Roots of Change (California)	(415) 391-0545	Jezra@rocfund.org
Chris Botsko	Community Science (Organizational Cluster Evaluation)	(301) 519-0722	cbotsko@communityscience.com
Jan Jernigan, PhD	CDC		jjernigan1@cdc.gov
Melissa King	Westat (FM Task Order 3 Project Director)	(240) 453-2721	melissaking@westat.com
Mary Butler, PhD	Westat (Lead Qualitative Analyst)	(301) 610-8862	marybutler@westat.com
Adam Chu, PhD	Westat (Sampling Statistician)	(301) 251-4326	CHUA1@westat.com
Barnali Das, PhD	Westat (Lead Quantitative Analyst)	(301) 279-4593	barnalidas@westat.com
Jane Schulman, PhD	Westat (Corporate Officer)	(301) 212-2108	janeschulman@westat.com
Stacy Miller	Westat (Executive Director, Farmers Market Coalition)	(304) 685-2669	stacy@farmersmarketcoalition.org
Susie McNutt	Westat (FM BPA Project Director)	(301) 738-3554	SusieMcNutt@westat.com
Sujata Dixit-Joshi	Westat (FM Task Order 1 Project Director)	(508) 435-0402	SujataDixit-Joshi@westat.com
Mustafa Karakus	Westat (FM Task Order 2 Project Director)	(301) 294-2874	MustafaKarakus@westat.com
Debra Tropp	AMS Branch Chief	(202) 720-8326	debra.tropp@ams.usda.gov
Ed Ragland	AMS Economist	(202) 690-1327	Edward.Ragland@ams.usda.gov
Mark Byron	FNS (Supplemental Foods Program, WIC)	(703) 305-2733	mark.byron@fns.usda.gov

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Melissa King
File Modified	0000-00-00
File Created	2021-01-30

FMIPS OMB Supporting Statement B 08032012_103112_REV