Attachment 4a - TWG MeetingSummary December 2011

Attachment 4a- TWG Meeting Summary December 2011

[Intentionally Blank]

Attendees

TWG Members

Alyson Abrami – NY City Health Department, Healthy Bucks Program (New York)

Rachel Chadderdon – Fair Food Network (Michigan)

Anna Curtin – Portland Farmers’ Market (Oregon)

Jezra Thompson – Roots of Change (California)

FNS

Eric Sean Williams, PhD – Contracting Officer Representative

Kelly Kinnison, PhD – Contracting Officer Representative

Mark Byron – Supplemental Foods Program, WIC

AMS

Debra Tropp – Agricultural Marketing Service, Branch Chief

Ed Ragland – Agricultural Marketing Service, Economist (by teleconference)

Community Science

Chris Botsko – Organizational Cluster Evaluation

CDC

Jan Jernigan, PhD (by teleconference)

Westat

Melissa King – FM Incentive Programs, Task Order 3 Project Director

Mary Butler, PhD – Lead Qualitative Analyst

Adam Chu – Sampling Statistician

Emily Liu, PhD – Lead Quantitative Analyst

Jane Schulman, PhD – Corporate Officer

Stacy Miller – Executive Director, Farmers Market Coalition

Stephen Leard – Research Assistant

Susie McNutt – Farmers Market BPA Project Director (by teleconference)

Sujata Dixit-Joshi- Farmers Market Task Order 1 Project Director (by teleconference)

Mustafa Karakus- Farmers Market Task Order 2 Project Director

The inaugural meeting of the Farmers Market Incentive Program Study (FMIPS) Technical Working Group (TWG) took place on Thursday, December 1, 2011 at Westat’s Conference Center in Rockville, Maryland. The purpose of the meeting was to discuss the research plan for FMIPS, including the study design, methodology, and implementation plans, with the TWG. Valuable input was received to enable Westat to refine the research plan to assist in ensuring maximal study outputs. This document summarizes the topics that were addressed and the related considerations or conclusions of the discussions. The membership roster of the TWG is included in the table below. The table also indicates the required area of expertise that each member represents for Task Order 3.

Category	Member	Contact Information
Umbrella/National Incentive Resource Providers	Rachel Chadderdon,	Fair Food Network Program Director for Double Up Food Bucks Ann Arbor, MI RChadderdon@fairfoodnetwork.org 734.717.0050
Regional or State Umbrellas Overseeing Incentive Programs	Jezra Thompson	Roots of Change Program Manager California Farmers Market Consortium San Francisco, CA jezra@rocfund.org 415.391.0545
Farmers Market Incentive Operators	Anna Curtin	Portland Farmers Market Education and Outreach Specialist 240 N Broadway, Suite 129 Portland, OR anna@portlandfarmersmarket.org 503.241.0032
Local Incentive Funders	Alyson Abrami	Farmers Market Programs NYC Dept. of Health and Mental Hygiene Physical Activity and Nutrition Program Queens, NY aabrami@health.nyc.gov 347.396.4221

Overview and Introduction

Dr. Williams, the Task Order 3 Contracting Officer’s Representative, welcomed the TWG and provided a brief background on the Food and Nutrition Service’s (FNS) goal for the FMIPS. He explained that FNS seeks to better understand the composition of organizations involved in the Supplemental Nutrition Assistance Program (SNAP) incentive programs, how they work, the relationships between different types of organizations in the support of farmers markets, and ultimately how FNS can better serve its clients through both farmers markets and incentive programs at farmers markets

He further explained that FNS is exploring how to assemble systematic data about incentive programs, specifically in terms of their operations, decision making processes, and means of self evaluation. The TWG’s help is needed to ensure FMIPS includes questions of “what we don’t know” or haven’t considered. Thus, the work of the TWG throughout the meeting is not necessarily to answer the research questions that have been put forth, but to tell us if we are missing questions and as importantly, the best way to ask the questions. Dr. Williams informed the TWG that while there would likely be discussions on many aspects of incentive programs in the meeting, all ideas proposed or raised by the TWG moving forward would need to fall within the scope of the study plan of Task 3 in order to be seriously considered or adopted into the research plan.

TWG Mission and Introduction to FMIPS

Ms. King reviewed the mission statement for the TWG and stressed the opportunity that was being presented for them, as key stakeholders, to participate in the FNS research. The TWG meetings will offer a forum for them to be heard and provide input on FNS’s research, and to provide Westat with input on the research approach and methods. Westat is seeking guidance not only on content area and practical issues of conducting the study – participation, building relationships and collaboration with organizations – but also in discussing the question that may arise, “why should organizations participate in the study?”

Ms. King asked the TWG to think, as representatives of the community, about the critical areas of interest for organizations would be once the data is collected and analyzed. Indicators and theories providing by the study and the participating organizations will enable the development of a variety of incentive program models and an indication of which of these various models work in what settings. Knowing there will be no “one size fits all” model, Ms. King explained that the ultimate goal is to compare and describe different models for incentive program organizations and try to assess which work better. She used the example that a model that worked in an urban community may not work in a rural community, or depending on the location of the markets there would likely need to be different stakeholders and/or community organizations involved for success.

The qualitative and quantitative objectives of the study were presented. Ms. King introduced the agenda item later in the day to discuss the issue of data sharing by organizations and the obstacles as well as the benefits to providing the study with self evaluation data. A benefit is that it would provide the opportunity to examine the types of data that are out there, and the quality and comparability of the data (apples to apples vs. apples to oranges). She stressed the importance of getting organizations to share their data and the need for TWG input on means to encourage data sharing. Ms. King reviewed the timeline for the key milestones of the study. She explained that due to the need for Office of Management and Budget clearance, data collection would not begin until Fall of 2012 with the final report available in Summer 2013. FNS concurred with Ms. King that it was the intention to share preliminary findings prior to Summer 2013, as is possible.

Design and Methods

Qualitative Research and Analysis

Dr. Butler presented the plans for the qualitative research. She said the goal of her review of the interview plans with the TWG was to be able to understand the feasibility and acceptability of what Westat proposes, as well as to adopt a design that will be satisfactory to the people participating in the study.

The goal of the qualitative component of Task 3 is to gain as great an understanding as possible of how incentive programs work and ultimately promote the purchase of fresh fruits and vegetables. The basic unit of study in the qualitative component of the FMIPS are the organizations directly or indirectly involved in the administration of incentive programs (basic unit of study = organization). Dr. Butler explained that the qualitative component of Task 3 is purposive, and therefore not intended for making inferences to a larger population. This distinction allows for the selection of the best respondents possible, i.e. those with the potential to contribute and provide the most illustrative and meaningful data. It is assumed that organizations of Types I, II, and III will interact and overlap to a wide and varied degree. She described the current plan to approach up to 15 Type I organizations, 40 Type II organizations, and 45 Type III organizations to participate in telephone interviews. Dr. Butler noted that the telephone interview format for administration of the qualitative instrument is extremely flexible, only bounded by the time limit imposed on the interview.

Dr. Butler explained that the qualitative data will be compiled into a database for two principal reasons – to provide a means for other organizations to analyze the findings (a reliability check), and to make the information publicly available for review. Dr. Williams stated that all three FNS tasks will have some publicly available data, and that personal identification information will necessarily be removed.

The question was raised whether Westat would ascertain the mission and governance of the selected organizations. Ms. King responded that this is a key organizational characteristic that we want to capture. She elaborated on the intent of Task 3 to elucidate organizational characteristics to create a profile, or typology, for each of the participating organizations. Dr. Dixit-Joshi, Westat’s Task Order 1 Project Director, stated that the mission and governance information are not being collected in Task 1, but that typology information (e.g. characteristics of supporting organizations) will be collected and available to Task Order 3. Ms. King confirmed that Task 3 would use as much of the organizational information as possible for establishing the frame of organizations for selection. It was noted that the input of TWG participants to identify the most salient characteristics for establishing an effective organizational typology is valuable and on the agenda to discuss in more detail later in the day.

Quantitative Research and Analysis

Dr. Liu explained that once the qualitative data were collected, they will be coded to allow for further analysis quantitatively. This analysis will be conducted in conjunction with the ALERT data as part of the FMIPS’ second objective. The ALERT dataset contains every EBT transaction, which are recorded by card number. Further, the location and total purchase of individual SNAP transactions can be determined (e.g. at a farmers market or grocery store). The quantitative analysis is intended to evaluate the 2012 performance, outcomes, and impacts of each incentive program on increasing SNAP redemptions at farmers markets and to assess the similarities and differences across programs with respect to performance and impact.

Dr. Liu reviewed some of the proposed analyses but pointed out that there were many potential questions to be posed and that the specific questions ultimately addressed are up for discussion. She noted that unlike the qualitative work, the planned unit of analysis in the quantitative analyses will be farmers markets. Westat proposes to collect and examine data at the market level (e.g. SNAP per household derived from ALERT data), which can then be subsequently matched to sociodemographic variables. Business and/or community characteristics will be available from Task Orders 1 and 2 and this information will be available for markets with incentive programs.

A discussion ensued regarding the details and feasibility of various intended comparisons (e.g. social vs. economic), and the specific definitions of key terms such as “market level”. “Market level” according to the ALERT data would not necessarily represent one individual market but could include a group of markets that are in the system as one entity. The TWG was reminded that the purpose of the quantitative analysis is to systematically measure the qualitative data collected and to use ALERT data to identify any changes in farmers market use based on incentive programs. In response, Ms. Miller pointed out that redemption rates do not necessarily equate to success. Instead changes in habit and perception are the goals. SNAP clients may not return to the farmers market, but might be making healthier choices elsewhere and we will not capture that trend. Ms. Abrami also questioned what data on SNAP clients was available related to behavior change, additive impact, and specific purchases. Dr. Williams reiterated that the mission of FNS is to serve its clients, the users of the SNAP program, and that many of the questions that the TWG participants were interested in regarding SNAP client behavior will be addressed by Task Order 2. It was noted by Ms. Tropp that change can be measured in many ways. She said AMS would be interested in future more complex analyses to address other measures, such as job growth, one of the main charges of her organization.

Presentations by TWG Member Organizations and Programs

Each of the TWG members provided an overview of their respective organizations, focusing on: goals; organizational mission; history of the organization and the incentive program; sources of funding; challenges faced; and their methods of implementation and evaluation of their incentive programs.

Dr. Kinnison raised the issue that there are often difficulties in finding systematic documentation on the impact of incentive programs, despite the apparent successful results that are reported. She pointed out that the ability to see how incentive dollars are distributed and the criteria used for distribution on a national scale will be extremely useful. In addition to the goal to gather data at the Federal and national level but ideally, to be able to develop simple criteria for collecting quality data by farmers markets. Having good, consistent, simple data from local markets is important. Ms. Thomson concurred with the need for the markets to have solid data. She stated that a desired focus of Roots of Change is to empower clients to develop capacity and one of the only ways this will happen is if they have data. Ms. Miller reported on the Farmers Market Coalition’s progress with a reporting system that will allow member organizations to select “a la carte” a set of indicators for assessing performance. Given the system would be relatively rudimentary, she, too, expressed an interest in having data that can be used based on broader or perhaps more complex yet useful indicators that could be compiled for local constituent markets. Ms. Abrami said a major area where development of capacity is needed is in real-time reporting. One impediment to this is that any rigorous data collection process is challenging in the majority of farmers market environments. One area where further analysis would be useful is the extent of the incentive compared to the rate of effect on sales. For example, a two dollar incentive may be just as useful as a five dollar incentive. Although this is outside the scope of Task 3, FNS reiterated the importance of making the FMIPS results accessible for further study so that issues like this could be addressed.

Organizational Characteristics

There was extensive discussion on the challenges of developing a feasible and effective typology of the different organizations to be sampled in Task 3. Ms. King described the various dimensions that the Westat team felt important. These included the organization’s mission and goals; its size; history and age; its source of funding; and the organization’s stakeholders. The TWG agreed that these were critical dimensions but there was general consensus that the currently proposed three-type-of-organizations methodology will need extensive clarification to be successfully implemented. Not only will it be challenging to categorize some of the organizations using the proposed definitions, there would likely be overlap between categories. Social mapping was suggested to determine the appropriate organizational categories for Task 3. The TWG felt that instead of a hierarchy, we needed to make sure that we thought of our study as more of a web. Dr. Butler clarified that while we were using the organization definitions provided by FNS, the plan was to consider the organizations and their relationships to each other and the community as more of a cluster. Dr. Williams said that he understood there were limitations to the FNS categories but that they were needed to provide some structure to the audience we under study.

Issues related to organizational capacity and differences in community receptivity were raised by Ms. Tropp. She pointed out that there are several ways to stratify interactions within the farmers market universe, e.g. number of markets served, operational budget etc. Ms. Chadderdon suggested organizations be categorized and/or characterized by their funding and support (Type 1), the level of administration (Type 2), and the boots-on-the-ground market level (Type 3).

Dr. Williams raised the importance of the organization’s function versus the organization’s size. The TWG then discussed how to quantify function for the purposes of establishing a typology. The TWG members discussed how they would define size, with the general consensus that function of the organization was directly related to the organization’s capacity, the latter of which was also difficult to define. Further, the TWG pointed out that the differing missions of the various organizations and markets (e.g. to support farmers, to support economies, to support healthy food access) would have a direct impact on their function, size, and thus their capacity. Ms. Kinnison reminded the Westat team that some organizational typology information discussed by the TWG will be collected on the Task 1 Farmers Market Manager Survey. The Westat team agreed to go back and take a closer look at the organizational descriptors that would be available a priori and adjust the plans as needed with regard to the questions to be asked. However, it was felt that for now it would be best to stick with the organizational definitions provided by FNS but the input provided by the TWG clarified the limitations and where Westat would need to exercise caution in its thinking of the organizational typology.

Incentive Program Characteristics

Discussions focused on the actual content of the interview guides, specifically what questions or topic areas would be useful to understand the different kinds of incentive programs. Dimensions proposed by Westat to characterize incentive programs included: the selection process for providing SNAP Incentives – is there an “award algorithm” for the vendor to qualify for incentives; the limitations on the use of the funding – incentives only vs. administration (employees and machines); incentive amounts; monitoring and tracking of incentive funds; mechanisms for vendor redemption of incentives; communications, advertising, and PR for incentive programs; and the barriers to providing incentives and implementing programs in FMs. The TWG concurred with the proposed dimensions but when asked about what aspects of the programs were missing, an ideological discussion developed regarding the organization’s long term goals for its incentive program. This was considered an important dimension to explore given the different perspectives and attitudes that organizations have towards incentive programs (e.g. incentives vs. subsidies, etc.). To address this, Dr. Williams suggested three specific questions that could possibly be examined in Task 3:

• Which organizations see incentive programs as a temporary incentive, and which organizations view incentive programs as an alternative revenue stream?

• Which organizations’ missions align with the stated mission of SNAP?

• Which group is responsible for the accurate redemption of incentives (i.e. central, market managers or individual vendors)?

Westat will incorporate this dimension into its study plans.

The implications of political barriers related to conducting Task 3 were pointed out by Ms. Miller and other TWG members, such as the extensive policy process that had been occurring, and the history of the relationship between farmers markets and the USDA. Thus the landmark nature of the current FNS commitment to study the role of farmers markets in the food system was noted as a welcome and a praiseworthy sign of progress, with credit due to the FNS and the large number of farmers markets and supporting organizations.

Data Sharing and Collaboration Strategies

Ms. King noted that Westat anticipated some challenges for gaining organizational collaboration for the study. Given this, she asked the TWG to provide ideas on items of interest to the organizations that could be provided in return for study participation and data sharing. The recently developed CDC farmers market incentive program evaluation toolkit was discussed, and Dr. Jernigan stated that the CDC would be happy for FNS to use the toolkit. However, the TWG felt this was less than enticing given the toolkit would be publically available to organizations. The TWG stated a higher order, more illustrative and powerful analysis by FNS and Westat of individual organizational data would be helpful, not only internally but in the interest of obtaining future funding (e.g. in the rigorous application process required for obtaining funding). General reports provided by FNS as a result of the analysis were not considered to be as valuable to participating organizations. Ms. Thompson pointed out many organizations that have data generally don’t have the capability to conduct complex analyses and therein would be the major incentive for organizations to participate.

FNS pointed out the value-add component of Task 3 data sharing; participating organizations would be included in the report in return for the provision of their data. Dr. Kinnison described that the diversity of farmers markets has been a consistent benefit and challenge in the policy arena. She cited how national averages in the farmers market community are generally useless because the data is not of a high enough quality to be used to effect change. Results at a national level are more critical to federal policy than data provided by individual organizations or a small group of organizations. Ms. Tropp agreed and explained the growing requirements for transparency in data collection and evaluation. Federal research policy and practice (e.g. data.gov) are moving towards this transparency and thus the requirement for sharing data with the public (i.e. participating organizations). In this context, the value-added proposition of providing data for Task 3 with respect to political weight and policy-making leverage was much stronger than similar evaluations conducted internally by a participating organization. Ms. Miller as well as the TWG members found this argument to be insightful.

Interview Considerations

Practical and logistical issues regarding the actual interviews were discussed. The TWG felt that the proposed length of 1 hour for the qualitative telephone interviews was both sufficient and reasonable, and that interviewing three respondents within each organization was a good approach. It was suggested that when recruiting and describing the expectations on length to respondents that we should indicate they anticipate an hour for the interview even if we thought the interview could be as short as 45 minutes. This would provide a cushion in case you had a longer interview. Interviews with organization directors will be developed to be much shorter, 20 minutes or less, given the limitations on both their time and their ability to provide the type of detailed data we aim to capture. A caveat was made and agreed upon that there will be variability by organization on the number of staff sufficiently familiar with their organization’s incentive program, and therefore we should allow for some variability in the number of respondents within each organization. It was suggested that we may consider interviewing between 2 and 5 respondents depending on the organization.

The TWG provided suggestions of individuals that could be approached for the pretesting to be conducted on the interview guides prior to OMB submission. TWG members provided several recommendations for possible pilot test respondents, including: Ellee Igoe, Meredith Freeman, Amy Gilroy, Diane Eggert, Darlene Wolnik, Sabrina Baronberg, Richard McCarthy, Emery Van Hook, Michael Hurwitz, Randall Fogelman, and Brandy Brooks. As needed, Westat will follow-up with the TWG regarding contact information for some of these individuals.

Summary and Discussion

Future dates for the next two meetings of the TWG were discussed. There was a recommendation to circulate several options by calendar but it was agreed to tentatively plan the second meeting for March 2012 (prior to the OMB submission in April) and the third meeting to occur when study results are available in draft form. It was noted by TWG members that March might be a difficult month considering other farmers market conferences that will be taking place. Westat will circulate calendars and identify a date for the second TWG meeting.

Several communications channels will be available for disseminating information and raising awareness for Task 3. These channels included the Agricultural Marketing Service, Roots of Change, and the Farmers Market Coalition, each of which expressed different methods available to them for raising awareness of the study among their constituencies. A webinar disseminating information and handling Q&A was recommended.

Awareness regarding the study will begin later in 2012 so that there is no confusion of Task 3 with Tasks 1 or 2.

FNS announced a new email address that is being provided for use by its stakeholders, SNAP-FMstudies@fns.usda.gov.