Attachment J - Cognitive Testing Report

Attachment J Cognitive Testing Report

Report on Cognitive Testing for the Farm to School Census

Katherine Ralston

August 31, 2012

As part of development of the Farm to School Census, we worked with Farm to School Coordinators in 6 regional offices to identify States and School food Authorities with a range of characteristics that would be encountered in the data collection, including size of the district and whether they already purchase locally produced food. State Child Nutrition Directors were asked 15 questions about the feasibility of providing contact information for food service directors in the State and forwarding the invitation to participate in the Census, as well as their assessments of potential problems. SFA Food Service Directors were asked to complete the questionnaire and answer 7 questions about their experience responding.

State Child Nutrition Directors

In order to test the list frame development process, we spoke to Child Nutrition Directors in Texas, Arizona, Florida, Montana, West Virginia, and Michigan. Our questions and their responses are given below.

Question: We plan to ask State Child Nutrition Directors for contact information for all public school district School Food Authorities (SFAs) in the State, including primary contact name, complete mailing address, email address and phone number.

• Would you be able to provide none/some/all of that information?

• Could it be provided electronically?

• How much lead time would you need?

• Do you have data that identifies which SFAs are managed by a private Food Service Management Company?

• Do you have data that identifies which SFAs cover more than one school district?

Responses: All the Child Nutrition Directors we consulted indicated that they would be able to provide the information electronically, with lead time of at most one week. States that have SFAs that are managed by a private Food Service Management Company indicated they would be able to identify those districts. States that have SFAs that cover more than one school district are also able to identify those SFAs.

Question: We plan to send a memo through the PartnerWeb system, asking State Child Nutrition Directors to forward a request to all public school SFAs to complete the questionnaire.

• When you forward items to SFAs, do you usually send them by email, hard copy, or both?

• Would you be able to screen the transmittal so that it only goes to public school district SFAs?

• Would you be willing to send it multiple times over the year?

Responses: All the Child Nutrition Directors we consulted indicated that they send PartnerWeb memos by email and no longer send by hard copy. They are able to screen the transmittal to send to public school districts, and are willing to send multiple reminders.

Question: We plan to ask SFA food service directors to complete the questionnaire on-line, with an option to call in and complete the questionnaire by phone, or request a paper copy to fax back or mail it back.

• Do you think a lot of directors will find it difficult to complete the questionnaire on-line?

• Why?

All States indicated that email communication with SFA Food Service Directors was the main channel for communication, although the Child Nutrition Director from Montana described email contact with Food Service Directors is “getting better”. All States, including Montana, reported that Directors would not have a problem replying on-line, especially if the link to the survey is included in the email.

Question: We plan to make these data publicly available, which means that the responses won’t be confidential.

• Do you think respondents will be uncomfortable with that?

• Why? What proportion?

• What might help assure them that personal information will not be revealed?

All States reported that as long as the reasons for non-confidentiality were made clear and personal information were not included in the collection, Food Service Directors would accept the lack of confidentiality of the responses, and that many would be eager to share information about their Farm to School activities.

Question: We are looking for food service directors to pretest the Farm to School Census questionnaire. Pretest respondents should be available in the summer, and should include some that have no local purchasing or farm to school activities, and some that might be expected to find an on-line survey more difficult.

• Can you suggest a food service director that would be appropriate to pretest the questionnaire in your State?

• Can you provide an email address and phone number?

On the basis of discussions with the States, 8 Food Service Directors were selected to review the survey on-line, from an invitation similar to the one that will be used when the Farm to School Census is fielded in early 2013. Questions posed to the Food Service Directors about their experience with the questionnaire and their responses are described below.

Food Service Directors

Food Service directors from 8 school districts reviewed the questionnaire and provided feedback in response to 7 questions.

The districts were:

• Great Falls MT;

• Tempe, AZ;

• Deweyville, TX;

• Clint Independent School District, TX;

• Richardson Independent School District, TX ;

• Grand Rapids, MI;

• Benzie Central School District, MI;

• Fayette County WV.

The respondent from Montana was unable to complete the questionnaire due to technical difficulties with the on-line questionnaire. The respondent from Clint, TX did not respond to questions about the questionnaire. The remaining respondents answered questions about their experience with instrument, summarized below.

Question: How long did the questionnaire take?

Responses: Of the 2 SFAs that did not currently purchase locally produced food, one answered “Less than a minute”, and the other did not answer. Of the 4 SFAs that do currently purchase locally produced food, one answered 5 minutes, 2 answered 15 minutes, and one answered 30 minutes.

Question: How difficult would you rate the questionnaire?

Responses: Five answered “Easy”, 2 answered “Not too difficult”, and one did not respond.

Question: Which questions were difficult to answer, and why?

Responses: Two directors answered this question:

“How much spent on local - would have to dig out invoices to be accurate, and contact DOD to see if we could get a report from them. Did an estimate - but it did prompt me to set up a tracking system for this school year.”

“Some of the safety questions if you were not versed on the areas.”

Question: Do you think Food Service Directors will be less likely to respond if the questionnaire responses are not confidential?

Responses: Four respondents answered yes, 2 answered no. One “yes” responder added: “If you are not sure what is acceptable you may be wary. For example someone told me that someone told them that they couldn't feed kids food out of the hoop house.”

Question: Did you have to ask anyone else in order to answer the questions? If so, whom (give

title/position)?

Responses: Five respondents answered no, one answered: “To be exact on the cost, I would have had to involve my accounts person as well as outside vendors for local.”

Question: Did we not ask anything you think we should have? If so, what?

Responses: Three respondents offered suggestions:

• “I think working the procurement system is a big issue, and you might find out if that is a barrier.”

• “Maybe ask if there we [need] mentoring programs to help set up programs. I get a lot of people that ask to come and see our program because they are not sure how to proceed. I would like to see different regions work together maybe "adopt" a school miles away sharing what they get.”

• “type of procurement rules followed - local, state or federal”

Question: Any other comments?

Responses: Two respondents offered comments, reflecting opposing opinions about local food:

• “At my programs we have two missions one is to educate students on how to make healthy choices in food for a healthier life (exposing them to fruits and veggies of all shapes and colors). The other is to educate them on the economics of supporting [their] local farmers and area business. Thank you for the chance to share.”

• “[H]ope that the whole issue of "local" goes away... suppliers, growers need to be aggressive and active, just like the big boys to get noticed and more importantly, provide seamless services, quality products. Selection of products and services in this competitive business cannot use LOCAL unless there is meaningful value added. We chose our Local suppliers first and foremost because they met our immediate needs in service and quality. The LOCAL part is ultimately incidental.”

Conclusions

The major insights we took from cognitive testing were that:

• The plan to disseminate the invitation to participate in the survey through the FNS PartnerWeb email system is likely to work smoothly, as State Child Nutrition Directors already use this system for transmittal of information from FNS.

• The invitation should be worded carefully to address concerns about confidentiality.

• For respondents that have a Farm to School program, the questionnaire is likely to take 15 to 30 minutes.

• Questions about value of food purchases should be designed to request an estimate rather than exact figures.

• The questionnaire should make clear what is meant by “local” food, since many districts purchase some items locally because those are the products that meet their specifications, especially in the case of milk.

• Some districts, like the respondent from Montana, may have technical difficulties using the on-line questionnaire, and the data collection plan must include a method of allowing respondents to complete the questionnaire by an alternative method, such as fax.

4