Appendix G. Pretest

APPENDIX G

PRETEST

OMB Control Number: 0584-XXXX
Expiration Date: XX/XX/20XX

PRETEST DEBRIEF PROTOCOL

RESPONDENT NAME:
RESPONDENT TITLE:
SFA NAME:
STATE:

Thank you for taking the time to speak with us today for the pre-test of the SFA Survey of Supply Chain Disruption. Do you have a copy of the survey in front of you? I also have a copy of your survey responses too, for reference.

If no: It will help our discussion if you have it in front of you. I can give you a minute to get it. (I can also email you a new copy of the survey now if necessary.)

This call should take no more than 30 minutes. Our goal today is to get your feedback on the survey, particularly if anything was confusing or difficult to answer, and your suggestions for revisions. We will first talk about your reactions to the survey overall, and then we will ask about specific survey questions.

Do you have any questions for us before we begin?

1. About how long did it take you to answer the entire survey, including time spent gathering information?

   1. Did any questions take particularly long to complete?

2. Now I want to know what you thought of the questions overall.

   1. What did you think of the organization and flow of questions?

   2. If any questions left blank: I see you left questions [numbers] blank. Can you tell me why?

   3. Did the survey use any terms you recommend changing or that you think we should define?

      

      The Food and Nutrition Service (FNS) is collecting this information to better understand the impacts of supply chain disruption on school districts nationwide. This is a mandatory collection. FNS will use the information to develop tailored resources, tools and flexibilities to support school districts in serving students healthy and nutritious meals. This collection does not request personally identifiable information (PII) under the Privacy Act of 1974. According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-XXXX. The time required to complete this information collection is estimated to average 50 minutes (0.8333 hours) per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 1320 Braddock Place, 5th Floor, Alexandria, VA 22314, ATTN: PRA (0584-0607). Do not return the completed form to this address.

      
      
      

   4. Is there any information or instruction you think we should add to the introduction to help SFAs understand the purpose of the survey or how to complete it?

1. Looking at Question 1, did you have any trouble understanding what to enter for SFA ID? Are there alternate terms you would suggest we use?

2. [IF SELECTED NSLP/SBP + SSO, OR SFSP] On Question 2, why did you select [NSLP/SBP IN ADDITION TO THE SSO or SFSP]? If appropriate: Have you operated SFSP since the regular school year began?

3. Did you understand the differences between Questions 3 and 4? Probe: What types of items were you thinking of when you answered [Y/N] to Q3?

4. Looking at Question 5: How often is your SFA experiencing challenges procuring or receiving the follow items, did the available response options make sense to you (never – always)?

   1. If no: What options would you recommend?

   2. Are the challenges you are experiencing more or less of a problem now than they were at the beginning of the year or are they about the same?

   3. Did the item categories and examples provided make sense or were any of them confusing?

5. For Question 6, did the available item categories, groupings and examples make sense to you?

   1. If no: What changes would you recommend?

6. Did you have any trouble answering Question 7 with the information you have available?

   1. Were there any response options you did not understand or that you think we should add or remove? Probe if respondent added an option under “Other”.

7. What information did you use to answer Question 8? Did you have any trouble answering given the available response options?

   1. If yes: What response options would you suggest?

8. Did you have any trouble answering Questions 9 and 10 on USDA Foods?

   1. Are there any specific types of issues you would suggest we add to Question 10?

9. Moving on to Question 11, we asked about impacts of the supply chain-related challenges on your SFA. Did the impacts listed adequately capture the effects this situation is having on your SFA or are there other impacts we should include? Probe if respondent added items under “Other”.

   1. Were there any impacts listed that seemed odd or did not make sense or were difficult to distinguish (interpret)?

   2. Are the challenges your SFA is experiencing impacting all school types in your SFA similarly, or are the challenges affecting elementary, middle or high schools differently? Is this information that you think would vary by SFAs?

10. We have similar questions about Question 12: What methods has your SFA used to address these challenges: Did the methods listed adequately capture the methods your SFA has used to address supply chain challenges? Are there any particular methods you think we should add to the list? Probe if respondent added items under “Other”.

    1. Were there any impacts listed that seemed odd or did not make sense or were difficult to distinguish (interpret)?

11. If respondent answered Question 13: Did you have any trouble answering the question about whether products you are substituting were of higher, lower or similar quality?

    1. What information did you consider in making your response?

    2. Would this question be easier for you to answer if we asked it in terms of whether the substituted items are more, less or similarly preferred by students?

12. Let’s talk about the school food service account questions: Questions 14 through 17.

    1. How easy or challenging was it to complete these questions?

    2. How confident are you in your answers?

    3. Was the example calculation provided in Question 17 helpful or confusing? Were you able to provide a reasonable estimate with the information you have available at this time?

       1. If no: Do you have suggestions for how we could reword this to make it easier for you to give us an estimate of the size of the deficit you are currently experiencing?

13. If respondent reported a deficit for October 2021 in Question 16: Did the factors listed in Question 18 adequately capture the factors you think are contributing to the financial deficit your SFA is experiencing?

    1. Are there any factors you think we should add, remove, reword or define?

14. Any suggestions for improvement on Questions 19 or 20?

15. One final question: We need to report whose feedback we solicited when we submit this survey to OMB for approval to distribute to all SFAs. Is it okay if we include your name, title and contact information in our list of pretest participants when we submit that document?

Those are all the questions I had for you. Is there anything else you’d like to add?

Thanks again for your feedback. It will be very helpful as we make our revisions.

PRETESTED SFA SURVEY OF SUPPLY CHAIN DISRUPTION

INTRODUCTION

Throughout the pandemic, school nutrition professionals have met extraordinary challenges to ensure every child can get the food they need to learn, grow and thrive. But circumstances in local communities remain unpredictable, and supply chains for food and labor have been stressed and at times disrupted. This survey will assist USDA’s Food and Nutrition Service (FNS) and its partners to enhance the toolbox for school nutrition professionals working hard to make sure students have reliable access to healthy meals. Insight gained through survey responses will build on the insight that USDA needs to support school districts during this difficult time with tailored resources, tools and flexibility needed to serve students healthy and nutritious meals.

1. Please enter your SFA Name and ID:

   1. SFA Name: _________

   2. SFA ID: ______

2. Which Child Nutrition Programs has your SFA operated this school year? Select all that apply

   1. National School Lunch Program (NSLP)

   2. School Breakfast Program (SBP)

   3. NSLP Seamless Summer Option (SSO)

   4. NSLP Afterschool Snack Service

   5. Child and Adult Care Food Program (CACFP) At-Risk Afterschool Meals

   6. Fresh Fruit and Vegetable Program (FFVP)

   7. Special Milk Program (SMP)

   8. Summer Food Service Program (SFSP)

3. Is your SFA experiencing challenges procuring or receiving meal service (non-food) supplies or equipment for the school meals programs?

   1. Yes

   2. No

4. Is your SFA experiencing challenges with procuring or receiving the types or quantities of foods or beverages that were planned to be on the school meals program menus?

   1. No challenges procuring or receiving foods or beverages planned

   2. Yes, challenges procuring or receiving the types of foods or beverages planned

   3. Yes, challenges procuring or receiving the quantities of foods or beverages planned

   4. Yes, challenges procuring or receiving both the types AND quantities of foods or beverages planned

5. How often is your SFA experiencing challenges procuring or receiving the following items? [ASK IF Q3 OR Q4 = YES; matrix format with responses: Never, Rarely, Sometimes, Most of the Time, All of the time]

   1. Fruits

   2. Vegetables

   3. Vegetable subgroups

   4. Grains

   5. Whole grain-rich items

   6. Milk

   7. Meats/Meat alternates

   8. Low sodium foods

   9. Meal service supplies (e.g., food trays, disposable utensils, or packaging)

   10. Equipment (e.g., for meal preparation or storage)

6. Please indicate the top 3 food categories in which your SFA is experiencing challenges procuring or receiving menu items: [ASK IF Q4 = YES; allow up to 3 selections]

   1. Canned/frozen vegetables and beans

   2. Fresh vegetables

   3. Canned/frozen fruits

   4. Fresh fruits

   5. Bakery items (e.g. bread, rolls, muffins)

   6. Staple grain items (e.g. rice, pasta, flour, cereal)

   7. Fluid milk

   8. Other dairy items (e.g. cheese, yogurt)

   9. Chicken products

   10. Beef products

   11. Turkey products

   12. Pork products

   13. Fish/seafood products

   14. Egg products

   15. Mixed dishes (e.g. sandwiches, pizza, burritos)

7. To the best of your knowledge, are the challenges your SFA is experiencing with procuring or receiving foods or supplies for the school meals programs due to any of the following factors? [ASK IF Q3 OR Q4=YES; matrix format with responses: Yes, No, Don’t Know for each row below]

   1. Shipment delays or cancellations

   2. Orders arriving with missing items, reduced quantities, or product substitutions

   3. Limited product availability

   4. Increased cost of food items or supplies

   5. Labor shortages in the supply chain (e.g., warehouses, distributors, drivers)

   6. Contract cancelled by vendor

   7. Limited or restricted food delivery days or times

   8. Vendor requiring pick-up at centralized locations

   9. Limited or no vendors available

   10. Vendor surcharges or increased purchasing minimums

   11. Vendors giving preference to restaurants over schools

   12. Other (please specify)

8. How long do you expect the challenges listed below to continue? [DISPLAY ALL SELECTED IN Q7; matrix format]

   1. Less than one month

   2. 1-3 months

   3. 4-6 months

   4. More than 6 months

   5. Don’t Know

9. Is your SFA experiencing any challenges obtaining products through USDA Foods?

   1. Yes

   2. No

10. Please specify the types of issues you are experiencing obtaining products through USDA Foods: [ASK IF Q9 = YES; matrix format with responses: Yes, No, Don’t Know for each row below]

    1. USDA Foods direct delivery (brown box) product cancellations

    2. USDA Foods direct delivery (brown box) delivery delays

    3. USDA Foods diverted to processors product availability issues

    4. USDA Foods diverted to processors delivery delays or issues

    5. USDA DoD Fresh product availability issues

    6. USDA DoD Fresh delivery delays or issues

11. What impact(s) are the supply chain-related challenges your SFA is experiencing having on school meal operations in your SFA? Select all that apply

    1. Reduced program participation

    2. Increased program costs

    3. Inability to offer enough reimbursable meals to participating children

    4. Inability to meet meal modification requirements for children with food and nutrition related disabilities

    5. Inability to meet meal pattern requirements

    6. Inability to maintain compliance with regular procurement requirements

    7. Inability to offer afterschool snacks/suppers

    8. Other (please specify)

12. What methods has your SFA used to address these challenges? Select all that apply [DISPLAY ON SAME PAGE AS Q11]

    1. Increased use of local vendors

    2. Increased use of USDA Foods direct delivery (brown box)

    3. Increased use of USDA DoD Fresh

    4. Leverage state contracts

    5. Conduct emergency procurements

    6. Increased use of micro-purchases

    7. Use meal pattern waivers (as appropriate)

    8. Use waiver to offer the Seamless Summer Option (SSO) at higher reimbursement rates

    9. Partner with restaurants, catering companies or community partners (such as universities and hospitals)

    10. Limit or repeat weekly menu offerings

    11. Substitute products

    12. Increased bulk purchasing

    13. Other (please specify)

    14. No methods used

13. In your opinion, are the products you are substituting to address supply chain disruptions of lower, similar, or higher quality than the items you had planned to use, on average? [ASK IF Q12 = K]

    1. Lower quality

    2. Similar quality

    3. Higher quality

    4. Don’t know

14. In a typical year prior to the pandemic (e.g., SY 2018-19 or before), what was the status of your school food service account balance?

    1. Operated at a surplus (i.e., revenues exceeded costs)

    2. Broke even (i.e., revenues are about equal to costs)

    3. Operated at a deficit (i.e., costs exceeded revenues)

15. At the end of SY 2020-21, what was the status of your school food service account balance?

    1. Operated at a surplus (i.e., revenues exceeded costs)

    2. Broke even (i.e., revenues are about equal to costs)

    3. Operated at a deficit (i.e., costs exceeded revenues)

16. As of October 2021, what was the status of your school food service account balance? If unsure, please use your best estimate.

    1. Operating at a surplus (i.e, revenues exceed costs)

    2. Breaking even (i.e., revenues are about equal to costs)

    3. Operating at a deficit (i.e., costs exceed revenues)

17. As of October 2021, approximately what percent of your SFA’s total costs were covered by your total revenues? [ASK IF 16=C]

Example Calculation:

SFA Costs = $100,000

SFA Revenues = $85,000

Percent of costs covered by revenues = $85,000 / $100,000 = 0.85 or 85%

Enter estimated value: ________%

18. Which of the following factors are driving this deficit within your SFA? [ASK IF 16=C] Select all that apply

    1. Food costs

    2. School nutrition service labor costs

    3. Supply costs

    4. Equipment costs

    5. Storage costs

    6. Transportation costs

    7. Indirect costs (e.g., fringe benefits, accounting, payroll, purchasing, facilities managements and utilities)

    8. Increased Program participation

    9. Decreased Program participation

    10. Other, please specify

    11. Don’t know

19. Is your SFA, or are schools within your SFA, experiencing staffing shortages in the following job categories: [matrix: Y/N for each row]

    1. School nutrition managers

    2. Administrative or clerical staff

    3. Maintenance staff

    4. Warehouse staff

    5. Drivers

    6. Nutritionists

    7. Cooks/Food prep

    8. Cashiers

    9. Monitors

    10. Other (please specify)

20. What else would you like FNS to know about the challenges your SFA is experiencing providing meals to students this school year?