SUPPORTING STATEMENT - PART B for
OMB Control Number 0584-[NEW]:
Summer Food Service Program Integrity Study
Chanchalat Chanhatasilpa
Office of Policy Support
Food and Nutrition Service
US Department of Agriculture
3101 Park Center Drive
Alexandria, VA 22302
Phone: 703-305-2115
E-mail: chanchalat.chanhatasilpa@fns.usda.gov
Table of Contents
B1. respondent universe and sampling methods. 4
B2. data collection procedures 8
B3. methods to maximize response rates and deal with non-response. 13
B4. tests of procedures or methods to be undertaken 14
B5. individuals consulted on the study and individuals collecting and/or analyzing data 16
Tables
Table B2-1. TARGET NUMBER OF STATES PER SELECTION CRITERION 9
Table B2-2. TARGET NUMBER OF SPONSORS PER SELECTION CRITERION 10
Table B5-1. INDIVIDUALS CONSULTED AND INDIVIDUALS INVOLVED IN DATA COLLECTION AND/OR ANALYSIS 16
Appendices
Appendix A: Study Materials
A-1. The Improper Payments Elimination and Recovery Act of 2010
A-2. The Improper Payments Elimination and Recovery Improvement Act of 2012
A-3. Richard B. Russell National School Lunch Act (Section 13(a))
A-4. Richard B. Russell National School Lunch Act (Section 28(c))
A-5. IRB Approval Letter
A-6. FNS Controls Over the SFSP: Audit Report
A-7. Statement of Confidentiality and Nondisclosure
A-8. Burden Table
Appendix B: Recruitment Materials
B-1. State and Sponsor Instrument Pre-Test Request
B-2. Site Supervisor Instrument Pre-Test Request
B-3. Study Notification E-Letter from FNS to State CN Directors
B-4. FAQ for States and SFSP Sponsors
B-5. E-Letter to State CN Directors with Link to Web Survey
B-6. E-Letter Survey Reminder
B-7. Phone Script for Nonrespondent State Directors
B-8. E-Letter Thank You for Survey Completion
B-9. Survey Thank You E-Letter with Request to Schedule Phone Interview
B-10. State Interview Thank You E-Letter with Request to Contact Selected Sponsors
B-11. Study Notification E-Letter to Sponsors from State CN Agency
B-12. E-Letter to Sponsors with Request to Schedule Phone Interview
B-13. E-Letter Reminder to Schedule Sponsor Interview
B-14. Sponsor Interview Thank You E-Letter and Request to Contact Selected SFSP Sites
B-15. Study Notification E-Letter to Selected SFSP Sites from SFSP Sponsor
B-16. FAQ for SFSP Sites
B-17. E-Letter to Site Supervisor to Schedule Phone Interview
B-18. E-Letter Reminder to Schedule Site Interview
B-19. Site Supervisor Interview Thank You E-Letter
Appendix C: Data Collection Instruments
C-1. Pre-test Interview Protocol
C-2. State Director Web Survey
C-3. State Director Interview Guide
C-4. Sponsor Interview Guide
C-5. Site Supervisor Interview Guide
C-6. State Director Web Survey Screen Shots
C-7. SFSP Integrity Pre-test Memo
Appendix D: Public Comments Received
D-1. Public Comment from Jamie Doxey
D-2. Public Comment from the School Nutrition Association
D-3. Public Comment from Academy of Nutrition and Dietetics
D-4. Public Comment from the Food Research and Action Center
Appendix E: Response to Public Comments
E-1. FNS Response to Jamie Doxey
E-2. FNS Response to School Nutrition Association
E-3. FNS Response to Academy of Nutrition and Dietetics
E-4. FNS Response to the Food Research and Action Center
E-5. NASS Comments
Respondent Universe
The respondent universe for this study includes all 54 State-level agencies that administer the Summer Food Service Program (SFSP), and all SFSP sponsors and sites. The number of sponsors and sites that operate the SFSP varies each year; in 2017 there were 7,858 sponsors and 50,363 sites.1 All 54 State-level Child Nutrition (CN) Directors that administer the SFSP will be included in the study (including three U.S. territories and the District of Columbia) and asked to complete a web survey, and a sample of 18 States will be asked to complete a follow-up telephone interview. We are conducting a survey of all State-level Directors to ensure we have a complete picture of the strategies States use to administer and oversee the SFSP, which can vary based on a variety of factors that are not readily captured in a subsample (e.g., State size, staffing levels, State administrative structure2, SFSP performance relative to National School Lunch Program (NSLP) participation, number of sponsors and sites, etc.).
Sampling Methods and Response Rates
This study uses quantitative (i.e., closed-ended survey questions) and qualitative methodologies (i.e., open-ended survey questions and telephone interviews) to address the research questions. All 54 State Directors will be included in the sample to complete the State Director Web Survey (Appendix C-2), with none expected to refuse participation. Based on FNS experience with other studies, State CN Directors typically cooperate with such requests, especially when the request is made by FNS and with sufficient advance notice, as we are planning to do.
The expected response rate varies significantly by respondent type. We do not expect that any State will decline to participate in the survey, and the survey will be fielded for six weeks, offering maximum flexibility. However, scheduling challenges and competing priorities may require that up to two of the twenty States selected for follow-up telephone interviews decline to participate. The expected response rate for the interviews with SFSP sponsors and sites is low, based on our experience with recruitment for a similar study, the Summer Food Service Program Participant Characteristics and Meal Analysis of Quality (“SFSP PC MAQ”)(OMB control number 0584-0635, expiration 2/28/2021). The study team will oversample sponsors by a factor of four,3 and sites by a factor of five,4 to allow replacements. We also anticipate that the States may advise us not to consider certain sponsors, and that sponsors may advise us not to consider certain sites based on additional information they provide upon notification (e.g., the State agency is conducting a review of a sponsor during the timeframe for sponsor data collection).
State agency support of the study will help achieve the desired response rate among sponsors selected for interviews; similarly, sponsor support of the study will help to achieve the desired response rate among sites selected for interviews. We do not expect any State-level key staff to be nonrespondents once the State CN Director agrees to participate in the interview, because we anticipate the CN Director will only identify key staff to participate after taking into account staff availability and expertise. The same is true of the key staff at the selected sponsor organizations. Finally, the study team will contact respondents to schedule the telephone interviews well in advance, and maintain flexibility to accommodate the availability of respondents to help ensure their participation. Table B1-1 shows the response rates for each data collection activity in the study, and the overall response rate for the study. We believe the response rates listed in Table B1-1 are reasonable given that Section 28 of the Richard B. Russell National School Lunch Act (42 U.S.C. 1769i) requires organizations participating in the Child Nutrition Programs to cooperate with FNS studies and evaluations.
Respondent |
Universe |
Sample Size |
Respondents |
Response rate (%) |
State Director Survey (CN Directors) |
54 |
54 |
54 |
100 |
State Interviews (CN Directors) |
54 |
18 |
18 |
100 |
State Interviews (Key staff) |
108 |
36 |
36 |
100 |
Subtotal: State Respondents |
162 |
108 |
108 |
100 |
Sponsor Interviews (Directors) |
7,858 |
192 |
48 |
25a |
Sponsor Interviews (Key staff) |
7,858 |
48 |
48 |
100 |
Subtotal: Sponsor Respondents |
7,858 |
240 |
96 |
40 |
Site Interviews (Site Supervisors) |
50,3635 |
240 |
48 |
20b |
Overall response rate |
|
588 |
252 |
43 |
a Rows 23-26 and 42-45 in the burden table (Appendix A-8) show that we will contact 192 sponsors, but ultimately only 48 will participate in an interview.
b Rows 33-36 and 52-55 in the burden table (Appendix A-8) show that we will contact 240 sites, but ultimately only 48 will participate in an interview.
Research suggests that six interviews for each respondent type will yield the fundamental thematic elements for analysis, and a saturation point is attained at twelve interviews.6 A sample of 48 sponsors allows us to reach that saturation point of twelve interviews across all of the following overlapping sponsor selection criteria: sponsor type (i.e., school food authority (SFA), private nonprofit, government unit, camp), program size (i.e., small, medium, large), and experience with the SFSP (i.e., new to the program in 2019 vs. participated for two or more years). We expect that these interviews will allow us to conduct more targeted analyses and explore differences according to these selection criteria.
We will use purposive sampling to select a diverse group of 18 States for telephone interviews in order to provide the richest descriptive information on the strategies States use to administer and oversee the SFSP. Among other considerations, we will seek diversity based on the the following criteria:
Size of States’ SFSP;
Location and FNS region;
SFSP performance relative to NSLP participation; and
State administrative structure (i.e., where SFSP is administered relative to other CN programs).
We will obtain information for the State selection through the State Director Web Survey (Appendix C-2) and the latest annual report from the Food Research and Action Center (FRAC)7, which ranks State SFSP performance relative to the NSLP. We expect all States to participate in an interview, including both State CN Directors and State-level key staff.
When possible, the study team will use the lists of 2018 SFSP sponsors obtained from SFSP PC MAQ to select and recruit sponsors. When sponsor lists for particular States in the sample are not available, the study team will make a targeted request to the State agencies to provide information for a more limited number of sponsors representing the various types. The study team will oversample to allow replacements. As noted earlier, we expect to recruit up to 192 sponsors (96 public and 96 private) across the 18 States that participate in interviews, and we expect that 48 will participate in the in-depth interviews (sponsor participants include both sponsor directors and key staff). The key variable driving the sample size is sponsor type; we will enroll 12 sponsors of each type (i.e., SFAs, private nonprofit organizations, governmental units, and camps). Achieving 12 of each sponsor type will allow us to conduct more meaningful analyses. It is expected that the different types of sponsors may demonstrate different processes due to the nature of the organizations. We will strive to enroll two to three sponsors per State, but this may vary to allow us to achieve diversity by sponsor type and other key criteria. At a minimum, we will enroll one sponsor within each State interviewed. We will also work to identify and enroll sponsors of varying levels of performance, program size, types of SFSP sites, and sponsors’ years of experience with the SFSP.
To facilitate site selection, we will request from the 18 States participating in interviews a list of all of their sites and the sponsor organization with which each site is affiliated. Then, during the sponsor interviews, we will confirm the list of sites that we received from the State to ensure that it is a complete and accurate accounting of the sites that the sponsor oversees. Once we have a complete list of each sponsor’s sites, we will select our purposive sample of sites and enlist the sponsors to assist with recruiting the selected sites by emailing sites the Study Notification E-Letter to Selected SFSP Sites from SFSP Sponsor (Appendix B-15) along with the FAQ for SFSP Sites (Appendix B-16). We expect to recruit up to 240 sites, of which 48 are expected to participate in the in-depth interviews (participants include site supervisors). We will enroll one SFSP site for each of the 48 sponsors interviewed in order to obtain an on-the-ground perspective of how the SFSP is administered and overseen. To ensure diversity in perspective, the sites will be selected to represent different site types (i.e., open, closed enrolled, camp, etc.) and sizes.
Statistical methodology for stratification and sample selection
Estimation procedure
Degree of accuracy needed for the purpose described in the justification
Unusual problems requiring specialized sampling procedures; and
Any use of periodic (less frequent than annual) data collection cycles to reduce burden
As discussed in Section B.1, we will collect quantitative descriptive data and qualitative information from all State agencies through the State Director Web Survey (Appendix C-2). We will collect qualitative information from a purposive sample of States, sponsors, and sites through telephone interviews.
For these data collection activities, no statistical methodology, estimation procedure, or calculations of degrees of accuracy are needed because we are collecting data from all State Directors and only a purposive sample of States, sponsors, and sites. The data collected via telephone interviews will provide rich information about SFSP administration and oversight, but will not be nationally representative. We will use the following criteria to select the 18 States (see table B2-1) and 48 sponsors (see table B2-2):
Table B2-1. Target Number of States per Selection Criterion
State selection domain |
Target number of States |
Program Size |
|
Small |
6 |
Medium |
6 |
Large |
6 |
State administrative structure |
|
SFSP in same agency as the School Meal Programs |
12 |
SFSP in separate agency as the School Meal Programs |
6 |
Program performance relative to NSLP participation8 |
|
Lowest third |
6 |
Middle third |
6 |
Highest third |
6 |
Table B2-2. Target Number of Sponsors per Selection Criterion
Sponsor selection domain |
Target number of sponsors |
Sponsor type |
|
SFA |
12 |
Private Nonprofit Organization |
12 |
Governmental Unit |
12 |
Camp |
12 |
Program size |
|
Small |
16 |
Medium |
16 |
Large |
16 |
Types of sites |
|
Open/Restricted open |
14 |
Closed enrolled |
14 |
Camp |
14 |
Migrant* |
6* |
Experience with SFSP |
|
New to program in 2018 |
12 |
Participated for 2+ years |
36 |
* We will include migrant sites to the extent State agencies are able to identify these sites.
Finally, to ensure diversity in perspective from site respondents, the study team will select a purposive sample of SFSP sites for interviews from within the 18 States selected for interviews. The sites will be selected to ensure diversity by size and site type (e.g., open, closed enrolled, etc.), and we will oversample to allow replacements.
We will use the following procedures to collect these data:
FNS will inform State CN Directors of the study purpose, activities and expected timeframes via the Study Notification E-Letter from FNS to State Child Nutrition Directors (Appendix B-3). The study contractor will follow that with an E-Letter to State CN Directors with Link to Web Survey (Appendix B-5), and ask them to complete the State Director Web Survey (Appendix C-2). Both of those E-Letters will include the FAQ for States and SFSP Sponsors (Appendix B-4) as an attachment. We will send a weekly E-Letter Survey Reminder (Appendix B-6) and follow up via telephone using the Phone Script for Nonrespondent State Directors (Appendix B-7) to obtain all responses within the six week timeframe for survey completion. Six weeks is expected to be sufficient to accommodate State CN Director schedules. The time required to complete the survey will not exceed 20 minutes.
Following completion of the State Director Web Survey, the State Directors will receive one of two thank you emails: 1) E-Letter Thank You for Survey Completion (Appendix B-8) or 2) E-Letter Survey Thank You with Request to Schedule Phone Interview (Appendix B-9). The former will be sent to all State CN Directors not selected for a follow-up telephone interview, and will be the final communication they receive from the study. The latter will go to those State CN Directors selected for follow-up, in-depth telephone interviews. The telephone interviews will not exceed 90 minutes, and will be audio-recorded and led by a trained researcher using the State Director Interview Guide (Appendix C-3).
Following the completion of each State interview, we will notify the State CN Directors of the selected SFSP sponsors in the State Interview Thank You E-Letter with Request to Contact Selected Sponsors (Appendix B-10), and ask them to encourage participation by emailing the sponsors the Study Notification E-Letter to Sponsors from State CN Agency (Appendix B-11) in addition to the FAQ for States and SFSP Sponsors (Appendix B-4).
After the selected sponsor directors have been notified, we will send the E-Letter to Sponsors with Request to Schedule Phone Interview (Appendix B-12), with the FAQ for States and SFSP Sponsors (Appendix B-4) attached. This correspondence will also indicate that the directors may include key staff with appropriate expertise to join the interview. One week later, we will send to unresponsive sponsors the E-Letter Reminder to Schedule Sponsor Interview (Appendix B-13). The telephone interviews will not exceed 60 minutes, and will be audio-recorded and led by a trained researcher using the Sponsor Interview Guide (Appendix C-4). Following the completion of each sponsor interview, we will send a Sponsor Interview Thank You E-Letter and Request to Contact Selected SFSP Sites (Appendix B-14) to encourage the sites’ participation in interviews. That E-Letter will include as attachments the Study Notification E-Letter to Selected SFSP Sites from SFSP Sponsor (Appendix B-15) in addition to the FAQ for SFSP Sites (Appendix B-16).
Selected site supervisors will then receive the Study Notification E-Letter to Selected SFSP Sites from SFSP Sponsor (Appendix B-15) in addition to the FAQ for SFSP Sites (Appendix B-16) from their sponsor. We will then send site supervisors the E-Letter to Site Supervisor to Schedule Phone Interview (Appendix B-17), and attach the FAQ for SFSP Sites (Appendix B-16) to request their availability for a telephone interview. Site supervisors who are unresponsive will receive an E-Letter Reminder to Schedule Site Interview (Appendix B-18). The telephone interviews will not exceed 30 minutes, and will be audio-recorded and led by a trained researcher using the Site Supervisor Interview Guide (Appendix C-4). Following the completion of each site interview, we will send a Site Supervisor Interview Thank You E-Letter (Appendix B-19).
Unusual problems requiring specialized sampling procedures
There are no unusual problems requiring specialized sampling procedures.
Any use of periodic (less frequent than annual) data collection cycles to reduce burden.
We are only conducting each data collection activity once for the SFSP sponsors and sites. For those States selected for telephone interviews in addition to the online survey, we are minimizing confusion and spreading out the burden by handling each data collection activity at a discrete time.
We do not expect any issues of non-response with the State CN Directors. However, we will seek to minimize burden by providing six weeks to complete the online survey, which will alleviate any workload or scheduling conflicts. State CN Directors will also be able to delegate completion of portions of the survey to key staff within the agency, as needed.
As noted earlier, we expect a relatively high rate of non-response from among SFSP sponsors and sites, given our experiences on SFSP PC MAQ. To maximize response rates among SFSP sponsors, we will enlist the support of the State CN Directors to encourage their participation prior to recruitment attempts by the study team by sending a Study Notification E-Letter to Sponsors from State CN Agency (Appendix B-11) in addition to the FAQ for States and SFSP Sponsors (Appendix B-4). We expect to replace some of the sponsor directors initially selected to accommodate scheduling conflicts, non-response, and other issues which may arise. This is reflected in the initial sampling universe of 192 sponsor directors, of which 48 are expected to participate in interviews. Furthermore, we will maximize response rates by sending an E-Letter Reminder to Schedule Sponsor Interview (Appendix B-13), and by offering a flexible two-month window to conduct the interview.
Similar to sponsor recruitment, we will attempt to maximize response rates among sites by first enlisting the support of the sponsor directors. We will ask the sponsor directors to send the Study Notification E-Letter to Selected SFSP Sites from SFSP Sponsor (Appendix B-15) and the FAQ for SFSP Sites (Appendix B-16) to encourage their sites to participate in the telephone interviews. We expect to replace some of the site supervisors initially selected to accommodate scheduling conflicts, non-response, and other issues which may arise. This is reflected in the initial sampling universe of 240 site supervisors, of which 48 are expected to participate in interviews. We also oversample because most site supervisors are only available during the summer months, but that is also the busiest time to contact them. To accommodate sites’ schedules and maximize response rates, we will send an E-Letter Reminder to Schedule Site Interview (Appendix B-18), and offer a flexible two-month window to conduct the interview.
The amount of data collected is adequate for the study purpose. The study results will ultimately inform FNS resources, training, and technical assistance pertaining to the SFSP. Research tells us that major themes emerge with six interviews, and total saturation is attained at 12 interviews.9 The study team used that information to determine the number of State, sponsor, and site respondents needed, and we feel that FNS will have sufficient information from each type of respondent in order to answer the research questions and refine their SFSP resources and trainings.
We conducted pre-test interviews of the State Director Web Survey (Appendix C-2), the State Director Interview Guide (Appendix C-3), the Sponsor Interview Guide (Appendix C-4), and the Site Supervisor Interview Guide (Appendix C-5). The State Director Web Survey and State Director Interview Guide were pretested with staff from two FNS Regional Offices who had valuable insight into the workings of all States in their region and could provide a broad perspective on the instruments. We tested these instruments with FNS regional office staff because the survey is a census, and we did not want to test the instrument with State CN Directors who would later be asked to complete it. We also tested the State Director Web Survey and State Director Interview Guide with one State-level key staff from Kansas, who helps to oversee the SFSP but is not the State CN Director. The Sponsor Interview Guide was tested with three sponsors. The sponsors were selected to achieve diversity in geographic location and sponsor type (i.e., SFA, governmental unit, private nonprofit organization). Finally, the Site Supervisor Interview Guide was tested with three sites. We believe that three tests of each data collection instrument was sufficient, because respondents had similar comments on each set of instruments and additional testing would likely not have yielded new insights.
The State and Sponsor Instrument Pre-Test Request was sent to the local government State Key staff and the SFSP Sonsor Directors, as well as to the SFSP Sponsor Directors, to request their participation in the pre-test (Appendix B-1), while the Site Supervisor Instrument Pre-Test Request was sent to the local government and business SFSP Site Directors to request their participation (Appendix B-2). Everyone participating in the pre-test received the Pre-Test Interview Protocol (Appendix C-1).
All instruments were tested to ensure that the respondents interpreted the questions as intended and could easily respond. They were also tested to ensure that the interview guides were easy for the interviewer to administer, and to verify the burden estimates. In all cases, trained interviewers reviewed the instruments with the respondents, observed and documented any issues that arose for either respondents or interviewers, and discussed any points of difficulty with respondents.
Following the pretests, the interviewer and an analyst reviewed their notes from each interview and produced a list of themes and patterns within the interview data. In particular, staff focused on problems and issues with the instruments, including areas where the respondents demonstrated confusion, hesitation, uncertainty, and/or discomfort. Staff discussed the results of the analysis to validate the findings and confirm recommendations. Themes and patterns were organized, evaluated, synthesized, and summarized into report form. Pretest results are summarized in the SFSP Integrity Pre-test Memo (Appendix C-7). Findings and recommendations from the pretests were used to refine the survey and interview guides. Specifically, we simplified the wording of the Site Supervisor Interview Guide and removed all program acronyms, added response options to the State Director Web Survey, and re-ordered questions to improve flow across all instruments, among other changes. Cognitive testing verified the initial burden estimates for the State Director Web Survey, the Sponsor Interview Guide, and the Site Supervisor Interview Guide. Cognitive testing revealed that more time would be needed to administer the State Director Interview Guide; we increased the time burden from 60 minutes to 90 minutes.
The proposed study plan, data collection instruments, and recruitment materials were developed and reviewed extensively by FNS and the study team. Westat and its subcontractor, Insight Policy Research, will participate in data collection and analysis as well as development of reports. In addition, the four members of the study’s Peer Advisory Panel reviewed and commented on the draft study plan, and will be asked to review future study deliverables as well.
Table B5-1. Individuals Consulted and Individuals Involved in Data Collection and/or Analysis
Name |
Affiliation |
Title |
Contact information |
Melissa Rothstein |
Westat |
Senior Study Director |
301-315-5975 |
Laurie May |
Westat |
Vice President |
301-517-8068 |
Lindsay Giesen |
Westat |
Policy Researcher |
240-453-5693 |
Tracy Vericker |
Westat |
Associate Director |
301-251-4242 |
Allison Roeser |
Westat |
Study Director |
301-279-4537 |
Kevin Baier |
Westat |
Statistician |
301-279-4593 |
Claire Wilson |
Insight Policy Research |
Study Director |
703-504-9484 |
Carole Trippe |
Insight Policy Research |
Director |
703-504-9498 |
Chanchalat Chanhatasilpa |
USDA/FNS |
FNS Contracting Officer’s Representative |
703-305-2115 |
Dr. Edwin L. Anderson |
NASS |
Mathematical Statistician |
202-690-0270 edwin.anderson@usda.gov |
Kelly Chanay* |
Kansas Department of Education |
Assistant Director |
785-296-2276 |
Chris Greenwood* |
City of Rockford, IL |
Community Health and Prevention Coordinator |
779-348-7569 |
Keven Vicknair* |
Equal Heart-Texas |
Executive Director |
469-526-3645 |
Daniel Miller* |
Boston University |
Associate Professor |
617-353-3752 |
* Member of the Peer Advisory Panel
1 Source: FNS-418 data for Summer 2017, as of March 22, 2018.
2 SFSP may be administered by the same agency that administers the National School Lunch Program (NSLP) or by a different agency.
3 To achieve our target 48 sponsors, we will recruit 192.
4 To achieve our target 48 sites, we will recruit 240.
5 FNS administrative data provides the total number of sponsors (7,858) and sites (50,363) by State in July 2017.
6 Guest, G., Bunce, A. and Johnson, L. (2006) How Many Interviews Are Enough? An Experiment with Data Saturation and Variability. Field Methods, 18, 59-82.
8 We will examine the Food Research and Action Center’s (FRAC) annual rankings of State-level SFSP performance relative to NSLP participation. The 2018 report can be found here: http://frac.org/wp-content/uploads/2018-summer-nutrition-report.pdf
9 http://fmx.sagepub.com/content/18/1/59
Page
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
File Title | Microsoft Word - OCIO Guidelines for Preparing the SuptStmt |
Author | ckrutzler |
File Modified | 0000-00-00 |
File Created | 2021-01-13 |