Appendix Q. Mainland Study Sampling Plans

Q. MAINLAND STUDY SAMPLING PLANS

This page has been left blank for double-sided copying.

The sampling plan for the mainland study is designed to support national and subgroup-level estimates for SFAs, schools, students, and meals, using carefully designed stratification and probability-proportionate-to-size (PPS) sampling methods. Throughout our design plan, we have accounted for the effects of within-SFA school similarities and within-school student similarities and have balanced those effects with the costs of changing the number of SFAs and schools at each step. All of the results presented in this plan are based on implementing the proposed sample design on the sampling frame, selecting multiple simulated samples, and averaging the results to estimate the characteristics of the actual samples that would be selected. We used these results in combination with prior findings from SNMCS-I to prepare the precision estimates shown later. A summary of the sample design is shown in Figure Q.1.

Figure Q.1 Summary of the sample design for the mainland study

Notes: The overall sampling frame includes SFAs serving public schools in the contiguous United States and the District of Columbia. SFAs serving only institutional populations or operated by States or the Federal government are excluded.

Sample sizes reflect target completed sample sizes. The sizes of the samples initially selected will be expanded to allow for nonparticipation due to ineligibility or noncooperation, based on our experiences in SNMCS-I.

In assigning SFAs to Groups 1, 2, and 3, we plan first to identify the largest SFAs and to set these aside to participate in Group 2 and 3 data collections.

SFA = school food authority; SNM = school nutrition manager.

The sampling plan for the mainland study is designed so that the confidence half-width intervals will not exceed plus or minus 5 percentage points for national estimates and plus or minus 10 percentage points for subgroups making up 25 percent of the population of SFAs, schools, or students. However, as discussed in Section 7, in many cases such a level of precision will be obtained for many other subgroups as well. Our approach to sampling middle schools is an important example. Middle schools make up 17 percent of the schools in the United States. To meet the analytical objectives for estimates by school type (elementary, middle, and high schools), our design uses stratification to ensure that a sufficient sample of middle schools (and similarly, elementary and high schools) is obtained to support estimation at a precision level of +/−10 percentage points for each of these three subgroups.

Selecting the samples requires high quality sampling frames at each stage. To select the samples of SFAs, we will construct a frame that combines data from the most recently available FNS-742 Verification Collection Report (OMB Control Number 0584-0594, Expires 6/30/2019) list of SFAs, the U.S. Department of Education’s Common Core of Data (CCD) “Local Education Agency (School District) Universe Survey,” and a Census file (Small Area Income and Poverty Estimates Program) that contains school district–level estimates of school-age children in poverty.

In some cases, there may be multiple sampled SFAs that are managed by the same entity. This may lead to excessive burden for that entity and may limit the ability to disaggregate certain information on individual SFAs being managed by that entity. If such cases arise, we will first see if it is possible for the entity to provide information for all sampled SFAs under its control. If not, we would consider sampling one SFA from the multiple SFAs originally sampled to reduce respondent burden.

The frame for selecting schools within SFAs will be the CCD school-level file.¹ It contains enrollment figures, grades served, demographic information, and locating information. In some SFAs, the CCD may not be current, given recent school closures, mergers, or additions or may have inadequate information for constructing the school sampling frame. We will identify such changes during recruitment, and we will give any new schools identified after the CCD release a chance of selection into the sample to ensure complete coverage.

1. Sampling SFAs

After the sampling frame for SFAs has been prepared, we will select SFAs in four steps. First, we will identify the overall certainty SFA selections as discussed in Section 2, below.² Second, we will set aside any SFAs that serve only charter schools, so they can be assigned to Group 1. Third, we will stratify the overall frame of the remaining (noncertainty) SFAs and randomly assign using random selection methods to assign the remaining SFAs to the three sampling groups (or subframes). Fourth, we will then select the three samples of SFAs using a stratified random sampling approach with probability proportional to size.

2. Selecting the overall certainty SFA sample (largest SFAs)

In assigning SFAs to Groups 1, 2 and 3, we plan first to identify the largest SFAs in the United States and to set these aside to participate in the Group 2 and 3 data collections. Other SFAs may be selected with certainty (meaning they are included in the sample automatically rather than sampled at random) within each of the three groups. The total number of SFAs that will be selected with certainty will be determined after the sampling frame is constructed. The target completed sample sizes presented in the sections that follow include both the certainty and noncertainty selections.

3. Selecting the three groups of SFAs

Group 1 SFAs (n = 125). Overall, the Group 1 sample (with the frame containing one-third of the non-charter SFAs—except the largest—and all charter SFAs) is designed to add observations to the SFA Director Survey so that SFA characteristics across all three groups can be measured precisely, overall and across designated subgroups. Because there are disproportionally few SFAs with many schools (serving 20 or more) and/or students (serving 12,000 or more) and a relatively large number of charter SFAs (which are only included in Group 1), we could interview too few large SFAs or too many charter SFAs if we used simple random sampling. We also want to target, to a small degree, SFAs with students in poverty and to allocate the sample so that analyses can examine single-district SFAs versus supervisory unions of districts.

To address these concerns, we will use stratified sampling, coupled with a PPS selection process. The method used in SNMCS-I for selecting SFAs in Group 1 was not explicitly stratified; the stratification used in SNMCS-II will enhance the ability to examine SFA-level results at a more granular level without reducing our ability to use the combined SNMCS-I and -II data to analyze trends.

We will create 10 strata for Group 1 SFA selection based on all combinations of (1) three ranges in the number of schools and (2) three ranges in the number of students associated with the SFA, along with charter status at the 10th stratum.³ We will allocate the sample across these strata in a semi-proportional manner, relative to the number of SFAs in each stratum, so that in the smaller strata a higher sampling rate is applied to ensure enough SFAs are selected to meet subgroup-level analyses on the two dimensions. Within each stratum, we will sample SFAs using PPS methods so that SFAs with higher rates of students approved for free or reduced-price meals and supervisory union SFAs have a slightly higher chance of selection, to create a larger sample of these SFA types for analyses.⁴

As in SNMCS-I, we will use implicit stratification within each stratum to ensure the sample is balanced by FNS region and urbanicity. We will identify certainty selections first and remove these from the frame, adjust the sample allocation (these two steps are conducted in an interactive fashion so that all certainty cases are accounted for), and pull the final sample. Because Group 1 is designed to support SFA-level analysis, the measure of size (MOS) is designed around the SFA characteristics. These characteristics include the poverty status of students and whether the SFA is single- or multi-district. Because these characteristics do not have large ranges in values compared to the ranges in student enrollment, we expect to sample only a few, if any, certainty SFAs in this group.

Group 2 SFAs (n = 125). The Group 2 sample is designed to provide completed SFA Director Surveys and a sample of schools and students to participate in the SNM Survey, Principal Survey, Menu Survey, and other school-level data collection activities, as well as student- and parent-level data collection. As for Group 1, we will stratify SFAs by 9 strata (the 10th stratum used in Group 1 for charter SFAs is not needed for Group 2) based on the ranges in (1) the number of schools and (2) the number of students associated with the SFA, using an equivalent allocation plan, and apply PPS sampling within strata. Because Group 2 is also used to produce school-, principal-, and student/parent–level estimates, we plan to adjust the MOS values for PPS selection using an MOS based on the square root of the number of schools in the SFA, in conjunction with higher MOS values for SFAs in poverty areas or with single-district SFA/nonsupervisory union SFA status. Before selecting the SFA sample, we will identify and sample certainty SFAs. We will then explicitly stratify those not sampled with certainty into the nine non-charter strata for Group 2 and select a sufficient sample using PPS methods within each stratum to yield 125 SFAs.

Group 3 SFAs (n = 250). The Group 3 sample is designed to provide everything that the Group 2 sample provides other than student- and parent-level data, as well as data for estimation of meal costs, school food service revenues, and plate waste. Plate waste observations will be conducted in schools in a subset of Group 3 SFAs. The Group 3 sampling will follow the same design as Group 2. We will select a sufficient sample to yield 250 SFAs in Group 3.

Table Q.1 presents a summary of the SFA selection process. We provide estimates of the number of SFAs by the dimensions of stratum membership and the proposed sample allocation across the completed set of 500 SFAs in Groups 1 through 3 combined. Population and sample estimates are also presented for various subgroups.

Table Q.1 Target completed sample sizes for the mainland study: SFAs

Subgroups	SFAs
	Population		Sample
	Count	Percentage	Count	Percentage
Number of schoolsa
1–2	5,442	38	167	37
3–4	3,843	27	132	29
5 or more	5,219	36	156	34
Number of studentsa
1–350	3,798	26	128	28
351–1,200	4,134	29	140	31
More than 1,200	6,572	45	187	41
Urbanicity
Urban	5,983	35	158	32
Rural	11,153	65	342	68
Percentage of students in poverty
0 to 17 percent	10,175	59	300	60
More than 17 percent to 35 percent	5,993	35	175	35
More than 35 percent	968	6	25	5
Percentage of students approved for F/RP meals
0 to 45 percent	6,460	38	183	37
46 to 63 percent	6,302	37	196	39
64 percent or more	4,374	26	121	24
Charter SFA
Yes	2,632	15	45	9
No	14,504	85	445	91
Supervisory union status
LEA is not a component of a supervisory union	15,868	93	448	90
All other SFAs	1,268	7	52	10
FNS region
Mid-Atlantic	1,747	10	48	10
Midwest	4,248	25	118	24
Mountain	2,669	16	83	17
Northeast	2,149	13	60	12
Southeast	1,319	8	37	7
Southwest	2,393	14	77	15
West	2,611	15	77	15
Total	17,136	100	500	100

Source: Mathematica internal simulations based on the 2012–2013 FNS-742 file and 2011–2012 Common Core of Data File. These counts will be updated at the time of sampling using the latest versions of both the FNS-742 file and the Common Core of Data File.

Note: Because the subgroups presented are not mutually exclusive, the total number of cases across these subgroups will not sum to the total.

^aExcluding charter SFAs.

FNS = Food and Nutrition Service; F/RP = free or reduced-price; LEA = local education agency; SFA = school food authority.

4. Selecting schools within SFAs

We will sample schools from both Group 2 and Group 3 SFAs, targeting an average of two participating schools per SFA in Group 2 and three participating schools per SFA in Group 3.⁵ This will yield expected sample sizes of completed school-level data for 250 schools in Group 2 and 750 schools in Group 3. Before sampling, we will remove from the school-level frames charter schools; schools that serve only prekindergarten or kindergarten students; schools that do not participate in the NSLP; and schools that are residential or institutional (for example, correctional facilities). For Group 2, we will select the schools within each SFA using PPS sampling with the square root of enrollment as the MOS. In many cases, the number of schools in the SFA is small, as noted; therefore, many of the schools in the SFA will be selected with certainty. In addition, as noted, because some schools will have come into existence since the frame was compiled, SFAs within which we sample schools will be asked to identify such schools, and these new schools will be given a chance of selection into the sample.

Following SNMCS-I, we will stratify the schools in Group 3 SFAs by school type (elementary, middle, and high schools) and, as feasible, will select a sufficient sample in each stratum to yield one completed school of each type for the cost study data collection activities.⁶ Because some SFAs will not have all three school types, the final sample sizes for each type of school will not be equivalent. However, the selection process adopted for SNMCS-II increases the percentage of middle schools in the sample from 16.7 percent (based on the rate present in the school population) to 23.0 percent to improve subgroup-level analyses for middle schools. A summary of the school sampling process is presented in Table Q.2.

Table Q.2. Target completed sample sizes for the mainland study: Schools

	Population		Sample
Subgroups	Count or mean	Percentage	Group 2	Group 3	Groups 2 and 3	Percentage, Groups 2 and 3
School type
Elementary	56,678	60	101	342	443	44
Middle	15,887	17	61	169	230	23
High	22,733	24	88	239	327	33
Urbanicity
Urban	52,606	55	64	239	303	30
Rural	42,692	45	186	511	697	70
Racial/ethnic distribution of students (mean %)
Non-Hispanic Black	19.82	n.a.	19.1	18.0	18.3	n.a.
Non-Hispanic White	57.88	n.a.	57.3	63.0	61.6	n.a.
Hispanic	24.81	n.a.	21.8	21.2	21.4	n.a.
Other	20.46	n.a.	17.1	18.6	18.2	n.a.
Students approved for F/RP meals
0 to 45 percent	29,233	31	76	246	322	32
46 to 63 percent	29,215	31	92	266	359	36
64 percent or more	36,850	39	81	238	319	32
FNS region
Mid-Atlantic	10,292	11	22	68	90	9
Midwest	17,200	18	61	165	226	23
Mountain	11,144	12	47	124	170	17
Northeast	9,120	10	27	84	111	11
Southeast	16,172	17	21	72	93	9
Southwest	14,069	15	40	122	162	16
West	17,301	18	33	114	148	15
Total	95,298	100	250	750	1,000	100

Source: Mathematica internal simulations based on the 2012–2013 FNS-742 file and 2011–2012 Common Core of Data File. These counts will be updated at the time of sampling using the latest versions of both the FNS-742 file and the Common Core of Data File.

Note: Because the subgroups presented are not mutually exclusive, the total number of cases across these subgroups will not sum to the total.

FNS = Food and Nutrition Service; F/RP = free or reduced-price; n.a. = not applicable.

5. Sampling students

As shown in Table Q.3, we will select a random sequential sample of students to yield eight completes in each Group 2 participating school (250 schools, 2,000 total students) to participate in the student- and parent-level data collection for various subgroups.

Table Q.3. Target completed sample sizes for the mainland study: Students and student trays

	Sample
	Group 2 students	Group 3
Subgroups		Student lunch trays	Student breakfast trays
School type
Elementary	806	1,370	703
Middle	490	1,256	644
High School	704	1,274	653
Urbanicity
Urban	515	1,243	637
Rural	1,485	2,657	1,363
Race/ethnicity
Non-Hispanic Black	258	544	279
Non-Hispanic White	1,428	2,808	1,440
Hispanic	367	634	325
Approved for F/RP meals
Yes	936	1,794	920
No	1,064	2,106	1,080
FNS region
Mid-Atlantic	174	390	200
Midwest	488	889	456
Mountain	373	603	309
Northeast	214	567	291
Southeast	166	354	181
Southwest	318	634	325
West	266	463	237
Total	2,000	3,900	2,000

Source: Mathematica internal simulations based on the 2012–2013 FNS-742 file and 2011–2012 Common Core of Data File.

Note: Because the subgroups presented are not mutually exclusive, the total number of cases across these subgroups will not sum to the total.

FNS = Food and Nutrition Service; F/RP = free or reduced-price.

6. Sampling student trays for plate waste observations

In a subset of 130 of the 750 Group 3 schools, we will select student trays sequentially (every nth student throughout the meal periods) at random in the cafeteria line for lunch or breakfast. We will select a sufficient number of trays to yield 30 completed lunch tray observations per school and about 15 breakfast trays per school. This in turn will produce a total of 3,900 student trays (plate waste observations) for lunch and 2,000 for breakfast. If multiple serving lines are present, the observations will be allocated evenly across the serving lines.

This subset of schools will be selected using a two-stage stratified sample. First we will select a minimum of 65 SFAs from the 250 participating Group 3 SFAs and then we will select all or nearly all of the schools from the schools sampled in these SFAs, to yield an approximately equal number of elementary, middle, and high schools totaling 130. When setting the sampling interval for student trays, we will attempt to give all student trays a selection probability that is approximately equal across all meal periods (for breakfast and lunch separately) within a school.

To maintain consistency with SNMCS-I, schools sampled for the plate waste observations must meet the following criteria: (1) lunch and breakfast (if the school offers breakfast) must be served and consumed in the cafeteria and (2) schools must meet a minimum threshold for the number of reimbursable lunches served per day to help ensure that the target number of observations can be completed in each sampled school during a one-day site visit. We will screen for plate waste eligibility status across the Group 3 SFAs and schools. Overall, we expect we will sample 65 SFAs to reach a total of 130 Group 3 schools eligible to participate in the plate waste observations.

7. Statistical precision

This section presents expected precision levels for SFA-, school-, and student-level estimates in the mainland study, based on the target completed sample sizes. In Table Q.4, we present the expected precision levels in terms of a 95 percent confidence interval for a 30 percent characteristic⁷ for SFA-level estimates. As shown, the sample design results in an expected precision level of +/−4.7 percentage points for the overall sample of 500 SFAs and achieves precision levels of +/−10 percentage points (or better) for any subgroups that make up 25 percent or more of the population. On the basis of results from SNMCS-I, we estimate (not shown) the average design effect to be 1.38 as a result of PPS selection and the expected nonresponse adjustments for SFA-level estimates.

Table Q.4. Expected precision levels for SFA-level estimates in the mainland study

Subgroups	Completed sample sizes	CI half interval (percentage)
Number of schools
1–2a	167	8.2
3–4	132	9.2
5 or morea	156	8.5
Number of students
1–350	128	9.3
351–1,200	140	8.9
More than 1,200a	187	7.7
Urbanicity
Urbana	158	8.4
Rurala	342	5.7
Percentage of students in poverty
0 to 17 percenta	300	6.1
More than 17 percent to 35 percenta	175	8.0
More than 35 percent	25	21.1
FNS region
Mid-Atlantic	48	15.3
Midwesta	118	9.7
Mountain	83	11.6
Northeast	60	13.6
Southeast	37	17.4
Southwest	77	12.0
West	77	12.0
Charter SFA	45	15.8
Supervisory union status
LEA is not part of a supervisory union or charter schoolsa	448	5.0
All other	52	14.7
Percentage of students approved for F/RP meals
0 to 45 percenta	183	7.8
46 to 63 percenta	196	7.5
64 percent or morea	121	9.6
Total	500	4.7

Source: Mathematica internal simulations based on the 2012–2013 FNS-742 file and 2011–2012 Common Core of Data File.

Note: Confidence intervals are based on a 30 percent outcome.

^aSubgroup represents 25 percent or more of the population.

CI = confidence interval; FNS = Food and Nutrition Service; F/RP = free or reduced-price; LEA = local education agency; SFA = school food authority.

Expected precision levels for school-level estimates are presented Table Q.5. For the sample of 1,000 schools (Groups 2 and 3 combined) that will complete the Menu Survey, SNM Survey, Principal Survey, and other school-level data collection activities, the expected precision level is +/−4.8 percent­age points for the overall sample and +/−10 percentage points (or better) for any subgroups that make up 25 percent or more of the population. We also show in Table Q.5 an expected precision level of +/−4.9 percentage points for the overall sample of 750 Group 3 schools that will be included in the study of meal costs and school food service revenues and precision levels of +/−10 percentage points (or better) for any subgroups that make up 25 percent or more of the population.

Table Q.5. Expected precision levels for school-level estimates in the mainland study

	Groups 2 and 3 combined		Group 3 only
Subgroups	Completed sample sizes	CI half interval (percentage)	Completed sample sizes	CI half interval (percentage)
School type
Elementarya	443	7.2	342	7.3
Middle	230	10.1	169	10.3
High	327	8.4	239	8.7
Urbanicity
Urbana	303	8.7	239	8.7
Rurala	697	5.7	511	5.9
Racial/ethnic distribution of students (mean %)
Non-Hispanic Black	18.3	0.5	18.0	0.5
Non-Hispanic White	61.6	0.8	63.0	0.9
Hispanic	21.4	0.5	21.2	0.5
Other	18.2	0.5	18.6	0.5
Students approved for F/RP meals
0 to 45 percent	322	8.4	246	8.6
46 to 63 percenta	359	8.0	266	8.2
64 percent or morea	319	8.5	238	8.7
FNS region
Mid-Atlantic	90	15.6	68	16.3
Midwest	226	10.1	165	10.4
Mountain	170	11.6	124	12.1
Northeast	111	14.4	84	14.6
Southeast	93	15.7	72	15.8
Southwest	162	11.9	122	12.1
West	148	12.4	114	12.6
Total	1,000	4.8	750	4.9

Source: Mathematica internal simulations based on the 2012–2013 FNS-742 file and 2011–2012 Common Core of Data File.

Notes: Confidence intervals are based on a 30 percent outcome. The level of precision for school estimates for the combined Group 2 and Group 3 sample is only slightly better than that for the Group 3 sample alone. This is due to the fact that combining the two samples introduces an additional design effect at a final value of 2.48, relative to the design effect of 1.95 for the Group 3 sample alone (which incorporates the SFA design effect of 1.38). This phenomenon is a necessary consequence for meeting the sometimes competing precision requirements for each of the survey objectives and the associated study components.

^aSubgroup represents 25 percent or more of the population.

CI = confidence interval; FNS = Food and Nutrition Service; F/RP = free or reduced-price; SFA = school food authority.

Using the SNMCS-I meal cost data, we estimate the precision level for a school-based meal cost estimate of the national average meal cost of $2.36 (averaged over schools and accounting for the selection of SFAs), with a standard deviation of $0.98, to be +/−$0.105, which represents 4.4 percent of the meal cost estimate necessary for achieving the desired precision. We estimate the average design effect to be 2.23 in Group 3 and 2.83 in Groups 2 and 3 combined as a result of weighting methods described in Section 9. These estimates are based on the SNMCS-I procedures that smooth the school selection probabilities by treating each school selected and participating in the study as representative of all other schools of the same grade level.

In Table Q.6, we present the expected precision levels for the student- and parent-level estimates associated with Group 2 and the student- and tray-level estimates associated with Group 3. As shown, the sample design results in an expected precision level of +/−4.9 percentage points for the overall sample of 2,000 completed parent-student dyads in Group 2 and expected precision levels of +/−10 percentage points (or better) for any subgroups that make up 25 percent or more of the population.

For the plate waste observations in Group 3, the expected precision levels are +/−3.4 and +/−4.1 percentage points, respectively, for the overall samples of 3,900 lunch trays and 2,000 breakfast trays. Expected precision levels are +/−10 percentage points (or better) for any subgroup that represents 25 percent or more of the population.

Table Q.6. Expected precision levels for student- and tray-level estimates in the mainland study

			Group 3 plate waste observations
	Group 2 students and parents`		Lunch		Breakfast
	Completed sample sizes	CI Half interval (percentage)	Completed sample sizes	CI Half interval (percentage)	Completed sample sizes	CI Half interval (percentage)
School type
Elementarya	806	7.8	1,370	5.7	703	7.9
Middle	490	10.0	1,256	5.9	644	6.3
Higha	704	8.3	1,274	5.9	653	6.9
Urbanicity
Urbana	515	9.7	1,243	6.0	637	7.3
Rurala	1,485	5.7	2,657	4.1	1,363	5.0
Race/ethnicity
Non-Hispanic Black	258	13.8	544	9.0	279	11.0
Non-Hispanic Whitela	1,428	5.9	2,808	4.0	1,440	4.9
Hispanic	367	11.5	634	8.3	325	10.2
Approved for F/RP meals
Yesa	936	7.2	1,794	5.0	920	6.1
Noa	1064	6.8	2,106	4.6	1,080	5.6
FNS region
Mid-Atlantic	174	16.8	390	10.6	200	13.0
Midwest	488	10.0	889	7.0	456	8.6
Mountain	373	11.5	603	8.5	309	10.5
Northeast	214	15.1	567	8.8	291	10.8
Southeast	166	17.2	354	11.2	181	13.7
Southwest	318	12.4	634	8.3	325	10.2
West	266	13.6	463	9.8	237	11.9
Total	2,000	4.9	3,900	3.4	2,000	4.1

Source: Mathematica internal simulations based on the 2012–2013 FNS-742 file and 2011–2012 Common Core of Data File.

Note: Confidence intervals are based on a 30 percent outcome.

^aSubgroup represents 25 percent or more of the population.

CI = confidence interval; FNS = Food and Nutrition Service; F/RP = free or reduced-price.

8. Sampling to account for nonresponse

At each stage of the selection process, we will select a large enough sample to achieve the target completed sample sizes, considering nonresponse and ineligibility, on the basis of our experience in SNMCS-I. We will order the samples of SFAs and schools within SFAs randomly so that, in response to refusals, recruiters may recruit the next unit on the list until the desired number of cooperating SFAs is obtained. To account for SFAs in which the number of schools available is not sufficient to obtain the desired number of two participating schools per SFA in Group 2 and three in Group 3, we will increase the quota specified for other SFAs, based on expected response rates, to meet the overall target of 1,000 participating schools in Groups 2 and 3 combined.⁸

9. Weighting

We will construct analysis weights for each type of data collected to account for the probabilities of selection and observed differential response rates across various subgroups. We will also post-stratify weights for each component so that they total to benchmarks obtained from the most recent CCD and VCR-742 data by SFA-, school-, and student-level characteristics, taking into account the ineligibility of the units identified during the study. At a minimum, we will prepare a final set of nonresponse and post-stratified weights for the following study components:

• SFA weights

• School weights (Group 2 and 3 schools; may include multiple weights across study instruments)

• Cost study weights (Group 3 SFAs and schools)

• Student/parent weights (Group 2 schools)

• Plate waste observation weights (Subset of Group 3 schools)

To conduct the nonresponse adjustments for each component, we plan to use a propensity modeling procedure to predict the probability of responding to the component given the available data collected on the sampling frame. We will weight the responding cases by the inverse of the predicted probability of response using a weighting class methodology that divides the propensity scores into classes and assigns the average score within the class to each case. This approach, outlined by Wun et al.⁹, helps eliminate the need to make large adjustments to the survey weights to increase the precision of the estimates.

As a final process in preparing weights, we will adjust weights using calibration or post-stratification methods¹⁰ to ensure that weighted totals or proportions mimic those for which we have comparable data from the sampling frame or other published sources. To complete this task, we plan to use the SUDAAN WTADJX procedure to conduct the nonresponse adjustments, as well as subsequent trimming of weights as needed, and to implement the post-stratification procedures in a single step. This process will ensure timely and consistent production of the final sampling weights that incorporates a scientific strategy based on the distribution of the nonresponse adjusted weights. In situations in which the response rate drops below 80 percent, we will conduct a nonresponse bias study during the creation of the nonresponse adjustments.

For many of the mainland study components, the weights will be constructed by preparing an SFA weight and then supplemental school and, potentially, student weights to account for all stages of the selection process, along with nonresponse on each of these components. We will also create composite weights to combine results across groups (for example, the SFA Director Survey observations across Groups 1, 2, and 3).¹¹ Given that Group 3 schools within each SFA are stratified by elementary, middle, and high school status while Group 2 schools are not, to improve the statistical precision, we also plan to examine methods for smoothing the school weights to improve precision for the Group 2 and 3 combined estimates. Finally, we will construct a separate set of school-level weights that reflect school enrollment. These weights will allow FNS to examine, for example, the percentage of students that attend schools with specific characteristics.

10. Cost study weights

We will create three sets of weights for use in the estimation of meal costs, following the approach used in SNMCS-I. These weights for the Group 3 sample include (1) a set of school-level weights to be used to aggregate school-level cost component estimates up to the SFA level, (2) SFA-level weights needed to estimate national average costs and to support exploratory and confirmatory analyses of the relationships between SFA characteristics and SFA-level average meal costs, and (3) school-level weights needed to estimate national average costs at the school level and support exploratory analyses of the relationships between school characteristics and school-level average meal costs.

We require a weight for each responding school within a Group 3 SFA. These weights are used to aggregate school-level estimates of meal costs up to the SFA level. We know that participation in the school meal programs—and, therefore, meal cost—varies by school type (elementary, middle, and high schools). To account for this variation, we will calculate a weight for each school type following the approach developed in SNMCS-I. With this approach we estimate a proxy for meal costs by imputing the amount of Federal reimbursement each school would have received if all meals (lunches and breakfasts separately) and afterschool snacks served during the reference period were reimbursed at the free meal rate.¹²

Next, using CCD estimates of enrollment in each Group 3 SFA by school type, we will estimate the share of each Group 3 SFA’s per capita reimbursement for each school type. Then, for each school type in each SFA, we will divide this estimated share by the number of schools of that type with adequate cost data and multiply the result by a ratio that will adjust for the number of types of schools with adequate cost data in that SFA. If there is one qualifying school of each school type present in an SFA, their weights will simply be the shares of per capita reimbursements obtained in the second step. These school weights are multiplied by the school meal costs and then rolled up to produce an SFA-level estimate. The cost estimates for school-level meals will be weighted using the same methodology as for other school-level estimates.

11. Plate waste weights

For the plate waste observations collected in a subset of Group 3 SFAs (about 65) and schools (about 130 within the 65 SFAs), we will construct traditional selection weights. We will start with the Group 3 SFA weight and account for the probability of selection of an SFA into the sample for the plate waste study among SFAs that meet the eligibility criteria. We will then account for any SFAs that choose not to participate in the plate waste study and for the probability of selection of each school, among schools that meet the eligibility criteria within sampled and participating plate waste-selected SFAs, along with any schools that choose not to participate in this study component. We will then adjust for probabilities of selection of student trays for observation within schools, making use of data on the total number of reimbursable breakfasts and lunches served on the day of observation.

1 Specifically, a file from the Public Elementary/Secondary School Universe Study.

2 SFAs (and other units) are sampled with certainty when they have a probability of selection greater than or equal to one based on their measure of size. The measures of size to be used for each of the groups is discussed in Section 3.

3 Based on the recent combined CCD and FNS-742 database, we decided to divide the number of schools in the SFAs into three categories corresponding to the 30th and 60th percentiles (0 to 3 schools, 4 to 10 schools, and 11 or more schools) so that we could ensure that enough SFAs with a large number of schools were included in the sample. Similarly, for student SFA size, we used three categories. All possible combinations of these categories create 9 strata, and the charter SFAs will be placed in a separate 10th stratum.

4 Oversampling of one group through the use of a higher measure of size value may in turn reduce the number of units sampled on another dimension. We have worked to balance these conflicts to achieve samples that will support the aforementioned subgroup analyses.

5 In some SFAs, we will need to obtain four or five school participants to reach the overall sample goal given some SFAs may not have a school in each of the elementary, middle, and high school strata.

6 Schools in Group 3 will provide data to estimate meal costs. Because meal costs may vary in schools that serve students of different ages, the design calls for stratification of schools by school type (elementary, middle, and high). Because schools in Group 2 will not contribute data for the estimation of meal costs, stratification by school type is not necessary.

7 This criterion is based on an internal analysis of key characteristics in SNMCS-I, which found many to be around 30 percent.

8 Of note, in Group 3 we plan to select schools in pairs within the elementary, middle, and high school strata, as in SNMCS-I, to ensure we obtain a completed component from a school at each school level, to the extent possible.

9 Wun, L., T.M. Ezzati-Rice, R. Baskin, J. Greenblatt, M. Zodet, F. Potter, N. Diaz-Tena, and M. Touzani. “Using Propensity Scores to Adjust Weights to Compensate for Dwelling Unit Level Nonresponse in the Medical Expenditure Panel Survey.” Agency for Healthcare Research and Quality Working Paper No. 04004, October 2004. Available at http://www.ahrq.gov.

10 Deville, Jean-Claude, and Carl-Erik Särndal. “Calibration Estimators in Survey Sampling.” Journal of the American Statistical Association, vol. 87, no. 418, 1992, pp. 376–382.

11 The composite can be adjusted simply to the proportion of SFAs responding from each group. It also can reflect differences in design effects among the groups.

12 This approach implicitly assumes that the free meal reimbursement rate is a reasonable proxy for the average cost per meal.