NASS Comments and Responses

Appendix K: NASS Comments and Responses

3/28/2013

Michael Jacobsen

OMB DOCKET FOR THE NSLP/SBP DIRECT CERTIFICATION STUDY

I have reviewed the OMB docket for the Evaluation of Demonstrations of NSLP/SBP Direct Certification of Children Receiving Medicaid Benefits and have only a few comments.

Overall, the study is well-designed and important. The researchers have covered many of the major issues surrounding an intensive data collection effort such as this study. However, I have some questions about the methodology and the wording of the docket.

First, in Part A.1, you have separated you six selected states into two separate groups. The DC-M1 group focuses on states containing very large cities, while the DC-M2 group focuses on less-populous states. Could a demographic bias between the two groups affect your findings? I could see this as a question being asked when the reports are released.

Response:

The states were not grouped this way by design. The selection of DC-M1 or DC-M2 for each state was based on the state’s preferences and ability to conduct DC-M in a subset of districts or statewide, as indicated in its application and subsequent discussions between FNS and state officials. FNS encouraged states to participate in DC-M1 if possible.

The groups will not be combined in analyses, except in IA1, which relies on a simulation of DC-M, rather than the outcomes of DC-M as implemented by the states. In the IA2 analyses, the results from the experimental design (in DC-M1 states) will be heavily emphasized, because such a design provides a credible basis for causal inferences. In addition, because the states were not randomly selected, the results cannot be statistically generalized beyond the collection of states included in the demonstration and, more specifically, in the experimental component (DC-M1). Similarly, the samples of districts within DC-M1 states were systematically restricted, so the results cannot be statistically generalized to each state as a whole. Of particular relevance to this question about the populations of demonstration states and the presence of large cities in these states, the legislated limit on the percentage of the nation’s students certified for free or reduced-price meals within districts in the DC-M1 component of the demonstration resulted in the exclusion of Chicago and the five largest districts in Florida in Year 1 of the study. In New York, in contrast, only New York City was included in the state’s Year 1 application; other districts in the state were not. The reports will emphasize these points.

Second, in Part A.2, I was confused about the dissemination of the data. I know you mention later in the docket that you would be releasing initial and final reports. However, that was not made clear here. This section requires the researchers to justify the project the proposed in the previous section, including details on the final product of the study. More detail on the output from this study is needed in this part.

Response:

We will add the following text at the end of A.2: “The analyses derived from this data collection will be presented in two reports to Congress, one detailing the results for SY 2012-2013 of the demonstration and one incorporating the results for SY 2013-2014. Three additional reports will present analyses related to (1) the Access Evaluation, (2) the Match Validation Substudy (MVS), and (3) the NSLP Afterschool Snack Program (ASP) and the Special Milk Program (SMP).”

Third, in Part A.2, you mention in footnote 4 that Alaska will not be included in the Access Evaluation. How could that affect your analysis? Do you see any potential issues with not including Alaska in this project when you are going to analyze the data from the four different projects as a whole? In other words, why Alaska?

Response:

Alaska could not be included in the Access Evaluation due to lack of availability of the data needed for that component. This data availability issue ultimately resulted in the state determining that it could not conduct DC-M, and therefore Alaska withdrew from the demonstration entirely. Thus, this question is no longer relevant. We will remove the footnote from the final version of Part A.

Fourth, in part A.2, you mentioned that the second round of school district cost data collection will involve not only the initial six states but a second set of states. Do you know which states you will include? If not, how will you choose?

Response:

FNS issued a Request for Application in November 2012 for additional states to apply to participate in Year 2 of the demonstration. Based on the applications received, FNS selected Massachusetts and New York State to participate in DC-M1 beginning in Year 2. ¹

In Part B, where is the reference located? How can the OMB docket reviewer find it?

Response:

The document referenced is a memo on matching, random assignment, and sampling that was delivered to FNS in November 2012. The document is referenced in footnote number four of Part B.

1 Although New York City was included in DC-M1 in Year 1, the state applied to include additional districts in the demonstration in Year 2.

K.4