ED Response to OMB Passback

MEMORANDUM OMB # 1850-0755 v.16

DATE: March 30, 2015

TO: Shelly Martinez

Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Patrick Gonzales

National Center for Education Statistics

THROUGH: Kashka Kubzdela

National Center for Education Statistics

SUBJECT: Response to 3/12, 3/13, and 3/30 OMB passback on PISA 2015 main study

1. Who are the 2 states?

North Carolina and Massachusetts.

2. Who is paying for the sample boost and why?

As in past administrations of PISA, as well as other international assessments in which the United States participates (i.e., PIRLS and TIMSS), all states have the opportunity to participate in PISA 2015 through NCES, which serves as the U.S. national center for these studies, by contracting directly with the U.S. national contractor. Using the national contractor ensures that the assessment is carried out in accordance with the international procedures and in the same manner as in the national sample. Moreover, integrating the national and state administrations provides for greater efficiency and cost-savings for individual states opting to participate.

Each state is paying the cost of administering PISA within their state to a sample of 50 schools and a minimum of 1,500 students. These sample numbers are the minimum international requirements for an adjudicated region. The 50 schools are independent of any schools that were selected in the US national sample. Choosing to participate in PISA (for Massachusetts, this will be the second time) signals that these states see value in international comparisons and the information PISA provides. By assessing separate samples in these two states, the states will receive state-level PISA scores and distributions in all PISA subjects, as well as contextual data. Their results will be reported in the PISA international reports and data will be made available for secondary research so that the states and other researchers can further explore the PISA data and state-level performance.

3. How were they chosen?

The states were self-selected based on their own interest and desire for acquiring state-level PISA results. Both of these states have participated in international assessments previously. Massachusetts participated at the state level in PISA 2012 and also in TIMSS 2007. North Carolina participated at grade 4 in TIMSS 2011.

4. Where is the discussion of utility for this increased burden?

As indicated above, these two states have decided that they would like to obtain state-level PISA results. The PISA U.S. national sample is designed to provide national estimates of performance but not state-level results. Expanding the sample for a state seeking information on schools and students under its jurisdiction is an efficient approach to enabling this data to be collected and reported. To ensure comparability and operational consistency, these two states will be utilizing the national contractor to collect the data, using the exact same instruments and procedures that will be used for the national sample. Moreover, integrating the national and state administrations provides for greater efficiency and cost-savings.

The increased burden above what is required for (and allocated to) the PISA U.S. national sample will provide the Massachusetts and North Carolina state departments of education and the general public with state-level PISA results with which to compare the state education systems with other countries. For the federal government, the utility in having states participate in international assessments like PISA is the increased visibility of the assessment, which aids in national school recruitment, the availability of actual state-level PISA scores which can be used as a check on research attempting to project or estimate state-level performance on PISA based on other assessment data, and the availability of valid and reliable data at a finer level of detail (i.e., state) than is possible with the national data.

5. Please clarify whether “using the exact same procedures” includes using NCES’s confidentiality pledge.  Is it indistinguishable to the school or student which entity (NCES or state) is sponsoring the collection?  We are trying to figure out if, at the end of the day, the additional data collection for the two states is a Federally-sponsored data collection (eg, NCES’s legal authority, confidentiality pledge, and OMB control number) or not.   If it is, we don’t see how the states can be given access to confidential data, among other things.  If it is not, it would seem important to avoid misleading respondents and others by making clear that these are two distinct data collections, with different authorities, confidentiality pledges, etc. 

States do not have access to restricted use data from the PISA collection unless they apply for a restricted use data license, like any other user. They have access to public-use raw data files and aggregate state level results. Yes, the same confidentiality pledge and everything else are given to respondents in the state samples as in the national sample. Respondents do not know if they are a state or a national sample school. The same materials and procedures are used for all respondents, as in previous administrations of PISA and TIMSS, and to assure full comparability of results.