PISA Memo

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: June 13, 2011

TO: Shelly Martinez, OMB

FROM: Dana Kelly and Holly Xie, NCES

THROUGH: Kashka Kubzdela, NCES

RE: Summary of Changes to the PISA 2012 Main Study Recruitment and Administration (OMB# 1850-0755 v.12) Materials

Thank you for meeting with NCES on May 25, 2011 to discuss the PISA 2012 field test and our plans for the main study. The attached PISA 2012 Main Study clearance request outlines NCES’s plans for the implementation of the main study. The PISA 2012 Field Test Recruitment and data collection were approved on November 22, 2010 (1850-0755 v.10) and a change request was approved on March 16, 2011. The PISA 2012 field test was conducted in April-May 2011.

Below, we summarize changes to the approved request for the field test that are reflected in the attached request for the main study. The changes and our plans for the main study are consistent with what we discussed in our briefing with you and your colleagues. Also, you asked during the briefing that we indicate clearly that OMB does NOT need to review the enclosed field test questionnaires provided in Appendix C because these are the field test versions that were approved in March 2011. The final versions of the main study questionnaires will be a subset of the field test versions and will be submitted to OMB for approval in early 2012. We do explain in Part A that the 30-day public-comment period is the public’s opportunity to provide input into the main study versions of the questionnaires. Also below, we provide the major milestones for the finalization of the PISA 2012 instruments and for the remainder of its clearance process, as we discussed during our briefing on May 25.

Changes from PISA 2012 Field Test and Recruitment clearance request:

• In the PISA 2012 Field Test and Recruitment OMB clearance request we presented seven scenarios for the main study. These scenarios reflected the possible combinations of core and optional components. In the attached request, we describe NCES’s plans to implement the paper-based mathematics, science, and reading literacy assessments and the computer-based problem-solving assessment, which comprise the “international core,” and to implement the computer-based reading and mathematics assessments and the paper-based financial literacy assessment, which are “international options.” This combination was presented as scenario “MS-A” in the PISA 2012 Field Test and Recruitment OMB clearance request.

• In the PISA 2012 Field Test and Recruitment clearance request NCES received approval to conduct an incentive experiment at the school level to test two levels of incentives for schools and school coordinators. In the attached request (section A.9) we describe the outcome of the experiment and propose incentive amounts for the main study: $200 for schools, $200 for school coordinators, $25 for students taking the paper-based assessment, and an additional $15 for students subsampled to also take the computer-based assessment. Additionally, we describe our plan to have a second-tier incentive for schools ($800, to be implemented if by May 2012 we do not have a sufficient school response rate), and an incentive amount for students if we need to administer the assessment out of school hours ($35 for the paper-based assessment and $15 for students subsampled for the computer-based assessment).

• We have revised the burden table to show only the main study burden estimates (section A.12). We also include burden estimates for two states that may participate in PISA 2012 to receive state-level results (one has committed and one has indicated a strong interest), and combined estimates for the national and state implementation. We also have refined the estimates for main study student and school sample sizes from those presented for informational purposes in the field test request.

• We have adapted the approved field test recruitment materials for use in the main study. The text for the materials is included in Appendix A (letters, brochure, FAQs, summary of activities, and timeline) and Appendix B (parental consent letters and forms and a Facts about PISA document).

• We have included the U.S. versions of the PISA 2012 field test student and school questionnaires in Appendix C of the attached clearance request. This includes the four student questionnaire forms, background questions about financial literacy that were presented to students as part of the financial literacy field test booklets, and one school questionnaire form. These were approved by OMB for administration in field test so OMB does NOT need to review the questionnaires at this time. However, this is the public’s opportunity to provide comments as explained in Part A. The final main study questionnaires will comprise a subset of the items that were field tested; there may be minor changes in wording to some items but it is not expected that any new items will be added. In the attached request we describe the process and timeline for finalizing the main study questionnaires.

• In the PISA 2012 Field Test and Recruitment OMB clearance request we included that NCES is working on plans for a methodological study to examine the relationship between performance on PISA and other indicators, and that to facilitate future contact with PISA 2012 students we plan to obtain student contact information during the main study. The attached clearance request includes: NCES’s plans for obtaining student contact information during the main study from students (specific questions to be asked of students are presented in Appendix D) and from schools; for informing parents and students about follow-up contact with students (see Appendix B); and for ensuring the security and confidentiality of information collected for this purpose (see section A.10). Actual follow-up with students will be covered under a separate clearance. Under this clearance request we are only seeking approval to gather contact information to support the future study.

• We have updated the estimated costs (section A.14) to include costs for scoring which had been inadvertently omitted from the field test clearance request.

Schedule/plans for the finalization of instruments and OMB clearance:

The following are the major milestones for the finalization of the PISA 2012 instruments and for the remainder of the clearance process, as we discussed during our briefing on May 25.

• July 15: NCES submits field test data to international consortium

• August 22-September 19: review U.S. item statistics

  • On or about September 7: NCES meets with OMB

    • OMB provides comments on U.S. field test item statistics for questionnaires

• September 26-28: PISA Questionnaire Expert Group (QEG) meets to review international field test data and recommend instruments

• October 10-14: National Project Managers meeting, review proposed instruments and data

• October 24-26: PISA Governing Board meeting, review and approval of instruments

• December: Final versions of instruments (international) dispatched to countries

• December-February: NCES prepares U.S. versions of instruments

• March 2012: Main Study Administration Package (30-day public comment)

  • Final versions of U.S. main study questionnaires

  • Description of any changes since last clearance

• June 2012: Begin printing instruments

• September-November: Main study data collection period

2