Part B PISA 2015 Main Study

PISA 2015 assesses students’ knowledge and skills in science, mathematics, and reading as they near the end of their compulsory school experience. For international comparability, this is defined as students who are 15 years old, in grades 7 or higher. A range of exact birthdates is specified by the international coordinating committees based on the months in which the data will be collected: students must be between the ages of 15 years and 3 completed months and 16 years and 2 completed months at the beginning of the test period. The universe for the selection of schools is all types of schools in all states of the United States and the District of Columbia. Within sampled schools, students will be selected for participation by drawing a random sample among the 15-year-old students.

B.2 Statistical Methodology

The Technical Standards for main study PISA 2015 established by the international governing board include the following:

Standard 1.8. The student sample size must be a minimum of 5,250 assessed students, or the National Defined Target Population.

Standard 1.9. The school sample size must be a minimum of 150 schools or all schools that have students in the National Defined Target Population.

Standard 1.10. The target cluster size is typically 35 PISA eligible students, which upon agreement can be increased or reduced to a number not less than 20.

Standard 1.11. School response rates must be above 85 percent of sampled schools. If a response rate is below 85 percent then an acceptable response rate can still be achieved through agreed upon use of replacement schools. PISA establishes three response rate zones—acceptable, intermediate, and not acceptable. “Acceptable” refers to original school response rates above 85 percent and means that the country’s data will be included in all international comparisons. “Not Acceptable” refers to original response rates below 65 percent and means that the country’s data will be a candidate for not being reported in international comparisons unless considerable evidence is presented that nonresponse bias is minor. “Intermediate” refers to original school response rates of between 65 and 85 percent and means that a decision on whether or not to include the country’s data in comparisons must be made while taking into account a variety of factors, such as student response rates, quality control, etc. In addition, schools with less than 50 percent participation of students are not considered participating schools and neither that school nor those students that did participate are considered in the calculation of response rates.

Standard 1.12. The overall student response rates must be above 80 percent of sampled students.

In addition, NCES has a standard in which student response rate should be at least 85 percent, and the sampling design described below is based on that rate.

Overview

The design for this study will be self-weighting, stratified, consist of two stages, and will use probability proportional to size (PPS). There will be no oversampling of schools or students. Schools will be selected in the first stage with PPS and students will be sampled in the second stage yielding overall equal probabilities of selection.

Target Populations

The PISA target population is 15-year-old students attending education institutions located within the United States in grades 7 and higher. The plan is to implement the survey in the fall of 2015 (October-November). The specific definition of age eligibility that will be used in the survey is “…between 15 years and 3 (completed) months to 16 years and 2 (completed) months at the beginning of the testing window.”

Sampling Frame of Schools

The population of schools for PISA 2015 is defined as all schools enrolling any 15-year-olds in grades 7 through 12. As in previous PISA cycles, the school sampling frame will be developed from the most up-to-date NCES Common Core of Data (CCD) and Private Schools Survey (PSS). For the PISA 2015 main study, we will use the school sampling frame prepared for the National Assessment of Educational Progress (NAEP) 2014 which used the 2012-2013 CCD and the 2011-2012 PSS school data.

The grade structure of the school is a key stratification variable designed to reduce sampling error, but this is especially so in PISA because data analyses have shown that achievement is highly related to grade. Other stratification variables include public/private, region of the country, location (urban/suburban/town/rural, etc.), and enrollment by race/ethnicity.

School, Teacher, and Student Samples

The international minimum number of completed assessments—for the core computer-based assessments in science, mathematics, reading, and collaborative problem solving—is 5,250 students in 150 schools. In PISA, the United States typically assesses between 5,600 and 5,900 students in 165 schools. Assuming a similar response rate as PISA 2012, our initial target is a total sample of about 240 schools to yield about 165 participating schools. To achieve the target final school response rate, we will use replacement schools to complete the sample.

The sample of teachers will be based on an assumption of up to 25 teachers per school and an average of 24 teachers per school for a total of 3,882 sampled teachers and 3,300 responding teachers.

The student-per-school target for the core assessment is at least 36 completed student assessments per school. Assuming a within-school response rate of 85 percent (rates were 85 percent in 2000, 82 percent in 2003, 91 percent in 2006, 86 percent in 2009, and 89 percent in 2012), the original sample size of students within schools will be 42. The two states that are finalizing plans to participate in PISA 2015 will each have a sample of 50 schools and 2,100 students to yield 1,890 assessed students.¹ Puerto Rico will have a sample size of 50 schools and 2,100 students.

Gathering Contact Information to Support a Future Methodological Study

In PISA 2012, NCES collected student contact information for a follow-up study to validate PISA by relating student performance on PISA to other cognitive, education, and employment outcomes (the new PISA Validation Study, OMB# 1850-0900). NCES is again considering conducting a follow-up with PISA 2015 students. As in PISA 2012, PISA 2015 students will be asked to supply contact information so NCES can contact them in the future. The follow-up study, including any follow-up contact with students, would be carried out under a separate OMB clearance request. In this current request we are seeking approval only to gather student contact information as was done in PISA 2012. Appendix D contains items that will be used to ask students for their email, home address, and phone numbers, along with those of a relative or a close friend. The items are the same as those used to collect contact information from the PISA 2012 students (OMB# 1850-0755 v.13) with an additional question asking for the email of a relative or another contact.

Nonresponse Bias Analysis, Weighting, Sampling Errors

It is inevitable that nonresponse will occur at both levels of sampling: school and student. We will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards. After the international contractor calculates sample weights, sampling errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, clustering and stratification (see the already approved Appendix C for more detail on plans for a nonresponse bias analysis).

B.3 Maximizing Response Rates

Historically PISA has struggled to achieve adequate school-level response rates. PISA is a long assessment requiring over 3 hours of student time. Schools are reluctant to participate because of the commitment of time and loss of instructional time for students. Because PISA is administered to an age-based sample, students are sampled from across grades and classes, which schools find disruptive. We have reviewed our materials and conducted focus groups (OMB# 1850-0803) in order to understand the attitudes that school administrators have toward PISA and what materials work best in describing PISA to schools. The following table provides the school and student response rates for each PISA round.

Table B. Historical PISA school and student response rates for the United States

Year	Weighted Original school response rate	Weighted School response rate after replacement	Weighted Student response rate
2000	56	70	85
2003	65	68	83
2006	69	79	91
2009	68	78	87
2012	67	77	89

Our approach to maximizing school and student response rates in the main study includes the following:

Use of a fall test administration, to avoid conflicts with state testing;
Selecting and notifying schools during the academic year preceding PISA data collection;
Communicating with state officials early in the process and applying a more proactive approach with states to gain assistance with sampled schools, including working with NAEP staff and state recruitment personnel to reach out to districts and schools;
Assigning personal recruiters for specific schools;
Incentives for schools, school coordinators, teachers, and students (see Section A9);
Contact with schools and school coordinators at set intervals throughout the year preceding the assessment;
A summer meeting several months before the data collection (June 2015) for representatives from sampled schools to inform them about PISA and keep them engaged in the study; and
Provision of school-level results on PISA for schools that achieve a minimum student response rate of 85 percent.

These approaches are based on experience from previous PISA administrations and consultation with staff from other studies such as TIMSS and NAEP.

B.4 Tests of Procedures and Methods

The main study will collect data on selected students who will be assessed in science, mathematics, reading, collaborative problem solving, and financial literacy. These data, or their subset, depending on the international data release plans, will be reported via a U.S. National Report that will be released approximately one year after data collection, in December 2016, to correspond with the international data release. This report will be tailored to a general audience and will provide preliminary results about the overall performance of U.S. students relative to student performance in other countries. Following the release of the national report, additional data will be made available to secondary users in the form of the International Data Explorer (IDE), an online tool on the NCES website, and a U.S. public-release dataset. Also, those schools that qualify (based on response rate and sample size requirements) will receive a school report with basic comparisons of the performance means of students in the school with overall means for the United States, OECD countries, and other similar schools.

The U.S. PISA field test assessed 72 schools and 2,288 students. Given that PISA 2015 uses computer-based assessments (CBA), the field test employed a mode effect study to determine the effect, if any, of moving trend items, previously administered in paper-based form (PBA), to a computer-based mode. The mode effect study examined the differences in student performance between the paper-based versions and the computer-based versions of the same items. Overall, the large majority of items performed similarly between both modes. While there were differences in student performance on some items across countries, the level of difficulty of an item stayed largely the same between PBA parameters and the new CBA-based parameters. Where mode effects do occur, an item can be adjusted using a statistical model-based method to maintain item comparability between modes and allow trend measurement to be maintained.

Overall, the International Consortium of contractors concluded that the CBA mode is an improvement over the PBA mode. The CBA allows more variations in the order in which assessment items are presented to students across the sample thereby providing better measurement of the subject domains. CBA also shows less effect on the proportion correct and student non-response due to the order in which students see the items. Additionally, timing and process data obtained from computer system log files help evaluate and better understand student non-response, which was not possible with a paper-based administration of PISA.

NCES was concerned that the length of the core assessment with equipment set up and the need to hold a second session for financial literacy would require a second day in schools. This proved not to be the case in the field trial. We were able to assess both the core PISA assessment and financial literacy in a single day in all schools. PISA 2015 will administer school questionnaires and new teacher questionnaires online. These questionnaires were launched and administered successfully in the field test, achieving adequate response rates. We believe that we can sustain these response rates in the main assessment.

B.5 Individuals Consulted on Study Design

Many people at the OECD, ETS, and other organizations around the world have been involved in the design and development of PISA. Some of the lead people are listed in section 8 of Part A. Overall direction for PISA is provided by Drs. Dana Kelly and Patrick Gonzales, the PISA National Project Managers at the National Center for Education Statistics, U.S. Department of Education. Dave Kastberg and David Ferraro at Westat are responsible for the sample design, data collection, and reporting. Internationally, Irwin Kirsch and Claudia Tamassia at ETS have responsibility for overall project coordination.

1 This is based on an expected response rate in the state of 90 percent; response rate in the states has been historically slightly higher than the national sample. This is attributable to the fact that, for state PISA samples, states recruit and manage the participation of the schools and the students.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	PREFACE
Author	Janice Bratcher
File Modified	0000-00-00
File Created	2021-01-25