PISA 2012 Full Scale Part B

PISA 2012 assesses students nearing the "end of their compulsory school experience". For international comparability, this is defined as students who are 15 years old, regardless of their grade in school. A range of exact birthdates is specified by the international coordinating committees based on the months in which the data will be collected. However, students must be between the ages of 15 years and 3 completed months and 16 years and 2 completed months at the beginning of the test period. The universe for the selection of schools is all types of schools in all states of the United States and the District of Columbia. Within sampled schools, students will be selected for participation by drawing a random sample among the 15-year-old students.

B.2 Statistical Methodology

The Technical Standards for main study PISA 2012 established by the international governing board include the following:

Standard 1.8. The student sample size must be a minimum of 4,500 assessed students, or the National Defined Target Population.

Standard 1.9. The school sample size must be a minimum of 150 schools or all schools that have students in the National Defined Target Population.

Standard 1.10. The target cluster size is typically 35 PISA eligible students, which upon agreement can be increased or reduced to a number not less than 20.

Standard 1.11. School response rates must be above 85 percent of sampled schools. If a response rate is below 85 percent then an acceptable response rate can still be achieved through agreed upon use of replacement schools. PISA establishes three response rate zones—acceptable, intermediate, and not acceptable. “Acceptable” refers to original school response rates above 85 percent and means that the country’s data will be included in all international comparisons. “Not Acceptable” refers to original response rates below 65 percent and means that the country’s data will be a candidate for not being reported in international comparisons unless considerable evidence is presented that nonresponse bias is minor. “Intermediate” refers to original school response rates of between 65 and 85 percent and means that a decision on whether or not to include the country’s data in comparisons must be made while taking into account a variety of factors, such as student response rates, quality control, etc. In addition, schools with less than 50 percent participation of students are not considered participating schools and neither that school nor those students that did participate are considered in the calculation of response rates.

Standard 1.12. The overall student response rates must be above 80 percent of sampled students.

In addition, NCES has a standard in which student response rate should be 85 percent, and the sampling design described below is based on that rate.

Overview

The design for this study will be self-weighting, stratified, consist of two stages, and will use probability proportional to size (PPS). Schools defined as very small schools (based on the enrollment of 15-year-old students) will be undersampled at a rate to be determined by a small school analysis. This is to reduce the administrative burden of handling small schools which make up a relatively large proportion of schools but less than 5% of enrolled 15-year-old students. There will be no oversampling of students. Schools will be selected in the first stage with PPS and students will be sampled in the second stage yielding overall approximately equal probabilities of selection.

Target Populations

The PISA target population is 15-year-old students attending education institutions located within the United States in grades 7 and higher. The plan is to implement the survey in the fall of 2012. The specific definition of age eligibility that will be used in the survey is “…between 15 years and 3 (completed) months to 16 years and 2 (completed) months at the beginning of the testing window.” For PISA 2012, eligible students will be those born between July 1, 1996 and June 30, 1997 and who are in grade 7 or higher.

Sampling Frame of Schools

The population of schools for PISA 2012 is defined as all schools containing in the target population (described above). As in previous PISA cycles, the school sampling frame will be developed from the most up-to-date NCES Common Core of Data (CCD) and Private Schools Survey (PSS). For the PISA 2012 main study, we will use the school sampling frame prepared for the National Assessment of Educational Progress (NAEP) 2012 which used the 2008-2009 CCD and 2009-2010 PSS school data.

The grade structure of the school is a key stratification variable designed to reduce sampling error, but this is especially so in PISA because data analyses have shown that achievement is highly related to grade. Other stratification variables will include public/private, region of the country, location (city/suburban/town/rural), enrollment by race/ethnicity, enrollment by gender, and state.

School and Student Samples

Based on the U.S. experience in the field test and the expectation that internationally the field test will demonstrate the feasibility and quality of all components planned for PISA 2012, the United States intends to participate in the core components of PISA as well as the optional financial literacy and computer-based assessments (although for financial literacy it is contingent on securing funding). Thus, the United States is planning to administer the following components:

Paper-and-pencil mathematics, reading, and science,
Computer-based problem solving
Computer-based mathematics and reading
Financial literacy (paper-and-pencil) [pending funding]

The international minimum number of completed assessments—for the core paper-and-pencil mathematics, science and reading assessment—is 4,500 students in 150 schools. The United States PISA sample is typically approximately 5,600 students in 168 schools. Assuming the same response level as PISA 2009, our initial target is a total sample of about 240 schools to yield about 168 participating schools. To achieve the target final school response rate, we will use replacement schools to complete the sample.

The number of sampled students per school will depend on whether or not the United States participates in the financial literacy option. None of the other international options influence sample size (although they influence burden, see below and part A.12). The student-per-school target for the core assessment is at least 35 completed student assessments per school. The student-per-school target including financial literacy is 43. Assuming a within-school response rate of 85 percent (rates were 85 percent in 2000, 82 percent in 2003, and 91 percent in 2006, and 86 percent in 2009); the original sample size of students within schools will be 50 for the core assessment. All students would take the 2-hour paper-and-pencil assessment. A subset of 18 students would participate in a second 40-minute session for computer-based assessment of reading, mathematics, or general problem-solving.

Gathering Contact Information to Support a Future Methodological Study

NCES is planning a methodological study to validate PISA by relating student performance on PISA 2012 to other outcomes. To support the validation study, the PISA 2012 students will be asked to supply contact information so NCES can contact them in the future. The follow-up study, including any follow-up contact with students, will be carried out under a separate OMB clearance request (to be submitted at a later date). In this current request we are seeking approval only to gather student contact information. Appendix D contains items to be asked of students regarding their home address and phone numbers and the name, home address and phone numbers of a relative or close friend (the items mirror those used in past and current NCES longitudinal studies). To ensure we have as complete a database of student contact information as possible, we will also ask schools for publically available directory information for students that participated in PISA 2012. Parent consent and informational materials include information about possible future contact.

To assure the confidentiality of individuals, student contact information will be collected, transmitted, and stored separate from students’ responses to the test booklets and questionnaires. Contact information and responses to the 2012 instruments will be linked only by ID codes and no student names or contact information will ever be released. Appropriate laws regarding data confidentiality, as described in section A.10 of Part A of this clearance request, will be abided and cited in PISA 2012 materials.

Student Sampling

The KeyQuest sampling software provided by ACER will be used to select the student samples in each school. In each school 50 students will be sampled—42 for the standard mathematics, reading and science assessment and 8 for financial literacy. Among the 42 students taking the standard mathematics, reading and science assessment, 18 will be selected to return for a second session to take the computer-based assessment.

Nonresponse Bias Analysis, Weighting, Sampling Errors

It is inevitable that nonresponse will occur at both levels: school and student. We will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards. After the international contractor calculates weights, sampling errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, clustering and stratification.

B.3 Maximizing Response Rates

Our approach to maximizing school and student response rates in the main study includes the following:

Use of a fall test administration, to avoid conflicts with state testing;
Selecting and notifying schools at least a year in advance;
Approaching schools directly, and notifying states and school districts;
Assigning personal recruiters for specific schools;
Incentives for schools, school coordinators, and students (see Section A9);
Contact with schools and school coordinators at set intervals throughout the year preceding the assessment;
A summer conference for representatives from sampled schools several months before the data collection (main study only) to inform them about PISA and keep them engaged in the study; and
Provision of school-level results on PISA.

These approaches are based on recommendations from an NCES panel and experience with previous PISA administrations.

B.4 Purpose of Main Study and Data Uses

The main study will collect data of selected students assessed in mathematics, reading, science, and general problem solving. These data will be reported via a U.S. National Report that will be released approximately one year after data collection in December 2013. This report will tailored to a general audience and will report preliminary results about the overall performance of US students relative to student performance in other countries. Following the release of the national report, additional data will be available to secondary users in the form of the International Data Explorer, an online tool on the NCES website, and an Electronic Codebook and U.S. public-release dataset. Also, those schools that qualify (based on response rate and sample size requirements) will receive a school report with basic comparisons of the performance of students in the school with overall means for the United States, OECD countries and other similar schools.

B.5 Individuals Consulted on Study Design

Many people at OECD, ACER, and other organizations around the world have been involved in the design of PISA. Some of these were previously listed in section A8. In addition, Westat co-project directors David Kastberg and Christine Nord, as well as other key staff at Westat, have provided input into the study design and implementation in the United States. Overall direction for PISA is provided by Holly Xie and Dana Kelly, the PISA National Project Managers at the National Center for Education Statistics, U.S. Department of Education.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	PREFACE
Author	Janice Bratcher
File Modified	0000-00-00
File Created	2021-02-01