Change Memo

_MEMORANDUM NATIONAL CENTER FOR EDUCATION STATISTICS

Institute of Education Sciences

United States Department of Education

Date: February 23, 2024

To: Beverly Pratt OMB

Through: Carrie Clarady, OMB Liaison, NCES

From: Samantha Burg, NCES

Re: Program for International Student Assessment 2025 (PISA 2025) Main Study Recruitment and Field Test Update (OMB# 1850-0755 v.29)

This change memo outlines changes in the change request for the Program for International Student Assessment (PISA) 2025 These changes include a revised student video for field test as well as minor changes to the program as described below:

1. updates to the package with the student video for the field test

2. updates the field test burden with additional burden of 3 added schools to the school sample;

3. clarifies that the student incentives for the field test are planned to be distributed to students as cash cards instead of checks;

4. Adds a non-binary gender option to the Student Listing Form that is presented in the E-filing instructions in appendix A.

Changes to all documents are described below in the “Proposed Changes for PISA 2025 Field Test” section.

Background to PISA:

PISA is an international assessment of 15-year-olds, which focuses on assessing students’ reading, mathematics, and science literacy. That is, PISA assesses functional skills that students have acquired as they near the end of mandatory schooling (aged 15 years), and students’ knowledge and skills gained both in and out of school environments. It is sponsored by the Organization for Economic Cooperation and Development (OECD) and is conducted in the US by the National Center for Education Statistics (NCES) of the U.S. Department of Education.

The United States has participated in PISA since it was first administered in 2000 and we plan to participate in 2025. This allows us to track trends over time and to compare the performance of U.S. students with that of students in other education systems. In each administration of PISA, one of the subject areas (reading, mathematics, or science literacy) is the major domain and has the broadest content coverage, while the other two subjects are the minor domains.

In 2025, science literacy will be the major domain, reading and mathematics literacy will be the minor domains and an additional assessment of self-regulated learning called Learning in a Digital World will also be included. In addition to the cognitive assessments, PISA includes questionnaires administered to school principals and assessed students. To prepare for the 2025 main study, PISA countries will conduct a field test in 2024. The primary purpose of the field test is to evaluate newly developed assessment and questionnaire items but it also serves to test the assessment operations.

Proposed Changes for PISA 2025 Field Test:

The changes to all PISA 2025 documents are described below. Just as a reminder specific illustrated changes are marked in red; additions are in simple red font, while deletions are in red strikethrough.

Throughout the package the stated sample of 50 schools for the field test has been changed to 53 schools. The sample was increased by 3 schools where the UH (short assessment) option will be administered as part of the international sample design.

1. Specific material changes to Part A

Section A.9: The description of the method of incentive payments to students was changed from checks to cash cards.

All student incentives will be offered directly to the students. Parents will be informed of the amount of the payment the students will receive in the consent form/letter in advance of the assessment. The payments will be provided as a personal check, as has been successfully done since PISA 2009 cash card.

Section A.12:

Page 12-13 - The estimates of burden have changed. The added schools to the field test sample increased from 50 to 53 schools. The number of students changes from 2,600 to 2,756.

Table A-1. Burden estimates for PISA 2025 field test and main study

	Sample	Expected response rate	Number of respondents	Number of responses	Burden per respondent (minutes)	Total burden (hours)
FIELD TRIAL—Based on core + international options
Recruitment and Pre-Assessment Activity (includes Puerto Rico)
School Administrator (US sample)	50 53	1	50 53	50 53	90	75 80
Special Handling Districts IRB Staff Approval (US sample)	15	1	15	15	120	30
Special Handling Districts IRB Panel Approval (US sample)	90	1	90	90	60	90
School Coordinator (US sample)	50 53	1	50 53	50 53	240	200 212
School Administrator
Questionnaire (US sample)	50 53	1	50 53	50 53	45	38 47
Parent
Student Participation Consent	2,600 2,756	1	2,600 2,756	2,600 2,756	3	130138
Total School Burden Field Trial			2,855 3,020	2,855 3,020		563 596
Student
US national sample
Directions	2,600 2,756	0.8	2,080 2,205	2,080 2,205	10	347 367
Assessment	2,600 2,756	0.8	2,080 2,205	2,080 2,205	120	4,160 4,410
Student questionnaire (Main questionnaire)	2,600 2,756	0.8	2,080 2,205	2,080 2,205	35	1,213 1,286
Student questionnaire (ICT questionnaire)	2,600 2,756	0.8	2,080 2,205	2,080 2,205	15	520 551
Total Student Burden Field Trial			2,080 2,205	4,160 4,410		1,733 1,837
Total Burden Field Trial			4,935 5,225	7,015 7,430		2,296 2,433
MAIN STUDY —Based on core + international options
US national sample
Recruitment and Pre-Assessment Activity
School Administrator	298	0.86	256	256	90	384
Special Handling Districts IRB Staff Approval (US sample)	31	1	31	31	120	62
Special Handling Districts IRB Panel Approval (US sample)	187	1	187	187	60	187
School Coordinator	256	0.85	218	218	240	872
School Administrator
Questionnaire	218	1	218	218	45	164
Parent
Student Participation Consent	10,713	1	10,713	10,713	3	536
Student
Directions	10,713	0.8	8,570	8,570	10	1,428
Assessment	10,713	0.8	8,570	8,570	120	17,140
Student questionnaire (main questionnaire)	10,713	0.8	8,570	8,570	35	4,999
Student questionnaire (ICT questionnaire)	10,713	0.8	8,570	8,570	15	2,142
State samples (up to 3 states)
Recruitment and Pre-Assessment Activity
School Administrator (US states)	162	1	162	162	90	243
School Coordinator (US states)	162	1	162	162	240	648
School Administrator
Questionnaire (US states)	162	1	162	162	45	122
Parent
Student Participation Consent	8,424	1	8424	8424	3	421
Student
US states (includes up to 3)
Directions	8,424	0.83	6,992	6,992	10	1,165
Assessment	8,424	0.83	6,992	6,992	120	13,984
Student questionnaire (main questionnaire)	8,424	0.83	6,992	6,992	35	4,079
Student questionnaire (ICT questionnaire)	8,424	0.83	6,992	6,992	15	1,748
Total School Recruitment Burden – Main Study			36,095	51,657		16,607
Total Burden Requested in this Submission			41,030 41,320	58,672 59,087		18,903 19,041
NOTE: OMB Clearance Requested: Total Burden includes all burden associated with conducting the PISA 2025 Field Test and the PISA 2025 Main Study. The estimated PISA 2025 Main Study burden is conservatively high because the PISA 2025 Main Study may include up to 3 states, however the burden is held consistent with national sample schools because of potential variability between states. The total student burden does not include time for cognitive assessment and its associated instructions, because assessment is not subject to PRA. *Special note: For the national main study sample, we expect to draw an initial sample of 298 schools. Taking into account closed, merged, and ineligible schools (historically, around 14% of sampled schools), as well as the historical school-level response rate, we anticipate interacting with/recruiting about 256 of these schools, of which, we estimate, 218 will participate in the PISA 2025 main study. To obtain the required number of students, we will ask to sample up to 52 students in each school. However, some of the smaller schools will not have 52 students available. We estimate: 218 schools x 52 students sampled x 0.945 (a factor used to account for variations in student population sizes across the schools) = 10,713 starting student sample size that we will work to recruit. Based on historical student assessment rates (~80%), we estimate that, in the end, we will assess about 8,570 students (10,713 x 0.80), which will assure that we meet the minimum required 6,300 assessed students.

Page 14 - The hourly rates for staff and parents have changed a result of an updated version of the source BLS Occupation Statistics wage information. The total burden time cost increases from $246,929 to $279,827.

2. Specific material changes to Part B

Section B2.a

Page 2 – The number of sampled schools changed throughout from 50 to 53 schools, as reflected in the table in section A.12.

3. Specific material changes to Appendix A-1

Pages 93-113: Revised the student video for the PISA 2025 field test

Global changes: All references to PISA 2022 were changed to PISA 2025 Field Test.

Page 93 – Decreased the length of the video from 9 minutes to approximately 6 minutes.

Page 95. Shortened the script.

PREVIOUS: What is PISA? PISA is an acronym that stands for the Program for International Student Assessment. It's the brainchild of the OECD - and what's the OECD? It's another acronym that stands for the Organization for Economic Cooperation and Development. The OECD brings together 34 countries with the aim of developing better policies for better lives.

In the late-1990s, countries that are members of the OECD came up with the idea to measure whether 15 year-olds around the world are well-prepared to participate in society. We chose 15 year-olds, rather than 12 or 17 year olds, because most 15 year olds are about to complete their compulsory education.

Experts in the field of education from around the world work together to create a two-hour test that focuses on core subjects, like reading, mathematics, and science. Participating countries decided to administer this test every three years and to rotate the main focus of the test among the three core subjects.

All very well, but testing students is nothing new, so what's so special about PISA?

PISA surveys are designed to find out whether students can use what they have learned in school and apply that knowledge to real-life situations and problems. PISA is less interested in knowing whether the students can repeat - like parrots - what they have been taught in class. Rather, the survey is designed to find out whether, for example, students can use the reading skills they have learned at school to make sense of the information they find in a book, a newspaper, on a government form, or in an instruction manual.

But the point of PISA is not to tell each individual student how well he or she has mastered a set of skills. Instead, PISA results are analyzed and extrapolated to the national level.

Picture one student sitting at a desk in a classroom somewhere taking the PISA test. Now, zoom out as though you're on the space shuttle and you can see the entire country in which that student is sitting. That's what PISA does with its test results.

REVISED:

To get started, let’s look at a brief overview of PISA. PISA is an acronym that stands for the Program for International Student Assessment. It's the brainchild of the OECD. And what's the OECD? It's another acronym that stands for the Organization for Economic Cooperation and Development.

In the late-1990s, countries that are members of the OECD came up with the idea to measure whether 15 year-olds around the world are well-prepared to participate in society. They chose 15 year-olds, rather than 12 or 17 year olds, because most 15 year olds are about to complete their compulsory education.

Experts in the field of education from around the world work together to create a test that focuses on core subjects like reading, mathematics, and science. PISA surveys are designed to find out whether students can use what they have learned in school and apply that knowledge to real-life situations and problems. PISA is less interested in knowing whether the students can repeat what they have been taught in class.

Page 96:

PREVIOUS: Adding on to what you just heard….PISA is the world’s largest international education study with more than 500,000 students participating in each cycle.

It measures how well 15-year-old students apply what they’ve learned in and out of school to real-world problems in reading, science, mathematics, and a new assessment called Learning in a Digital World.

PISA provides information about how well students in the United States are performing in comparison to students in countries around the world.

REVISED: PISA is the world’s largest international education study with more than 500,000 students participating in each cycle.

But the point of PISA is not to tell each individual student how well he or she has mastered a set of skills. Instead, PISA results are analyzed and extrapolated to the national level.

You can think of PISA as an Olympics of the mind! PISA provides information about how well students in the United States are performing in comparison to students in countries around the world.

Page 98: Changed number of students selected from 6,000 to 3,000 reflecting the size of the FT sample.

Pages 100 and 101: Split the old screen into two to lessen the reading load:

PREVIOUS:

After the main assessment, you will be asked to complete questions about your attitudes and experiences in school and your familiarity with information and communication technology. This helps policymakers learn more about students’ experiences and attitudes and how they relate to students’ level of knowledge and skills.

Because no one student takes the entire assessment, unlike many tests you take, you will not get your own score on PISA.

PISA takes place at your school. You will be excused from class during the PISA assessment.

Now, let’s take a look at an example item from previous PISA tests, and what you can expect to see on assessment day.

REVISED:

After the main assessment, you will be asked to complete questions about your attitudes and experiences in school, and your familiarity with information and communication technology. This helps policymakers learn more about students’ experiences and attitudes and how they relate to students’ level of knowledge and skills.

Because no one student takes the entire assessment, unlike many tests you take, you will not get your own score on PISA.

PISA takes place at your school. You will be excused from class during the PISA assessment.

Page 102: Deleted a screen from the released item to shorten the presentation. The screen and text now focus on the left and right panels of the item screen showing the organization of the item material.

Page 103: Deleted the NCES screens with additional released items, keeping the information on the OECD PISA site, which has more current item information.

Page 105: Revised the last screen removing ambiguous information and prompting for questions from students.

Page 106: Added a new guide for school coordinators to use in facilitating a student meeting.

4. Change from Sex to Gender for PISA 2025 Field Test

In accordance with guidance from the OECD, OMB, and NCES standards, the 2025 PISA Field Test will use the following approach to collect gender information for students.

1. Roster information collected from schools will now collect information about student gender rather than student sex.

2. Student gender is a nonbinary category, with three possible response options: Female (1), Male (2) or Nonbinary/Another Gender (3).

3. The US version of the school questionnaire (Appendix C-2, p. 7 and p. 17) asks for total school enrollment, with a breakdown for the number of boys, number of girls, and number of students who identify as nonbinary or another gender.

4. The US version of the student questionnaire (Appendix C-2, p. 7 and p. 128) asks students to supply their own gender. That question allows only a binary response, female or male.