Part B PIRLS 2021 Field Test Recruitment

Progress in International Reading Literacy Study (PIRLS 2021) FIELD TEST RECRUITMENT

OMB# 1850-0645 v.12

Supporting Statement Part B

Submitted by:

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

December 2018

revised September 2019

2. COLLECTION OF INFORMATION EMPLOYING STATISTICAL INFORMATION

B.1 Respondent Universe

The respondent universe for the PIRLS field test is all students enrolled in grade 4 that have a mean age of at least 9.5 years of age during the 2019-2020 school year. The universe for the selection of schools is all types of schools in 15 populous states. A sample of 45 schools will be selected for the field test, with the goal of obtaining participation from a minimum of 40 schools. Within sampled schools, approximately 1,650 students will be selected for participation by drawing a random sample of two classes. Only intact classrooms of grade 4 students will be assessed. School administrators and teachers of the selected classrooms will also be asked to complete questionnaires.

B.2 Statistical Methodology

Field Test Sampling Plan and Sample

The purpose of the PIRLS field test is to test out a new PIRLS delivery system, assessment items, background questionnaire (BQ) items, and to ensure that classroom and student sampling procedures proposed for the main study are successful. In selecting a school sample for this purpose, it is important to minimize the burden on schools, districts, and states, to minimize the impact on these entities while also ensuring that the field test data are collected effectively.

As required by the PIRLS International Study Center, the field test sample is to consist of at least 45 schools with approximately 1,650 students to be selected for participation from a random sample of two classes from each school, which is estimated to yield a minimum of 1,400 students to be assessed. A probability sample of schools is not required for the field test because the field test is designed only to test items, questions, and procedures. However, the sample must include a broad range of schools covering such characteristics as public (including charter schools), large, small, urban, and rural schools, and schools from a variety of different states.

The field test sample will be drawn before the main study sample, and schools will be selected for the field test from the set of schools that may be included in the main study sample, though the chances of a school being selected for both samples are minimal. We will draw the field test sample from 15 states. (Typically California, Illinois, Virginia, and Georgia are among the selected states because of the variation in size and diverse demographics.) This approach will achieve the desired distribution of schools by region, poverty level, and ethnicity, and will inform the recruitment and data collection process for the main study.

For the field test sampling frame, schools in the selected states will be stratified by state and high/low poverty¹ resulting in 30 different strata. Serpentine sorting will be used to sort schools by locale (city, suburb, town, and rural), race/ethnicity status (“15 percent or above” or “below 15 percent” Black, Hispanic, Asian and Pacific Islander, and American Indian and Alaskan Native students), and fourth grade enrollment within each stratum. A purposive sample of 45 schools will be selected for the field test that allocates equally to the separate states, although purposive selection of schools within the states may be conducted to ensure that to the extent possible, the proportion of schools in the field test closely aligns with the proportion of schools in the main study school sampling frame on the margins of the stratification and sort characteristics described previously. In addition, we will select the PIRLS field test sample so as to minimize overlap with the NAEP sample. Two replacement schools will be selected for each of the 45 sampled schools from the same strata that will have the same sort characteristics as the corresponding sampled schools. Once the field test sample has been selected, a summary of the distribution of the characteristics of the selected schools will be prepared, showing the comparison with the national population of schools.

The student sampling procedures for the field test will correspond as closely as feasible to what is planned for the main study, so as to try out the operational procedures for student sample selection. The sample will be selected by selecting two classes per school, depending on the number of classes available in grade 4. Each participating school will be asked to submit an exhaustive list of classes (that is, a list that accounts for each student in the grade exactly once). Smaller classes will be combined to form “pseudo-classes” for the purposes of sampling. Once the list of classes is submitted, we will use a sampling algorithm in the sampling software provided by the IEA to select two classes (or pseudo-classes) with equal probability. The student sample will then consist of all students in the selected classes.

We plan to gather class and student lists from participating schools electronically using a secure electronic filing process (as explained in Part A). Electronic filing provides advantageous features, such as efficiency and data quality checks. Schools will access the electronic filing system through a web site.

Main Study Sampling Plan and Sample

The school sample design for the main study must be more rigorous than that for the field test. It must be a probability sample of schools that fully represents the entire fourth grade population in the United States. At the same time, to ensure maximum participation, it must be designed to minimize overlap with other NCES studies involving student assessment that will be conducted around the same time. The main study will take place in the spring of 2021, about two months after the NAEP 2021 assessment. NAEP 2021 will assess several thousand schools nationally, at grades 4 and 8. To be fully representative, the PIRLS sample may include some schools that will have participated in the Main NAEP 2021 at grade 4. However, this number will be kept to a minimum.

In order to assess the minimum required 5,000 students from 150 schools for the digitalPIRLS plus 1,500 students from 50 schools for the bridge study paperPIRLS, we will sample 285 schools and 6,900 students. For each original sample school, two replacement schools will also be identified. The sampling frame will be obtained from the most current versions of NCES’s Common Core of Data (CCD) and Private School Survey (PSS) files, restricted to schools having grade 4, and eliminating schools in Puerto Rico, U.S. territories, and Department of Defense overseas schools. The sample will be stratified according to school characteristics such as public/private, Census region, poverty status (as measured by the percentage of students in the school receiving free or reduced-price lunch in the National School Lunch Program (NSLP)). This will ensure an appropriate representation of each type of school in the selected sample of schools. The process used to determine school eligibility, student eligibility, and student sampling is described below.

Schools will be selected with probability proportional to size (PPS) sample, where the measure of size is based on the number of estimated students at grade 4. A PPS design ensures that all students have an approximately equal chance of selection because the same sample size will be selected from each school, regardless of the size of the school. It also improves cost-efficiency by increasing the number of students per school.

Student sampling will be accomplished by selecting up to two classes per school. Each grade 4 school will be asked to prepare a list of classes that is comprehensive and includes each grade 4 student in the school in one of the listed classes. As described above, schools will submit these classes and student lists via secure E-filing. Any class with fewer than ten students will be combined with another class to form a ‘pseudo-class’ with at least ten students in it. We will then select two classes (or pseudo-classes) from each school, with equal probability (unless only one class is possible), and all students in those classes/pseudo-classes will be included in the sample. If a school has only one class, then all the students in the grade will be included in the sample.

Nonresponse Bias Analysis, Weighting, and Sampling Errors

It is inevitable that nonresponse will occur at both levels: school and student. We will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards. After the calculation of weights, sampling errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, clustering and stratification (see Appendix B for more detail).

B.3 Maximizing Response Rates

The most significant challenge in recruitment for PIRLS has been engaging the schools and gaining their cooperation. Given that classrooms are selected, student participation is not as great of a challenge. Historically student participation rates have never fallen below 90 percent (see table 1). That said, it is important to U.S. PIRLS that students are engaged and try hard on the assessment.

Table 1. Historical PIRLS school and student participation rates

Year	School Participation Rate		Overall Student Participation Rate
	Before Replacement	After Replacement
2016 PIRLS	76	92	94
2016 ePIRLS	74	89	90
2011	80	85	96
2006	57	86	95
2001	61	86	96

Our approach to school recruitment is to:

• Obtain endorsements about the value of PIRLS from relevant organizations;

• Work with NAEP state coordinators;

• Inform Chief State School Officers and test directors about the sample of schools in their state. Enclose a sample letter of endorsement they can send to schools;

• Send letters and informational materials to schools and districts. These letters will be customized by the type of school;

• Train experienced school recruiters about PIRLS;

• Implement strategies from NAEP’s Private School Recruiting Toolkit. This toolkit, developed for NAEP, includes well-honed techniques used to recruit a very challenging type of schools;

• Follow-up mailings with telephone calls to explain the study and schools’ involvement, including placing the PIRLS assessment date on school calendars;

• Offer schools $200 for participation and, as a second-tier incentive, $800 to schools that are historically very difficult to recruit (as explained in Part A);

• Maintain continued contact until schools have built a relationship with the recruiter and fully understand PIRLS;

• Offer a $100 incentive to the individual at the school identified to serve as the school coordinator-; and

• Make in-person visits to some schools, as necessary.

Our approach to student recruitment is to:

• Send parental permission forms home to parents. Implied permission is encouraged, but written permission will be collected if required by the school district or school;

• Encourage the teacher to encourage student participation;

• Offer participating students a small gift valued at approximately $4. In PIRLS 2016, each participating student received a small wristwatch and a pencil. Comparably valued items will be distributed to participating students for the PIRLS 2021 data collection;

• Students will also receive a certificate with their name thanking them for participating in and representing the U.S. in PIRLS 2021; and

• When feasible, have the test administrator speak to the students prior to the scheduled test day to encourage participation.

Our approach to teacher recruitment is to:

• Send letters and informational materials to teachers;

• Provide the option of an electronic or hard-copy questionnaire;

• Offer a $20 incentive for participation; and

• Have the test administrator speak to the teacher on the day of the student session.

B.4 Purpose of Field Test and Data Uses

The central goals for the field test are to evaluate new assessment items and background questions and to ensure that classroom and student sampling procedures proposed for the main study are successful. The U.S. will implement the PIRLS assessment and will analyze data from the field test to inform decisions for implementing the main study.

Prior to the administration of PIRLS 2021, the IEA’s long-term plan was for PIRLS to become an online assessment with completely electronic administration instead of paper delivery. PIRLS 2021 is designed to bridge PIRLS from paper-based administrations to computer-based administrations by including a bridge study as part of the main study. The primary value of the U.S. participating in PIRLS 2021 is to evaluate the feasibility and validity of fully transitioning from paper to electronic delivery in PIRLS in the context specific to the U.S.

B.5 Individuals Consulted on Study Design

Overall direction for PIRLS is provided by Dr. Sheila Thompson, National Research Coordinator, National Center for Education Statistics, U.S. Department of Education.

The following persons are responsible for the statistical design of PIRLS:

• Pierre Foy. TIMSS and PIRLS International Study Center, Boston College (617-552-6253); and

• Sylvie LaRoche and Ahmed Almaskut from Statistics Canada (613-863-9480).

Contractors responsible for sampling and data analysis are to be determined.

Analysis and reporting will be performed by:

• TIMSS and PIRLS International Study Center, Boston College;

• the U.S. national contractor, Westat; and

• National Center for Education Statistics, U.S. Department of Education.

1 High poverty schools are defined as having 50% or more students eligible for participation in the National School Lunch Program (NSLP), and low poverty schools have less than 50% of students eligible for NSLP. In addition, in the main study, which includes private schools in the sample, all private schools are classified as low poverty because no NSLP information is available