Appendix B - NRBA Plan

Progress in International Reading Literacy Study

(PIRLS 2021) Main study RECRUITMENT and field test

OMB# 1850-0645 v.15

AppendiX B:

Non-Response Bias Analysis Plan

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

November 2020

Outline of the PIRLS 2021 Non-response Bias Analyses

We will conduct a non-response bias analyses (NRBA) for schools if the response rate for original schools is below 85 percent. Currently, it is assumed that the student response rate for PIRLS 2021 will be above 85 percent and thus, no NRBA at the student level will be required.

PIRLS 2021 School Non-Response Bias Analysis Outline

If needed, the NRBA will be conducted in the winter of 2022 before the release of the PIRLS 2021 national report in June of 2024. A summary of the findings will be included in the technical appendix of the U.S. national report and the full NRBA will be included in the U.S. technical report.

1. INTRODUCTION

NCES standards for assessment surveys stipulate that a nonresponse bias analysis is required at any stage of data collection reporting a weighted unit response rate less than 85 percent. If the U.S. PIRLS weighted school response rate is below 85 percent, NCES will require an investigation into the potential magnitude of nonresponse bias at the school level in the U.S. sample.

2. METHODOLOGY

The analysis will be conducted in three parts:

1. The distribution of the participating original school sample will be compared with that of the total eligible original school sample. The original sample is the sample before substitution. In each sample, schools will be weighted by their size-adjusted school base weights, excluding any non-response adjustment factor.

2. The distribution of the participating final sample, which includes the participating substitutes for schools from the original sample that did not participate, will be compared to the total eligible final sample. The final sample is the sample after substitution. Again, size-adjusted school base weights will be used for both the eligible sample and the participating schools.

3. The same sets of schools will be compared as in the second analysis but, this time, when analyzing the participating final schools alone, school nonresponse adjustments will be applied to the size-adjusted school base weights. The total eligible final sample will again be weighted by their size-adjusted school base weights.

The following categorical variables will be available for all schools:

• School control—public or private;

• Locality—urban-centric locale code, i.e., central city, suburb, town, rural;

• Census region; and

• Poverty level—for public schools, a high poverty school is defined as one in which 50 percent or more of the students are eligible for participation in the National School Lunch Program (NSLP), and a low poverty school is defined as one in which less than 50 percent are eligible; all private schools are treated as low poverty schools.

• School size—grade 4 enrollment of school (as shown on school frame) divided into three equally sized categories (small, medium, and large).

The following continuous variables will be available for all schools:

• Number of grade-eligible (grade 4) students enrolled;

• Total number of students;

• Mean percentage of students by race/ethnicity (White non-Hispanic, Black non-Hispanic, Hispanic, Asian non-Hispanic, American Indian or Alaska Native non-Hispanic, Native Hawaiian or other Pacific Islander non-Hispanic, and two or more races).

An additional continuous variable, the percentage of students eligible to participate in the NSLP, will be available only for public schools.

Two forms of analysis will be undertaken:

• A test of the independence of each school characteristic and participation status, and

• A logistic regression in which the conditional independence of these school characteristics as predictors of participation will be examined.

For categorical variables, the distribution of frame characteristics for participants will be compared with the distribution for all eligible schools. The hypothesis of independence between the characteristic and participation status will be tested using a Rao-Scott modified Chi-square statistic at the 5 percent level (Rao and Thomas 2003). For continuous variables, summary means will be calculated and the difference between means will be tested using a t test. In addition to these tests, logistic regression models (including all characteristics) will be used to provide a multivariate analysis in which the conditional independence of these school characteristics as predictors of participation will be examined.

3. RESULTS

For each categorical or continuous variable, a table will be shown giving the percentage (or mean) for the participating and eligible populations along with the bias, relative bias, and the p-value of the test. Text summaries of the results will also be provided. The logistic regression results will be shown giving the parameter estimate, standard error, t test, and p-value. The results will be given for each analysis.

3.1 Original Respondent Sample

Categorical Variables

Continuous Variables

Logistic Regression Model

3.2 Participating Final Sample with Substitutes (Final Sample)

Categorical Variables

Continuous Variables

Logistic Regression Model

3.3 Nonresponse-adjusted Final Sample with Substitutes

Categorical Variables

Continuous Variables

4. CONCLUSIONS

A summary of the results will be presented along with a conclusion on the effect of substitutes and the non-response weighting adjustment.