Part A PIRLS 2021 Field Test Recruitment

The Progress in International Reading Literacy Study (PIRLS) is an international assessment of fourth-grade students’ achievement in reading. PIRLS reports on four benchmarks in reading achievement at grade 4 (Advanced, High, Medium, and Low) and on a variety of issues related to the education context for the students in the sample, including instructional practices, school resources, curriculum implementation, and learning supports outside of school. Since its inception in 2001, PIRLS has continued to assess students every 5 years (2001, 2006, 2011, and 2016), with the next PIRLS assessment, PIRLS 2021, being the fifth iteration of the study. Participation in this study by the United States at regular intervals provides data on student achievement and on current and past education policies and a comparison of U.S. education policies and student performance with those of the U.S. international counterparts. In PIRLS 2016, 58 education systems participated. The U.S. will participate in PIRLS 2021 to continue to monitor the progress of its students compared to that of other nations and to provide data on factors that may influence student achievement.

PIRLS is coordinated by the International Association for the Evaluation of Educational Achievement (IEA), an international collective of research organizations and government agencies that create the assessment framework, the assessment instrument, and background questionnaires. The IEA decides and agrees upon a common set of standards and procedures for collecting and reporting PIRLS data, and defines the studies’ timeline, all of which must be followed by all participating countries. As a result, PIRLS is able to provide a reliable and comparable measure of student skills in participating countries. In the U.S., the National Center for Education Statistics (NCES) conducts this study, with support from U.S. Department of Education contractors, and works with the IEA to ensure proper implementation of the study and adoption of practices in adherence to the IEA’s standards. Participation in PIRLS allows NCES to meet its mandate of acquiring and disseminating data on educational activities and student achievement in the U.S. compared with foreign nations [The Educational Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., §9543].

Compared to PIRLS 2016, PIRLS 2021 differs in that it offers a redesigned digital format, referred to as digitalPIRLS to distinguish it from the traditional paper-and-pencil format, called paperPIRLS, which will still be available to countries. As the U.S. will be administering the digital version of PIRLS, in this submission, all discussion of PIRLS 2021 refers to digitalPIRLS unless otherwise specified. The other difference to note is that, in an effort to reduce burden on schools and students, the assessment will be administered for one day only (rather than over two days as was done in 2016 when ePIRLS was administered for the first time independently). Given the transition to a digitally based assessment (DBA) for PIRLS, a bridge study between the two PIRLS formats will be conducted by the IEA during the main study. This design will require that approximately 1,500 students will be administered the paperPIRLS booklets, which will increase the sample size required during the main study.

In preparation for the PIRLS 2021 main study, all countries are asked to implement a field test in 2020. The purpose of the PIRLS field test is to evaluate new assessment items and background questions, to ensure practices that promote low exclusion rates, and to ensure that classroom and student sampling procedures proposed for the main study are successful. In selecting a school sample for this purpose, it is important to minimize the burden on schools, districts, and states, while also ensuring that the field test data are collected effectively. PIRLS staff will work to help respondents understand the study’s value relative to burden imposed, and to ensure a high level of school participation.

Data collection for the field test in the U.S. will occur from March through April 2020. The U.S. plans to recruit sampled public and private schools and assess about 1,000 students. The student samples will be obtained by selecting two classes from each school. The U.S. PIRLS main study will be conducted March through June 2021, and will involve a nationally-representative sample of approximately 6,500 students in the target population from 200 schools.

This submission describes the overarching plan for all phases of the data collection, including the 2021 main study.¹ In addition to the supporting statements Parts A and B, Appendix A provides field test recruitment materials consisting of letters to state and district officials and school principals, text for a PIRLS field test brochure, “Frequently Asked Questions,” a “Summary of Activities”, and parental consent materials. Appendix B provides the non-response bias analysis plan, and Appendix C provides the PIRLS 2016 background questionnaires which will be provided as examples during recruitment per district or school request.

Because PIRLS is a collaborative effort among many parties, the U.S. must adhere to the international schedule set forth by the IEA, including the availability of draft and final questionnaires. In order to meet the international data collection schedule for the spring 2020 field test, recruitment activities are scheduled to begin in May 2019. Recruitment for the main study will begin in March of 2020 to align with recruitment for other NCES studies [e.g., the National Assessment of Education Progress (NAEP)], and for schools to put the assessment on their calendars. We expect the main study materials and procedures to be very similar to those used in the field test. However, as NCES does not yet have a national contractor and the international contractor has not yet issued guidance on PIRLS 2021 sampling or draft instruments, this submission requests approval only for the PIRLS 2020 field test recruitment. A second submission will follow in July 2019, with a 30-day public comment period, to request approval for PIRLS 2020 field test data collection and PIRLS 2021 main study recruitment. Additionally, should there be any revisions to recruitment materials or changes from the past field test sampling requirements, we will submit to OMB a change request with the final materials and recruitment details in early 2019.

A. Justification

A.1 Importance of Information

Benchmarking of U.S. student achievement against other countries continues to be of high interest to education policymakers, and informs policy discussions of economic competitiveness and workforce and post-secondary preparedness. PIRLS provides a unique opportunity to compare U.S. students’ reading knowledge and skills at fourth grade with that of their peers in countries around the world. The continuation of U.S. participation allows for the study of past and current education policies that have shaped reading achievement over the past 20 years. Furthermore, participating countries are not only able to obtain information about students' knowledge and abilities, but also about the cultural environments, teaching practices, curriculum goals, and institutional arrangements that are associated with student achievement.

PIRLS complements what we learn from national assessments such as the National Assessment of Educational Progress (NAEP) by identifying the strengths and weaknesses of student reading achievement relative to participating countries around the world. It provides valuable benchmarking information about educational polices enacted in other countries and policies that could be applied to U.S. educational practices.

Based on earlier PIRLS data releases, it is likely that the results of these studies will draw great attention in the U.S. and elsewhere. It is therefore expected that PIRLS will contribute to ongoing national and international debates and efforts to improve reading learning and achievement.

A.2 Purposes and Uses of Data

PIRLS assesses reading knowledge and skills at grade 4. PIRLS is designed to align broadly with curricula in the participating countries. The results, therefore, suggest the degree to which students have learned concepts and skills likely to have been taught in school. PIRLS also collects background information on students, teachers, schools, curricula, and official education policies in order to allow cross-national comparison of educational contexts that may be related to student achievement.

Data compiled and collected from PIRLS 2021 allows for evidence-based decisions to be made for the purposes of educational improvement. Each successive participation in PIRLS provides trend information about student achievement in reading relative to other countries, as well as indicators that show how this achievement relates to demographic and curricular, school, teacher, and student factors that provide the educational context for achievement. These high-quality, internationally comparative trend data are key in informing education policy discussions.

Through the participation in PIRLS and other international assessment programs, NCES is able to provide comparative indicators on student performance and school practices across countries in order to benchmark U.S. student performance, and to suggest hypotheses about the relationship between student performance and factors that may influence performance as well as areas in which students have strengths or weaknesses. The international studies identify differences among countries over time in instructional practices, school policies, and opportunity-to-learn that can inform discussions about how to improve students’ ability to read.

This collection of data is consistent with the NCES mandate, which specifies that "The purpose of the Center [NCES] shall be to collect and analyze and disseminate statistics and other information related to education in the U.S. and in other nations." [§406 of the General Education Provisions Act, as amended (20 U.S.C. 1221e-1)]. ESRA 2002 (20 U.S.C., §9543) also mandates that NCES shall collect, report, analyze, and disseminate statistical data related to education in the U.S. and in other nations, including acquiring and disseminating data on educational activities and student achievement in the U.S. compared with foreign nations.

PIRLS 2021 Components

The reading assessment is organized around a content dimension that specifies the subject matter to be assessed and a cognitive dimension that specifies the thinking processes to be assessed. PIRLS assesses two purposes for reading that fourth-grade students typically engage in: reading for literary experience and reading to acquire and use information. PIRLS also assesses four broad processes of comprehension predominantly used by fourth-grade readers: focusing on and retrieving explicitly stated information; making straightforward inferences; interpreting and integrating ideas and information; and evaluating and critiquing content and textual elements. The PIRLS 2021 framework is similar to 2016, but has been slightly updated to provide more specificity for item writers, and to better reflect current curricula in participating countries. There were no revisions to the content domains or cognitive domains, nor were there changes to the target percentages for either domain.

Assessment Instruments

To minimize burden and ensure broad subject-matter coverage, PIRLS will use a matrix sampling approach where the reading items are organized into sets for administration. For the bridge study to be conducted during the main study, the paper-and-pencil reading test will also use a matrix sampling approach with the items organized in a set of test booklets, with each student receiving only one booklet. This design is consistent with past practice.

Questionnaires

The background questionnaires for PIRLS 2021 were developed to address the issues outlined in the PIRLS context questionnaire framework. The U.S. is adapting the questions to fit the U.S. education context, as appropriate, including adding a few questions, such as about the race/ethnicity of students.

School Questionnaire. A representative from each participating school will be asked to provide information on reading resources, teacher availability and retention, principal leadership, school emphasis on academic success, school climate, and parental involvement in school activities. These questionnaires will be offered online, with a paper-and-pencil backup.

Teacher Questionnaire. Reading teachers of students in selected classes will be asked to complete a teacher questionnaire, which will include questions about teacher preparation and experience, reading instruction, instructional resources and technology, instructional time, instructional engagement, and classroom assessment. These questionnaires will be offered online, with a paper-and-pencil backup.

Student Questionnaire. Student information will be collected about home resources, motivation, self-concept, self-efficacy, and student characteristics such as gender and race/ethnicity. It should be administered to all students who have received parental permission to participate in PIRLS. Students will only receive a paper-and-pencil questionnaire.

A.3 Improved Information Technology (Reduction of Burden)

Each participating nation is expected to adhere to the internationally prescribed design. In the U.S., the school and teacher questionnaires will be made available to school administrators and teachers online as the main mode of administration, with a paper-and-pencil backup to facilitate user preference for participation. The online questionnaires will be provided on the secure NCES server, so that NCES will be able to control access to the data to ensure confidentiality and minimize disclosure risk.

A communication website will be used for PIRLS 2021 during the field test and main study in order to provide a simple, single source of information to engage and maintain high levels of school involvement. This portal will be used throughout the assessment cycle to inform schools of their tasks and to provide them with easy access to information tailored for their anticipated needs. We plan to gather class and student lists from participating schools electronically using a secure electronic filing process. Electronic filing is an electronic system for submitting lists of student information, including student background information in school records. The electronic filing system provides advantageous features such as efficiency and data quality checks.

A.4 Efforts to Identify Duplication

In the U.S., reading achievement is systematically assessed at (1) the Federal level, where trend data have been collected on a fairly regular basis since 1971 through the National Assessment of Educational Progress (NAEP); (2) the state level, where data are routinely collected as part of state testing programs, though they vary across the states in terms of the frequency of testing, age/grades tested, and types of cognitive items administered; and (3) the district level, where data are collected through the use of commercially or locally developed standardized tests as well as tests developed in conjunction with the instructional programs used in schools. PIRLS 2021 does not duplicate these assessments.

PIRLS 2021 is part of a program of international cooperative studies of educational achievement supported and funded, in part, by the U.S. Department of Education. These studies represent the U.S. participation in international studies involving 55 to over 70 countries each. As part of international cooperative studies, the U.S. must collect the same information at the same time as the other nations for purposes of making both valid international comparisons with other countries and with the previous PIRLS data. While some studies in the U.S. collect similar, though not identical, kinds of information (e.g., NAEP), the data from those studies cannot be substituted for the information collected in PIRLS in that they do not allow for comparisons outside the U.S. Furthermore, the data collected through PIRLS is based on a unique framework that is not shared by any other state, national, or international data collection effort. In order to participate in these international studies, the U.S. must agree to administer the same core instruments that are administered in the other countries. Because the items measuring reading achievement have been developed with intensive international coordination, any changes to the instruments require international coordination and approval.

A.5 Minimizing Burden for Small Entities

The school samples for PIRLS contain small-, medium- and large-size schools, including private schools, selected based on probability proportionate to their size. All school sizes are needed to ensure an appropriate representation of each type of school in the selected sample of schools. Burden will be minimized wherever possible. In addition, staff will conduct all test administrations, and will assist with parental notification, sampling, and other tasks as much as possible within each school. The assessment will be administered to intact classes to minimize disruption to school schedules.

A.6 Frequency of Data Collection

The field test data collection is scheduled for March 1 through April 15, 2020, and the main study data collection for February through May, 2021. This schedule is prescribed by the international collective for PIRLS, and adherence to this schedule is necessary to establish consistency in survey operations among participating countries as well as to maintain trend lines.

A.7 Special Circumstances

None of the special circumstances identified in the Instructions for Supporting Statement apply to the PIRLS study.

A.8 Consultations outside NCES

Consultations outside NCES have been extensive and will continue throughout the life of the project. The IEA studies are developed as a cooperative enterprise involving all participating countries. An international panel of reading and measurement experts provide substantive and technical guidance for the study and National Research Coordinators participate in extensive discussions concerning the projects, usually with advice from national subject matter and testing experts. In addition, the IEA convened a panel of reading experts from around the world to develop cognitive items.

The majority of the consultations (outside NCES) have involved the TIMSS & PIRLS International Study Center at Boston College in the U.S. Key to these ongoing consultations are: Dirk Hastedt (Executive Director of the IEA), Michael Martin, Ina V.S. Mullis, and Pierre Foy, all of whom have extensive experience in developing and operating international education surveys (especially related to PIRLS).

A.9 Payments or Gifts to Respondents

In order to achieve acceptable school response rates, schools have historically been offered incentives to participate and as a thank you for the time they invest in and the space they make available for the international assessments. High response rates are required by both IEA and NCES, and are difficult to achieve in school-based studies. The U.S. has historically had difficulties in achieving sufficient participation levels. As in PIRLS 2016, in PIRLS 2021 schools will be offered $200 for participation and, to address challenges that may be encountered with securing school cooperation, we will utilize a second-tier incentive that will allow us to offer up to $800 to schools that are historically very difficult to recruit. This second-tier incentive will be offered only to:

Private schools in the original sample or substitute private schools (e.g., in ICILS 2018, only 40% of original private schools participated, with the overall rate of about 57% after substitutes were added).
Public schools that are selected for and participate in NAEP 2021 and are also selected for PIRLS 2021 (typically these schools participate only in NAEP because it is required, and refuse participation in PIRLS; there are about 40 such schools nationwide).
All substitute schools (typically, refusals by original schools occur in the mid to late fall prior to the assessment in the spring, at which point substitute schools are recruited; recruitment of substitutes is typically very challenging (less than 25% participate) because recruitment is late, after the school year has begun and school calendars are set; also, some states refuse to recruit schools this late in the school year).
Refusing original school if necessary to meet international participation requirements.

A similar second-tier recruitment strategy has been used in other international studies conducted by NCES. Most recently, in ICILS 2018, we began offering the second-tier incentive very late, in the middle of data collection, and attempted to turn around schools that had already refused, as well as newly activated substitutes and schools that were not doing any work to prepare for the assessment. We were able to successfully recruit about 20% of these schools even with this late start, which brought us much closer to our target recruitment percentage. We expect the extra incentive strategy to be more successful in PIRLS if we begin it during the fall, before data collection begins.

The school staff serving as School Coordinators will receive $100 for their time and effort in coordinating PIRLS assessment, plus $50 for running the PIRLS system check, and assisting with computer setup (these components may be delegated to a school IT coordinator if necessary). The School Coordinator serves a critical role in data collection, functioning as the central school contact, and facilitating arrangements for the assessments. They are asked to file class and student listing forms; arrange the date, time, and space for the assessment; and disseminate information to parents and students.

Consistent with prior administrations of PIRLS, as a token of appreciation for their participation, the students will receive a small gift valued at approximately $4. In PIRLS 2016, each participating student received a small sports wrist watch; a pencil imprinted with “USA”; and a certificate with their name, thanking them for participating in and representing the Unites States in PIRLS 2021. Similarly priced items will be distributed to participating students for the PIRLS 2021 data collection. Some schools also offer recognition parties with pizza or other treats for students who participate; however these are not reimbursed by NCES or the contractor.

Teachers will be offered $20 for completing the PIRLS teacher questionnaire. Historically, participation is high among school administrators without offering incentives; therefore, no incentive will be offered for completion of the school administrator questionnaire.

A.10 Assurance of Confidentiality

Data security and confidentiality protection procedures have been put in place for PIRLS to ensure that the PIRLS contractor for the U.S. and its subcontractors comply with all privacy requirements, including:

The statement of work of PIRLS contract;
Privacy Act of 1974 (5 U.S.C. §552a);
Family Educational and Privacy Act (FERPA) of 1974 (20 U.S.C. §1232(g));
Privacy Act Regulations (34 CFR Part 5b);
Computer Security Act of 1987;
U.S.A. Patriot Act of 2001 (P.L. 107-56);
Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9573);
Confidential Information Protect and Statistical Efficiency Act of 2002;
E-Government Act of 2002, Title V, Subtitle A;
Cybersecurity Enhancement Act of 2015 (6 U.S.C. §151);
The U.S. Department of Education General Handbook for Information Technology Security General Support Systems and Major Applications Inventory Procedures (March 2005);
The U.S. Department of Education Incident Handling Procedures (February 2009);
The U.S. Department of Education, ACS Directive OM: 5-101, Contractor Employee Personnel Security Screenings;
NCES Statistical Standards; and
All new legislation that impacts the data collected through the inter-agency agreement for this study.

Furthermore, the contractor will comply with the Department’s IT security policy requirements as set forth in the Handbook for Information Assurance Security Policy and related procedures and guidance, as well as IT security requirements in the Federal Information Security Management Act (FISMA), Federal Information Processing Standards (FIPS) publications, Office of Management and Budget (OMB) Circulars, and the National Institute of Standards and Technology (NIST) standards and guidance. All data products and publications will also adhere to the revised NCES Statistical Standards, as described at the website: http://nces.ed.gov/statprog/2012/.

The laws pertaining to the collection and use of personally identifiable information are clearly communicated in correspondence with states, districts, schools, teachers, students, and parents. Letters and information materials will be sent to parents and school administrators describing the study, its voluntary nature, and the extent to which respondents and their responses will be kept confidential (see copies in appendix A).

Letters to teachers, school coordinators, and supporting materials will read:

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Letters to states, districts, and schools and parent notification letters and supporting materials will read:

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information provided by school staff and students may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

The following statement will appear on the login page for PIRLS and the front cover of the printed questionnaires (the phrase “search existing data resources, gather the data needed” will not be included on the student questionnaire):

The National Center for Education Statistics (NCES), within the U.S. Department of Education, conducts PIRLS in the United States as authorized by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0645. The time required to complete this information collection is estimated to average [XX] minutes per [respondent type], including the time to review instructions [, search existing data resources, gather the data needed,] and complete and review the information collection. If you have any comments or concerns regarding the accuracy of the time estimate(s), suggestions for improving the form, or questions about the status of your individual submission of this form, write directly to: Progress in International Reading Literacy Study (PIRLS), National Center for Education Statistics, Potomac Center Plaza, 550 12^th Street, SW, 4^th floor, Washington, DC 20202.

OMB No. 1850-0645, Approval Expires xx/xx/2022.

The PIRLS 2021 confidentiality plan includes signing confidentiality agreements and notarized nondisclosure affidavits by all contractor and subcontractor personnel and field workers who will have access to individual identifiers. Also included in the plan is personnel training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses; controlled and protected access to computer files under the control of a single data base manager; built-in safeguards concerning status monitoring and receipt control systems; and a secured and operator-manned in-house computing facility. Data files, accompanying software, and documentation will be delivered to NCES at the end of the project. Neither names nor addresses will be included on any data file. In ePIRLS, students log into the automated assessment using non-identifying ID’s and thus the resulting data are collected and stored with only the non-identifying PIRLS assigned ID.

NCES understands the legal and ethical need to protect the privacy of the PIRLS respondents and has extensive experience in developing data files for release that meet the government’s requirements to protect individually identifiable data from disclosure. The contractor will conduct a thorough disclosure analysis of the PIRLS 2021 data when preparing the data files for use by researchers, in compliance with ESRA 2002 (20 U.S.C. §9573). Schools with high disclosure risk will be identified and, to ensure that individuals may not be identified from the data files, a variety of masking strategies will be used, including swapping data and omitting key identification variables (i.e., school name and address) from both the public- and restricted-use files (though the restricted-use file will include an NCES school ID that can be linked to other NCES databases to identify a school); omitting key identification variables such as state or ZIP Code from the public-use file; and collapsing or developing categories for continuous variables to retain information for analytic purposes while preserving confidentiality in public-use files.

A.11 Sensitive Questions

The questionnaires do not have items considered to be of sensitive nature.

A.12 Estimates of Burden

This package shows estimated burden to respondents for all PIRLS 2021 activities, and requests approval for burden to respondents for field test recruitment (demarcated in black font in Table A.1). The minimum sample size for the field study is 25 schools and 800 students and for the main study 150 schools and 4,000 students. The burden table assumes exceeding the minimum requirements and is based on a target yield of approximately 1,000 students in the field test, 5,000 students in the main study, and 1,500 in the bridge study. The time required for students to respond to the assessment (cognitive items) portion of the study and associated directions are shown in gray italicized font and are not included in the totals because they are not subject to the PRA. The burden estimates also include burden for (1) contacting states, districts, schools, and parents in order to recruit for the PIRLS field test and main study, including (a) sending recruitment letters to districts and schools selected for each study, (b) contacting and seeking research approvals from special handling districts, where applicable, and (c) notifying parents of sampled students about their participation in either study; and (2) collecting data. Burden estimates for the field test data collection and all aspects of the main study data are not requested at this time and are provided in table A.1 for information purposes.

The district and school contact letters for the field test and main study are assumed to impose small burden on all contacted parties, both those that refuse and those that agree to participate in the PIRLS studies. Thus the burden hours for this activity are based on the total number of districts and schools contacted rather than the total number agreeing to participate.

The special handling districts are those known to require completion of a research application before they will allow schools under their jurisdiction to participate in a study. Based on an initial assessment of previous PIRLS data collections, we estimate that there may be up to 10 special handling districts in the sample. Contacting special handling districts begins with updating district information based on what can be gleaned from online sources. Calls are then placed to verify the information about where to send the completed required research application forms, and, if necessary, to collect contact information for this process. During the call, inquiry is also made about the amount of time the districts spend reviewing similar research applications. The estimated number of such districts represents those with particularly detailed application forms and lengthy processes for approval. This operation should begin in the spring of the year preceding the start of the data collection to allow sufficient time for special handling districts’ review processes. We will continue to work with these districts until we receive a final response from each district (approval or denial of request) up until March 1, 2020 for the field test and March 1, 2021 for the main study.

The total district and school response burden estimate for the main study recruitment and field test is based on 10 minutes for districts to read materials and respond, and 20 minutes for schools to read materials and respond. The total response burden estimate for IRB approvals is based on 120 minutes for staff approval and 60 minutes for panel approval. The total response burden for parental notification is based on 10 minutes for reading and reviewing recruitment materials and notification forms. The total response burden for the field test data collection is based on 35 minutes for students and 40 minutes for teachers and a school administrator to complete their respective questionnaires.

Please note that the questionnaire burden for students in Table A.1. is estimated at 50 minutes because experience has shown that administering the 35-minute student questionnaire after the assessment often takes longer due to time for bathroom breaks after the assessment and getting the class of students resettled to focus on the questionnaire.

Based on the estimated hourly rates for principals/administrators, school coordinators, teachers, and parents of $46.85, $29.25, $29.25, and $24.34, respectively^², and the federal minimum wage of $7.25 as the hourly rate for the students, and based on the estimated total of 295 burden hours for PIRLS field test recruitment, the associated estimated respondent burden time cost is $8,554.

Table A.1. Burden estimates for PIRLS 2021 Field Test and Main Study.

Activity	Sample size	Expected response rate	Number of respondents	Number of responses	Per respondent (minutes)	Total burden (hours)
Field Test Recruitment
Contacting Districts	40	1.00	40	40	10	7
Contacting Schools	40	1.00	40	40	20	14
District IRB Staff Study Approval	5	1.00	5	5	120	10
District IRB Panel Study Approval	30	1.00	30	30	60	30
Parental notification	1,400	1.00	1,400	1,400	10	234
Total Field Test Recruitment			1,515	1,515		295
Field Test Data Collection
Student
Assessment	1,100	0.93	1,023	1,023	160	2,728
Directions	1,100	0.93	1,023	1,023	10	171
Student Questionnaire	1,100	0.93	1,023	1,023	50	853
School Staff
School Administrator	40	0.95	38	38	40	26
Teacher (2 per school)	80	0.95	76	76	40	51
School Coordinator	40	1.00	40	40	240	160
Total Field Test Data Collection			1,177	3,223		3,989
Total Field Test			2,692	4,738		4,284
Main Study Recruitment
Contacting Districts	215	1.00	215	215	10	36
School Recruitment (Original Schools)	215	0.70	151	151	20	51
School Recruitment (Replacement Schools)	70	0.50	35	35	20	12
District IRB Staff Study Approval	30	1.00	30	30	120	60
District IRB Panel Study Approval	180	1.00	180	180	60	180
Parental notification	10,306	1.00	10,306	10,306	10	1,718
Total Main Study Recruitment			10,917	10,917		2,057
Main Study Data Collection
Student
Assessment	5,300	0.96	5,088	5,088	160	13,568
Bridge study paperPIRLS assessment	1,600	0.96	1,536	1,536	160	4,096
Directions	6,900	0.96	6,624	6,624	10	1,104
Student Questionnaire	6,900	0.96	6,624	6,624	50	5,520
School Staff
School Administrator	215	0.95	205	205	40	137
Teacher (2 per school)	400	0.95	380	380	40	254
School Coordinator	215	1.00	215	215	300	1,075
Total Main Study Data Collection			7,424	20,672		25,754
Total Main Study			18,341	31,589		27,811
Total requested in this submission			1,515	1,515		295

Note: Burden in black font represents the activities for which burden approval is sought in this submission. Student burden during data collection has been inflated to account for the total time students might be away from their classrooms (including getting seated, taking the assessment, and taking bathroom breaks) based on prior data collection experience. Burden for the activities shown in gray font will be requested in future submissions. Burden for student assessments is shown italicized font and are not included in the burden total because cognitive assessments are not subject to PRA. The totals in the “Number of respondents” column do not include duplicate counts of respondents.

A.13 Total Annual Cost Burden

No cost to respondents is anticipated beyond the estimated burden cost described in Section A.12.

A.14 Annualized Cost to Federal Government

The total cost to the federal government for conducting PIRLS 2021 full scale has not been determined as the national contract for this study has not yet been awarded. However, based on past administrations, the cost for the field test is estimated to be about $1,873,000. All direct and indirect costs for this study are valued at an estimated $4,000,000 over five years, from March 2019 to March 2024. The final cost details and totals will be provided in this section once available.

A.15 Program Changes or Adjustments

With regards to content, PIRLS 2021 differs from PIRLS 2016 in that PIRLS 2021 will be administered as a DBA (digitalPIRLS). Additionally, for the main study, some students will be administered digitalPIRLS and some students will be administered the booklet form of paperPIRLS for the purpose of the bridge study. Furthermore, PIRLS 2021 will be administered to students on one day, rather than on two as was done in 2016. Because this is a reinstatement of a previously approved collection, all requested burden reflects a burden increase.

A.16 Plans for Tabulation and Publication

The PIRLS field test is designed to provide a statistical review of the performance of items on the cognitive assessment and questionnaires in preparation for the main study data collection.

Based on the data collected in the main study, the TIMSS & PIRLS International Study Center will prepare a report to be released in December 2021. As has been customary, NCES plans to also release a report at the same time as the international report is released, interpreting the results for the U.S. audience. NCES reports on initial data releases are generally limited to simple bivariate statistics. There are currently no plans to conduct complex statistical analyses of PIRLS 2021 dataset. An example of the past PIRLS report can be found at https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2018017. In the spring of 2022, the International Study Center will also prepare a technical report, describing the design and development of the assessment as well as the scaling procedures, weighting procedures, missing value imputation, and analyses. After the release of the international data, NCES plans to release the national data and an accompanying User’s Guide for the study.

Electronic versions of each publication are made available on the NCES website. Schedules for tabulation and publication of PIRLS 2021 results in the U.S. are dependent upon receiving data files from the international sponsoring organization. With this in mind, the expected data collection dates and a tentative reporting schedule are as follows:

Dates	Activity
April—December 2019	Prepare data collection manuals, forms, assessment materials, questionnaires
May 2019—February 2020	Contact and gain cooperation of states, districts, and schools for field test
February 2020—March 2020	Select student samples
March 2020—April 2020	Collect field test data
May 15, 2020	Deliver raw data to international sponsoring organization
July 2020—August 2020	Review field test results
March 2020—February 2021	Prepare for the main study/recruit schools
February 2021—June 2021	Collect main study data
June 2021	Receive final data files from international sponsors
June 2021—December 2021	Produce report

A.17 Display OMB Expiration Date

The OMB expiration date will be displayed on all data collection materials.

A.18 Exceptions to Certification Statement

No exceptions to the certifications are requested.

1 The materials that will be used in the 2021 main study will be based on the field test materials. Additionally, this submission is designed to adequately justify the need for and overall practical utility of the full study and to present the overarching plan for all of the phases of the data collection, providing as much detail about the measures to be used as is available at the time of this submission. As part of this submission, NCES is publishing a notice in the Federal Register allowing first a 60- and then a 30-day public comment period. For future submission covering the details of the field test data collection and all aspects of the main study, NCES will publish a notice in the Federal Register allowing an additional 30-day public comment period.

2 The average hourly earnings of principals/education administrators in the May 2017 National Occupational and Employment Wage Estimates sponsored by the Bureau of Labor Statistics (BLS) is $46.85, for school coordinators and teachers is $29.25, and of parents is $24.34. Where mean hourly wage was not provided, it was computed from the mean annual wage assuming 2,080 hours per year. The student wage is based on the federal minimum wage. Source: BLS Occupation Employment Statistics, http://data.bls.gov/oes/ data type: Occupation codes: Education Administrators, Elementary and Secondary Schools (11-9032), Education School Teachers, Except Special Education (25-2021), and all employees (00-0000); accessed on December 7, 2018.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Calvin Choi
File Modified	0000-00-00
File Created	2021-01-20

Part A PIRLS 2021 Field Test Recruitment

Part A PIRLS 2021 Field Test Recruitment.docx

Progress in International Reading Literacy Study (PIRLS 2021) Field Test Recruitment

OMB: 1850-0645

TABLE OF CONTENTS