Volume I NAEP 2015 Wave 3 Rev

This document contains supplemental information pertaining to the 2014-2016 National Assessment of Educational Progress (NAEP) clearance proposal. NAEP is administered by the National Center for Education Statistics (NCES) in the Institute for Education Sciences of the U.S. Department of Education. The National Assessment Governing Board (henceforth referred to as the Governing Board) sets policy for NAEP and determines the content framework for each assessment. The system clearance was approved in March 2013 (OMB# 1850-0790 v.36), and the terms of clearance state that NCES will publish a notice in the Federal Register allowing a 30-day public comment period on the details of each collection concurrent with submission to OMB.

This third wave of materials includes two special studies that will be administered as part of the 2015 main NAEP administration.¹ These two special studies are:

NAEP – Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011) Link Study, and

Computer Familiarity Study

Both of these special studies include the administration of an additional student questionnaire to a sub-sample of students participating in the 2015 main NAEP administration. Questions for both special studies have been tested through cognitive interviews and revised based on the interview results. See appendices B (NAEP-ECLS-K:2011 Link Study) and C (Computer Familiarity Study) for cognitive interview reports.

Wave 3 contains the descriptions, burden, and questionnaires for these two special studies. As these supplemental questionnaires will be administered as part of the 2015 main NAEP administration, the sampling and administration plans described in Waves 1 and 2 are also pertinent to these special studies.

2 . Overview of 2015 Special Studies in Wave 3

NAEP – ECLS-K:2011 Link

The ECLS-K:2011 is a longitudinal study conducted by NCES that provides descriptive information on children’s status at entry to school, their transition into school, and their progression through the elementary grades. Amongst other research questions, the ECLS-K:2011 can be used to investigate the relationships between a child’s academic performance and social development, as the ECLS-K:2011 provides both direct and indirect assessments of students’ academic achievement and these measures can be used to quantify growth in reading, mathematics, and science. NAEP, in contrast, is a large-scale, cross-sectional, assessment survey of academic achievement of American public- and private-school students. In 2015, both assessments will be administered to fourth-graders, and thus, in order to expand the usefulness of both datasets, NCES plans to link the two assessments.

We anticipate that in the main NAEP 2015 sample, approximately 300 grade 4 ECLS-K:2011 students will be naturally sampled for each of the following assessments: operational math, reading, and science (900 students total). In the schools where ECLS-K:2011 students are sampled, we will expand the student sample to include all other ECLS-K:2011 students (estimated at approximately 700 additional students). All of the supplemental ECLS-K:2011 students will participate in NAEP reading. Both the naturally occurring and the supplemental ECLS-K:2011 students will contribute to both operational NAEP and the NAEP – ECLS-K:2011 link analyses.

At the conclusion of the NAEP assessment, all ECLS-K:2011 students will be given an additional questionnaire comprised of SES (socioeconomic status)-related questions² (see Volume II of this submission). The student responses to the SES-related questions will be compared to parents’ responses on similar questions given as part of the ECLS-K:2011 data collection. As such, NAEP will have the opportunity to examine the accuracy of grade 4 students’ responses. The additional NAEP SES questionnaire will require approximately 5-10 minutes of additional student assessment time. Since almost all selected students will complete this questionnaire, we expect the sample size to be approximately 1,600 students.

Computer Familiarity Study

As NAEP transitions from paper-and-pencil to technology-based assessments (TBA), an area of desired research involves the degree to which all children are ready for such a transition. Do all students have the same access and experience with the technologies that will be used to collect the data (computers and tablets)? What is the relationship between access and experience with these technologies and performance on NAEP assessments?

A special study is being developed by the NAEP Validity Studies panel (see appendix A for list of members) to gain information regarding familiarity with these technologies. The study, which will be undertaken as part of the 2015 NAEP reading, mathematics, and science assessments, will analyze a core set of items to measure access to and familiarity with technology-based equipment that has been used by NAEP or might be used in future NAEP assessments. The goal is to build reliable composites that measure technology access and familiarity and then to analyze these measures in relation to NAEP mathematics, reading, and science performance. The study contains a supplemental survey questionnaire related to computer familiarity and access (see Volume II³) in both paper-and-pencil and TBA versions. The questionnaire will require approximately 10 minutes at grade 4, and 15 minutes at grades 8 and 12.

A subset of 50 paper-and-pencil administration schools and 50 TBA schools participating in the 2015 main NAEP administration will participate in this study. This subset of schools will be drawn so that there is not overlap with the subset participating in the NAEP – ECLS-K:2011 link study discussed above. All students in this subset of schools will be given the Computer Familiarity Study questionnaires. The paper-and-pencil schools will yield responses from approximately 4,000 students per grade, and the TBA schools will yield approximately 2,000 students per grade. Thus, each grade will yield responses from 6,000 students, for a total of 18,000 student respondents.

3. 2015 Wave 3 Burden Information

The burden for the Wave 3 materials is shown in Table 1.

Table 1. Burden for NAEP – ECLS-K:2011 Link and Computer Familiarity Studies

Study	Number of Respondents*	Time per Respondent	Burden Hours
NAEP – ECLS-K:2011	1,600	10 minutes	267
Computer Familiarity Study	Grade 4 - 6,000	Grade 4 - 10 minutes	4,000
	Grade 8 - 6,000	Grade 8 – 15 minutes
	Grade 12 - 6,000	Grade 12 - 15 minutes
Overall Totals	19,600*		4,267

* Note: The students in these studies have already been accounted for in the Waves 1 and 2 counts. However, the additional responses and burden hours for these supplemental questionnaires were not part of the Waves 1 and 2 submissions.

4. Consultations Outside the Agency

Educational Testing Service (ETS)

ETS serves as the NAEP Item Development (ID) contractor and as the Design, Analysis, and Reporting (DAR) contractor. ETS staff will be involved in item development and analysis activities for the special studies questionnaire data.

Westat

Westat is the Sampling and Data Collection (SDC) contractor for NAEP. Westat will administer the special study questionnaires for the NAEP – ECLS-K:2011 link and the Computer Familiarity studies.

American Institutes for Research (AIR)

AIR is a not-for-profit research organization that has maintained the NAEP Validity Studies (NVS) panel under contract to NCES since 1995. The Computer Familiarity Study described here was commissioned by the NVS panel on behalf of NCES. As such, AIR’s role includes the development and analysis of the items used in the study.

Pearson is the NAEP Materials, Distribution, Processing, and Scoring (MDPS) contractor and will be involved in the printing and processing of the paper-and-pencil materials for the study.

Fulcrum IT LLC (Fulcrum IT)

Fulcrum IT is the NAEP contractor responsible for the development and ongoing support of NAEP computer-based assessments for NCES. Fulcrum will be involved in the collection and processing of TBA data related to the studies.

5. Estimates of Costs to the Federal Government

Table 2 provides estimates for the Wave 3 portion of the 2015 administrations.

Table 2: Administration Cost Estimates (Wave 3 Studies)

Component	Provider	Estimated Cost
Item development, data analysis, reporting, coordination costs	ETS and AIR	$435,000
Administration and data collection	Westat	$95,000
System production, printing and processing	Pearson and Fulcrum	$445,000
Total		$975,000

6. Assurance of Confidentiality

NAEP confidentiality complies with the CIPSEA legislation (Confidential Information Protection provisions of Title V, Subtitle A, Public Law 107-347 and the National Assessment of Educational Progress Authorization Act). NAEP confidentiality policies and procedures are detailed in the main NAEP 2014-2016 system clearance submittal (OMB# 1850-0790 v.36), approved in March 2013. This assurance is applicable for the Computer Familiarity Study. However, since the ECLS-K:2011 is a longitudinal study where students are tracked over time, the confidentiality being cited for the NAEP-ECLS-K:2011 Link Study is the ESRA citation. This language indicates that participants will be notified that their participation is voluntary and that their responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002 (ESRA, 20 U.S.C. §9573).

Appendix A: Committee Lists

NAEP Socio-Economic Status Panel

Name Affiliation

Charles Cowan Analytic Focus LLC, Alexandria, VA

Robert Hauser University of Wisconsin-Madison

Robert Kominski U.S. Census Bureau, Washington, DC

Hank Levin Columbia University, New York City, NY

Sam Lucas University of California-Berkeley

Stephen Morgan Cornell University, Ithaca, NY

Margaret Beale Spencer University of Chicago

Chris Chapman (ex-officio) National Center for Education Statistics, Washington, DC

NAEP Validity Studies Panel

Name Affiliation

Peter Behuniak University of Connecticut

George Bohrnstedt American Institutes for Research, Washington, DC

Jim Chromy Research Triangle Institute, Research Triangle Park, NC

Phil Daro University of California/SERP/Pearson

Lizanne DeStefano University of Illinois

Richard Duran University of California

David Grissmer University of Virginia

Larry Hedges Northwestern University, Evanston, IL

Gerunda Hughes Howard University, Washington, DC

Robert Linn University of Colorado at Boulder

Ina Mullis Boston College

Scott Norton Council of Chief State School Officers, Washington, DC

Gary Phillips American Institutes for Research, Washington, DC

Lorrie Shepard University of Colorado at Boulder

1 These materials were not available for inclusion in the Wave 2 submission.

2 See appendix A for list of members of the SES questionnaire advisory panel.

3 Note: one question (When did you first use a tablet?) differs in one of its response options (I have never used one versus Never until this assessment) depending on whether administered in paper-and-pencil or TBA mode.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	NATIONAL ASSESSMENT OF
Author	joconnell
File Modified	0000-00-00
File Created	2021-01-30