TIMSS_PIRLS_2011 Full Scale Part A

Trends in International mathematics and science study (TIMSS) and progress in international reading literacy study (PIRLS) 2011 Data Collection

REQUEST FOR OMB REVIEW

Supporting Statement Part A

Prepared by:

National Center for Education Statistics

U.S. Department of Education

Washington, DC

November 2, 2010

TABLE OF CONTENTS

PREFACE

A. JUSTIFICATION

A.1 Importance of Information 1

A.2 Purposes and Uses of the Data 5

A.3 Improved Information Technology (Reduction of Burden) 7

A.4 Efforts to Identify Duplication 8

A.5 Minimizing Burden for Small Institutions 9

A.6 Frequency of Data Collection 9

A.7 Special Circumstances 10

A.8 Consultations Outside NCES 10

A.9 Payments or Gifts to Respondents 10

A.10 Assurance of Confidentiality 11

A.12 Estimates of Burden 14

A.13 Total Annual Cost Burden 16

A.14 Annualized Cost to Federal Government 17

A.15 Program Changes or Adjustments 17

A.16 Plans for Tabulation and Publication 17

A.17 Display OMB Expiration Date 19

A.18 Exceptions to Certification Statement 19

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL INFORMATION

B.1 Respondent Universe and Response Rates 21

B.2 Procedures for Collection of Information 23

B.3 Maximizing Response Rates 25

B.4 Tests of Procedures 25

B.5 Individuals Consulted on Statistical Design 25

C. APPENDIX A: RECRUITMENT LETTERS AND DOCUMENTS (Approved; OMB #1850-0645 v.6)

D. APPENDIX B: PARENTAL PERMISSION LETTERS AND SUPPORTING MATERIALS (Approved; OMB #1850-0645 v.6)

PREFACE

The Office of Management and Budget (OMB) approved in April 2010 (OMB# 1850-0645 v.6) recruitment for the 2011 Trends in International Mathematics and Science Study (TIMSS) and the Progress in International Reading Literacy Study (PIRLS) full-scale data collection, along with the state benchmarking study and a waiver of the 60-day federal register notice for the clearance of the full-scale study PIRLS/TIMSS 2011 instruments. The National Assessment of Education Progress (NAEP) Alignment with TIMSS Study was approved by OMB in July 2010 (OMB# 1850-0790 v.29).

This submission is a request for OMB’s approval of the final versions of the 2011 PIRLS/TIMSS full-scale non-cognitive data collection instruments. A description of the modifications that have been made to the previously approved field test instruments (OMB# 1850-0645 v.5) is also attached. The Supporting Statement Parts A and B are the same as those ap­prov­ed in April 2010 (OMB# 1850-0645 v.6), with the following changes: the requested respondent burden in section A.12, the annualized cost in A.14, the explanation in A.15, the timeline in section A.16, and the description of the sample in section B.1 have been updated to accurately reflect minor modifications since the field test, the increased demands of the NAEP Alignment with TIMSS Study, and to reflect the addition of Florida as a state benchmarking participant in the 2011 PIRLS/TIMSS full-scale study at grade 4 and North Carolina as a state benchmarking participant in the 2011 TIMSS full-scale study at grade 4 (a paragraph regarding these additions has also been included on page ii of this Preface).

During the 2010-11 school year, the Trends in International Mathematics and Science Study (TIMSS) and the Progress in International Reading Literacy Study (PIRLS), sponsored by the International Association for the Evaluation of Educational Achievement (IEA) and conducted in the United States under the auspices of the National Center for Education Statistics (NCES), will be administered in more than 50 other countries. TIMSS will assess the mathematical and science knowledge of fourth- and eighth-grade students while PIRLS will assess the reading literacy of fourth-grade students. TIMSS and PIRLS are major resources for the United States to evaluate the academic achievement of elementary and middle school students and for NCES to meet its mandate of “acquiring and disseminating data on educational activities and student achievement…in the United States compared with foreign nations” [The Educational Sciences Reform Act of 2002 (HR 3801, Part C, Sec.153)].

TIMSS and PIRLS are collaborative international studies, and as such are developed by participating countries’ representatives through an extensive series of international meetings and working groups. These international meetings and working groups (assisted by expert panels, researchers, and the IEA support staff) developed frameworks¹ used to develop the assessment and the background questionnaires and the studies’ timeline. They also reached agreement on the common standards and procedures for collecting and reporting data. All participating countries must follow the studies’ timeline and the common standards and procedures. As a result, TIMSS and PIRLS are able to provide a reliable and comparable measure of student skills in participating countries.

Although TIMSS and PIRLS are on different data collection cycles (TIMSS collects data every 4 years and PIRLS collects data every 5 years), their two cycles coincide in the 2010-11 school year, necessitating a coordinated approach to data collection. The international schedule calls for the full-scale data collection of TIMSS and PIRLS to occur in April–May 2011.

Based on recruitment efforts for the spring 2010 field test, NCES combined the grade 4 samples for PIRLS and TIMSS main data collection. Response rates and feedback from schools indicated that administering PIRLS and TIMSS in the same schools is feasible and does not have a negative effect on participation. The decision to proceed with a combined TIMSS and PIRLS grade 4 sample was reviewed and approved by the NCES Chief Statistician.

Also since the field test, the state of Florida contacted the IEA to request that Florida participate in the 2011 PIRLS/TIMSS full-scale study as a benchmarking participant at grade 4, and the state of North Carolina contacted the IEA to be a state benchmarking participant in the 2011 TIMSS full-scale study at grade 4. Both requests were approved by the IEA and NCES has subsequently drawn a state sample for Florida to participate in PIRLS/TIMSS at grade 4 and a state sample for North Carolina to participate in TIMSS at grade 4. North Carolina’s grade 8 schools were already sampled as part of the National Assessment of Education Progress (NAEP) Alignment with TIMSS Study, which was approved by OMB in July 2010 (OMB# 1850-0790 v.29).

In the grade 4 samples, schools with three or four grade 4 classes will have two classes randomly assigned to one assessment, and the remaining classes assigned to the other assessment. Schools with two grade 4 classes will have one class randomly assigned to one assessment and the other class assigned to the other assessment. Schools with one grade 4 class will be randomly assigned to participate in only one assessment. Compared with having two independent samples, this approach reduces costs and total burden time on schools by reducing the number of schools that will need to be contacted and will need to fill out a questionnaire.

1. JUSTIFICATION

A.1 Importance of Information

TIMSS and PIRLS provide a unique opportunity to compare U.S. students’ reading, mathematics, and science knowledge and skills at the fourth- and eighth-grade with that of their peers in countries around the world. TIMSS complements what we learn from national assessments by identifying the strengths and weaknesses of student science and math performance relative to students in participating countries around the world. The results from TIMSS and PIRLS inform national discussions about education and international competitiveness. They provide valuable benchmarking information on how U.S. students and schools compare to their counterparts around the world—information which allows educators and policymakers to examine other education systems for effective practices that may have application to the United States, and contributes to ongoing discussions of ways to improve the quality of education for all students.

To provide a more complete understanding of the TIMSS and PIRLS studies, the remainder of this section explains in detail what information each study has provided in the past by reviewing both studies’ history, their approach to student measurement, the contextual information they provide, and their organization and direction.

Trends in International Mathematics and Science Study (TIMSS)

First administered in 1995, the Trends in International Mathematics and Science Study (TIMSS) was the largest and most ambitious international study of student achievement ever conducted. It was conducted at five grade levels in more than 53 countries (the third, fourth, seventh, and eighth grades, and the final year of secondary school). Students were tested in mathematics and science, and extensive information about the teaching and learning of mathematics and science was collected from students, teachers, and school principals. TIMSS also investigated the mathematics and science curricula of the participating countries through an analysis of curriculum guides, textbooks, and other curricular materials.

In 1999, TIMSS conducted a second assessment designed to measure mathematics and science achievement of eighth-grade students and collected information from students, teachers, and school principals about mathematics and science curricula, instruction, home contexts, and school characteristics and policies. Of the 38 participating countries, 26 had participated in the 1995 TIMSS assessment, which enabled these countries to measure changes in their children's mathematics and science achievement and in school and home contexts for learning.

TIMSS 2003 built on the success of TIMSS 1995 and 1999 and offered the opportunity to study trends in eighth-grade mathematics and science achievement at three points over an eight-year period. It also allowed for the study of fourth grade trends at two points, between 1995 and 2003. TIMSS 2003 administered assessments of student achievement in mathematics and science at the fourth and eighth grades and context questionnaires to students, teachers, and principals. TIMSS was again administered most recently in 2007, this time in 65 countries.

As per international specifications, TIMSS will be administered to students near the end of the school year in which they have had, respectively, four years and eight years of formal schooling (fourth- and eighth-grade in the United States). In the Southern Hemisphere, full-scale data collection for TIMSS will be conducted in October through December 2010, and in most of the participating Northern Hemisphere countries, including the United States, it will be conducted in April and May 2011. A total of 68 countries are expected to participate in TIMSS 2011. The initial release of the TIMSS 2011 results is planned for December 2012.

Progress in International Reading Literacy Study (PIRLS)

First administered in 2001 in 35 countries, the Progress in International Reading Literacy Study (PIRLS) was designed to provide a trend measure to the 1991 Reading Literacy Study and also to serve as a starting point for a more regular, continuing cycle of assessments of reading literacy across the world. Next, in 2006, PIRLS was administered in 41 countries. The 2011 PIRLS assessment will be the third cycle, with 55 countries participating in the assessment (49 of which are also participating in the 2011 TIMSS).

Viewed as a foundation for learning across all subjects, reading literacy equips young children with the ability to participate fully in their communities and the larger society. PIRLS focuses on grade 4 because it is considered an important transition point in children’s development as readers. Typically, at this point, students know how to read and are developing more advanced reading skills to further their learning.

Like TIMSS, in addition to the development of the assessment, PIRLS is also designed to collect a range of contextual data from school administrators, teachers, and students about reading curricula and classroom instruction, home contexts, and school characteristics related to the reading abilities of students.

Measurement of Student Achievement in TIMSS and PIRLS

The overriding principle in constructing tests for both studies is to produce assessment instruments that will generate achievement data that are reliable and valid for the purposes for which they are to be used. Based on curriculum frameworks, the TIMSS and PIRLS assessments are developed through an international consensus-building process involving input from experts in education, reading, mathematics, science, and measurement. The assessments contain questions requiring students to select appropriate responses or to solve problems and answer questions in an open-ended format. With each cycle, TIMSS and PIRLS release test questions to the public and then replace these with newly developed questions. The newly developed mathematics, science, and reading questions for the 2011 assessments will be field tested to ensure that they accurately measure U.S. students’ knowledge and skills. Field testing the new questions is necessary to eliminate wording or topics that would put U.S. students at a disadvantage when assessing their knowledge relative to students in other countries.

The TIMSS and PIRLS tests have been designed to maximize reading, mathematics and science content coverage yet minimize the burden on individual students. The tests are based on a matrix design, whereby blocks of items have been distributed across multiple test booklets and the booklets are distributed across students in a country. Each student completes only one test booklet and, therefore, no student is requested to complete all cognitive items developed for TIMSS or PIRLS.

TIMSS will have 28 total blocks of cognitive items per grade, 14 for math and 14 for science. These will be distributed across 14 booklets at each grade level. As in the past, the assessment time for the booklets will be 72 minutes at grade 4 and 90 minutes at grade 8, per student. PIRLS will have items divided into ten 40-minute blocks of passages and items. These 10 blocks will be distributed across 13 booklets. The assessment time for the booklets will be 80 minutes. More detailed information on the TIMSS assessment framework and design can be found at:

http://timssandpirls.bc.edu/timss2011/frameworks.html. The PIRLS assessment framework can be found at http://timssandpirls.bc.edu/pirls2011/framework.html.

Contextual Information Provided by TIMSS and PIRLS

Instructional, school, and student background variables are important for understanding international differences in student achievement in mathematics and science. For this reason, TIMSS includes student, teacher, and school questionnaires to collect contextual information about instruction and learning. Previously, TIMSS has gathered a large amount of data related to a variety of topics including instructional strategies, classroom activities, school characteristics and resources, attitudes, parental expectations, and many demographic characteristics. Since TIMSS 2003, the international organization through its expert panels has focused on reducing the number of questions asked of schools, teachers, and students. Indeed, the estimate is that the questionnaires have been reduced by an average of one third from the 1999 TIMSS questionnaires, and approximately 10-15 percent from the 2007 versions of the instruments. The student questionnaire for both fourth and eighth grade will be completed by each student participating in the assessment and requires approximately 30 minutes to finish. The teacher and school questionnaires each require approximately 30 minutes to finish.

PIRLS also follows this model, collecting contextual information from students, teachers, and schools. The focus of the questionnaires is primarily on topics related to reading instruction, learning practices and habits, classroom activities, school characteristics and resources, and demographic characteristics. The PIRLS student questionnaire for fourth grade requires approximately 30 minutes to complete, while the teacher and school questionnaires each require approximately 30 minutes to complete.

Because TIMSS and PIRLS will be administered internationally at the same time in 2011, the International Study Center has made a concerted effort to reduce burden between the instruments for each study by consolidating the separate TIMSS and PIRLS questionnaires into a single questionnaire that has retained those items found to be of value for research and eliminated items that resulted in no or limited value to research in the past. Because some questions on the student background questionnaire are about learning specific subject matter, the new consolidated student background questionnaire has a separate subject-specific module at the end that asks questions relevant only to students taking TIMSS or PIRLS. Because no fourth-grade class in the United States will receive both TIMSS and PIRLS, separate TIMSS and PIRLS questionnaires are required for fourth-grade teachers and students. However, in schools where TIMSS and/or PIRLS will be administered at grade 4, the administrators of these schools will receive a single, combined TIMSS and PIRLS school questionnaire.

Organization and Direction of TIMSS and PIRLS

TIMSS and PIRLS 2011 are sponsored by the International Association for the Evaluation of Educational Achievement (IEA) and conducted in the United States under the auspices of the National Center for Education Statistics (NCES) in the Institute of Education Sciences of the U.S. Department of Education. The TIMSS and PIRLS International Study Center at Boston College directs the study at the international level under the oversight of the IEA, and is responsible for the study design, operations, analysis, and reporting. The International Study Center works directly with the contract organizations (data collectors) in each participating country to carry out the various aspects of the study. In the United States, Westat has been awarded the contract to collect data for TIMSS and PIRLS 2011.

The IEA Data Processing Center (DPC) in Hamburg, Germany, which serves as the data processing unit of the IEA, is responsible for all aspects of data processing including verifying the accuracy and comparability of the data and constructing the international databases for TIMSS and PIRLS. The Center is once again handling all aspects of data processing for TIMSS and PIRLS 2011. In addition, it is responsible for sampling, including planning the sample designs, consulting with participating countries about their sample, and ensuring that procedures are implemented correctly. The U.S. sampling plan will be submitted to the DPC for review and approval. The DPC will also be responsible for scaling the TIMSS and PIRLS data once collected.

Each participating country is represented by a National Research Coordinator (or agency). The National Research Coordinators are responsible for implementing TIMSS and PIRLS in their countries in accordance with the international standards. In the United States, the NRC for TIMSS is Dr. Patrick Gonzales, a staff member of NCES. Dr. Gonzales has been the U.S. TIMSS NRC since 1999. The NRC for PIRLS is Dr. Stephen Provasnik, also a staff member of NCES. Dr. Provasnik has been the PIRLS NRC since 2008.

Funding for the international coordination of TIMSS and PIRLS is provided by the National Center for Education Statistics, Institute of Education Sciences, of the U.S. Department of Education. Each of the countries participating in TIMSS and PIRLS provides funding for implementation of its own national data collection, and also contributes to the international coordinating costs.

A.2 Purposes and Uses of the Data

Learning more about potentially effective practices in reading, mathematics, and science teaching is an ongoing challenge for educators and researchers. The continuing purpose of TIMSS and PIRLS is to obtain information about students' knowledge and abilities in the specified subjects, and about the cultural environments, teaching practices, curriculum goals, and institutional arrangements that are associated with student achievement in the respective subject areas. Additionally, TIMSS and PIRLS are designed to measure cross-national change in these areas by administering cyclical data collections using common test and contextual items.

The outcomes of these international studies are of great interest to national and local policymakers, industrial leaders, and educators, specifically because they permit direct comparisons across countries as well as international benchmarking of students’ reading, mathematics, and science knowledge. These studies identify differences between countries and within countries over time in instructional practices, school policies, and access to opportunity-to-learn that can lead to discussions of how school systems organize instruction.

Based on earlier TIMSS and PIRLS data releases, it is likely that the results of these studies will draw great attention in the United States and elsewhere. It is therefore expected that TIMSS and PIRLS will contribute to ongoing national debate and efforts to improve reading, mathematics, and science learning and achievement across the nation.

The studies assist educators who seek to relate teaching practices and other influences to the attainment of satisfactory levels of reading, mathematics and science achievement. U.S. analysis of the international data set will enable researchers to identify elements of education programs that can be tailored to improve achievement levels among U.S. students. The studies will also provide NCES with the opportunity to explore the relationship between achievement on the National Assessment of Educational Progress (NAEP) and TIMSS. This will be achieved by having six states participate in grade 8 TIMSS as if they were separate countries (referred to in TIMSS as “Benchmarking” participants). Data from the six states will be used to validate a separate study, the NAEP Alignment with TIMSS Study (already approved; OMB# 1850-0790 v.29), that seeks to equate scores on the mathematics portion of grade 8 NAEP to the mathematics portion of grade 8 TIMSS.

U.S. participation in TIMSS and PIRLS is entirely consistent with the NCES mandate. The enabling legislation of the National Center for Education Statistics [Section 406 of the General Education Provisions Act, as amended (20 U.S.C. 1221e-1)] specifies that "The purpose of the Center [NCES] shall be to collect and analyze and disseminate statistics and other information related to education in the United States and in other nations." The Educational Sciences Reform Act of 2002 (HR 3801, Part C, Sec.153) also specifies that NCES “shall collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations, including … (6) acquiring and disseminating data on educational activities and student achievement … in the United States compared with foreign nations.” Apart from being essential for any international perspective on reading, mathematics, and science knowledge and skills, U.S. participation fulfills both the national and international aspects of NCES' mission.

Consequences of Not Collecting the Data

Failure to collect reliable TIMSS and PIRLS data in the full-scale study would end the regular data collection and reporting cycle of these two studies in the United States that are critical to fulfilling NCES’ mandate, and would deprive U.S. policymakers and educators of national trend and international comparison information. Moreover, failure to collect full-scale TIMSS data for the nation and selected states will prevent the NAEP-TIMSS Linking Study and compromise the ability of NCES to investigate the relationship between student achievement on NAEP and TIMSS.

A.3 Improved Information Technology (Reduction of Burden)

The design and procedures for TIMSS and PIRLS 2011 are prescribed internationally and data collection involves paper and pencil student assessments and questionnaires, as well as optional online or paper and pencil school and teacher questionnaires. Each participating nation is expected to adhere to the internationally prescribed design.

As in past administrations, TIMSS and PIRLS sample intact school classes rather than selecting students from a number of different classes within a school. This effectively reduces burden across sampled schools and minimizes disruption to school schedules. For TIMSS at grade 8, two eighth-grade classrooms per school will be randomly selected. One of these two classes will be administered TIMSS; the other class will be administered the “braided” booklet that is part of the NAEP-TIMSS Linking Study design. Based on discussions with the international organizing body and the contractor, an increase in the school sample size to 600 will counter the effect of administering TIMSS to only one class, rather than two classes. Details about the portion of the NAEP-TIMSS Linking Study that will be carried out during the NAEP testing window were provided through the OMB review and clearance process established for NAEP and approved by OMB in July 2010 (OMB# 1850-0790 v.29). Because TIMSS and PIRLS coincide in 2011, the United States will combine the grade 4 school samples and administer both assessments in some of the same schools (as noted in the Preface and described in detail in Section B). In brief, up to four classrooms within a school may be selected and assigned randomly to either TIMSS or PIRLS at grade 4. While the number of classes would be greater in larger schools, the total number of schools required in the sample would be smaller, thus reducing cost and the total burden time within and across schools

To further reduce the burden on schools and school staff, test administration will be handled by field data collectors hired, trained, and supervised by Westat, under a contract with NCES. Consequently, school personnel will not be asked to code, package, or mail any of the survey materials. All clerical procedures will be handled by contractor field staff.

The DPC is supporting the administration of school and teacher questionnaires over the Internet as an alternative option to the paper and pencil version of both questionnaires. The United States has fielded the use of the online school and teacher questionnaires and, based on this experience, will implement the online questionnaires in the main assessment as well. NCES has remedied the problem of confidentiality assurance and disclosure risk by hosting the online school and teacher questionnaires on the secure NCES servers. School principals and teachers of the students assessed will be provided with the option of responding to the background questionnaire items online or in paper-and-pencil format. Each principal and teacher will be assigned a unique ID and password that can be used to access the online questionnaires. The online questionnaires will appear exactly as the paper-and-pencil versions, and will allow a user to suspend and restart data entry at any point in the process. All data entry completed by the respondent is automatically saved so that the next time s/he accesses the online questionnaires, the previously entered responses are available. By collecting the data on the secure NCES server, NCES will be able to control access to the data to ensure confidentiality and minimize disclosure risk.

Neither TIMSS nor PIRLS will use electronic assessment instruments, as they would be inappropriate for these studies given the different levels of technology available to the nations involved in the project and the requirement that all nations implement the studies in a similar manner. The uniform testing method requirement is consistent with the findings of recent field test trials for PISA 2006, where assessment instruments administered on laptop computers showed significantly different results than those administered in the pencil and paper versions.

A.4 Efforts to Identify Duplication

In the United States, reading, mathematics, and science achievement is systematically assessed at (1) the Federal level, where trend data have been collected on a fairly regular basis since 1971, through the National Assessment of Educational Progress (NAEP); (2) the state level, where data are routinely collected as part of state testing programs, though they vary across the states in terms of the frequency of testing, age/grades tested, and types of cognitive items administered; and (3) the district level, where data are collected through the use of commercially or locally developed standardized tests as well as tests developed in conjunction with the instructional programs used in schools. However, TIMSS and PIRLS 2011 do not duplicate these assessments.

TIMSS and PIRLS 2011 are part of a program of international cooperative studies of educational achievement supported and funded, in part, by the U.S. Department of Education. These studies represent the U.S. participation in international studies involving approximately 65 nations in TIMSS and 55 nations in PIRLS. As part of international cooperative studies, the United States must collect the same information at the same time as the other nations for purposes of making both valid international comparisons with other countries and comparisons with the previous TIMSS and PIRLS data. While some studies in the United States collect similar, though not identical, kinds of information (e.g., NAEP), the data from those studies cannot be substituted for the information collected in TIMSS or PIRLS. Furthermore, the data collected through TIMSS and PIRLS are based on unique frameworks that are not shared by any other state, national, or international data collection effort. In order to participate in these international studies, the United States must agree to administer the same core instruments that are administered in the other countries. Because the items measuring reading, mathematics, and science achievement have been developed with intensive international coordination, any changes to the instruments would also require international coordination and approval.

NCES is conducting a study that will link grade 8 TIMSS mathematics with grade 8 NAEP, through an equating study known as the NAEP Alignment with TIMSS Study (OMB# 1850-0790 v.29) or the “NAEP-TIMSS Linking Study.” To validate the linking study, additional data will be collected in eight states at grade 8 only. If successful, the Linking Study (to be conducted separately) will provide NCES with the ability to estimate TIMSS scores for all states that participate in NAEP grade 8. This, then, would effectively reduce duplication of effort by providing all states with both NAEP scores and international achievement estimates. Details about the portion of the NAEP-TIMSS Linking Study that will be carried out during the NAEP testing window were provided through the OMB review and clearance process established for NAEP.

A.5 Minimizing Burden for Small Institutions

The school samples for TIMSS and PIRLS contain small-, medium- and large-size schools, including private schools, selected based on probability proportionate to their size. It is necessary to include small and private schools so that the students attending such schools are represented in the data collection. Burden will be minimized wherever possible for all institutions participating in the data collection. For the 1995 and 1999 TIMSS studies, the U.S. school sample involved the selection of a sample of geographic PSUs and then school sampling within the selected PSUs. For the 2003 and 2007 TIMSS and 2001 and 2006 PIRLS, the sample was not clustered. An un-clustered design will be used again for TIMSS and PIRLS 2011. This reduces the instances of requests for multiple schools from individual districts and reduces overlap with schools selected for NAEP samples. Other examples of ways in which burden will be minimized are sampling of intact classes to minimize disruption to school schedules and having test administration handled by data collectors hired, trained, and supervised by the national data collection contractor.

A.6 Frequency of Data Collection

The full-scale data collection is scheduled for April-May 2011. This schedule is prescribed by the international sponsoring organization for TIMSS and PIRLS, and adherence to this schedule is necessary to establish consistency in survey operations among participating countries as well as to maintain trend lines.

A.7 Special Circumstances

No special circumstances exist in the data collection plan for TIMSS or PIRLS that would necessitate unique or unusual manners of data collection. None of the special circumstances identified in the Instructions for Supporting Statement applies to the TIMSS or PIRLS studies.

A.8 Consultations Outside NCES

Consultations outside NCES have been extensive and will continue throughout the life of the project. The IEA studies are developed as a cooperative enterprise involving all participating countries. An International Steering Committee has general oversight of the study as a whole and, within most participating nations. A National Steering Committee has advisory responsibility for the conduct of the national study. Each National Research Coordinator participates in extensive discussions concerning the projects, usually with advice from national subject matter and testing experts. In addition, the IEA convened separate panels of reading, mathematics, and science experts from around the world to develop the cognitive items for both studies.

The majority of the consultations (outside NCES) have involved the TIMSS and PIRLS International Study Center at Boston College in the United States. Key to these ongoing consultations are the following individuals: Hans Wagemaker (executive director of the IEA); Michael Martin, Ina V.S. Mullis, Alka Arora, Ebru Erberber, Pierre Foy, and Ann Kennedy (all of Boston College); Barbara Malak (IEA); Marc Joncas (Statistics Canada); Oliver Neuschmidt (IEA Data Processing Center); Graham Ruddock (National Foundation for Educational Research, UK); and Christine O’Sullivan (K-12 Consulting), all of whom have extensive experience in developing and operating international education surveys (especially related to TIMSS and PIRLS).

A.9 Payments or Gifts to Respondents

For TIMSS and PIRLS 2011, compensation in the full-scale study will be the same as in the approved field test (OMB# 1850-0645 v.5). Compensation will be offered to schools and school coordinators, and small tokens of appreciation to students. In order to achieve acceptable school response rates, schools have historically been offered compensation for the time they invest in and the space they make available for the international assessments. Based on compensation provided in past administrations of TIMSS, as well as compensation currently offered to schools participating in international assessments, schools will be compensated $200 for their time.

The school staff serving as School Coordinators will be compensated $100 for their time and effort. The School Coordinator serves a critical role in data collection, functioning as the central school contact and facilitating arrangements for the assessments. They are asked to file class and student listing forms; arrange the date, time and space for the assessment; and disseminate information to parents and students.

Consistent with prior administrations of TIMSS and PIRLS, as a token of appreciation for their participation, students will receive a small gift valued at approximately $4. In TIMSS 2007, each participating student received a small watch/stop watch that could be clipped securely (with an attached karabiner) to a backpack or belt loop. The same product will be distributed to participating students for the TIMSS and PIRLS 2011 data collection. Students will also receive a certificate with their name thanking them for participating and representing the United States in TIMSS or PIRLS 2011. Some schools also offer recognition parties with pizza or other treats for students who participate; however these are not reimbursed by NCES or the contractor.

A.10 Assurance of Confidentiality

TIMSS and PIRLS 2011 will conform to all relevant federal regulations – specifically, the Privacy Act of 1974 (5 U.S.C. 552a), the Education Sciences Reform Act of 2002 (20 U.S.C. 9573 the Family Educational and Privacy Rights Act (20 U.S.C. 1232g), Protection of Pupil Rights Act (20 U.S.C. 1232h), Title V, subtitle A of, the E-Government Act of 2002 (P.L. 107-347), and the NCES Statistical Standards and Policies. The plan for maintaining confidentiality includes signing confidentiality agreements and notarized nondisclosure affidavits obtained from all personnel who will have access to individual identifiers. Also included in the plan is personnel training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses; controlled and protected access to computer files under the control of a single data base manager; built-in safeguards concerning status monitoring and receipt control systems; and a secured and operator-manned in-house computing facility.

Data will be presented in aggregate statistical form only. All information identifying the individual respondents will be kept confidential, in compliance with the law (U.S. C. 9573), which states that:

(b) The Director shall ensure that all individually identifiable information about students, their academic achievements, their families, and information with respect to individual schools, shall remain confidential…

( c ) (2) No person may

(i) use any individually identifiable information furnished under the provisions of this section for any purpose other than a research, statistics, or evaluation purpose under this subchapter;

(ii) make any publication whereby the data furnished by any particular person under this subchapter can be identified; or

(iii) permit anyone other than the individuals authorized by the Director to examine the individual reports."

The laws pertaining to the collection and use of personally identifiable information are clearly communicated in correspondence with states, districts, schools, teachers, and students, per NCES requirements (see recruitment letters and supporting materials in Appendix A of the accompanying documentation; approved by OMB in April 2010, OMB# 1850-0645 v.6). Letters will be sent to parents and school administrators describing the voluntary nature of this survey. The material sent will include materials to describe the study and to convey the extent to which respondents and their responses will be kept confidential (see parental permission letters and supporting materials in Appendix B; approved by OMB in April 2010, OMB# 1850-0645 v.6). In addition, all contractor and subcontractor staff and field workers involved in data collection and analysis are required to sign the confidentiality pledge that appears in exhibit 1.

Data files, accompanying software, and documentation will be delivered to NCES at the end of the project. Neither names nor addresses will be included on any data file.

Exhibit 1. Affidavit of Nondisclosure

Affidavit of Nondisclosure

______________________________________________________________________

(Job Title)

______________________________________________________________________

(Date Assigned to Work with NCES Data)

______________________________________________________________________

(Organization, State or Local Agency Name)

______________________________________________________________________

(Organization or Agency Address)

____________________________________________________________

(NCES Individually Identifiable Data)

I, __________________________________ , do solemnly swear (or affirm) that I will not –

1. make any disclosure or publication whereby a sample unit or survey respondent (including students and schools) could be identified or the data furnished by or related to any particular person or school under these sections could be identified;

2. or use or reveal any individually identifiable information furnished, acquired, retrieved or assembled by me or others, under the provisions of Section 183 of the Education Sciences Reform Act of 2002 (P.L. 107-279) and Title V, subtitle A of the E-Government Act of 2002 (P.L. 107-347) for any purpose other than statistical purposes specified in the NCES survey, project or contract.

___________________________________

(Signature)

[The penalty for unlawful disclosure is a fine of not more than $250,000 (under 18 U.S.C. 3571) or imprisonment for not more than five years (under 18 U.S.C. 3559), or both. The word "swear" should be stricken out when a person elects to affirm the affidavit rather than to swear to it.]

A.11 Sensitive Questions

Currently, there are no intended questions in the PIRLS and TIMSS questionnaires that are of a sensitive nature. While NCES intends to augment the attached questionnaires with national options (e.g., questions related to race/ethnicity), these additional questions are considered standard practice in survey research and will conform to all existing laws regarding sensitive information. As soon as the final versions of the full-scale test questionnaire instruments are available, NCES will update all documents to include all items to be asked of respondents.

A.12 Estimates of Burden

The cost/burden to respondents for the TIMSS and PIRLS 2011 full-scale study using the combined sample at grade 4 is calculated for the estimated time required of students and school staff (teachers, principals, school administrator, and school coordinators) to complete pre-assessment and assessment activities (see table 1). Burden calculations are also provided at the bottom of table 1 for (a) the eight states that will participate in the NAEP-TIMSS Linking Study (hereafter referred to as “state benchmarking”) at grade 8, assuming that funding for this portion of the study is made available, (b) for Florida’s participation in TIMSS and PIRLS as a state at grade 4, and (c) for North Carolina’s participation in TIMSS as a state at grade 4. Pre-assessment activities include the time involved to decide to participate, completing class and student listing forms, distributing parent consent materials, distributing school and teacher questionnaires, and arranging assessment space. Assessment activities include the time involved to complete student, teacher, and school administrator questionnaires, as well as the time for assessment directions, but they do not include students’ time responding to the assessment (cognitive items) portion of the study.

For the full-scale study, the average response burden for students is based on a 30-minute questionnaire and 20 minutes for instructions and break (students have 2x40 (80 total) minutes to complete the PIRLS cognitive assessment portion, 2x36 (72 total) minutes for the 4^th grade TIMSS, and 2x45 (90 total) minutes for the 8^th grade TIMSS). At an estimated $5.85 per hour cost to students, the dollar cost of the full-scale study for students is estimated at $221,107. The average response burden for the school administrator is based on a 30-minute school questionnaire and 90 minutes during the recruitment process. At an estimated $50.00 per hour cost to administrators, the dollar cost of the full-scale study for school administrators is estimated at $89,350. The average response burden for the teachers is based on a 30-minute teacher questionnaire. At an estimated $35.00 per hour cost to teachers, the dollar cost of the field test for teachers is estimated at $77,945. The response burden for the school coordinator is based on an estimated average of 4 hours to collect class rosters, handle consent requests, and negotiate time and space for the assessments. At an estimated $35.00 per hour cost to coordinators, the dollar cost of the full-scale study for school coordinators is estimated at $125,020.

Table 1. Burden estimates for TIMSS and PIRLS 2011 Full-scale Study.



The dollar cost of state benchmarking at grade 8 is estimated for students at $114,414; for school administrators at $54,050; for teachers at $18,900; and for school coordinators at $75,600. The total number of respondents for the full-scale study using the combined sample at grade 4 and including all state benchmarking at grade 4 and 8 is estimated at 78,651 respondents (2,866 during the recruitment period plus an additional 75,785 respondents during the data collection period), with a total burden time of 68,721 hours (7,882 during the recruitment period plus an additional 60,839 hours during the data collection period).

The total number of respondents for the full-scale study without state benchmarking at grade 8 is estimated at 52,486 with a total burden time of 45,382 hours.

A.13 Total Annual Cost Burden

Other than the burden associated with completing these pre-assessment activities and questionnaires (estimated above in Section A.12), the study imposes no additional cost to respondents nor the imposition of any record-keeping requirement.

A.14 Annualized Cost to Federal Government

The total cost to the federal government for conducting the TIMSS and PIRLS 2011 full-scale study is estimated to be $9.9 million. These estimates include all direct and indirect costs of the project, including sampling ($79,000), data collection ($1.3 million), data analysis ($742,088), and reporting ($672,000) phases.

The total cost to the federal government for conducting 2011 TIMSS grade 8 state benchmarking in eight states (for the NAEP Alignment with TIMSS Study) is estimated to be $5.45 million. This estimate includes all direct and indirect costs of the project, including sampling ($60,000), data collection ($1,800,000), data analysis ($75,000), and a national report ($112,500), methodological analyses on the effectiveness of the linkages and the potential for applying the methodology to subsequent data collections and other international assessments ($900,000), and revamping the International Data Explorer to include the state-level results ($1,100,000). There are no costs to the federal government for the state participation of Florida or North Carolina at grade 4, except for the inclusion of the data in the International Data Explorer.

The costs of these studies are spread out over a 3-year period, at the total annual cost to the federal government of $5,116,667.

A.15 Program Changes or Adjustments

There is an increase in burden in relation to the burden approved in the current record, because the last approval was for the field test and full-scale recruitment only (OMB# 1850-0645 v.6), while this clearance request is for full-scale recruitment and data collection activities.

The central change to TIMSS and PIRLS from previous cycles is the effort to combine the grade 4 sample and administer both TIMSS and PIRLS in some of the same schools. Compared with having two independent samples, this approach will reduce costs and total burden time on schools by reducing the number of schools that will need to be contacted and will need to fill out a questionnaire.

A.16 Plans for Tabulation and Publication

The TIMSS and PIRLS International Study Center (ISC) will prepare separate reports for mathematics and science in Grade 4 and Grade 8 for TIMSS, and a separate report for PIRLS describing reading literacy at grade 4. The states participating in TIMSS state benchmarking at grade 8 and Florida and North Carolina at grade 4 will appear in the ISC international TIMSS reports, and Florida will also appear in the ISC international PIRLS report. The first of these reports will be released in December 2012. As has been customary, NCES will also release a report for each study at the same time as the international reports are released, interpreting the results for the U.S. audience. NCES reports on initial data releases are generally limited to simple bivariate statistics. There are currently no plans to conduct complex statistical analyses of either dataset. (Examples of past reports on TIMSS and PIRLS data can be found at http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2009001 and http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2008017) In the spring of 2013, the International Study Center will also prepare technical reports for both TIMSS and PIRLS describing the design and development of both TIMSS and PIRLS and detailing the scaling procedures, weighting, missing value imputation, and analyses. After the release of the international data, NCES plans to release the national data and an accompanying User’s Guide for each study.

Electronic versions of each publication are generally made available on the NCES website. Schedules for tabulation and publication of TIMSS and PIRLS 2011 results in the United States are dependent upon receiving data files from the international sponsoring organization. With this in mind, the expected data collection dates and a tentative reporting schedule are as follows:

April 2010	Began recruiting schools for the full-scale study.
July 2010 – February 2011	Prepare for the full-scale study phase
April – May 2011	Collect full-scale study data (Northern hemisphere)
August 31, 2011	Deliver full-scale study raw data to international sponsoring organization
May 2012	Receive final data files from international sponsors
June – December 2012	Produce General Audience Report and Technical Report for the United States

A more detailed production schedule is provided below. This shows the expected dates from the field test activities currently underway through the full-scale study activities.

PIRLS/TIMSS Production Schedule

November 2009	Final national version of field test questionnaires approved by the international consortium.
December 2009	Submitted to RIMS a request for regular clearance for TIMSS/PIRLS full-scale study school recruitment, 60-day notice waiver for full-scale study collection instruments clearance, and submit the finale versions (as approved by the IEA) of field test collection instruments.
March - April 2010	Collected field test data (approved under previous clearance).
April 2010	Finalized grade 4 sampling design; 2. Selected grades 4 and 8 samples (as described in Supporting Statement Part B).
April - May 2010	1. Contacted states with full-scale sample schools; 2. Contacted districts with sample schools to comply with all of their collection recruitment requirements; 3. Began to recruit schools and obtain classroom rosters for generating the classroom sample;. 4. Delivered raw field test data to international sponsoring organization.
June 2010	Received Field Test Report from the international consortium.
August 2010	Received final international version of PIRLS full-scale study questionnaires.
September 2010	Received final international version of TIMSS full-scale study questionnaires.
November 2010	Submitted to NCES/RIMS full-scale study documents for OMB clearance.
November 2010	Final national version of PIRLS full-scale study questionnaires approved by international sponsoring organization.
November 2010	Final national version of TIMSS full-scale study questionnaires approved by international sponsoring organization.
November –December 2010	Receive OMB clearance for finalized full-scale study materials.
April – May 2011	Collect full-scale study data (Northern hemisphere)
August 2011	Deliver raw data to international sponsoring organization
May 2012	Receive final data files from international sponsors
June - December 2012	Produce General Audience Report and Technical Report for the United States

A.17 Display OMB Expiration Date

The OMB expiration date will be displayed on all data collection materials.

A.18 Exceptions to Certification Statement

No exceptions to the certifications are requested.

1 The TIMSS frameworks can be downloaded from http://timss.bc.edu/timss2011/frameworks.html#. Chapter 1 lays out the mathe­matics framework for grades 4 and 8; chapter 2 lays out the science framework for grades 4 and 8; and chapter 3 explains the contextual framework used to develop the background questionnaire. The PIRLs frameworks can be downloaded from http://timssandpirls.bc.edu/pirls2011/framework.html. Chapters 1 and 2 explain the reading process and assessment framework; chapter 3 explains the framework for the background questionnaire.