TIMSS_PIRLS 2011 Part A

Trends in International mathematics and science study (TIMSS) and progress in international reading literacy study (PIRLS) 2010 and 2011 Data Collections

REQUEST FOR OMB REVIEW

Supporting Statement Part A

Prepared by:

National Center for Education Statistics

U.S. Department of Education

Washington, DC

September 24, 2009

TABLE OF CONTENTS

PREFACE i

A. JUSTIFICATION 1

A.1 Importance of Information 1

A.2 Purposes and Uses of the Data 5

A.3 Improved Information Technology (Reduction of Burden) 7

A.4 Efforts to Identify Duplication 8

A.5 Minimizing Burden for Small Institutions 9

A.6 Frequency of Data Collection 9

A.7 Special Circumstances 9

A.8 Consultations Outside NCES 9

A.9 Payments or Gifts to Respondents 10

A.10 Assurance of Confidentiality 11

A.12 Estimates of Burden 14

A.13 Total Annual Cost Burden 16

A.14 Annualized Cost to Federal Government 17

A.15 Program Changes or Adjustments 17

A.16 Plans for Tabulation and Publication 17

A.17 Display OMB Expiration Date 19

A.18 Exceptions to Certification Statement 19

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL INFORMATION 21

B.1 Respondent Universe and Response Rates 21

B.2 Procedures for Collection of Information 22

B.3 Maximizing Response Rates 25

B.4 Tests of Procedures 25

B.5 Individuals Consulted on Statistical Design 25

C. APPENDIX A: RECRUITMENT LETTERS AND DOCUMENTS

D. APPENDIX B: PARENTAL PERMISSION LETTERS AND SUPPORTING MATERIALS

PREFACE

During the 2010-11 school year, the Trends in International Mathematics and Science Study (TIMSS) and the Progress in International Reading Literacy Study (PIRLS), sponsored by the International Association for the Evaluation of Educational Achievement (IEA) and conducted in the United States under the auspices of the National Center for Education Statistics (NCES), will be administered in more than 50 other countries. TIMSS will assess the mathematical and science knowledge of fourth- and eighth-grade students while PIRLS will assess the reading literacy of fourth-grade students. TIMSS and PIRLS are major resources for the United States to evaluate the academic achievement of elementary and middle school students and for NCES to meet its mandate of “acquiring and disseminating data on educational activities and student achievement…in the United States compared with foreign nations” [The Educational Sciences Reform Act of 2002 (HR 3801, Part C, Sec.153)].

Although TIMSS and PIRLS are on different data collection cycles (TIMSS collects data every 4 years and PIRLS collects data every 5 years), their two cycles coincide in the 2010-11 school year, necessitating a coordinated approach to data collection. As part of this coordinated approach, the United States will field test a sampling plan that will employ a design with a combined sample of schools at grade 4 in order to ascertain the feasibility of administering TIMSS and PIRLS in the same schools in the main assessment. In this approach, schools with three or four grade 4 classes will have two classes randomly assigned to one assessment, and the remaining classes assigned to the other assessment. Schools with one or two grade 4 classes will be randomly assigned to participate in only one assessment. Compared with having two independent samples, this approach should reduce costs and total burden time on schools by reducing the number of schools that will need to be contacted and will need to fill out a questionnaire. In addition, it should reduce the amount of variance found in U.S. schools as compared to when only one of two classrooms in a school was chosen for an assessment¹.

The international schedule calls for the joint TIMSS and PIRLS 2011 field test data collection in the United States to occur between March 1 and April 15, 2010, with the full-scale data collection scheduled to occur in April–May 2011. Included with this request are copies of the non-cognitive data collection instruments, which may be subject to minor editing, all of which will be finalized by the IEA by November 1. Other materials required for OMB approval, such as recruitment letters, brochures, and other advance materials explaining the studies are included in Appendixes A and B of the accompanying documentation. Additional information, such as burden estimates for the main study and the state benchmarking study, are also included for informational purposes, but not for approval.

NCES is requesting that the Office of Management and Budget (OMB) approve on an emergency basis a six-month clearance (from November 2–May 1) for the PIRLS/TIMSS March-April 2010 field test, including recruitment of selected schools, school districts, and state education agencies for the field test starting in November 2009. In order to have a reasonable chance to meet the minimum number of schools for the PIRLS and TIMSS field test, NCES must initiate contact with states and school districts as soon as it receives the list of sample schools chosen by the IEA, in early November.

Increasingly, school districts require formal approval of school-based studies, a process which includes submission of study documents and plans and, in some cases, attendance at special meetings. These district-level approvals typically take 3 weeks or more before permission is given to contact schools. Failure to receive approval to begin contacting states and districts by November 2009 may jeopardize NCES' ability to conduct meaningful field tests of the instruments and operation procedures in spring 2010.

1. JUSTIFICATION

A.1 Importance of Information

TIMSS and PIRLS provide a unique opportunity to compare U.S. students’ reading, mathematics, and science knowledge and skills at the fourth- and eighth-grade with that of their peers in countries around the world. TIMSS complements what we learn from national assessments by identifying the strengths and weaknesses of student science and math performance relative to students in participating countries around the world. The results from TIMSS and PIRLS inform national discussions about education as well as international competitiveness. They provide valuable benchmarking information on how U.S. students and schools compare to their counterparts around the world—information which allows educators and policymakers to examine other education systems for effective practices that may have application to the United States, and contributes to ongoing discussions of ways to improve the quality of education for all students.

To provide a more complete understanding of the TIMSS and PIRLS studies, the remainder of this section explains in detail what information each study has provided in the past by reviewing both studies’ history, their approach to student measurement, the contextual information they provide, and their organization and direction.

Trends in International Mathematics and Science Study (TIMSS)

First administered in 1995, the Trends in International Mathematics and Science Study (TIMSS) was the largest and most ambitious international study of student achievement ever conducted. It was conducted at five grade levels in more than 53 countries (the third, fourth, seventh, and eighth grades, and the final year of secondary school). Students were tested in mathematics and science, and extensive information about the teaching and learning of mathematics and science was collected from students, teachers, and school principals. TIMSS also investigated the mathematics and science curricula of the participating countries through an analysis of curriculum guides, textbooks, and other curricular materials.

In 1999, TIMSS conducted a second assessment designed to measure mathematics and science achievement of eighth-grade students and collected information from students, teachers, and school principals about mathematics and science curricula, instruction, home contexts, and school characteristics and policies. Of the 38 participating countries, 26 had participated in the 1995 TIMSS assessment, which enabled these countries to measure changes in their children's mathematics and science achievement and in school and home contexts for learning.

TIMSS 2003 built on the success of TIMSS 1995 and 1999 and offered the opportunity to study trends in eighth-grade mathematics and science achievement at three points over an eight-year period. It also allowed for the study of fourth grade trends at two points, between 1995 and 2003. TIMSS 2003 administered assessments of student achievement in mathematics and science at the fourth and eighth grades and context questionnaires to students, teachers, and principals. TIMSS was again administered most recently in 2007, this time in 65 countries.

As per international specifications, TIMSS will be administered to students near the end of the school year in which they have had, respectively, four years and eight years of formal schooling (fourth- and eighth-grade in the United States). In the Southern Hemisphere, main data collection for TIMSS will be conducted in October through December 2010, and in most of the participating Northern Hemisphere countries, including the United States, it will be conducted in April and May 2011. A total of 68 countries are expected to participate in TIMSS 2011. The initial release of the TIMSS 2011 results is planned for December 2012.

Progress in International Reading Literacy Study (PIRLS)

First administered in 2001 in 35 countries, the Progress in International Reading Literacy Study (PIRLS) was designed to provide a trend measure to the 1991 Reading Literacy Study and also to serve as a starting point for a more regular, continuing cycle of assessments of reading literacy across the world. Next, in 2006, PIRLS was administered in 41 countries. The 2011 PIRLS assessment will be the third cycle, with 55 countries participating in the assessment (49 of which are also participating in the 2011 TIMSS).

Viewed as a foundation for learning across all subjects, reading literacy equips young children with the ability to participate fully in their communities and the larger society. PIRLS focuses on grade 4 because it is considered an important transition point in children’s development as readers. Typically, at this point, students know how to read and are developing more advanced reading skills to further their learning.

Like TIMSS, in addition to the development of the assessment, PIRLS is also designed to collect a range of contextual data from school administrators, teachers, and students about reading curricula and classroom instruction, home contexts, and school characteristics related to the reading abilities of students.

Measurement of Student Achievement in TIMSS and PIRLS

The overriding principle in constructing tests for both studies is to produce assessment instruments that will generate achievement data that are reliable and valid for the purposes for which they are to be used. Based on curriculum frameworks, the TIMSS and PIRLS assessments are developed through an international consensus-building process involving input from experts in education, reading, mathematics, science, and measurement. The assessments contain questions requiring students to select appropriate responses or to solve problems and answer questions in an open-ended format. With each cycle, TIMSS and PIRLS release test questions to the public and then replaces these with newly developed questions. The newly developed mathematics, science, and reading questions for the 2011 assessments will be field tested to ensure that they accurately measure U.S. students’ knowledge and skills. Field testing the new questions is necessary to eliminate wording or topics that would put U.S. students at a disadvantage when assessing their knowledge relative to students in other countries.

The TIMSS and PIRLS tests have been designed to maximize reading, mathematics and science content coverage yet minimize the burden on individual students. The tests are based on a matrix design, whereby blocks of items have been distributed across multiple test booklets and the booklets are distributed across students in a country. Each student completes only one test booklet and, therefore, no student is requested to complete all cognitive items developed for TIMSS or PIRLS.

TIMSS will have 28 total blocks of cognitive items per grade, 14 for math and 14 for science. These will be distributed across 14 booklets at each grade level. As in the past, the assessment time for the booklets will be 72 minutes at grade 4 and 90 minutes at grade 8, per student. PIRLS will have items divided into ten 40-minute blocks of passages and items. These 10 blocks will be distributed across 13 booklets. The assessment time for the booklets will be 80 minutes. More detailed information on the TIMSS assessment framework and design can be found at http://timssandpirls.bc.edu/TIMSS2007/ frameworks.html. The PIRLS assessment framework can be found at http://timssandpirls.bc.edu/

pirls2011/framework.html.

Contextual Information Provided by TIMSS and PIRLS

Instructional, school, and student background variables are important for understanding international differences in student achievement in mathematics and science. For this reason, TIMSS includes student, teacher, and school questionnaires to collect contextual information about instruction and learning. Previously, TIMSS has gathered a large amount of data related to a variety of topics including instructional strategies, classroom activities, school characteristics and resources, attitudes, parental expectations, and many demographic characteristics. Since TIMSS 2003, the international organization through its expert panels has focused on reducing the number of questions asked of schools, teachers, and students. Indeed, the estimate is that the questionnaires have been reduced by an average of one third from the 1999 TIMSS questionnaires, and approximately 10-15 percent from the 2007 versions of the instruments. The student questionnaire for both fourth and eighth grade will be completed by each student participating in the assessment and requires approximately 30 minutes to finish. The teacher and school questionnaires each require approximately 30-40 minutes to finish.

PIRLS also follows this model, collecting contextual information from students, teachers, and schools. The focus of the questionnaires is primarily on topics related to reading instruction, learning practices and habits, classroom activities, school characteristics and resources, and demographic characteristics. The PIRLS student questionnaire for fourth grade requires approximately 30 minutes to complete, while the teacher and school questionnaires each require approximately 30-40 minutes to complete.

Because TIMSS and PIRLS will be administered at the same time in 2011, the International Study Center has made a concerted effort to reduce burden between the instruments for each study, by consolidating the separate TIMSS and PIRLS questionnaires into a single questionnaire that has retained those items found to be of value for research and eliminated items that resulted in no or limited value to research in the past. Because some questions on the student background questionnaire are about learning specific subject-matter, the new consolidated student background questionnaire has a separate subject-specific module at the end that asks questions relevant only to students taking TIMSS or PIRLS.

Organization and Direction of TIMSS and PIRLS

TIMSS and PIRLS 2011 are sponsored by the International Association for the Evaluation of Educational Achievement (IEA) and conducted in the United States under the auspices of the National Center for Education Statistics (NCES) in the Institute of Education Sciences of the U.S. Department of Education. The TIMSS and PIRLS International Study Center at Boston College directs the study at the international level under the oversight of the IEA, and is responsible for the study design, operations, analysis, and reporting. The International Study Center works directly with the contract organizations (data collectors) in each participating country to carry out the various aspects of the study. In the United States, Westat has been awarded the contract to collect data for TIMSS and PIRLS 2011.

The IEA Data Processing Center (DPC) in Hamburg, Germany, which serves as the data processing unit of the IEA, is responsible for all aspects of data processing including verifying the accuracy and comparability of the data and constructing the international databases for TIMSS and PIRLS. The Center is once again handling all aspects of data processing for TIMSS and PIRLS 2011. In addition, it is responsible for sampling, including planning the sample designs, consulting with participating countries about their sample, and ensuring that procedures are implemented correctly. The U.S. sampling plan will be submitted to the DPC for review and approval. The DPC will also be responsible for scaling the TIMSS and PIRLS data once collected.

Each participating country is represented by a National Research Coordinator (or agency). The National Research Coordinators are responsible for implementing TIMSS and PIRLS in their countries in accordance with the international standards. In the United States, the NRC for TIMSS is Dr. Patrick Gonzales, a staff member of NCES. Dr. Gonzales has been the U.S. TIMSS NRC since 1999. The NRC for PIRLS is Dr. Stephen Provasnik, also a staff member of NCES. Dr. Provasnik has been the PIRLS NRC since 2008.

Funding for the international coordination of TIMSS and PIRLS is provided by the National Center for Education Statistics, Institute of Education Sciences, of the U.S. Department of Education. Each of the countries participating in TIMSS and PIRLS provides funding for implementation of its own national data collection, and also contributes to the international coordinating costs.

A.2 Purposes and Uses of the Data

Learning more about potentially effective practices in reading, mathematics, and science teaching is an ongoing challenge for educators and researchers. The continuing purpose of TIMSS and PIRLS is to obtain information about students' knowledge and abilities in the specified subjects, and about the cultural environments, teaching practices, curriculum goals, and institutional arrangements that are associated with student achievement in the respective subject areas. Additionally, TIMSS and PIRLS are designed to measure cross-national change in these areas by administering cyclical data collections using common test and contextual items.

The outcomes of these international studies are of great interest to national and local policymakers, industrial leaders, and educators, specifically because they permit direct comparisons across countries as well as international benchmarking of students’ reading, mathematics, and science knowledge. These studies identify differences between countries and within countries over time in instructional practices, school policies, and access to opportunity-to-learn that can lead to discussions of how school systems organize instruction.

Based on earlier TIMSS and PIRLS data releases, it is likely that the results of these studies will draw great attention in the United States and elsewhere. It is therefore expected that TIMSS and PIRLS will contribute to ongoing national efforts and debate to improve reading, mathematics, and science learning and achievement across the nation.

The studies assist educators who seek to relate teaching practices and other influences to the attainment of satisfactory levels of reading, mathematics and science achievement. U.S. analysis of the international data set will enable researchers to identify elements of education programs that can be tailored to improve achievement levels among U.S. students. The studies will also provide NCES with the opportunity to explore the relationship between achievement on the National Assessment of Educational Progress (NAEP) and TIMSS. This will be achieved by having four states participate in grade 8 TIMSS as if they were separate nations (referred in TIMSS as “Benchmarking” participants). Data from the four states will be used to validate a separate study, the TIMSS-NAEP Linking Study, that seeks to equate scores on the mathematics portion of grade 8 NAEP to the mathematics portion of grade 8 TIMSS.

U.S. participation in TIMSS and PIRLS is entirely consistent with the NCES mandate. The enabling legislation of the National Center for Education Statistics [Section 406 of the General Education Provisions Act, as amended (20 U.S.C. 1221e-1)] specifies that "The purpose of the Center [NCES] shall be to collect and analyze and disseminate statistics and other information related to education in the United States and in other nations." The Educational Sciences Reform Act of 2002 (HR 3801, Part C, Sec.153) also specifies that NCES “shall collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations, including … (6) acquiring and disseminating data on educational activities and student achievement … in the United States compared with foreign nations.” Apart from being essential for any international perspective on reading, mathematics, and science knowledge and skills, U.S. participation fulfills both the national and international aspects of NCES' mission.

Consequences of Not Collecting the Data

Failure to collect the TIMSS and PIRLS field test data would compromise the ability of the United States to ensure that its students are not at a disadvantage with regard to both studies’ cognitive and non-cognitive items. This could compromise the reliability of the main study. The failure to collect reliable TIMSS and PIRLS data in the main study would, in turn, end the regular data collection and reporting cycle of these two studies in the United States that are critical to fulfilling NCES’ mandate, and deprive U.S. policymakers and educators of national trend and international comparison information. Moreover, failure to collect main TIMSS data for the nation and four states will prevent the TIMSS-NAEP Linking Study and compromise the ability of NCES to investigate the relationship between student achievement on NAEP and TIMSS.

A.3 Improved Information Technology (Reduction of Burden)

The design and procedures for TIMSS and PIRLS 2011 are prescribed internationally and data collection involves paper and pencil responses. Each participating nation is expected to adhere to the internationally prescribed design.

As in past administrations, TIMSS and PIRLS sample intact school classes rather than selecting students from a number of different classes within a school. This effectively reduces burden across sampled schools and minimizes disruption to school schedules. For TIMSS at grade 8, a maximum of two eighth-grade classrooms per school will be assessed. Since TIMSS and PIRLS fall in the same year, the United States will use the field test to examine the possibility of combining the grade 4 school samples and administer both assessments in the same schools. The sampling plan for this is described in detail in Section B. In brief, the field test will examine the feasibility of selecting up to four classrooms within a school and assigning classes randomly to either TIMSS or PIRLS. While the number of classes would be greater in larger schools, the total number of schools required in the sample would be smaller, thus reducing cost and the total burden time within and across schools. This plan has been reviewed and approved within NCES, and will be revisited once the field test data have been collected to determine, among other things, whether there are any negative effects on participation rates or operations.

To further reduce the burden on schools and school staff, test administration will be handled by field data collectors hired, trained, and supervised by Westat, under a contract with NCES. Consequently, school personnel will not be asked to code, package, or mail any of the survey materials. All clerical procedures will be handled by contractor field staff.

The DPC is exploring the possibility of administering school and teacher questionnaires over the Internet as an alternative option to the paper and pencil version of both questionnaires. It is not yet clear if the United States will pursue this option for the field test due to confidentiality and disclosure risk concerns at NCES. However, if the United States adopts this option, NCES would gather questionnaire responses centrally via a secure web link in which respondents would log on and complete their specific version of the questionnaire. NCES is continuing its discussions with the IEA to address U.S. concerns regarding the collection of data online. For other participating countries, the DPC will gather on-line questionnaire responses centrally via a secure web link.

Neither TIMSS nor PIRLS will use electronic assessment instruments, as they would be inappropriate for these studies given the different levels of technology available to the nations involved in the project and the requirement that all nations implement the studies in a similar manner. The uniform testing method requirement is consistent with the findings of recent field test trials for PISA 2006, where assessment instruments administered on laptop computers showed significantly different results than those administered in the pencil and paper versions.

A.4 Efforts to Identify Duplication

In the United States, reading, mathematics, and science achievement is systematically assessed at (1) the Federal level, where trend data have been collected on a fairly regular basis since 1971, through the National Assessment of Educational Progress (NAEP); (2) the state level, where data are routinely collected as part of state testing programs, though they vary across the states in terms of the frequency of testing, age/grades tested, and types of cognitive items administered; and (3) the district level, where data are collected through the use of commercially or locally developed standardized tests as well as tests developed in conjunction with the instructional programs used in schools. However, TIMSS and PIRLS 2011 do not duplicate these assessments.

TIMSS and PIRLS 2011 are part of a program of international cooperative studies of educational achievement supported and funded, in part, by the U.S. Department of Education. These studies represent the U.S. participation in international studies involving approximately 65 nations in TIMSS and 55 nations in PIRLS. As part of international cooperative studies, the U.S. must collect the same information at the same time as the other nations for purposes of making both valid international comparisons with other countries and comparisons with the previous TIMSS and PIRLS data. No other study in the United States will use the instruments developed by the international sponsoring organization. Therefore, while some studies in the United States collect similar, though not identical, kinds of information (e.g., NAEP), the data from those studies cannot be substituted for the information collected in TIMSS or PIRLS. Furthermore, the data collected through TIMSS and PIRLS are based on unique frameworks that are not shared by any other state, national, or international data collection effort. In order to participate in these international studies, the United States must agree to administer the same core instruments that are administered in the other countries. Because the items measuring reading, mathematics, and science achievement have been developed with intensive international coordination, any changes to the instruments would also require international coordination and approval.

NCES is currently planning a study that will link grade 8 TIMSS mathematics with grade 8 NAEP, through an equating study known as the TIMSS-NAEP Linking Study. To validate the linking study, additional data will be collected in four states at grade 8 only (through wholly separate samples). If successful, the Linking Study (to be conducted separately) would provide NCES with the ability to estimate TIMSS scores for all states that participate in NAEP grade 8. This, then, would effectively reduce duplication of effort by providing all states with both NAEP scores and international achievement estimates.

A.5 Minimizing Burden for Small Institutions

The school samples for TIMSS and PIRLS contain small-, medium- and large-size schools, including private schools, selected based on probability proportionate to their size. It is necessary to include small and private schools so that the students attending such schools are represented in the data collection. Burden will be minimized wherever possible for all institutions participating in the data collection. For the 1995 and 1999 TIMSS studies, the U.S. school sample involved the selection of a sample of geographic PSUs and then school sampling within the selected PSUs. For the 2003 and 2007 TIMSS and 2001 and 2006 PIRLS, the sample was not clustered. An un-clustered design will be used again for TIMSS and PIRLS 2011. This reduces the instances of requests for multiple schools from individual districts and reduces overlap with school selected for NAEP samples. Other examples of ways in which burden will be minimized are sampling of intact classes to minimize disruption to school schedules and having test administration handled by data collectors hired, trained, and supervised by the national data collection contractor.

A.6 Frequency of Data Collection

The field test data collection for TIMSS and PIRLS 2011 is scheduled for March 1 through April 15, 2010, with the full scale data collection scheduled for April-May 2011. This schedule is prescribed by the international sponsoring organization for TIMSS and PIRLS, and adherence to this schedule is necessary to establish consistency in survey operations among participating countries.

A.7 Special Circumstances

No special circumstances exist in the data collection plan for TIMSS or PIRLS that would necessitate unique or unusual manners of data collection. None of the special circumstances identified in the Instructions for Supporting Statement applies to the TIMSS or PIRLS studies.

A.8 Consultations Outside NCES

Consultations outside NCES have been extensive and will continue throughout the life of the project. The IEA studies are developed as a cooperative enterprise involving all participating countries. An International Steering Committee has general oversight of the study as a whole and, within most participating nations, a National Steering Committee has advisory responsibility for the conduct of the national study. Each National Research Coordinator participates in extensive discussions concerning the projects, usually with advice from national subject matter and testing experts. In addition, the IEA convened separate panels of reading, mathematics, and science experts from around the world to develop the cognitive items for both studies.

The majority of the consultations (outside NCES) have involved the TIMSS and PIRLS International Study Center at Boston College in the United States. Key to these ongoing consultations are the following individuals: Hans Wagemaker (executive director of the IEA); Michael Martin, Ina V.S. Mullis, Alka Arora, Ebru Erberber, Pierre Foy, and Ann Kennedy (all of Boston College); Barbara Malak (IEA); Marc Joncas (Statistics Canada); Oliver Neuschmidt (IEA Data Processing Center); Graham Ruddock (National Foundation for Educational Research, UK); and Christine O’Sullivan (K-12 Consulting), all of whom have extensive experience in developing and operating international education surveys (especially related to TIMSS and PIRLS).

A.9 Payments or Gifts to Respondents

For TIMSS and PIRLS 2011, compensation will be offered to schools and school coordinators, and small tokens of appreciation to students. In order to achieve acceptable school response rates, schools have historically been offered compensation for the time they invest in and the space they make available for the international assessments. Based on compensation provided in past administrations of TIMSS, as well as compensation currently offered to schools participating in international assessments, schools will be compensated $200 for their time.

The school staff serving as School Coordinators will be compensated $100 for their time and effort. The School Coordinator serves a critical role in data collection, functioning as the central school contact and facilitating arrangements for the assessments. They are asked to file class and student listing forms; arranging the date, time and space for the assessment; and disseminating information to parents and students.

Consistent with prior administrations of TIMSS and PIRLS, as a token of appreciation for their participation, students will receive a small gift valued at approximately $4. There are many possible products that could be obtained for this price. For example, for TIMSS 2007, each participating student received a small watch/stop watch that could be clipped securely (with an attached karabiner) to a backpack or belt loop. A similar product will be chosen for the TIMSS and PIRLS 2011 data collection. Students will also receive a certificate with their name thanking them for participating and representing the United States in TIMSS or PIRLS 2011. Some schools also offer recognition parties with pizza or other treats for students who participate, however these are not reimbursed by NCES or the contractor.

A.10 Assurance of Confidentiality

TIMSS and PIRLS 2011 will conform to all relevant federal regulations – specifically, the Privacy Act of 1974 (5 U.S.C. 552a), the Education Sciences Reform Act of 2002 (20 U.S.C. 9573 the Family Educational and Privacy Rights Act (20 U.S.C. 1232g), Protection of Pupil Rights Act (20 U.S.C. 1232h), Title V, subtitle A of, the E-Government Act of 2002 (P.L. 107-347), and the NCES Statistical Standards and Policies. The plan for maintaining confidentiality includes signing confidentiality agreements and notarized nondisclosure affidavits obtained from all personnel who will have access to individual identifiers. Also included in the plan is personnel training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses; controlled and protected access to computer files under the control of a single data base manager; built-in safeguards concerning status monitoring and receipt control systems; and a secured and operator-manned in-house computing facility.

Data will be presented in aggregate statistical form only. All information identifying the individual respondents will be kept confidential, in compliance with the law (U.S. C. 9573), which states that:

(b) The Director shall ensure that all individually identifiable information about students, their academic achievements, their families, and information with respect to individual schools, shall remain confidential…

( c ) (2) No person may

(i) use any individually identifiable information furnished under the provisions of this section for any purpose other than a research, statistics, or evaluation purpose under this subchapter;

(ii) make any publication whereby the data furnished by any particular person under this subchapter can be identified; or

(iii) permit anyone other than the individuals authorized by the Director to examine the individual reports."

The laws pertaining to the collection and use of personally identifiable information are clearly communicated in correspondence with districts, schools, teachers, and students, per NCES requirements (see recruitment letters and supporting materials in Appendix A of the accompanying documentation). Letters will be sent to parents and school administrators describing the voluntary nature of this survey. The material sent will include materials to describe the study and to convey the extent to which respondents and their responses will be kept confidential (see parental permission letters and supporting materials in Appendix B). In addition, all contractor and subcontractor staff and field workers involved in data collection and analysis are required to sign the confidentiality pledge that appears in exhibit 1.

Data files, accompanying software, and documentation will be delivered to NCES at the end of the project. Neither names nor addresses will be included on any data file.

Exhibit 1. Affidavit of Nondisclosure

Affidavit of Nondisclosure

______________________________________________________________________

(Job Title)

______________________________________________________________________

(Date Assigned to Work with NCES Data)

______________________________________________________________________

(Organization, State or Local Agency Name)

______________________________________________________________________

(Organization or Agency Address)

____________________________________________________________

(NCES Individually Identifiable Data

I, __________________________________ , do solemnly swear (or affirm) that I will not –

1. make any disclosure or publication whereby a sample unit or survey respondent (including students and schools) could be identified or the data furnished by or related to any particular person or school under these sections could be identified;

2. or use or reveal any individually identifiable information furnished, acquired, retrieved or assembled by me or others, under the provisions of Section 9573 of the Education Sciences Reform Act of 2002 (P.L. 107-279) and Title V, subtitle A of the E-Government Act of 2002 (P.L. 107-347) for any purpose other than statistical purposes specified in the NCES survey, project or contract;

___________________________________

(Signature)

[The penalty for unlawful disclosure is a fine of not more than $250,000 (under 18 U.S.C. 3571) or imprisonment for not more than five years (under 18 U.S.C. 3559), or both. The word "swear" should be stricken out when a person elects to affirm the affidavit rather than to swear to it.]

A.11 Sensitive Questions

Currently, there are no intended questions in the PIRLS and TIMSS questionnaires that are of a sensitive nature. While NCES intends to augment the attached questionnaires with national options (e.g., questions related to race/ethnicity), these additional questions are considered standard practice in survey research and will conform to all existing laws regarding sensitive information. As soon as the final versions of the field test and main test questionnaire instruments are available, NCES will update all documents to include all items to be asked of respondents.

A.12 Estimates of Burden

The cost/burden to respondents for the TIMSS and PIRLS 2010 field test is calculated for the estimated time required of students and school staff (teachers, principals, school administrator, and school coordinators) for pre-assessment activities and assessment activities (see table 1). The pre-assessment activities include the time involved to decide to participate, completing class and student listing forms, distributing parent consent materials, distributing school and teacher questionnaires, and arranging assessment space. The assessment activities include the time involved to complete student, teacher, and school administrator questionnaires, as well as the time for assessment directions, but they do not include students’ time responding to the assessment (cognitive items) portion of the study.

For the field test, the average response burden for students is based on a 30-minute questionnaire and 20 minutes for instruction and break. At an estimated $5.85 per hour cost to students, the dollar cost of the field test for students is estimated at $29,607. The average response burden for the school administrator is based on a 30-minute school questionnaire and 90 minutes during the recruitment process. At an estimated $50.00 per hour cost to administrators, the dollar cost of the field test for school administrators is estimated at $8,600. The average response burden for the teachers is based on a 30-minute teacher questionnaire. At an estimated $35.00 per hour cost to teachers, the dollar cost of the field test for teachers is estimated at $7,210. The response burden for the school coordinator is based on an estimated average of 4 hours to collect class rosters, handle consent requests, and negotiate time and space for the assessments. At an estimated $35.00 per hour cost to coordinators, the dollar cost of the field test for school coordinators is estimated at $12,040.

The total number of respondents for the field test is estimated at 6,739, with a total burden time of 5,783 hours.

Table 1. Burden estimates for TIMSS and PIRLS 2011 Field Test

Respondent	Study	Grade	Sample Size	Expected response rate	# of Respondents	Per Respondent (Minutes)	Total Burden Hrs.
Student	PIRLS	4	2,190	93%	2,036	50	1,697
Student	TIMSS	4	2,190	93%	2,036	50	1,697
Student	TIMSS	8	2,150	93%	2,000	50	1,667
School	PIRLS	4	10	99%	10	30	5
School	PIRLS & TIMSS	4	30	99%	30	30	15
School	TIMSS	4	10	99%	10	30	5
School	TIMSS	8	36	99%	35	30	18
Teacher	PIRLS	4	146	97%	142	30	71
Teacher	TIMSS	4	146	97%	142	30	71
Teacher	TIMSS	8	65	97%	63	30	32
Teacher	TIMSS	8	65	97%	63	30	32
Recruitment and Pre-Assessment Activity
School Administrator	PIRLS & TIMSS	4 & 8	120	72%	86	90	129
School Coordinator	PIRLS & TIMSS	4 & 8	86	99%	86	240	344

The burden to respondents for the TIMSS and PIRLS 2011 full-scale study is calculated for the estimated time required of students and school staff to complete pre-assessment and assessment activities (see table 2). Again the time for students to complete the cognitive instruments is excluded. Burden calculations are also provided at the bottom of table 2 for the four states (TBD) that will participate in TIMSS State Benchmarking Study in grade 8.

For the full-scale study, the average response burdens and costs per hour for students, school administrators, teachers, and schools coordinators are the same as for the field test. Given these average times and costs per hour, the dollar cost of the full-scale study without state benchmarking is estimated for students at $206,739; for school administrators at $55,250; for teachers at $77,910; and for school coordinators at $77,280. The dollar cost of the full-scale study’s state benchmarking is estimated for students at $39,002; for school administrators at $20,000; for teachers at $7,000; and for school coordinators at $28,000.

Table 2. Burden estimates for TIMSS and PIRLS 2011 Full-Scale Study

Respondent	Study		Grade	Sample Size		Expected response rate		# of Respondents			Per Respondent (Minutes)	Total Burden Hrs.
FULL SCALE STUDY
Student		PIRLS	4		16,800		93%		15,624	50		13,020
Student		TIMSS	4		16,800		93%		15,624	50		13,020
Student		TIMSS	8		12,000		93%		11,160	50		9,300
School		PIRLS	4		52		99%		51	30		26
School		PIRLS & TIMSS	4		302		99%		300	30		150
School		TIMSS	4		52		99%		51	30		26
School		TIMSS	8		152		99%		150	30		75
Teacher		PIRLS	4		1,650		97%		1,600	30		800
Teacher		TIMSS	4		1,650		97%		1,600	30		800
Teacher Science		TIMSS	8		645		97%		626	30		313
Teacher Math		TIMSS	8		645		97%		626	30		313
Recruitment and Pre-Assessment Activity
School Administrator		PIRLS & TIMSS	4 & 8		650		85%		552	90		828
School Coordinator		PIRLS & TIMSS	4 & 8		552		99%		552	240		2,208
STATE BENCHMARKING
Student		TIMSS	8		8,600		93%		8,000	50		6,667
School		TIMSS	8		202		99%		200	30		100
Teacher Science		TIMSS	8		206		97%		200	30		100
Teacher Math		TIMSS	8		206		97%		200	30		100
Recruitment and Pre-Assessment Activity
School Administrator		PIRLS & TIMSS	4 & 8		236		85%		200	90		300
School Coordinator		PIRLS & TIMSS	8		202		99%		200	240		800

The total number of respondents for the full-scale study, including state benchmarking, is estimated at 57,516 (1,504 during the recruitment period plus additional 56,012 during the assessment period), with a total burden time of 48,946 hours (4,136 during the recruitment period plus additional 44,810 during the assessment period).

A.13 Total Annual Cost Burden

Other than the burden associated with completing these pre-assessment activities and questionnaires (estimated above in Section A.12), the study imposes no additional cost to respondents nor the imposition of any record-keeping requirement.

A.14 Annualized Cost to Federal Government

The cost to the federal government for conducting the TIMSS and PIRLS 2010 field test is estimated to be $2.8 million, combined. This estimate includes all direct and indirect costs of preparing for and conducting the field test including sampling ($68,000), data collection ($321,000) scoring ($804,000) and data set preparation ($559,000).

The total cost to the federal government for conducting the TIMSS and PIRLS 2011 full-scale study is estimated to be $9.9 million. These estimates include all direct and indirect costs of the project, including sampling ($79,000), data collection ($1.3 million), data analysis ($742,088), and reporting ($672,000) phases.

The total cost to the federal government for conducting 2011 TIMSS grade 8 state benchmarking in four states is estimated to be $2.0 million. This estimate includes all direct and indirect costs of the project, including sampling ($40,000), data collection ($800,000), data analysis ($50,000), and reporting ($75,000).

The costs of these studies are spread out over a 4-year period, at the total annual cost to the federal government of $3,675,000.

A.15 Program Changes or Adjustments

The central change to TIMSS and PIRLS from previous cycles is the effort to combine the grade 4 sample and administer both TIMSS and PIRLS to the same schools for the field test. If this is deemed feasible and acceptable response rates are obtained during the field test, this sampling procedure will be carried through to the full scale study. If the combined sample is not deemed feasible, then two independent grade 4 school samples will be drawn for TIMSS and PIRLS full scale data collections. A decision on sampling for grade 4 schools for the full scale assessment will be made after the conclusion of the field test in spring 2010.

A.16 Plans for Tabulation and Publication

The TIMSS and PIRLS International Study Center will prepare separate reports for mathematics and science in Grade 4 and Grade 8 for TIMSS, and a separate report for PIRLS describing reading literacy at grade 4. The four states participating in grade 8 TIMSS will also appear in the international TIMSS reports published by the International Study Center. The first of these reports will be released in December 2012. As has been customary, NCES will also release a report for each study at the same time as the international reports are released, interpreting the results for the U.S. audience. NCES reports on initial data releases are generally limited to simple bivariate statistics. There are currently no plans to conduct complex statistical analyses of either dataset. (Examples of past reports on TIMSS and PIRLS data can be found at http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2009001 and http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2008017.) In the spring of 2013, the International Study Center will also prepare technical reports for both TIMSS and PIRLS describing the design and development of both TIMSS and PIRLS and detailing the scaling procedures, weighting, missing value imputation, and analyses. After the release of the international data, NCES plans to release the national data and an accompanying User’s Guide for each study.

Electronic versions of each publication are generally made available on the NCES website. Schedules for tabulation and publication of TIMSS and PIRLS 2011 results in the United States are dependent upon receiving data files from the international sponsoring organization. With this in mind, the expected data collection dates and a tentative reporting schedule are as follows:

October – December 2009	Prepare data collection manuals, forms, and other assessment materials for field test
November 2009 – February 2010	Gain cooperation of states, districts, schools for field test
March – April 2010	Collect field test data
April 30, 2010	Deliver field test raw data to international sponsoring organization
June 2010	Receive Field Test Report from international sponsors
July 2010 – February 2011	Prepare for the Main Study phase
April – May 2011	Collect Main Study data (Northern hemisphere)
August 31, 2011	Deliver main study raw data to international sponsoring organization
May 2012	Receive final data files from international sponsors
June – December 2012	Produce General Audience Report, Survey Report, and Technical Report for the United States

PIRLS/TIMSS Production Schedule

October 1, 2009	Submit to OMB request for emergency clearance for TIMSS/PIRLS field test.
November 1, 2009	Receive OMB clearance for field test school recruitment and field test administration.
November 1, 2009	Final national version of field test questionnaires approved by the international consortium.
November 2, 2009	Submit to OMB request for regular clearance for TIMSS/PIRLS full-scale study school recruitment, 60-day notice waver for full-scale study collection instruments clearance, and submit the finale versions (as approved by the IEA) of field test collection instruments.
Early November, 2009	List of sample schools released by the international consortium.
Early November, 2009	Contact states with field test sample schools, a week later contact the districts and comply with all of their collection recruitment requirements, then begin to recruit schools and obtain classroom rosters for generating the classroom sample.
October - December  2009	Prepare data collection manuals, forms, and assessment materials for field test.
January - February 2010	Print and proof all field test materials.
February - March, 2010	Receive OMB clearance for full-scale school recruitment and 60-day notice waver for full-scale study collection instruments clearance.
March 1 - April 15, 2010	Collect field test data.
April 2010	1. Contact states with full-scale sample schools; 2. contact districts with sample schools and comply with all of their collection recruitment requirements; 3. begin to recruit schools and obtain classroom rosters for generating the classroom sample.
May 14, 2010	Deliver raw data to international sponsoring organization.
June 2010	Receive Field Test Report from the international consortium.
August 2010	Receive final international version of PIRLS full-scale study questionnaires.
September 2010	Receive final international version of TIMSS full-scale study questionnaires.
October 2010	Submit to NCES/RIMS full-scale study documents for OMB clearance.
October 2010	Final national version of PIRLS full-scale study questionnaires approved by international sponsoring organization.
November 2010	Final national version of TIMSS full-scale study questionnaires approved by international sponsoring organization.
November –December 2010	Receive OMB clearance for finalized full-scale study materials.
April – May 2011	Collect Full-Scale Study data (Northern hemisphere)
August 31, 2011	Deliver raw data to international sponsoring organization
May 2012	Receive final data files from international sponsors
June - December 2012	Produce General Audience Report, Survey Report, and Technical Report for the United States

A.17 Display OMB Expiration Date

The OMB expiration date will be displayed on all data collection materials.

A.18 Exceptions to Certification Statement

No exceptions to the certifications are requested.

1 Data collected in TIMSS 2007 for Grade 4 in the United States showed that the between-school variance along with the between-classroom variance accounted for roughly 25 percent of the total population variance. Of this, between-school variance accounted for roughly 60 percent, leaving 40 percent due to the between-classroom variance.

i