TIMSS_PIRLS_2011 Full Scale Part B

Trends in International mathematics and science study (TIMSS) and progress in international reading literacy study (PIRLS) 2011 Data Collection

REQUEST FOR OMB REVIEW

Supporting Statement Part B

Prepared by:

National Center for Education Statistics

U.S. Department of Education

Washington, DC

November 2, 2010

2. COLLECTION OF INFORMATION EMPLOYING STATISTICAL INFORMATION

B.1 Respondent Universe and Response Rates

The respondent universe for TIMSS and PIRLS full-scale study is all students enrolled in grade 4 (TIMSS and PIRLS) and grade 8 (TIMSS) during the 2010-2011 school year. National probability samples of schools containing a grade 4 or grade 8 were drawn in June 2010 and state probability samples for North Carolina schools containing a grade 4 were drawn in August 2010. State probability samples for Florida will be drawn in November, 2010. All selected students will be asked to participate in a combined mathematics and science assessment (TIMSS) or a reading literacy assessment (PIRLS).

As noted in Supporting Statement Part A, NCES pursued a combined sample for TIMSS and PIRLS at grade 4, based on response rates and feedback from the spring 2010 field test. The fielding of a combined sample in the field test indicated no adverse effect on response rates at grade 4. Furthermore, recruitment staff received positive feedback from schools based on this approach. The decision to proceed with a combined TIMSS and PIRLS grade 4 sample has been reviewed and approved by the NCES Chief Statistician.

At grade 4, a national sample of 450 schools was selected with the goal of obtaining participation from a minimum of 383 schools. The universe for the selection of schools was all types of schools containing a grade 4 in the 50 states and the District of Columbia. A state sample of schools will be drawn for Florida to produce a total state sample (including those Florida schools in the national sample) of 75 schools with the goal of obtaining participation from a minimum of 64 schools. A state sample of schools was drawn for North Carolina to produce a total state sample (including those North Carolina schools in the national sample) of 70 schools with the goal of obtaining participation from a minimum of 60 schools.

The TIMSS and PIRLS full-scale study combines TIMSS and PIRLS in the schools selected at fourth grade as follows: In schools with three or more fourth-grade classes, two classes will be randomly selected and assigned to either TIMSS or PIRLS, with the remaining classes (up to four) assigned to the other assessment. In schools with two fourth-grade classes, both classes will be selected; one class will be randomly assigned to TIMSS and the other to PIRLS. In schools with one fourth-grade class the class will be selected and randomly assigned to either TIMSS or PIRLS. Again, only students in the selected intact classrooms will be assessed.

For grade 8, a sample of 600 schools was selected for the TIMSS national sample, with the goal of garnering participation from a minimum of 510 schools. The universe for the selection of schools was all types of schools containing a grade 8 in the 50 states and the District of Columbia. Students will be selected by drawing a sample of two intact mathematics classrooms (in which grade 8 students are enrolled) in each sampled school. As described in Supporting Statement Part A, one of these two classes will be randomly assigned to participate in TIMSS; the other class will receive the “braided” booklets that are part of the NAEP-TIMSS Linking Study design. Based on discussions with the international organizing body, the international sampling referee and the contractor, an increase in the school sample size to 600 will counter the effect of administering TIMSS to only one class, rather than two classes. In schools that have only one mathematics classroom at grade 8, that one classroom (instead of a sample of two classrooms) will be selected to participate in TIMSS. Only students in the selected intact classrooms will be assessed for TIMSS at grade 8. Details about the portion of the NAEP-TIMSS Linking Study that will be carried out during the NAEP testing window were provided to OMB through the clearance process established for NAEP and were approved by OMB in July 2010 (OMB# 1850-0790 v.29).

It is expected that, for schools, an overall response rate of 85-90 percent will be achieved after replacement, and that, within participating schools, between 87 and 90 percent of the students will respond. Recent field work in schools has shown that response rates of 90 percent or higher are increasingly difficult to achieve.

For the TIMSS data collection in 2007, school participation at grade 4 was 70 percent before replacement and 89 percent after replacement (see table 2). At grade 8, it was 68 percent before replacement and 83 percent after replacement. The overall student participation rate was 95 percent at grade 4 and 93 percent at grade 8. In TIMSS 2003, school participation in the United States at grade 4 was 70 percent before replacement and 82 percent after replacement. At grade 8 in 2003, school participation was 71 percent before replacement and 78 percent after replacement. The overall student participation rate was 95 and 94 percent in grades 4 and 8, respectively. This was a decline from TIMSS 1999, when the school participation rate in the United States for grade 8 was 83 percent before replacement and 90 percent after replacement. The overall student response rate within those schools was 94 percent. In 1995, the U.S. grade 4 sample achieved a school response rate of 86 percent and a student response rate of 94 percent. No replacement schools were used in grade 4 in 1995.

The school response rates for PIRLS in 2006 were 57 percent before replacement and 86 percent after replacement. The overall student participation was 96 percent. In 2001, school participation was 61 percent before replacement and 86 percent after replacement. The overall student participation rate was 96 percent.

Table 2. Recent TIMSS and PIRLS school and student participation rates

B.2 Procedures for Collection of Information

Statistical Methodology

For TIMSS and PIRLS 2011 at grades 4 and TIMSS 2011 at grade 8, the sample design consists of a stratified national sample of schools having the relevant grade. In addition, for the eight states participating in TIMSS grade 8 state benchmarking, as described in Part A, the sample design consists of a stratified state sample of schools having an eighth grade. The school sampling frames were obtained from the Common Core of Data (CCD) and Private School Universe Survey (PSS) files. The selection of schools was with probability proportional to size, where the size measure is an estimate of the number of students enrolled in the relevant grade, obtained from the CCD and PSS files.

In both the national and state samples, two substitute schools were identified for each original school, in accordance with the international procedures. These numbers allow for the patterns of school nonresponse that have been observed in the most recent TIMSS and PIRLS studies. Stratification variables used in the design for the national sample were public/private status, region of the country, type of location, minority enrollment, and school size. Stratification variables used in the design for the state benchmarking sample were public/private status, type of location, minority enrollment, and school size. In both cases, the two substitute schools associated with each original selection were those schools that are adjacent to the original selection on the sample frame when sorted by the stratification variables. Thus the substitute schools matched the original selection as closely as possible with regard to all of the stratification variables.

The international protocol called for the school sample to be selected by the IEA International Study Center, using a list of schools supplied by the U.S. National Research Coordinator. However, the actual process was a collaborative one, where Westat selected the TIMSS and PIRLS U.S. national and state school samples, and the sample was verified by the International Study Center. This was consistent with past practice.

For the classroom sampling, the key steps are as follows: (1) obtain a comprehensive list of all eligible classes, ensuring that each student enrolled in grade 4, or grade 8, is enrolled in exactly one listed class; (2) sort the classes in the order specified by the international procedures; (3) determine the number of classes listed, and develop a procedure to ensure that these data are available on files for weighting; (4) select a maximum of four classrooms at random, using the international procedures; (5) assign identification numbers to the students and teachers from the selected classroom and record on the appropriate international survey forms. Names are associated with unique student identification numbers in lists given to the School Coordinator to ensure that missing information can be obtained through follow-up sessions, if necessary, and so that teachers know, for attendance purposes, which students participated in the TIMSS or PIRLS assessments. It is important to note that the names of students do not leave the school and under no circumstances are the names of students or teachers included in the international or national database, nor is this information forwarded to any organization. After all data collection is complete, the School Coordinator is instructed to destroy the list of names associated with the unique IDs to ensure complete confidentiality and privacy of respondents, per NCES practice. Neither the contractor nor NCES retains these lists.

Estimation

All estimation and weighting procedures for the TIMSS and PIRLS data are prescribed by and are the responsibility of the international sponsoring agency and the TIMSS and PIRLS International Study Center. The United States, and all other participating countries, will comply with these procedures and policies by delivering the raw data, and documentation of sampling variables, to the study center for weighting and processing. (Note, however, that NCES will conduct a disclosure analysis prior to submitting the data to the international contractor so as to comply with current federal law.) The weighted data will then be returned to each country for national analyses. All data delivered to the TIMSS and PIRLS International Study Center will be devoid of any data that could lead to the identification of individuals.

There are no anticipated problems that would require specialized sampling procedures, nor is there any use of periodic data collection cycles to reduce burden.

B.3 Maximizing Response Rates

To ensure a high response rate among teachers and principals, we have built interest in the project through individual contact and recruitment. Our estimated response rate will be at least 85 percent. Appropriate channels are followed for securing cooperation of schools (e.g., contacting school system and obtaining permission from relevant school organizations). We developed an invitational package to use with sample schools and also to inform their district and state level educators. As included in appendix B of the accompanying materials (approved in April 2010; OMB# 1850-0645 v.6), the invitational package includes a letter requesting participation, a brochure describing the projects and the types of questions that will be asked, a timeline showing the central activities for principals, school coordinators, teachers, and students, and a key information sheet providing more detail about events of the study.

B.4 Tests of Procedures

The United States participated in a field test for TIMSS and PIRLS 2011 in spring 2010. The central goals of the field test were to evaluate the approach of assessing the same schools for both TIMSS and PIRLS in grade 4 and collecting item data to ascertain the reliability and utility of all variables to be collected and procedures to be used in the studies. As noted in Supporting Statement Part A and above, NCES determined that the fielding of a combined grade 4 sample in the field test had no adverse effect on response rates. Furthermore, recruitment staff received positive feedback from schools based on this approach. The decision to proceed with a combined TIMSS and PIRLS sample at grade 4 was reviewed and approved by the NCES Chief Statistician.

B.5 Individuals Consulted on Statistical Design

The following persons are responsible for the statistical design of TIMSS:

• Pierre Foy, TIMSS and PIRLS International Study Center, Boston College (617-552-6253)

• Marc Joncas, Statistics Canada (613-951-0007)

Westat is the contractor responsible for sampling and data analysis:

• David Kastberg, Project Director, Westat (301-294-3811)

• David Ferraro, Senior Statistician, Westat (301-251-4261)

Analysis and reporting will be performed by:

• TIMSS International Study Center, Boston College

• David Kastberg, Westat, under contract with NCES

• National Center for Education Statistics, U.S. Department of Education