TIMSS_PIRLS 2011 Part B

Trends in International mathematics and science study (TIMSS) and progress in international reading literacy study (PIRLS) 2010 and 2011 Data Collections

REQUEST FOR OMB REVIEW

Supporting Statement Part B

Prepared by:

National Center for Education Statistics

U.S. Department of Education

Washington, DC

September 18, 2009

draft

2. COLLECTION OF INFORMATION EMPLOYING STATISTICAL INFORMATION

B.1 Respondent Universe and Response Rates

The respondent universe for TIMSS and PIRLS field test is all students enrolled in grade 4 (TIMSS and PIRLS) and grade 8 (TIMSS) during the 2010-2011 school year. National probability samples of these students will be drawn. All selected students will be asked to participate in a combined mathematics and science assessment (TIMSS) or a reading literacy assessment (PIRLS).

For grade 4, a sample of 50 schools will be selected for the field test, with the goal of obtaining participation from a minimum of 45 schools. The universe for the selection of schools is all types of schools in 5 populous states (California, Connecticut, Florida, Ohio and Texas). The field test will introduce an attempt to combine TIMSS and PIRLS in 30 of the schools selected at fourth grade. In these schools, up to 4 classes will be selected; the first two classes will be randomly assigned as a pair to either TIMSS or PIRLS, with the remaining classes assigned to the other assessment. The remaining 20 schools will be small schools with only one or two fourth-grade classes, which will be randomly assigned to TIMSS or PIRLS at the selection stage. Two fourth-grade classrooms will be selected in these smaller schools unless there is only one fourth grade class in the school. In that case, the single class will be selected. Again, only students in the selected intact classrooms will be assessed.

For grade 8, a sample of 50 schools will be selected for TIMSS, with the goal of garnering participation from a minimum of 45 schools. The universe for the selection of schools is all types of schools in the same five states indentified for the grade 4 sample--California, Connecticut, Florida, Ohio and Texas. Students will be selected by drawing a sample of two intact mathematics classrooms (in which grade 8 students are enrolled) in each sampled school. In schools that have only one mathematics classroom at grade 8, that one classroom (instead of a sample of two classrooms) will be selected to participate. Only students in the selected intact classrooms will be assessed for TIMSS at grade 8.

It is expected that, for schools, an overall response rate of 85-90 percent will be achieved after replacement, and that, within participating schools, between 87 and 90 percent of the students will respond. Recent field work in schools has shown that response rates of 90 percent or higher are increasingly difficult to achieve. For the TIMSS data collection in 2007, school participation at grade 4 was 70 percent before replacement and 89 percent after replacement. At grade 8, it was 68 percent before replacement and 83 percent after replacement. The overall student participation rate was 95 percent at grade 4 and 93 percent at grade 8.

In TIMSS 2003, school participation in the United States at grade 4 was 70 percent before replacement and 82 percent after replacement. At grade 8 in 2003, school participation was 71 percent before replacement and 78 percent after replacement. The overall student participation rate was 95 and 94 percent in grades 4 and 8, respectively. This was a decline from TIMSS 1999, when the school participation rate in the United States for grade 8 was 83 percent before replacement and 90 percent after replacement. The overall student response rate within those schools was 94 percent. In 1995, the U.S. grade 4 sample achieved a school response rate of 86 percent and a student response rate of 94 percent. No replacement schools were used in grade 4 in 1995.

The school response rates for PIRLS in 2006 were 57 percent before replacement and 86 percent after replacement. The overall student participation was 96 percent. In 2001, school participation was 61 percent before replacement and 86 percent after replacement. The overall student participation rate was 96 percent.

B.2 Procedures for Collection of Information

Statistical Methodology

For TIMSS and PIRLS 2011 at grades 4 and TIMSS 2011 at grade 8, the sample design will consist of a stratified national sample of schools having the relevant grade. For the four states participating in TIMSS grade 8 state benchmarking, the sample design will consist of a stratified state sample of schools having an eighth grade. The school sampling frames will be obtained from the Common Core of Data (CCD) and Private School Universe Survey (PSS) files. The selection of schools will be with probability proportional to size, where the size measure is an estimate of the number of students enrolled in the relevant grade, obtained from the CCD and PSS files.

In both the national and state samples, two substitute schools will be identified for each original school, in accordance with the international procedures. These numbers allow for the patterns of school nonresponse that have been observed in the most recent TIMSS and PIRLS studies. Stratification variables to be used in the design for the national sample will be public/private status, region of the country, type of location, minority enrollment, and school size. Stratification variables to be used in the design for the state benchmarking sample will be public/private status, type of location, minority enrollment, and school size. In both cases, the two substitute schools associated with each original selection will be those schools that are adjacent to the original selection on the sample frame when sorted by the stratification variables. Thus the substitute schools will match the original selection as closely as possible with regard to all of the stratification variables.

The international protocol calls for the school sample to be selected by the IEA International Study Center, using a list of schools supplied by the U.S. National Research Coordinator. However, the process is to be a collaborative one, where Westat will select the TIMSS and PIRLS U.S. national and state school samples, and have the sample verified by the International Study Center. This is consistent with past practice.

As mentioned, the field test sampling plan will employ a design that provides a combined sample of schools at grade 4 in order to ascertain the feasibility of administering TIMSS and PIRLS in the same schools in the full scale collection. The field test sample at grade 4 will consist of 50 schools from across 5 populous states (California, Connecticut, Florida, Ohio and Texas). This is somewhat of a purposive sample as the goals of the field test are to understand response to the combined option and also to evaluate assessment items, but to do so in a cost efficient way.

Each state will have 10 sampled schools and 2 substitute schools for each sampled school. Within this set of 50 schools and across the 5 states, 30 schools will be “large” schools, defined as having 3 or more fourth-grade classes. These large schools will be asked to supply 3 to 4 classes, depending on the number that can be selected. In the case of schools with 3 classes, one class will be randomly assigned to one study and the remaining two to the other. In the case of schools with four classes, two classes will be assigned to TIMSS and two to PIRLS.

The remaining 20 schools will be small schools with less than 3 classes. These schools will be randomly assigned to TIMSS or PIRLS. Only one study will be administered in these schools. The distribution of large and small schools across states will be approximately equal.

The effectiveness of the combined approach at grade 4 will be evaluated by examining the school response rates by state and comparing these with the rates from TIMSS 2007 and PIRLS 2006. If the rates from the field test are similar, then consideration will be given to employing a combined approach in the full scale study. If school response in the field test does not resemble the rates from the previous cycles, or other major problems present themselves, then two independent samples will be drawn at grade 4 for TIMSS and PIRLS.

The TIMSS field test school sample at grade 8 will select 50 schools from the same five states, with each state also contributing 10 sampled schools, each with 2 substitutes. Only up to 2 classes will be selected from the grade 8 schools.

The four states participating in the TIMSS grade 8 state benchmarking will not participate in the field test; they will participate in the full scale data collection only.

For the classroom sampling, the key steps are as follows: (1) obtain a comprehensive list of all eligible classes, ensuring that each student enrolled in grade 4, or grade 8, is enrolled in exactly one listed class; (2) sort the classes in the order specified by the international procedures; (3) determine the number of classes listed, and develop a procedure to ensure that these data are available on files for weighting; (4) select a maximum of four classrooms at random, using the international procedures; (5) assign identification numbers to the students and teachers from the selected classroom and record on the appropriate international survey forms. Names are associated with unique student identification numbers in lists given to the School Coordinator to ensure that missing information can be obtained through follow-up sessions, if necessary, and so that teachers are cognizant of which students participated in the TIMSS or PIRLS assessments. It is important to note that the names of students do not leave the school and under no circumstances are the names of students or teachers included in the international or national database, nor is this information forwarded to any organization. After all data collection is complete, the School Coordinator is instructed to destroy the list of names associated with the uniquie IDs to ensure complete confidentiality and privacy of respondents, per NCES practice. Neither the contractor nor NCES retains these lists.

Estimation

All estimation and weighting procedures for the TIMSS and PIRLS data are prescribed by and are the responsibility of the international sponsoring agency and the TIMSS and PIRLS International Study Center. The United States, and all other participating countries, will comply with these procedures and policies by delivering the raw data, and documentation of sampling variables, to the study center for weighting and processing. (Note, however, that NCES will conduct a disclosure analysis prior to submitting the data to the international contractor so as to comply with current federal law.) The weighted data will then be returned to each country for national analyses. All data delivered to the TIMSS and PIRLS International Study Center will be devoid of any data that could lead to the identification of individuals.

There are no anticipated problems that would require specialized sampling procedures, nor will there be any use of periodic data collection cycles to reduce burden.

B.3 Maximizing Response Rates

To ensure a high response rate among teachers and principals, we will build interest in the project through individual contact and recruitment. Our estimated response rate will be at least 85 percent. Appropriate channels will be followed for securing cooperation of schools (e.g., contacting school system and obtaining permission from relevant school organizations). We have developed an invitational package to use with sample schools and also to inform their district and state level educators. As included in appendix B of the accompanying materials, the information will include a letter requesting participation, a brochure describing the projects and the types of questions that will be asked, a timeline showing the central activities for principals, school coordinators, teachers, and students, and a key information sheet providing more detail about events of the study.

B.4 Tests of Procedures

The United States is participating in a full field test for TIMSS and PIRLS 2011. The central goals of the field test are to evaluate the approach of assessing the same schools for both TIMSS and PIRLS in grade 4 and collecting item data to ascertain the reliability and utility of all variables to be collected and procedures to be used in the studies. If deemed necessary, NCES may request the contractor to conduct cognitive laboratories to ascertain how students (or teachers and principals) respond to a limited number of non-cognitive items for which additional information is sought. The cognitive laboratories would involve no more than 9 respondents per identical item.

B.5 Individuals Consulted on Statistical Design

The following are responsible for the statistical design of TIMSS:

• Pierre Foy, TIMSS and PIRLS International Study Center, Boston College (617-552-6253)

• Marc Joncas, Statistics Canada (613-951-0007)

Westat will be the contractor responsible for sampling and data analysis:

• David Kastberg, Project Director, Westat (301-294-3811)

• David Ferraro, Senior Statistician, Westat (301-251-4261)

Analysis and reporting will be performed by:

• TIMSS International Study Center, Boston College

• David Kastberg, Westat, under contract with NCES

• National Center for Education Statistics, U.S. Department of Education