Memo

MEMORANDUM OMB # 1850-0645 v.6

DATE: April 7, 2010

TO: Shelly Martinez

Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Patrick Gonzales

Stephen Provasnik

National Center for Education Statistics

THROUGH: Kashka Kubzdela

National Center for Education Statistics

SUBJECT: Differences between the TIMSS/PIRLS 2011 clearance package submitted for review (OMB# 1850-0645 v.6) and the current active record (# 1850-0645 v.5)

To aid OMB’s review, this memo outlines differences between Trends in International Mathematics and Science Study (TIMSS) and the Progress in International Reading Literacy Study (PIRLS) materials approved under filed test (FT) emergency clearance on November 3, 2009 (OMB# 1850-0645 v.5) and those included in the current regular submission package for full-scale (FS) study recruitment (OMB# 1850-0645 v.6).

In this submission, NCES is requesting that OMB (a) approve recruitment for the PIRLS/TIMSS 2011 full-scale study and state benchmarking study, including recruitment materials and burden times for recruiting selected schools, school districts, and state education agencies starting in May 2010, and (b) grant a waiver of the 60-day federal register notice for the clearance of the main study instruments for the PIRLS/TIMSS April-May 2011 data collection given that the main study questionnaires are not expected to deviate significantly from the approved field test questionnaires. However, because the field test data collection activities will not be completed until May 2010, we are also asking to carry over with this request the total number of respondents and burden hours cleared under OMB# 1850-0645 v.5. In June 2010, after the field test activities are completed, we will submit a change request to reduce the cleared burden hours to only those needed for the full scale recruitment as described in the attached supporting statement Part A. Additionally, included with this request are copies of the final version of the non-cognitive data collection instruments cleared on November 3, 2009 (OMB# 1850-0645 v.5).

Differences between the FT and FS OMB clearance packages

1. The request for approval for the field test was changed to request for approval of recruitment materials and activities for the full scale study (p.ii). Additionally, information regarding the full scale study has been included so that the 60-day federal register notice can be waived at the time of the FS study clearance. As a result, language that mentioned the field test was changed to refer to the full scale study throughout Part A and B and in the recruitment letters and parental notification letters.

2. Two boilerplate endorsement letters from the Secretary of Education were added to appendix C as part of the recruitment materials.

3. Additional background information about PIRLS and TIMSS was added to the Preface of Part A (p.i).

4. In section A1, the explanation of the combined sample was finalized and made more explicit with the following sentences added to p.4 before the header “Organization and Direction of TIMSS and PIRLS”: “Because no fourth-grade class in the United States will receive both TIMSS and PIRLS, separate TIMSS and PIRLS questionnaires are required for fourth-grade teachers and students. However, in schools where both TIMSS and PIRLS are administered at grade 4, the administrators of these schools will receive a single, combined TIMSS and PIRLS school questionnaire.”

5. Throughout the discussion of the ‘TIMSS-NAEP’ Linking Study, the name of the study was changed to the NAEP-TIMSS Linking Study, and the number of states to be used in the study was revised upward from four to eight .. The decision to increase the number of states was made after considerable discussion with contractors and other experts about the design of the proposed NAEP-TIMSS Linking Study. It was decided that the sample should be increased to 8 states in order to provide adequate data to validate the outcome of the NAEP-TIMSS Linking Study, resulting in 8 rather than only 4 state estimates. NCES has requested $3.45 million for the state benchmarking effort to be done during the TIMSS testing window and $5 million for the braided assessment administration during the NAEP window [to be cleared under a separate OMB package], printing of all of the braided test booklets, scoring and linking of all of the NAEP blocks, and for contracting the NAEP state coordinators to recruit the additional 8^th grade schools where braided assessment will be administered during either the TIMSS or the NAEP testing window. Based on the PIRLS/TIMSS field test and ensuing discussions with NCES staff and contractors, NCES designed the state samples so as to require fewer schools per state than originally estimated and arranged to coordinate school recruitment for the TIMSS/NAEP state benchmarking portion with the help of NAEP coordinators, thereby reducing the expected per-school cost.  As a result, NCES has determined that sampling 8 states is feasible within the requested budget.

6. In section A3 (p.7), the following sentence was revised as follows (with bolded text added): “The design and procedures for TIMSS and PIRLS 2011 are prescribed internationally and data collection involves paper and pencil responses student assessments and questionnaires, as well as optional online or paper and pencil school and teacher questionnaires.”

7. In section A12, the estimates of number of respondents, burden hours, and costs were revised as a result of more precise estimation of the number of respondents and schools and the increase in the number of states in the NAEP-TIMSS Linking Study. Table 1 has also been revised. The number of respondents, burden hours, and costs for independent grade 4 samples was eliminated, as it has been decided to pursue a combined grade 4 sample for TIMSS and PIRLS. In addition, we have revised the number of respondents, burden hours, and costs for the grade 8 sample and the grade 8 state benchmarking samples to reflect the inclusion of eight states. These changes reflect final decisions made about the sample design. The samples have now been selected by the contractor based on these finalized designs.

8. Part B was revised to reflect the finalized sampling design for grades 4 and 8. The changes are as follows:

   1. As noted in the Supporting Statement Part A, NCES will pursue a combined sample for TIMSS and PIRLS at grade 4, based on response rates and feedback from the spring 2010 field test. The fielding of a combined sample in the field test indicated no adverse effect on response rates at grade 4. Furthermore, recruitment staff received positive feedback from schools based on this approach. The decision to proceed with a combined TIMSS and PIRLS grade 4 sample has been reviewed and approved by the NCES Chief Statistician.

   2. The combined grade 4 sample to be used for the full-scale study will require a sample of 450 schools selected with the goal of obtaining participation from a minimum of 383 schools. The universe for the selection of schools is all types of schools containing a grade 4 in the 50 states and the District of Columbia. The TIMSS and PIRLS full-scale study will combine TIMSS and PIRLS in 350 of the schools selected at fourth grade. In these schools, up to four classes will be selected; the first two classes will be randomly assigned as a pair to either TIMSS or PIRLS, with the remaining classes assigned to the other assessment. The remaining 100 schools will be small schools with only one or two fourth-grade classes, which will be randomly assigned to TIMSS or PIRLS at the selection stage. Two fourth-grade classrooms will be selected in these smaller schools unless there is only one fourth grade class in the school. In that case, the single class will be selected. Again, only students in the selected intact classrooms will be assessed.

   3. For grade 8, a sample of 600 schools will be selected for the TIMSS national sample, with the goal of garnering participation from a minimum of 510 schools. The universe for the selection of schools is all types of schools containing a grade 8 in the 50 states and the District of Columbia. Students will be selected by drawing a sample of two intact mathematics classrooms (in which grade 8 students are enrolled) in each sampled school. One of these two classes will be randomly assigned to the actual TIMSS assessment; the other class will be assigned to the “braided” booklets that are part of the NAEP-TIMSS Linking Study design. Based on discussions with the international organizing body, the international sampling referee and the contractor, an increase in the school sample size to 600 will counter the effect of administering TIMSS to only one class, rather than two classes. In schools that have only one mathematics classroom at grade 8, that one classroom (instead of a sample of two classrooms) will be selected to participate in TIMSS. Only students in the selected intact classrooms will be assessed for TIMSS at grade 8. Details about the portion of the NAEP-TIMSS Linking Study that will be carried out during the NAEP testing window will be provided through the OMB review and clearance process established for NAEP.

9. The discussion of the expected response rates in Part B was revised to reflect the full-scale study and table 2 was added to help clarify the discussion.

10. In the final version of the non-cognitive data collection instruments cleared on November 3, 2009 (OMB# 1850-0645 v.5), no changes were made except for all occurrences in the student questionnaire of “e.g.” being replaced with “for example.”