ED Response to OMB Comments

1. We are confused as to how NCES can be prepared to begin recruiting now if it doesn’t yet have a final sample design.  Exactly when will the sample design be finalized? How can recruiting begin before the sample is drawn?  Please provide a specific schedule of key dates from now until the sample is drawn, is approved by the international body, and recruiting contacts begin.

We have by now finalized the sample design and have revised the relevant text throughout the current clearance package’s Supporting Statement Parts A and B and the memo outlining differences between this package and the one approved in November 2009 under emergency clearance (all revisions are shown in tracked changes in the attached three documents). The schedule has been updated on page 19 of Part A.

2. When will NCES decide whether to offer online questionnaires to teachers and students?  NCES mentions confidentiality concerns but we also would be concerned about operational and logistical issues.  Therefore, we would be concerned about any decision that would not be in time to implement it in the field test.

This issue has been resolved and is addressed in a revision on pages 7-8 of Part A.

3. Who will make the decision about what the children will receive as their “token?”  We typically approve a specific item(s) rather than simply an amount to spend unless the money is going to a school official to spend as they wish.  In the latter case, it is considered a school incentive rather than a student one.

This topic is addressed in a revision on page 11 of Part A.

4. NAEP-TIMSS Linking Study -- Why does NCES believe that there needs to be an increase from four to six states in this study?  What is the total additional cost of increasing by approximately 100 schools and 4000 students, and what share of the increase is each program bearing?

As is now reflected in the revisions in the package and in the accompanying memo, the decision to increase the number of states was made after considerable discussion with contractors and other experts about the design of the proposed NAEP-TIMSS Linking Study. These decisions have been finalized only recently, influenced by the newness of this project and its scope, timing of the initial decisions to undertake it, coordination needed among program areas and contractors, and results of the field test with regards to combining assessments in single schools. It was decided that the sample should be increased to 8 states in order to provide adequate data to validate the outcome of the NAEP-TIMSS Linking Study, raising the number of resulting state estimates from 4 to 8. NCES has requested $3.45 million for the state benchmarking effort to be conducted during the TIMSS testing window and $5 million for the braided assessment administration during the NAEP window [to be cleared under a separate OMB package], printing of all of the braided test booklets, scoring and linking of all of the NAEP blocks, and for contracting the NAEP state coordinators to recruit the additional 8th grade schools where braided assessment will be administered during the TIMSS and the NAEP testing windows. Based on the PIRLS/TIMSS field test and ensuing discussions with NCES staff and contractors, NCES designed the state samples so as to require fewer schools per state than originally estimated and arranged to coordinate school recruitment for the TIMSS/NAEP state benchmarking portion with the help of NAEP coordinators, thereby reducing the expected per-school cost.  As a result, NCES has determined that sampling 8 states is feasible within the requested budget.

Please note that the burden table has changed to reflect both the decision to administer PIRLS and TIMSS in the same schools at 4^th grade when possible, to include the braided assessment in the same schools as the national TIMSS assessment at 8^th grade when possible (approximately 50% of classroom will receive TIMSS only and 50% braided assessment), and increase the number of states used for state benchmarking (TIMSS only). Please also note that this package only requests clearance for the TIMSS background questionnaire burden associated with the braided assessment that will be administered during the TIMSS window, while the burden associated with the NAEP background questionnaire that will be given in association with the braided assessment during the NAEP window will be submitted for clearance under a separate NAEP package. The NAEP package will also explain in more details the NAEP-TIMSS Linking Study as a whole.

5. Now that the decision to do the joint sampling for TIMSS and PIRLS have been made, along with the additional of NAEP content, please provide documentation that describes the quality assurance process related to the spiraling of hard copy assessments.

As described in the supporting documents, TIMSS and PIRLS will be administered in the same schools at grade 4 (where the number of students permits). However, because TIMSS and PIRLS sample intact classes of students rather than students from across a school, the TIMSS and PIRLS assessment instruments will be administered to separate classes of students. The contractor will utilize the sampling software to randomly assign intact classes of students to either TIMSS or PIRLS, but not both. The designation as a TIMSS or PIRLS class at grade 4 will be clearly noted on the resulting school sampling and classroom sampling forms, which are used by the field staff.  Field staff are required to verify that the correct instruments are distributed to the selected class of students, and that the students who are expected to be in attendance are in fact in attendance (that is, no switching between TIMSS and PIRLS classes). Once the data collection is completed, all instruments are processed by Pearson at a central facility. As part of its quality control, the IEA (the international sponsoring organization) reviews the data to make sure it conforms to the approved sampling plan. 

 

The original TIMSS and PIRLS booklets will receive close scrutiny to ensure that they meet NCES and international standards. Specifically, once the contractor has assembled the booklets, per the instructions from the IEA, draft copies of the instruments (cognitive and non-cognitive) will be reviewed by NCES and ESSI (Education Statistics Services Institute). Assuming the instruments need no corrections, the contractor will direct the printer to create final digital and hardcopy proofs of all instruments for sign-off.  These documents will be thoroughly checked again for errors before printing.  NCES has made it a condition of sign-off that it receive printed copies of the instruments, rather than just the digital proofs, to make sure that no errors are introduced in the final printing.  These procedures will be followed for the original PIRLS (grade 4) and TIMSS instruments (grades 4 and 8).

 

As also described in the supporting documents, the grade 8 sample calls for sampling two intact mathematics classes from each of the 600 sampled schools. One of the intact classes will receive the TIMSS instruments; the other intact class will receive the hybrid "braided" instruments that include both TIMSS and NAEP item blocks.  PIRLS is not administered at grade 8. During the TIMSS data collection window, the "braided" booklets will contain 4 blocks of items: three blocks of TIMSS and one block of NAEP.  The NAEP blocks will be rotated between the first and second block position in the booklet to mimic their position in the original NAEP instruments.  The third and fourth blocks will always be TIMSS blocks.  A detailed design of the "braided" booklets--to account for all item blocks from both assessments and their original positions in the original assessments--has been created and reviewed by NCES, experts, and contractors. The original NAEP mathematics assessment has 10 blocks of items; the original NAEP science assessment has 9 blocks of items. The original TIMSS assessment has 14 blocks of mathematics items and 14 blocks of science items. To ensure that each NAEP and TIMSS item block is rotated among the four positions in the "braided" booklets, a total of 38 "braided" booklets will be created. This effectively accounts for all the positions of the item blocks in the original assessments, allowing for detailed analyses comparing performance of the items across the original and "braided" booklets as well as to investigate potential item fatigue.

 

Each original NAEP and TIMSS item block has a unique ID. Each of the 38 "braided" booklets has a map of which NAEP and TIMSS items blocks are to be included, as well as the exact position of each block (1 through 4). This detailed map of the "braided" booklets will be used by ETS to guide the creation of the booklets.  As a quality control measure, NCES will review each of the 38 "braided" booklets to ensure that every item block is correctly placed and that no items have been deleted or reordered.  Each student in the sampled intact mathematics classes will receive a predetermined booklet (that is, the "braided" booklets will be pre-assigned to students in the classes so that the booklets are spiraled among the students). As with the original assessments, field staff will be required to document that each student in the "braided" booklet sessions received the correct booklet.

 

Similar to the QC process for the original TIMSS and PIRLS booklets, the "braided" booklets will also receive close scrutiny to ensure that they meet NCES and international standards. The contractor will assemble the booklets, per the detailed map for the 38 "braided" booklet. Draft copies of the "braided" booklets will be reviewed by NCES and ESSI (Education Statistics Services Institute). Assuming the instruments need no corrections, the contractor will be directed to create final digital and hardcopy proofs of all instruments for sign-off.  These documents will be thoroughly checked again before the final print run. NCES has made it a condition of sign-off that it receive printed copies of the instruments, rather than just the digital proofs, to make sure that no errors are introduced in the final print run.