EDIT Supporting Statement B

Assessing Early Childhood Teachers’ Use of Child Progress Monitoring to Individualize Teaching Practices

OMB Generic Information Collection Request for Pre-testing
0970-0355

OMB Supporting Statement
Part B


September 2015

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

7th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

Project Officer:

Laura Hoard

CONTENTS

OMB SUPPORTING STATEMENT B 1

B1. Respondent universe and sampling methods 1

B2. Procedures for collection of information 1

B3. Methods to maximize response rates and deal with nonresponse 2

B4. Tests of procedures or methods to be undertaken 3

B5. Individuals consulted on statistical aspects and individuals collecting and/or analyzing data 3

OMB SUPPORTING STATEMENT B

B1. Respondent universe and sampling methods

A purposive sample of nine preschool classrooms will be selected based on specific characteristics including classrooms with English-speaking teachers¹ who are using both a curriculum-embedded ongoing assessment system and a standard progress monitoring tool (for example, general outcomes measures). We have a list of preschool programs in the northeast US that are implementing standard progress monitoring tools and will begin by reaching out to those nearest to our Washington DC and Princeton, NJ office locations. When we speak with each program, we will ask about the use of both a both a curriculum-embedded ongoing assessment system and a standard progress monitoring tool (The recruitment call script is in Attachment B).

The sample of nine teachers will allow the research team to advance the development of the EDIT measure. The sample of teachers at each recruited program will be selected by the program director. We expect the response rate for this work will be 100 percent. The EDIT measure will be completed by EDIT team members who can confer and ensure there is no item non-response. The teacher questionnaire will be completed by the teacher at the end of the visit. The EDIT team members will review the teacher questionnaire before leaving the site, and if there is item non-response, the team will inquire whether the teacher left the item blank intentionally.

The EDIT team will examine the distribution of ratings and frequencies of ratings for each item.

B2. Procedures for collection of information

As depicted in Figure B.1, project activities begin with recruitment of the setting (Attachments A and B include the advance letter and recruitment script) followed by the collection of consent from teachers and children in a recruited classroom (Attachments C and D include memos with a description of the consent process to the program and the teacher, respectively; the teacher consent form is provided in Attachment E; the parent consent form for the children in the classroom is Attachment F). In collaboration with a panel of experts (see Table B1), three data sources have been identified as necessary for coding the EDIT measure. These three data sources are: (1) pre-existing classroom documents assembled by the classroom teacher, (2) video recordings of ongoing assessment and instruction activities with focal children collected by the teacher, and (3) an in-person teacher interview.

The lead teacher in each recruited classroom is asked to select two consented children as focal children, one who is doing well in preschool and one who is experiencing some difficulty (Attachment G includes a brief memo describing the focal child selection and project activities for the program and Attachment H includes this description in more detail for the teacher). The teacher is asked to video record a combination of one assessment and two language or literacy activities with each of the focal children (illustrated instructions on use of the video equipment for teachers is in Attachment I). The teacher is also asked to gather assessment and instruction documents about the focal children from the previous two months.

Data collection procedures include site visits by EDIT team members to complete the EDIT measure based on the three required data sources. Prior to the visit, we will call the program to confirm the visit (Attachment J contains the reminder call script). During the visit, the EDIT team members review the gathered documents and video-recordings, and interview the teacher for one hour using a semi-structured protocol (Attachment K). These three sources provide the evidence needed to rate how teachers use assessment and individualize based on the criteria in the EDIT measure (Attachment J). Teachers are asked to complete the teacher questionnaire at the end of the visit (Attachment M) to provide information about the teacher. The research team also asks teachers to participate in a brief phone call after the site visit to discuss their experiences in participating in the EDIT using a semi-structured protocol (protocol is in Attachment N; reminder call script is in Attachment J).

Figure B.1. Sequence of EDIT activities

The full EDIT development team will debrief after three site visits have been conducted (see Figure A.1). This debriefing will allow for a review of the EDIT items and processes which may lead to a refinement of the measure. If any changes are made to EDIT items or procedures based on the debriefing sessions we will send updated materials to OMB as a nonsubstantive change.

B3. Methods to maximize response rates and deal with nonresponse

Expected response rates

The proposed sample is a purposive sample of nine preschool classroom teachers. We will seek programs with preschool classrooms implementing both a curriculum-embedded ongoing assessment system and a standard progress monitoring tool. We understand that assembling the documents, recording the videos, and participating in the teacher interview requires time from each participating teacher and that some teachers we approach may decline to participate. Thus, we plan to provide tokens of appreciation – as described in Supporting Statement A – to acknowledge the effort teachers put forth for participation in the pre-test. We will continue to recruit until we reach nine classroom teachers that agree to participate.

Maximizing response rates and dealing with nonresponse

Because this is a purposive sample, we will continue recruiting until nine preschool classrooms agree to participate in the EDIT. The team will use reminder phone calls to confirm the scheduled site visits with the setting point person (Attachment J). We will collect the teacher questionnaire in person at the conclusion of the teacher interview to ensure that all nine teachers have completed it. We will also schedule a debriefing call with the teacher at that time. We will use reminder phone calls to encourage teachers to keep their debriefing appointments (Attachment J). A token of appreciation will be provided to centers who agree to participate as well as for teachers that are part of the site visit and debriefing call. We expect these tokens of appreciation will encourage centers and teachers to participate in this important pre-testing activity.

B4. Tests of procedures or methods to be undertaken

We have pre-tested the EDIT measure and procedures in nine classrooms and have incorporated results from these pre-tests into the measure being submitted to OMB (see Attachment K). The previous pretest involved visits to classrooms followed by an internal team debrief and revision to the measure three times.

B5. Individuals consulted on statistical aspects and individuals collecting and/or analyzing data

Mathematica Policy Research and its partners have the responsibility for the data collection, refinements to the EDIT instrument, and reporting. With a small purposive sample and refinements to the items throughout the data collection, Mathematica will examine the availability of evidence for making ratings on the instrument, identify any need for additional clarifications or behavioral descriptors in the measure, and consider whether the observed variation is captured in the current items. The EDIT raters consist of Mathematica Policy Research and its partners Judith Carta, Ph.D. (University of Kansas) and Barbara Wasik, Ph.D. (Temple University). The EDIT raters’ experiences using the EDIT will be used to refine the items and procedures.

Consultants for the project include members of the expert panel, in Table B.1.

Table B.1. EDIT expert panel membership

Name	Affiliation
Stephen Bagnato	University of Pittsburgh
Linda Broyles	Southeast Kansas Community Action Program
Virginia Buysse	FPG Child Development Institute
Lynn Fuchs	Vanderbilt University
Leslie Nabors Oláh	Educational Testing Service
Sheila Smith	Columbia University
Patricia Snyder	University of Florida

1 Teachers proficient in English as well as additional languages are eligible.