Supporting Statement B: Early Childhood Training and Technical Assistance (T/TA) Cross-System Evaluation Project

Early Childhood Training and Technical Assistance (T/TA) Cross-System

Evaluation Project

OMB Information Collection Request- Formative Data Collection for Informing Policy Research

0970-0356

Supporting Statement

Part B

May 2016

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

330 C Street, SW

4th Floor

Washington, DC 20201

Project Officer:

Ann Rivera, Ph.D.

B1. Respondent Universe and Sampling Methods

This project will provide preliminary information to the Administration for Children and Families (ACF), Office of Planning, Research and Evaluation (OPRE) for the design of a future evaluation of the Early Childhood Training and Technical Assistance (EC T/TA) System and finalization of a conceptual framework of the system. The current information collection request is for gathering preliminary information from the leadership of the nine national centers (Centers) and the key T/TA system stakeholders within the twelve regions that make up the Head Start Regional T/TA network and the ten regions that constitute the Child Care State Capacity Building Center. The results will inform the evaluation design and data collection instrument development for the EC T/TA cross-system evaluation. In addition, the study team will present a draft conceptual framework to these two groups and the leadership of the Centers to obtain their critical feedback. The study team will collect information for developing future dissemination and evaluation design options, supports and strategies.

The study team will use information provided by ACF to identify and recruit respondents for the study from the nine National Centers, the Office of Head Start’s (OHS) Regional T/TA Coordinators, as well as the Office of Child Care’s (OCC) Child Care State Capacity Building Center Project Director and T/TA State System Specialists. Our goal is for this process to result in a census of OHS and OCC national and regional T/TA providers.

Our preference is to interview a single representative of each team, but if any of these participants request, the contractor will interview another member of their team along with or instead of the invited participant (e.g., in addition to the lead evaluator of a National Center, we would interview the Center Director or an additional member of the evaluation team). The relevant, knowledgeable respondents may vary across the nine Centers and Regional-level T/TA teams. Some, but not all, teams may have other staff who could answer the questions or who want to contribute to the interviews about goals, activities and evaluation efforts. We will not be able to identify these other individuals or know exactly how many will want additional staff involved without communicating first with the respondents identified above (i.e., Center lead evaluators, Regional T/TA Coordinators, Child Care State System Specialists). The number of respondents we have estimated in this request reflect these unknowns by including what we believe to be the maximum number of respondents likely.

These individuals also will be invited to provide feedback on the study’s conceptual framework, along with other key stakeholders from the national and regional levels of the EC T/TA system.

B2. Procedures for Collection of Information

Study teams composed of two researchers will contact potential respondents for telephone interviews and focus groups, and senior researchers will conduct the telephone interviews and focus groups, with another researcher taking notes. Senior researchers leading these activities are experienced in the process of gathering information for purposes of designing evaluation studies and resources. In addition, all team members will receive training to ensure that potential respondents are engaged in a consistent manner and data are collected and recorded consistently.

The remainder of this section describes the study team’s procedures for engaging potential respondents and collecting information.

As a part of the recruitment process for the telephone interviews, the study team will send an initial recruitment email to each potential respondent (i.e., Center evaluators, OHS Regional T/TA Coordinator, OCC T/TA State System Specialists and the Project Director of the Child Care State Capacity Building Center) inviting him/her to participate in a 90-minute telephone interview (Appendix A). If s/he is interested, we will schedule a time for the interview. If a potential participant does not reply to the initial email, the study team will send a follow-up email one week later. If a potential participant does not reply to the second recruitment email within a week, we will call him/her. If s/he still does not respond after the follow-up call, we will send an email to ask for an alternate respondent. These follow-up contacts will include the same information provided in the initial email. The study team will provide a teleconference line for participants to call at the scheduled time for their interview. A senior researcher experienced in leading qualitative interviews will conduct the telephone interview utilizing a semi-structured interview protocol (Appendix B), and another researcher will take notes.

In addition, the study team will present a draft conceptual framework of the EC T/TA System during a series of six 90-minute focus groups (in-person and via webinar). We will send recruitment emails to invite individuals from three different groups of T/TA system stakeholders (Appendix C) to participate in a 90-minute focus group. We will offer both virtual and in-person options for providing feedback. We will hold focus groups via webinar and/or in-person during national meetings that many of the key T/TA system stakeholders are likely to attend. Senior researchers experienced in conducting focus groups will use a semi-structured discussion guide (Appendix D) to facilitate discussions and solicit feedback on the conceptual framework, while another researcher will take notes.

This mode of data collection was selected to minimize the burden on respondents. Because responses can vary, the study team chose this mode to be able to adapt the interview protocols and discussion guide based on the study team’s analysis of existing materials and during the discussion to reduce the length of the interview or focus group to the minimum amount of time required. In addition, this mode prevents T/TA system stakeholders from having to write lengthy responses and the need for follow-up to clarify the written responses.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

Expected Response Rates

We expect that at least one evaluator from each Center, one T/TA Coordinator from each OHS region, one State System Specialist from each OCC region, and the Project Director of the Child Care State Capacity Building Center will participate in virtual and/or in-person focus groups on the study’s conceptual framework. However, we acknowledge that constraints may warrant a lower participation rate. We expect that our previous interactions with the national and regional T/TA providers will facilitate recruitment in this data collection effort.

Dealing with Nonresponse

Although we hope to gather information from each of the Center’s evaluators, the T/TA Coordinator from each OHS region, the State System Specialist from each OCC region, and the Project Director of the Child Care State Capacity Building Center, we recognize that a few may not participate in the interviews or focus groups. There is no prior data on which to base estimates of expected nonresponse, though it is possible that stakeholders who have used or are more interested in evaluation design and practice may be more likely to participate. When reviewing the findings from this data collection, we will report the participation rate and consider nonresponse bias when summarizing the information gathered. We will address potential concerns about nonresponse bias by clearly reporting any limitations in interpreting findings due to significant numbers of missing responses from important subgroups of respondents.

Maximizing Response Rates

The study team will be accommodating of respondents’ schedules. We can be flexible about the dates and times for telephone interviews and focus groups conducted via webinar. To maximize response rates, project staff will conduct interviews and focus groups during hours most convenient for participants, which we expect to be between typical working hours of 8am-5pm EST, Monday through Friday. We will accommodate the normal working schedule of individuals on the west coast by extending interview availability until 8pm EST. We may also interview individuals outside of the regular work day, if needed, to maximize participation. We will conduct telephone interviews and focus groups via webinar in order to eliminate any travel burden on participants. Any in-person focus groups will take place on-site at national professional meetings that many stakeholders are likely to attend.

B4. Tests of Procedures or Methods to be Undertaken

To test the semi-structured interview protocols, the study team will internally pilot the protocols between two project staff. In addition, one OHS and one OCC region will each act as a pilot, such that after this first interview, any minor yet necessary changes that need to be made to the semi-structured interview protocols will be made. Consistent with this practice, the semi-structured discussion guide for gathering feedback on the conceptual framework will be modified, if necessary, after completing the first focus group. If we modify approved instruments, we will submit the updated version(s) to OMB through a nonsubstantive change.

B5. Individual Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The interview team is led by Drs. Carrie Markovitz and Marc Hernandez, co-principal investigators from ACF’s federal contractor, NORC at the University of Chicago. Other team members include Patricia Ruggles, Senior Fellow; Dr. Carol Hafford, Principal Research Scientist; Dr. Terese Schwartzman, Research Scientist; Dr. Eileen Graf, Research Scientist; and Jennifer Scolese, Research Analyst. Focus groups on the conceptual framework are to be directed by Drs. Carrie Markovitz and Marc Hernandez, co-principal investigators and Dr. Carol Hafford, Principal Research Scientist.

3