SSB - BIAS - NG GenIC for Qualitative and Descriptive Quantitative Implementation Research Data Collection for Two EHS/HS Sites

ACF Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) Project

Generic Information Collection for Qualitative and Descriptive Quantitative Implementation Research Data Collection for Two Early Head Start/Head Start Sites

OMB Information Collection Request

0970 – 0502

Supporting Statement

Part B

February 2024

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Kim Clum

Marie Lawrence

Part B

B1. Objectives

Study Objectives

Behavioral Interventions to Advance Self-Sufficiency-Next Generation (BIAS-NG) consists of a series of mixed-methods studies to identify, develop, and test interventions related to social services and benefit receipt. This Generic Information Collection (GenIC) request pertains to implementation research in two sites in the Early Head Start/Head Start (EHS/HS) domain, Wayne County Early Head Start/Head Start programs Starfish and Matrix. The Wayne County Early Head Start/Head Start (WC EHS/HS) study features an implementation study (the IC proposed here) and an impact study consisting of a randomized controlled trial (RCT) to assess the impact of a behaviorally-informed messaging intervention on child attendance. The information collected in this GenIC is specific to the implementation study and is intended to inform understanding of how the interventions being evaluated by the RCT in this phase of the BIAS-NG study are being implemented. The GenIC request includes survey, interviews and focus groups that will collect information about the treatment and control groups and about caregivers’ and practitioner’s perspectives on the intervention materials.

Generalizability of Results

This implementation study GenIC is intended to present an internally valid description of the implementation of an intervention in chosen sites, not to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

This implementation study involves a survey, interviews, and focus groups with Starfish and Matrix staff and caregivers. These methods will allow the research team to ask questions about the implementation of behaviorally informed messaging interventions that cannot be answered through analysis of administrative data.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target Population

The target population to be included in this GenIC include:

• Caregivers with children enrolled in center-based Early Head Start or Head Start programs at Starfish or Matrix who have a mobile phone number on record and have opted to receive messaging from the program through ChildPlus.

• Starfish and Matrix staff. There are 30 centers across these two programs- each with approximately one center leader and one or two family service workers. Each site has over a hundred teachers and support staff.

Sampling

The target respondents to be included in this GenIC include:

• Up to 312 caregivers with children in EHS/HS across multiple Matrix and Starfish centers. This includes a short survey offered to all 630 caregivers in the intervention group, with an expected response rate of 40%. Additionally, up to 30 caregivers in the intervention group and 30 caregivers in the control group will be interviewed. Depending on the caregivers’ availability, some caregivers may be interviewed individually and some may participate in focus groups. We aim to gather informants from a mix of center sizes (for example, larger centers, medium-sized centers and smaller centers) and demographics (specifically, racial and ethnic background of families).

• Up to 40 Family Service Workers (FSWs). Because the interventions are focused primarily on affecting child attendance, we will aim to interview many of the 40 FSWs across Matrix/Starfish who engage directly with families about attendance. Additionally, we will aim to gather informants who reflect a mix of case experiences--for example, staff who have been with the program for many years and new staff. Depending on schedules and availability, some staff may be interviewed individually and some may participate in focus groups. 

• Up to 30 Teachers from up to 10 Matrix EHS/HS centers and up to 10 Starfish EHS/HS centers. We aim to include teachers from a mix of center sizes (for example, larger centers, medium-sized centers and smaller centers) and from different racial and ethnic backgrounds.  Additionally, we will recruit informants who reflect variation on other relevant dimensions--for example, staff who have been with the program for many years and new staff.  Depending on schedules and availability, some staff may be interviewed individually and some may participate in focus groups. 

• Up to 30 Key Program Staff including administrators and leadership. Because the interventions are focused primarily on affecting child attendance, we will aim to interview additional key program staff with different perspectives on child attendance at the center and program-wide levels. Depending on schedules and availability, some staff may be interviewed individually and some may participate in focus groups. 

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

The data collection instruments will not be pre-tested at scale but are very similar to other individual ICs under this generic clearance that have been used successfully to address the three central research questions:

1. To what extent was the intervention implemented with fidelity?

2. What are caregiver perspectives on the intervention, program staff, and fellow families?

3. What are staff perspectives on their work for the program, families, and families’ response to the intervention?

B4. Collection of Data and Quality Control

To collect qualitative data, study teams will conduct focus groups and interviews with program staff and caregivers. Research staff members are experienced in the process of implementation research. Each interview or focus group will begin with an introduction that explains the purpose and goals of the BIAS-NG project. The facilitator will inform staff or caregivers that the conversation will be audio-recorded but that we will stop the recording at any time they do not feel comfortable being recorded. The facilitator will begin the conversation with the staff or caregiver[s] after receiving verbal consent from each interview or focus group participant.

At the end of the interviews and focus groups, caregivers and staff will receive a gift card worth $40.

The instruments guiding these interviews and focus groups are provided in Instruments 1, 2, and 3.

To recruit caregivers, we will use data from ChildPlus, EHS/HS’ data management software to provide a record of families within the control and intervention groups. We will work with staff from Starfish and Matrix to recruit participants from each group. The research team will interview these participants in person at a convenient location or via video-conference or phone if preferred.

To recruit program staff for interviews and focus groups, we will similarly work with program leadership to connect us with all program workers and to identify and connect us to a sample of workers who are available for an interview or focus group. The instrument guiding the program staff interviews is provided in Instrument 3.

Additionally, we plan to field a short online survey sent via text and email to the entire intervention group, up to 630 caregivers. Study participants can refuse to complete the survey or refuse to answer any of the questions on the survey, and will not be penalized in any way.  Instrument 4 details the survey questions.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The data collection activities in this IC request are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion.

For qualitative data collection, the research team will work closely with administrators and staff to recruit caregivers and EHS/HS staff for us to interview. Participants in the interviews and focus groups will be identified through a convenience sample identified by program staff, though we will aim to gather informants who reflect variation on relevant dimensions—for example, intervention group participants who had a range of child attendance records.

As focus group and interview respondents are not randomly sampled, we will not calculate a response rate. We will collect a response rate for respondents of the survey. To further increase the likelihood of participation, we will offer focus group and interview participants gift cards as tokens of appreciation, as discussed in Supporting Statement Part A.

One aspect of our quantitative implementation research data collection from caregivers is an online survey. Response rates for surveys are typically 40% or lower. We do not expect this population’s response rate to be different. As a result, we do not view the survey as large-scale data collection but rather as opportunistically gathered data to help us understand how the intervention was delivered. In the Building Bridges and Bonds study funded by HHS, response rates via text message have ranged from 20-30 percent.

Only participants who received the intervention will receive the survey. We will work with Starfish and Matrix staff to send out a link to the survey via their existing program messaging capabilities.

Non-Response

As participants will not be randomly sampled and findings are not intended to be representative, interview and focus group non-response bias will not be calculated. As the survey will not be linked to administrative data, non-response bias will not be calculated.

B6. Production of Estimates and Projections

Findings will only present an internally valid description of the implementation of an intervention in chosen sites. Results will be included in the final report. The data collected from this IC will not be used to make policy decisions.

B7. Data Handling and Analysis

Data Handling

Qualitative data will be analyzed using qualitative analysis methods, such as coding interviews for themes relevant to behavioral barriers to children’s attendance and experience with the intervention. With study participant permission, we will audio record conversations and use these records to minimize errors due to coding and data processing.

The intervention group survey will be programmed in an online survey management system (i.e., Qualtrics) and then the link to the survey will be sent to families in the intervention group through text message and email.

Data Analysis

Interviewers will take detailed notes during the focus group interviews, which will also be recorded. Interviewers will write summary notes after each interview to record early impressions and emerging patterns. These notes will not be considered project data to be analyzed, but they will serve as a record of initial (emerging) themes that will then be followed up on by a more thorough analysis to better understand the intervention fidelity and contrast along with staff and client perceptions. The analysis will focus on answering our four primary implementation research questions, listed in section B3.

The research team will use one short, specific caregiver survey to ask caregivers in the intervention group about their perception of the messages, as well as about their sense of connection to the program and awareness of resources offered by the program. It will not be possible to link survey results in this test to ChildPlus administrative records given families do not know ChildPlus IDs and administrative data currently shared with BIAS NG research staff do not include identifiable information that could be used to fuzzy match families/children to ChildPlus. As such, survey data will be used to understand broad trends and sentiments across intervention families.

Data Use

The goal of the BIAS-NG Generic Clearance is to conduct qualitative and descriptive quantitative research to identify and understand the psychological and behavioral factors that can affect the effectiveness of human service programs. This IC will conduct an implementation study to describe and document the intervention, how it operated, and provide information about the contrast in treatment between the research groups—both whether the planned contrast between the treatment group and the control group occurred (implementation fidelity) as well as how the treatments implemented actually differed from the status quo (treatment contrast). This information will be important for interpreting the findings of the impact study. Interviews and focus groups are our sole data sources to learn about staff and caregiver perceptions of the interventions, to better understand the mechanisms and effects of this experiment in this context.

Aside from summarizing findings in public reports as part of the project’s dissemination activities, no other documentation will be released regarding this data collection. Our dissemination goals prioritize sharing findings with practitioners, decisionmakers, and academic scholars at the federal, state, and local levels. We will document the limitations of the resulting data when making anything public.

B8. Contact Persons

Caitlin Anzelone, MDRC, Project Director – caitlin.anzelone@mdrc.org

Emily Hanno, MDRC, Site Lead – emily.hanno@mdrc.org

Kim Clum

Senior Social Science Research Analyst

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

330 C St. SW, 4th Floor

Washington, DC 20201

(202) 205-4922

Attachments

• Instrument 1 – Intervention Group Participant Interview and Focus Group Protocol

• Instrument 2 – Control Group Interview and Focus Group Protocol

• Instrument 3 – EHS/HS Staff Interview and Focus Group

• Instrument 4 – Intervention Group Participant Survey