Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
ACF Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) Project
Generic Information Collection for Qualitative and Descriptive Quantitative Implementation Research Data Collection for Two Early Head Start/Head Start Sites
OMB Information Collection Request
0970 - 0502
Supporting Statement
Part A
February 2024
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officer:
Kim Clum
Marie Lawrence
Part A
Executive Summary
Type of Request: This data collection is part of the Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) project. This is a new generic information collection (GenIC) under the umbrella BIAS-NG project. More information on the context of the BIAS-NG project can be found in Section A.1.1 – Study Background.
Description of Request: This GenIC pertains to the implementation research for Matrix and Starfish, two ACF grantee sites in Wayne County, Michigan within the Early Head Start/Head Start (EHS/HS) domain. The Wayne County Early Head Start/Head Start (WC EHS/HS) study features an implementation study (the IC proposed here) and an impact study consisting of a randomized controlled trial (RCT) to assess the impact of behaviorally informed messages on child attendance in EHS/HS programs at 2 ACF grantee sites. Over the last several years, the research team has partnered with these sites to define and diagnose a problem and consider how behavioral science might be applied to support program operations. The team designed and prototyped the intervention before launch. Prior work to better understand the program process from both the administrative and client perspectives was informed by work approved under this generic in December 2021. The information collected in this GenIC is specific to the implementation study and is intended to inform understanding of how the interventions being evaluated by the RCT are being implemented. This GenIC request includes surveys, interviews, and focus groups that will collect information about the intervention and control conditions and about participants’ and practitioners’ perspectives on the intervention materials. This GenIC is intended to yield an internally valid description of the implementation of an intervention in the EHS/HS sites, not to promote statistical generalization to other sites or service populations.
A1. Necessity for Collection
The Office of Planning, Research, and Evaluation (OPRE) at the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS), seeks Office of Management and Budget (OMB) approval to conduct surveys, interviews, and focus groups with administrators, staff, and caregivers in Matrix and Starfish Early Head Start and Head Start (EHS/HS) programs to understand the mechanisms and effects of an intervention that is informed by behavioral science and intended to improve program outcomes. This information collection (IC) is planned as part of ACF’s Generic Clearance for the Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) project and builds on previous work in these two EHS/HS programs. The goal of the BIAS-NG Generic Clearance is to conduct qualitative and descriptive quantitative research to identify and understand the psychological and behavioral factors that can affect the effectiveness of human service programs, specifically Child Welfare (CW), Temporary Assistance for Needy Families (TANF), and Early Head Start/Head Start (EHS/HS). Earlier work with these two sites, which was approved in 2021 under this generic, was designed to provide a better understanding of the points in the processes used for outreach and delivery of services, or in the client’s experiences of those processes, that are most amenable to a behavioral intervention geared towards improving program outcomes. This request builds on that work.
There are no legal or administrative requirements that necessitate this GenIC. ACF is undertaking the collection at the discretion of the agency.
Study Background
The BIAS-NG project builds on a prior OPRE project, the Behavioral Interventions to Advance Self-Sufficiency (BIAS) project, which relied exclusively on administrative data to test the short-term impact of small “nudge” interventions in human services programs. Going beyond the work conducted for BIAS, the BIAS-NG project is testing new interventions in more domains and collecting a wider range of data.
The study described in this IC is an implementation study as part of Phase 4 in collaboration with the Starfish and Matrix, two EHS/HS grantees. As indicated in the overarching generic clearance for the BIAS-NG project, Phase 3 is to conduct behavioral diagnosis and design and Phase 4 is the Testing phase, consisting of both an impact study and an implementation study. The information collected in Phase 3 (data collection approved under this generic in December 2021) was critical to understanding the program process from both the administrative and client perspectives. The diagnosis activities directly contributed to designing interventions that will be evaluated by a randomized controlled trial (RCT) in this next phase of the BIAS-NG study.
The impact study for Phase 4 of this specific study relies solely on available administrative data and tests the effects of behaviorally informed messages sent by the EHS/HS programs through their existing data management and communication software system, Child Plus, to improve child attendance. For this GenIC, the implementation study for Phase 4 will collect qualitative information from program staff and families, as well as quantitative information from a brief caregiver survey and from the ChildPlus data management system to better understand the mechanisms and effects of behavioral interventions and the implementation of the intervention.
A2. Purpose
Purpose and Use
The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.
To develop the study described here, the research team has been working with Starfish and Matrix EHS/HS programs on the sites’ goals of improving child attendance and family engagement. The behavioral diagnosis research allowed the team to gather structured in-depth information to understand the program process from both the administrative and client perspectives. The BIAS-NG team has triangulated: the insights of behavioral science; information from observations of on-the-ground implementation of programs; and information on parent and staff experiences. In Phase 3, the diagnostic and design phase of the study, the research team identified the following behavioral barriers to child attendance: (1) reliance on reactive attendance communication; (2) limited support to help caregivers navigate competing obligations; (3) dispersed family-staff connections; (4) weak family-to-family connections; and (5) over-reliance on communications to a primary caregiver.1
For Phase 4, the research team designed a positive, proactive messaging intervention, to be delivered weekly over a 3-month period in the second half of School Year 2023-24, intended to promote more consistent daily attendance by (a) underscoring the importance of attendance for child development, (b) helping families feel more connected to the program, and (c) highlighting supports available to help families get to the program. This intervention is designed to overcome several of the behavioral barriers identified in the diagnosis process. The impact study is a randomized controlled trial using administrative data to document key baseline traits and outcomes like child attendance.
The goal of the BIAS-NG Generic Clearance is to conduct qualitative and descriptive quantitative research to identify and understand the psychological and behavioral factors that can affect the effectiveness of human service programs. To understand the factors related to this intervention in Wayne County EHS/HS, the BIAS-NG team is conducting an impact study based on available administrative data and an implementation study (instruments included in this request package).
Wayne County EHS/HS Impact Study (based on available administrative data)
The Wayne County EHS/HS impact study is a two-arm randomized controlled trial that tests the impact of behaviorally informed messaging on child attendance.
EHS/HS caregivers who have opted-in to general messaging from the program are randomly divided into two groups.
Caregivers in the two groups experience the following:
Control Group: receive regular program communication
Intervention Group: receive additional positive, proactive messaging from program staff using the ChildPlus data management system.
The implementation research component (this GenIC) provides key insights into better understanding the intervention, by better understanding the randomized controlled trial’s fidelity, caregiver perceptions, and staff perceptions. The implementation study will describe and document the intervention, how it operated, and provide information about the contrast in intervention between the research groups—both whether the planned contrast between the intervention and the control conditions occurred (implementation fidelity), as well as how the intervention implemented differed from the status quo (intervention contrast). This information will be important for interpreting the findings of the impact study.
Research Questions or Tests
This specific GenIC is for data collection related to phase 4, implementation research, of the study described in the overarching generic clearance. BIAS-NG is also conducting an impact study based solely on administrative data, and data from that work will be used in conjunction with the data collected through this GenIC. More information about the data to be used from the implementation study is provided in the following subsection.
Implementation Research Questions:
To what extent were the interventions implemented with fidelity?
What are participant perspectives on the intervention, program staff and fellow families?
What are staff perspectives on their work for the program, families, and families’ response to the intervention?
Implementation Study Design:
This GenIC will conduct an implementation study to describe and document the intervention, how it operated, and provide information about the contrast in intervention between the research groups—both whether the planned contrast between the intervention conditions and the control condition occurred (implementation fidelity) as well as how the intervention implemented differed from the status quo (intervention contrast). This information will be important for interpreting the findings of the impact study.
We will gather information to address these questions through surveys, focus groups and interviews with Starfish and Matrix families and staff. The qualitative data collection activities are essential to conducting implementation research. Please see Instruments 1, 2, 3, and 4 for survey, focus group and interview questions, and Table 1 for details.
Study Design
Table 1: Research Question and Instrument Matrix
Data Collection Activity |
Instruments |
Respondent, Content, Purpose of Collection |
Mode and Duration |
Intervention Group Participant Focus Group/ Interview |
Instrument 1 |
Respondents: Caregivers in the Intervention Group
Research Questions:
Purpose: Understand participant experiences with the intervention. |
Mode: Focus group or interview
Duration: 1 hour |
Control Group Participant Focus Group/Interview |
Instrument 2 |
Respondents: Caregivers in the Control Group
Research Questions:
Purpose: Understand participant experiences without the intervention. |
Mode: Focus group or interview
Duration: 1 hour |
EHS/HS Program Staff Focus Group/ Interview |
Instrument 3 |
Respondents: EHS/HS Program Staff
Research Questions:
Purpose: Understand Wayne County EHS/HS staff experiences with the intervention. |
Mode: Focus group or interview
Duration: 1 hour |
Intervention Group Participant Survey |
Instrument 4 |
Respondents: Caregivers in the Intervention Group
Research Questions:
Purpose: Understand participant experiences with the intervention |
Mode: Online survey distributed via messages
Duration: 8 minutes |
In the rest of this document and in Supporting Statement B, we include a description of:
Planned implementation research data collection. Instruments include guides for surveys, focus groups and interviews of program staff and caregivers (intervention groups and control group).
Planned qualitative analyses. Audio recordings and notes from focus groups and interviews will be analyzed for patterns and themes.
Administrative data that the agencies are already collecting and that the study will utilize.
Other Data Sources and Uses of Information
Administrative data will supplement the information collected in participant and staff focus groups/interviews to further understand: 1) the extent to which the interventions were delivered as intended; and 2) the extent to which the implemented interventions differed from what was offered/delivered to the control group. The data will be provided by ChildPlus, an existing data management and communication software system used by these programs.
ChildPlus Administrative Data
Content: ChildPlus data includes records of ChildPlus messages sent to intervention group sample and control group sample.
Purpose: The goal of collecting ChildPlus data is to understand whether participants assigned to the intervention group receive the correct materials and that control group participants do not receive the intervention materials. Additionally, the data will help to the research team document how applicants experienced the interventions.
The information collected through this implementation study will be used in conjunction with information gleaned from the impact study, which relies solely on administrative data. The impact study aims to address the following research question: What is the effect of offering EHS/HS caregivers positive, proactive messaging on child attendance?
We will conduct a two-arm test that will assess the impact of random assignment to the intervention group involving receiving positive, proactive messages. Existing administrative data from ChildPlus will be used to complete the analysis for the impact study.
A3. Use of Information Technology to Reduce Burden
Caregiver (participant) and staff focus groups and interviews will be scheduled at convenient times or when they are already planning to be at a program center. If a phone or video interview is easier for caregiver or staff schedules, we may conduct interviews in that mode. Interviews and focus groups will be recorded, with permission from the interviewees.
A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency
We have worked carefully with Starfish and Matrix to understand the data the program routinely collects. None of the data currently collected by the programs would allow us to understand caregiver and staff perceptions of and responses to the intervention materials. In addition, the study team will not collect information that is available from existing public sources.
A5. Impact on Small Businesses
We do not anticipate any small organizations to be affected by this GenIC. Nonetheless, we will schedule interviews at times that are convenient to caregivers to minimize disruption of daily activities.
A6. Consequences of Less Frequent Collection
Rigorous evaluation of innovative initiatives is crucial to building evidence of what works and how best to allocate scarce government resources. Not collecting information about the implementation and effect of the intervention would hinder the government’s ability to learn how interventions were implemented and why and to what degree the interventions achieved desired outcomes.
A7. Now subsumed under 2(b) above and 10 (below)
A8. Consultation
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register, first announcing the agency’s intention to request OMB review of the umbrella generic for initial approval, then for updates to the umbrella, and later for an extension. These notices all invited public comment on the proposed generic. Public comment related to the initial approval of the umbrella generic was requested through two comment periods, the initial 60-day period (82 FR 23572) and a following 30-day period (82 FR 34530). Public comment related to updates to the umbrella was requested through a 30-day comment period (84 FR 33947). Finally, public comment related to a request to extend approval of the umbrella generic was requested through two comment periods, the initial 60-day period (87 FR 9629, published February 22, 2022) and a following 30-day period (87 FR 35557, published June 10, 2022). No substantive comments were received during any of the notice and comment periods.
The research team consulted with leading behavioral science experts on the intervention design and on select items in the data collection protocols.
A9. Tokens of Appreciation
Starfish and Matrix caregivers and staff who participate in interviews will receive a gift card of $40. We intend for the gift card to help offset potential out of-pocket costs to respondents for time spent on the interview, additional cell-phone data or phone minutes, or child care costs associated with interviews.
Based on MDRC’s experience with this population, a $40 gift card will help to offset these incidental costs associated with participation. As detailed in the approved BIAS-NG umbrella generic clearance, based on experiences in the field to date, the study team has found that a $20 gift card may not be sufficient to support an adequate response rate. This is likely to be especially true when the study team asks clients to attend a separate meeting to participate in interviews or focus groups and/or when the client is a parent with young children. For example, in the Allegheny County child welfare site, only four respondents out of 13 scheduled completed a client interview, even after several reminder calls, as $20 was not enough to offset the costs.
Tokens of appreciations of this amount have been used in prior research activities for BIAS-NG and approved by the MDRC Institutional Review Board (IRB) and OMB for the BIAS-NG project. We do not believe this token of appreciation is so high as to be coercive for caregivers.
A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing
Personally Identifiable Information
Caregivers are assigned a family ID by the programs. The study team receives de-identified, individual level data using the caregivers’ family ID as the identifier. Caregivers’ and staff members’ name and contact information (email or phone number) may be collected for purposes of scheduling and conducting the interviews and focus groups.
Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier in a way that triggers the Privacy Act of 1974, as amended (5 U.S.C. 552a).
Assurances of Privacy
Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.
All interview and focus group respondents who participate in research under this clearance will be read a statement that will explain the study and will inform individuals that their participation is voluntary and of the extent of their privacy as respondents (informed consents are included in each of the Instruments). Caregivers will be told verbally that their conversations will not be shared in a form that identifies the individual with anyone outside the research team. As ACF’s prime contractor, MDRC plans to implement all data collection activities with sub-contractor MEF Associates. Information will be kept private to the extent permitted by law and in accordance with current Federal information security standards and other applicable regulations.
Data Security and Monitoring
MDRC researchers will conduct interviews in person, which will be recorded with the participant’s permission, or use Zoom for government to conduct and record videoconference interviews. Per MDRC policy, project team members store these data files and recordings in designated secure folders within MDRC’s secure environment. MDRC employees are required to maintain and process quantitative and qualitative data in designated project folders on the MDRC network. With the exception of the temporary storage of data during onsite collection, MDRC employees are not allowed to download, keep, or process individual-level data on the hard drives of their MDRC workstations or any other storage.
The project Data Manager will organize BIAS-NG project folders and will supervise storage of BIAS-NG data files. All reports, tables, and printed materials are limited to presentation of aggregate numbers. Original project notes and recordings will be stored in secure folders with limited access rights for use only by authorized implementation study researchers. These original documents will be stored until the published report, to refer to these sources for fact-checking. MDRC will destroy all paper records and electronic records containing personally identifiable information when no longer needed for research purposes in accordance with funder and contractual requirements, as well as MDRC retention policies.
We plan to share the data we collect and protect the privacy of the individual data collected in the form of Restricted Access Files (RAF) archived with the Inter-university Consortium for Political and Social Research (ICPSR). RAFs deposited with ICPSR are restricted to approved users who have signed a legal agreement tightly limiting their acceptable use, analysis, and disclosure of the data. Per MDRC standard procedure, the Data Librarian and project Data Manager will verify that all incoming files are accounted for at the end of the project – deleted or permanently archived, per agreement with funder and data providers.
A11. Sensitive Information 2
We are asking some questions that may cover sensitive topics in this GenIC in terms of caregiver relationships with staff and vice versa. For example, we plan to ask caregivers in an interview to characterize their experience with Starfish or Matrix staff. These answers will help the study team address core implementation research questions around caregivers’ experience with the intervention. To encourage honest responses, we assure caregivers that program staff will not see their responses, and staff that caregivers will not see their responses, in any way that can be linked back to them.
MDRC’s IRB has approved the overall BIAS-NG impact study and similar implementation research protocols. The IRB will also formally review study protocols for this data collection.
A12. Burden
Explanation of Burden Estimates
Table 2 provides details about how this estimate of burden hours and costs were calculated.
We expect to conduct interviews/focus groups with a total of up to 60 caregivers (approximately 30 from the intervention group and 30 from the control group), and up to 100 EHS/HS staff (approximately 40 Family Service Workers, 30 teachers, and 30 additional program staff). We also plan to field a short survey to up to 630 intervention group participants (30 of whom also participated in the interviews/focus groups). The estimate below represents an upper bound on potential burden.
Table 2: Burden Hours for Wayne County
Activity |
No. of Respondents (total over request period) |
No. of Responses per Respondent (total over request period) |
Avg. Burden per Response (in hours) |
Total Burden (in hours) |
Average Hourly Wage Rate |
Total Annual Respondent Cost |
Intervention Group Participant Interview/Focus Group (Instrument 1) |
30 |
1 |
1 |
30 |
$10.10 |
$303 |
Control Group Participant Interview/Focus Group (Instrument 2) |
30 |
1 |
1 |
30 |
$10.10 |
$303 |
EHS/HS Teachers Interview/Focus Group (Instrument 3) |
30 |
1 |
1 |
30 |
$28.60 |
$858 |
EHS/HS Key Program Staff Interview/Focus Group (Instrument 3) |
30 |
1 |
1 |
30 |
$28.60 |
$858 |
EHS/HS Family service worker Interview/ Focus Group (Instrument 3) |
40 |
1 |
1 |
40 |
$28.60 |
$1,144 |
Intervention Group Participant Survey (Instrument 4) |
252 |
1 |
.13 |
33 |
$10.10 |
$333 |
Total |
412 |
|
|
193 |
|
$3,799 |
Estimated Annualized Cost to Respondents
We estimate the average hourly wage for Starfish and Matrix staff to be the average hourly wage of “educational and library occupations” in the Detroit-Warren-Dearborn Area taken from the U.S. Bureau of Labor Statistics, May 2022 National Occupational Employment and Wage Estimates ($28.60). To compute the total estimated cost for participants, the total burden hours were multiplied by $10.10, the Michigan minimum wage as of January 1, 2023, for both small and large employers.
A13. Costs
The data collection proposed under this generic IC involve imposing time burdens on very busy administrative and frontline staff in human services agencies. Based upon our experience in the field to date under this package, we propose offering a modest honorarium of $40 to program staff participating, in recognition of the time and professional expertise they contribute to the studies. These honoraria are intended to both encourage staff participation and recognize their efforts to support a timely and high-quality data collection.
A14. Estimated Annualized Costs to the Federal Government
The total cost for the implementation research data collection, analysis, and reporting activities under this current IC request will be approximately $73,000. Annual costs to the Federal government will be approximately $36,500 (the annual cost over the course of 2 years). There will be no notable costs beyond normal labor costs for staff.
Cost Category |
Estimated Costs |
Implementation Research Field Work |
$40,000 |
Publications/Dissemination (Implementation Research section of final report) |
$33,000 |
Total costs over the request period |
$73,000 |
Annual costs |
$36,500 |
A15. Reasons for changes in burden
This is an individual IC under the BIAS-NG Generic Clearance (0970-0502).
A16. Timeline
Phase 4: Evaluation: Phase 4 consists of implementing the behavioral intervention and evaluating it and collecting data on outcomes. Implementation data from interviews and focus groups will begin following OMB approval. We also collect administrative data about the implementation of the interventions. We expect data collection will conclude by the end of the school year in May 2024.
Phase 5: Dissemination: Dissemination efforts during the time of this clearance include site-specific reports, infographics, products aimed at practitioners, sharing findings at conferences, and publicizing our findings and our work on social media. Dissemination efforts are expected to begin after analysis concludes (about 8 months after the May 2024 end of the school year).
A17. Exceptions
No exceptions are necessary for this information collection.
Attachments
Instrument 1 – Intervention Group Participant Interview and Focus Group Protocol
Instrument 2 – Control Group Participant Interview and Focus Group Protocol
Instrument 3 – EHS/HS Program Staff Interview and Focus Group Protocol
Instrument 4 – Intervention Group Participant Survey
1 The current intervention does not target behavioral barrier 5. Conversations with site leadership illuminated several feasibility barriers that would make sending messages to secondary caregivers challenging. For example, permissions structures typically cover only the primary caregiver, as secondary caregivers do not personally opt in to receive messages from the programs.
2 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Jordan Engel |
| File Modified | 0000-00-00 |
| File Created | 2024-09-27 |