Supporting Statement B - Fathers and Continuous Learning in Child Welfare Project

The Administration for Children and Families (ACF) Office of Planning, Research, and Evaluation’s (OPRE) within the U.S. Department of Health and Human Services (HHS) seeks approval under the umbrella generic Formative Data Collections for Program Support (OMB #0970-0531) to conduct a 12-month pilot study with as many as six child welfare agencies implementing a continuous quality improvement process called the Breakthrough Series Collaborative (BSC) to improve father and paternal relative (FPR) engagement in the child welfare system for the Fathers and Continuous Learning in Child Welfare (FCL) Project.

The objectives of this pilot study are to understand the feasibility of implementing the BSC within a child welfare agency and to identify strategies that may be evaluated in a later stage of the project. The study will allow ACF to assess whether sites engaged in the BSC as planned and contribute to understanding the feasibility of implementing the BSC in child welfare settings. This study will involve the delivery of targeted assistance and refinement related to program implementation as well as use rapid-cycle testing activities to strengthen programs in preparation for summative evaluation. ACF’s overarching goals for the FCL project are to (1) build knowledge about how collaborating with system partners and continuously using data for improvements in engagement strategies can create a culture that “thinks about and engages fathers and paternal relatives” and (2) contribute to the evidence base on father and paternal relative engagement strategies by evaluating potential promising strategies. ACF will use the findings to inform future project activities, including an evaluation study.

Generalizability of results

This study is intended to present an internally valid description of the implementation of the BSC in participating child welfare agencies, not to promote statistical generalization to other sites or service populations.

Appropriateness of study design and methods for planned uses

The pilot study is a descriptive process study. To understand and assess each site’s implementation plan and progress, both qualitative and quantitative data sources will be used to capture the experience of participating stakeholders (for example, agency leaders, frontline staff, agency supervisors, agency partners, FPRs).

We will use the information we collect to answer the research questions and provide insight into the overarching study objective, documenting the implementation of FPR engagement strategies using the BSC methodology. These findings will help identify promising FPR engagement strategies that ACF will consider for a future evaluation study. In addition, findings from the pilot study may be shared with the sites.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and design

Target population

The universe of programs eligible for the pilot study includes local and state child welfare agencies in the United States and their key partners. The target respondents include staff and fathers and paternal relatives from as many as six child welfare agencies and the agencies’ key partners. As many as 21 people from each site will be asked to participate in data collection, for a total of 126 respondents across the six sites.

Sampling and site selection

The FCL project team worked with federal partners, experts, and stakeholders familiar with child welfare agencies to identify 10 promising candidate sites for inclusion in this pilot study (OMB Control No. 0970-0356; approved March 26, 2018). The FCL project team worked with federal partners to select six sites for the pilot study from the 10 candidate sites. Factors influencing the selection of the pilot study sites included: (1) interest in and an identified champion of change for the BSC (based on phone and site visits), (2) range and depth of FPR engagement strategies of interest to the site, (3) capacity and willingness to collaborate with system partners, and (4) organizational capacity for BSC. These criteria are intended to help ACF select sites with a high likelihood of successful BSC implementation; sites are not intended to be representative of child welfare agencies nationally.

B3. Design of data collection instruments

Development of data collection instruments

The data collection instruments for the site visits (Instruments 1 through 4) were developed based upon the essential data needed to answer the priority research questions. Table 1 presents a crosswalk between the data collection instruments and study’s objectives.

The instruments for the BSC implementation (Instruments 5 through 16) were derived from the BSC implementation evaluation study, Culture of Continuous Learning Project: A Breakthrough Series Collaborative for Improving Child Care and Head Start Quality (CCL) (0970-0507). The BSC methodology expert who designed CCL’s BSC implementation instruments tailored CCL’s instruments to meet FCL’s objectives.

All instruments were closely examined to confirm that they were streamlined and not collecting duplicative data.

Table 1. Crosswalk between data collection instruments and study objectives

Instrument	Understand the feasibility of implementing the BSC within a child welfare agency	Identify strategies that may be evaluated in a later stage of the project
Site visit instruments
Instrument 1: Improvement Team Group Interview	X	X
Instrument 2: Focus Group with Engagement Strategy Staff	X	X
Instrument 3: Interview with Child Welfare Agency Leaders	X	X
Instrument 4: Group Interview of Fathers on the Improvement Team	X	X
BSC implementation instruments
Instrument 5: Improvement Team Information Form	X
Instrument 6: Data Collection Planning Worksheet	X	X
Instrument 7: Discussion Forum Prompts	X
Instrument 8: Learning Session Day 1 Evaluation	X
Instrument 9: Action Planning Form	X
Instrument 10: Learning Session Overall Evaluation	X
Instrument 11: Site Self-Assessment	X	X
Instrument 12: Plan, Do, Study, Act (PDSA) Worksheet	X	X
Instrument 13: Implementation Assessment	X
Instrument 14: Speed Sharing for Learning Session 2	X
Instrument 15: Elevator Speech Assignment	X
Instrument 16: Spreading and Sustaining the Work	X	X

B4. Collection of data and quality control

Site visits

A two-person team conducted each in-person site visit and will conduct each virtual site visit. During the site visits, we will conduct interviews and focus groups with agency leadership, agency staff, agency partners, and FPRs. To identify and recruit agency and partner staff for site visit interviews and focus groups, we will request organizational charts and information about staff roles at the site and its partner organizations for those participating in the BSC. We will work with the key site contact to set the schedule for all interviews and focus groups.

We will work with our key site contact to contact staff, partners, and FPRs to participate in the focus groups and interviews. We will ensure that focus group and interview participants have at least a minimal level of involvement in designing or implementing the strategies. To obtain an in-person focus group size of about six individuals, we asked the key site contact to recruit 10, to account for potential no-shows. To obtain a virtual focus group size of about five individuals, we will ask the key site contract to recruit 7, to account for potential no-shows. This smaller group for virtual focus groups and interviews will help us manage the conversation. We will work with the key contact at each site for both in-person and virtual interviews and focus groups to discuss barriers to participation and how to overcome them (for example, transportation for fathers, union rules for child welfare staff, access to web cameras). For in-person site visits, we also asked our contacts to assist us in securing a location for the meetings and group discussions that is convenient for participants, likely in the agency office. For virtual site visits, we will ask our contacts to assist us in ensuring staff are available to meet virtually considering their changing workload during COVID-19 and staff access the video conferencing tool, WebEx.

The instruments’ introductory sections include the consent statement for in-person and virtual site visit interviews and focus groups. For in-person site visit information collection, project staff read the consent statement to respondents before beginning interviews and focus groups. The consent statement provides assurances that the information shared will be kept private and reported in a manner that will not identify individual respondents. Participants were asked to provide verbal consent after the consent statement is read. For virtual site visits information collection, the project staff will read the consent statement to respondents before beginning interviews and focus groups. We will display the consent statement using the WebEx’s screen sharing feature.

To support data collection for virtual site visits during COVID-19, we will use a PowerPoint developed for each interview and focus group to display the interview questions. This will help interviewees keep track of the questions we are asking.

To ensure data quality, the site visit staff will receive training that includes a careful review of:

the objectives of the pilot study and the research questions that the interviews and focus groups will address;
each topic guide with an emphasis on the intent of each question to ensure interviewers have a keen sense of how to properly probe, which is critical due to the open-ended nature of interviews and focus groups.

We will create templates for the interview and focus group summaries to ensure that site visitors capture information consistently. The training will also include a review of these templates.

BSC implementation instruments

The BSC implementation instruments include self-administered paper instruments, instruments in Microsoft Word that are stored on a secure SharePoint site, and forms on a secure SharePoint site. These instruments will be distributed to the Improvement Team members. The FCL study team will hand out the paper instruments at in person meetings and will distribute the instruments stored on a secure SharePoint site by providing the SharePoint location to the Improvement Team members. The BSC experts working with the Improvement Teams will provide an overview of each instrument prior to administration and be available to answer questions as they arise. Data monitoring is an inherent part of the BSC methodology, so the experts will review instruments for data quality and completeness.

B5. Response rates and potential nonresponse bias

Response rates

ACF expects greater than 90 percent participation for site visits and BSC implementation instruments. The information collected will allow ACF to describe child welfare agencies’ experiences implementing FPR engagement strategies using the BSC methodology. ACF expects a high response rate given the extensive site engagement process conducted to date, and the regular communication the FCL project team will have with sites, allowing the FCL project team to encourage participation. We anticipate that agency staff, their partners, and FPRs will be willing to share information with the FCL project team.

We still expect greater than 90 percent participation for virtual site visits.

Nonresponse

Given that participating agencies were selected based, in part, on their willingness to participate in information collection about their experience, we expect little to no nonresponse to the information collection.

We also expect little to no nonresponse to the information collection due to the COVID-19 pandemic.

B6. Production of estimates and projections

We will not produce estimates for this study, because analyses of the quantitative data will be limited to descriptive statistics.

B7. Data handling and analysis

Data handling

We will review quantitative data for completeness and accuracy of data entry.

Data analysis

Qualitative data

We will use standard qualitative procedures to analyze and summarize information from the Improvement Teams and system partners. We will organize, code, triangulate, and identify themes. For each qualitative data collection activity, we will use standardized templates to organize and document the information and then apply the codes. We will search the coded text to gauge consistency and triangulate across participants and data sources. This process will reduce large volumes of qualitative data to a manageable number of topics/themes/categories (Yin 1994; Coffey and Atkinson 1996) that we can then analyze to address the research questions.

To code the qualitative data for key themes and subtopics, we will first develop a coding scheme that builds from the interview/focus group questions and aligns with the Collaborative Change Framework and strategies for each site. The principal investigator will refine the initial coding scheme by reviewing codes and a preliminary set of data output to make adjustments and ensure alignment with the topics that emerge from the data. Two FCL project team members will be trained to code the data using NVivo, a qualitative analysis software package. To ensure reliability across coders, the two FCL project team members will code an initial document and compare codes to identify and resolve discrepancies. In addition, as coding proceeds, the principal investigator will continue to review samples of coded data to check reliability.

Having coded the data, we will be able to look across staff and organizations (within agencies and among partnering organizations) by searching on codes. Using NVivo will also enable us to retrieve data on particular codes by type of study participant (for example, organizational leadership or Improvement Team member,). To compare information, we may retrieve data for subsets of agencies, such as those that implement the same FPR engagement strategy, serve similar target populations, or have similar staffing structures.

Quantitative data

We will summarize quantitative data using basic descriptive methods. The FCL project team will identify the analytic variables to be constructed and specify a plan for their development after the selected pilot sites begin identifying the FPR strategies they intend to test.

Process analysis methods

Table 1, above, documents the proposed data sources mapped to the study’s objectives. Our analysis will first focus on documenting and describing BSC implementation. We will then examine fidelity to the BSC model and facilitators and barriers to implementation fidelity.

Although the strategies, outputs, key actors, and the data we collect about them will vary by site, we expect that some analyses will be similar. Across all sites, we will document key supports for BSC implementation and components of the child welfare system that facilitate implementation and those that do not. Factors may include the influence of a larger organization or agency; access to staff training/professional development or a national or state training and technical assistance provider; and the organization’s experience using data for decision-making.

The BSC implementation instruments can track progress toward the BSC outcomes and global aim. For example, a metric that tracks staff attitudes and beliefs about the benefits of FPR engagement can be used to understand progress toward the creation of father-friendly culture of the child welfare agency. Output indicators may include staff and father reports of enhanced relationships, as well as progress in FPR engagement in the child welfare system. We will examine how these metrics vary between baseline and about one year after implementing the BSC.

Data use

The FCL project team will use the information collected to answer the research questions and provide insight into the overarching study objectives, documenting the implementation of FPR engagement strategies using the BSC methodology. The FCL project team will use the collected and analyzed data to complete a final report. The report is intended for internal ACF use, but may be shared with the sites as well.

Written products will clearly state that this study is intended to present an internally valid description of the implementation of the BSC in participating child welfare agencies, not to promote statistical generalization to other sites or service populations.

We will not generate a document for public use on how to properly interpret, analyze, and evaluate information from this collection because we are not creating a public-use file.

B8. Contact person(s)

Mathematica is conducting this project under contract number HHSP233201500035I/HHSP23337025T. Mathematica developed the plans for this data collection. Matthew Stagner, mstagner@mathematica-mpr.com, Roseana Bess, rbess@mathematica-mpr.com, and Jennifer Bellamy, Jennifer.Bellamy@du.edu can answer questions about the statistical aspects of the data collection and questions about the BSC.

Staff from Mathematica, the University of Denver, and JRA Consulting, Ltd. will be responsible for collecting, processing, and analyzing the information for OPRE.

Attachments

Instrument 1: Semi-Structured Protocol: Improvement Team Group Interview

Instrument 2: Semi-Structured Protocol: Focus Group with Engagement Strategy Staff

Instrument 3: Semi-Structured Protocol: Interview with Child Welfare Agency Leaders

Instrument 4: Semi-Structured Protocol: Group Interview of Fathers and Paternal Relatives on the Improvement Team

Instrument 5: Improvement Team Information Form

Instrument 6: Data Collection Planning Worksheet

Instrument 7: Discussion Forum Prompts

Instrument 8: Learning Session Day 1 Evaluation

Instrument 9: Action Planning Form

Instrument 10: Learning Session Overall Evaluation

Instrument 11: Site Self-Assessment

Instrument 12: Plan, Do, Study, Act (PDSA) Worksheet

Instrument 13: Implementation Assessment

Instrument 14: Speed Sharing for Learning Session 2

Instrument 15: Elevator Speech Assignment

Instrument 16: Spreading and Sustaining the Work

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Mathematica FCL Team
File Modified	0000-00-00
File Created	2021-01-13