Instrument 1: Interview topic guide

Anchor

Engaging Fathers and Paternal Relatives in Child Welfare Breakthrough Series Collaborative

This topic guide will help the Fathers and Continuous Learning in Child Welfare (FCL) project team understand (1) the experiences of key staff and partners who were involved in efforts to enhance engagement of fathers and paternal relatives, including the Breakthrough Series Collaborative (BSC), , and (2) the community perceptions about the agency’s efforts to engage fathers and paternal relatives. The respondents will include senior leaders; team managers and supervisors; frontline staff; partner leadership; partner frontline staff; and community stakeholders.

The average estimated public reporting burden for this collection of information is about 90 minutes per interview. Providing information is voluntary, and all responses that are collected are kept private to the extent permitted by law.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is XXXX-XXXX, and the expiration date is XX/XX/20XX.

Staff and Stakeholder Interview: Topic Guide

Fathers and Continuous Learning in Child Welfare (FCL) project

Staff and Stakeholder Interview: Topic Guide

Note to interviewer

The following detailed script should not be read verbatim. As the interviewer, you must be familiar enough with the script to introduce the study and the focus of the interview without reading word-for-word from the script. Please familiarize yourself thoroughly with the text before you conduct interviews.

Background

1. Introduce the moderator and co-facilitator

Thank you for taking the time to speak with us today. My name is [NAME,] and my colleague is [NAME]. We are from Mathematica, an independent research firm, and we are here to learn about your experiences with the [CHILD WELFARE AGENCY].

[If virtual] We especially appreciate your willingness to participate in this site visit virtually given these circumstances.

2. Explain the project and purpose of discussion

You should have received a copy of a Consent Information Form by email. If you did not, please let us know and we will send you that information. I’m going to review the content of that form before we begin.

We are conducting the Fathers and Continuous Learning in Child Welfare evaluation for the Office of Planning, Research, and Evaluation in the Administration for Children and Families at the U.S. Department of Health and Human Services. This project is designed to examine the use of the Breakthrough Series Collaborative (BSC) methodology to strengthen fathers’ and paternal relatives’ engagement with children involved in child welfare, and to add to the evidence base on engagement strategies for fathers and paternal relatives in child welfare.

The BSC is a collaborative learning approach in which multiple agencies test and spread promising practices to help themselves improve in a focused topic area. The BSC emphasizes the rapid use of data, feedback, and quality improvement to create organizational change and sustainable practices. Five agencies participated in the BSC. Each agency was represented in the BSC by staff with diverse roles who participated in a team-based learning approach, attended multiple in-person trainings, and received site-specific consultation about developing, testing, and adjusting their engagement strategies.

We are interested in hearing about your experiences working with [CHILD WELFARE AGENCY] and your efforts to enhance the engagement of fathers and paternal relatives with children involved in child welfare. We are interviewing key staff, partners, and other community stakeholders to learn about your agency’s efforts to engage fathers and paternal relatives. These efforts may include both distinct “strategies” and more general “approaches” to increasing father and paternal relative engagement. Strategies are distinct changes to policy or practice, such as enhanced father locating procedures. We think of approaches more broadly, including things [CHILD WELFARE AGENCY] has done to make it more welcoming to fathers and paternal relatives, such as staff training, knowledge sharing, and changes to the physical environment; approaches may also include multiple related strategies that together supported engagement. Both strategies and approaches may be important for increasing engagement among fathers and paternal relatives and changing agency culture, so we are interested in learning your perspectives on both.

3. Privacy and recording

We expect this discussion to take up to 90 minutes. Before we start, I want to let you know that your participation in this discussion is voluntary. We will use the information you share with us to write a summary of what we have learned. We will not connect your name to any of your responses, so please feel free to talk openly about your opinions. We will keep your identity private to the extent permitted by law.

We want to record the conversation so we make sure to capture the information you share accurately when we write reports. We will destroy the recording at the end of the project. If you want to say anything that you do not want recorded, please let me know, and I will be glad to pause the recorder. [Site visitors: if recording is declined, please take notes.]

Do we have your permission to record the conversation?

There are no consequences if you choose not to participate in this discussion. If you do not know the answer to a question, please say so, and we will simply move on. You do not have to answer any questions that you don’t want to answer.

If in person: We also ask that you keep the discussion private, and do not share what we discuss here with others outside this room.

• If virtual: We ask that you keep this discussion private, and do not share the details of this conversation with anyone who isn’t on the call today. We encourage you to find a quiet, private place where no one will overhear. [If necessary:] Also, we recognize that we are all working from home, and that is challenging for all of us. If you need to take a break or turn off your video to deal with any interruptions, don’t worry, we understand. Does that sound okay? Do you have any questions or suggestions before we get started? If there are no other questions, I’ll start the recording.

Paperwork Reduction Act Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is XXXX-XXXX, and it expires X/XX/20XX.

Note to interviewer

Before conducting interviews (either on site or virtually), and based on your current understanding of the site’s strategies, tailor this topic guide to be sure you capture information about the site’s engagement strategies and adjust the question phrasing as necessary to make sure respondents understand the questions.

Not all agencies will have respondents whose roles correspond to the ones in the upcoming tables. Some roles might be filled by more than one person or multiple roles might be filled by one person. Tailor the topic guide based on the following definitions:

• Senior leader: High-level child welfare administrator at each child welfare agency who oversaw the work of the Improvement Team

• Team manager and supervisors: Program manager, mid-level manager, and/or supervisors at each child welfare agency, responsible for supporting the work of the senior leader and Improvement Team members and overseeing frontline staff

• Frontline staff: Case managers or staff at the child welfare agency who work directly with fathers and paternal relatives and might have been a part of implementing strategies; this could include fathers and paternal relatives who participated on the Improvement Team and are formally employed by the agency

• Partner leadership: High-level administrator at partner agencies, such as court systems or fatherhood programs, who participated in formulating strategies and/or oversaw the work of partner frontline staff responsible for implementing strategies

• Partner frontline staff: Case managers or staff at the partner agency who work directly with fathers and paternal relatives and may have been a part of implementing strategies; this could include fathers and paternal relatives who participated on the Improvement Team and are formally employed by a partner

• Community stakeholder: community member whose role has intersected with the child welfare agency and has an interest in father and paternal engagement with the child welfare system, but was not formally part of formulating or implementing strategies.

Table 1. Number of topics covered, by staff role

Topic	Senior leaders	Team managers and supervisors	Frontline staff	Partner leader-ship	Partner frontline staff	Comm’ty stake-holder
A. Selecting strategies	4	3	0	4	0	2
B. Implementing strategies	6	7	5	8	4	6
C. Monitoring implementation	2	4	3	3	2	0
D. Shifting culture	6	6	6	6	6	3
E. Building on the BSC	9	3	0	5	0	4
F. Interview wrap-up	4	4	2	4	2	2
Total	31	26	16	28	14	17

Table 2. Topics to cover in interviews

Topic	Senior leaders	Team managers and supervisors	Frontline staff	Partner leader-ship	Partner frontline staff	Comm’ty stake-holder
A. Selecting strategies
In your opinion, what are the three or four main things that [child welfare agency] does to engage fathers and paternal relatives? For those involved in the BSC: For each strategy mentioned: Was [strategy] something you conducted a PDSA on in the BSC? If yes: What did you learn in using PDSAs (Plan, Do, Study, Act) to test the strategy? If no: Was [strategy] something that [child welfare agency] did before participating in the BSC, or something that it developed after the BSC was completed? For community stakeholders: Has this always been your experience? If no: How has this changed; what was it like before, and when did this change? If yes: How long have you been involved or aware of [the child welfare agency’s] work to engage fathers and paternal relatives?						
For each strategy mentioned in Question 1: Who was involved in the decision to [continue with/develop] [strategy]? What roles do these people play in [child welfare agency or community]? What is their involvement in implementing [strategy]? What kind of input did they provide in the decision to continue with [strategy]?				
For each strategy mentioned in Question 1: What information did you use when considering whether to [continue with/develop] [strategy]? (Probe for examples: BSC metrics, administrative data, staff feedback, participant feedback.) How did you decide to combine [strategies] together into a bundle? Did the information or data you used have any limitations or shortcomings? What would you have liked to know about [strategy] before you decided to continue it, but didn’t?				
What other things that you haven’t mentioned above does [child welfare agency] do to engage fathers and paternal relatives or cultivate racial equity?						
B. Implementing strategies
Please walk me through how you engage fathers and paternal relatives and cultivate racial equity for fathers and paternal relatives in your daily work. How is this different from how you worked with and viewed fathers and paternal relatives before [child welfare agency] participated in the BSC? How is this different from how you worked with and viewed fathers and paternal relatives [when you started at agency/at the start of your career]? How is this different from how [child welfare agency] has addressed racial inequity in the past? 1a. For community stakeholders: How does [child welfare agency] cultivate racial equity for fathers and paternal relatives in their work? How has this changed, if at all, from the way the child welfare agency worked before?						
For each strategy or approach mentioned in Section A: Is [strategy/approach] implemented the way it was originally intended? [If no] To your understanding, what was the original plan for how [strategy/approach] was supposed to work? In your opinion, what are the key differences in how [strategy/approach] is implemented now, compared to how it was implemented when you first started? Specifically, what is your role in [strategy/approach] and how has it changed from the original plans? Why did the plans change?				
How has [child welfare agency’s] approach to engaging fathers and paternal relatives grown over time? For example, have more staff, offices and/or partners become involved? Did you collect any information or feedback about how any particular strategies were working? Which ones? If so, did that information contribute to changes in the implementation plan?						
For each strategy/approach mentioned in Section A: We’ve talked a little about your role in [strategy/approach]. What other roles are involved in implementing [strategy/approach] now? What are their responsibilities? (for example, interacting directly with fathers and paternal relatives, monitoring fidelity, collecting data, providing supervision and oversight) Have there been any changes in these roles and responsibilities over time?				
For each strategy/approach mentioned in Section A: On the whole, how easy or hard would you say it has been to implement and grow [strategy]? In your opinion, how easy or hard has it been to grow [strategy/approach] within [child welfare agency]? What factors within [child welfare agency/partner organization] have made it harder to implement or grow [strategy/approach]? (For example, lack of staff buy in, additional data collection burdens, bureaucratic inertia.) What factors within [child welfare agency/partner organization] have helped the implementation or growth of [strategy/approach]? (For example, engaged and invested leadership, funding, media attention.)				
On the whole, how easy or hard would you say it has been to engage fathers and paternal relatives in your daily work? What factors within [child welfare agency/partner organization] have made it harder to engage fathers and paternal relatives? (For example, lack of staff buy in, additional data collection burdens, bureaucratic inertia.) What factors within [child welfare agency/partner organization] have helped engage fathers and paternal relatives? (For example, engaged and invested leadership, funding, media attention.)					
Thinking back, how did the COVID-19 pandemic disrupt or affect father and paternal relative engagement? Were there other environmental or community issues that affected father and paternal relative engagement? If yes, in what ways? For community stakeholders: How did COVID-19 affect the fathers and paternal relatives you work with? How did it affect their interactions with the child welfare system?						
Now, I’d like you think about public policy. Was there any legislation, administrative rules, funding requirements, or federal, state, or local policies that affected implementation? If so, in what ways? For community stakeholders: What laws or other policies affect the extent to which the fathers you work with engage with the child welfare system?						
If you had to pick the greatest success of father and paternal relative engagement, what would it be? Why? What would you say the greatest challenge has been, and why?						
Since starting to implement enhancements to the way [child welfare agency] engages fathers and paternal relatives developed under the BSC, have you noticed any changes in outcomes related to the engagement of fathers and paternal relatives? If so, what changes have you seen? To what extent do you think these are the result of the strategy, or something else?						
C. Monitoring implementation
What are [child welfare agency’s] goals for engaging fathers and paternal relatives and cultivating racial equity for fathers and paternal relatives? How would you know whether you were on the right track for meeting those goals? For each strategy/approach mentioned in Section A: What would it take for you to be able to say that [strategy/approach] has been a success?	
For each strategy/approach mentioned in Section A: Thinking about where you are now, and given [child welfare agency’s] goals, would you say that [strategy/approach] has been successful? Why or why not? [If strategy/approach has not been successful] What needs to change for [strategy/approach] to be successful?				
For each strategy/approach mentioned in Section A: How do you know whether [strategy/approach] is implemented as intended? Who is responsible for collecting this information? How frequently is it collected? How do you use the information once it is collected? What other practices does [child welfare agency] have for monitoring the [strategy/approach]?				
For each strategy/approach mentioned in Section A: What are the key client-level outcomes that [strategy/approach] is intended to affect? How do you know if [strategy/approach] is making a difference in these outcomes? What data do you collect on these outcomes? How frequently are data collected? Where are these data stored? Who is responsible for collecting and entering these data? Who is responsible for reviewing these data? Can you think of an example of how you have used these data to assess the success of [strategy/approach] or improve it?					
What has made it hard to collect or use data to assess the success of strategies and approaches to engage father and paternal relatives and cultivate racial equity for fathers and paternal relatives? (for example, lack of an accessible, centralized data system, lack of a process for checking the reliability of the data, data collection is burdensome, strategy is not designed to directly affect client outcomes)					
How do you assess the completeness and accuracy of data you collect about strategies and approaches to engage fathers and paternal relatives and cultivate racial equity for fathers and paternal relatives? In your opinion, how complete and accurate are these data? What processes have you put in place to improve the quality of the data? What would be helpful to you to improve the quality of the data?			
D. Shifting culture
In your own words, what are the core values of [child welfare agency]? What messages are you given by senior leadership about how you do your job? What messages do you try to convey to your staff about how they should approach their work? About engaging fathers and paternal relatives? About racial equity? Ask for each value: In your own opinion, how central is [father and paternal relative engagement/cultivating racial equity] to [child welfare agency’s] core values? Why? To what extent do your supervisor or other leaders encourage you to involve fathers and paternal relatives in cases? To what extent has your supervisor or other leaders explicitly discussed racial equity with you and your colleagues? Ask for each value: To what extent does [child welfare agency] recognize or reward staff for their efforts to [improve father and paternal relative engagement/promote racial equity]? Ask for each value: What changes have you noticed to [child welfare agency’s] organizational culture related to [fathers and paternal relatives/cultivating racial equity] since beginning to implement [strategy]? (for example: changes in mission and values statements, how staff talk about fathers and paternal relatives and/or racial equity, formal changes in policies and practices, environmental changes, changes in who is hired and represented on staff For community stakeholders: In what ways are the core values communicated to you and other community stakeholders? How do these core values align with your/your organization’s core values?						
How do you think that the father and paternal engagement strategies or approaches to cultivating racial equity for fathers and paternal relatives we’ve discussed have contributed to any of the changes you have observed in organizational culture at [child welfare agency]?						
What training or professional development have you been offered about fathers and paternal relatives or racial equity? For each training: When was it offered? Was this a one-time opportunity or something that has been offered multiple times? Was it optional for staff, or required? If optional, did you participate? If respondent participated: How helpful was the training or professional development opportunity? Why? If respondent did not participate: What kept you from being able to participate, or why did you choose not to participate?					
Aside from trainings or professional development, what other supports or resources does [child welfare agency] provide about father and paternal relative engagement or racial equity? (For example, manuals or guides, coaching, literature/tip sheets. Or for community stakeholders, community forums or other engagement events.) Have you used these supports? If no: Why not? If yes: How helpful have you found them, and why?						
E. Building on the BSC
Prior to participating in the BSC, what was [child welfare agency’s] process for planning and implementing improvements to program operations? What continuous quality improvement processes were already in place?	
How has participating in the BSC changed the way [child welfare agency] approaches continuous quality improvement?				
Are there specific BSC elements that [child welfare agency] has continued using since the BSC ended in March 2021? (for example: convening an improvement team, using an online shared learning environment, using a rapid-cycle model for improvement, such as PDSAs) If yes: Which ones, and why? Can you give an example or situation when the BSC element has been used? If no: What kept [child welfare agency] from continuing to use BSC elements? What has made it easy or hard to continue using BSC elements? If applicable: Why were some elements not continued?		
One of the core elements of the BSC was the creation of a multilevel inclusive team to lead change in an organization. While it was up to each child welfare agency to determine the makeup of this team, they were encouraged to include fathers and/or paternal relatives themselves. How successful do you think you have been in involving fathers and paternal relatives as decision makers? Why or why not? What role did individual fathers and paternal relatives play in the BSC? Since the BSC ended, how have you been able to involve fathers and paternal relatives? Going forward, what do you think the ideal involvement of fathers and paternal relatives would be? What would make it hard to get that level of involvement? Do you currently have any plans to involve fathers and paternal relatives as decision makers?				
How, if at all, have community partners been involved in engaging fathers and paternal relatives? Who are your most important community partners? How do you expect they will continue to be involved in efforts to engage fathers and paternal relatives?						
What do you think are the biggest practice or service challenges facing [child welfare agency] today?						
What thoughts or discussions have you had about how to address those challenges? What elements of the BSC do you think would be particularly helpful in addressing those challenges? (for example, using PDSAs to test strategies, collecting and analyzing data)						
Who are the most important stakeholders to involve in addressing these challenges?						
F. Interview wrap-up
Thinking back, what has been the most rewarding part of participating in the BSC? What was most challenging part of participating?				
What are the biggest lessons that you have learned from participating in the BSC? What would you say has been the biggest influence that participation has had on [child welfare agency]?				
Overall, reflecting on [child welfare agency’s] approach to father and paternal engagement, what would you say have been the biggest benefits to fathers and paternal relatives?						
Is there anything else that you think is important for us to understand about how the child welfare agency has changed as a result of BSC participation? Personally, what have been the biggest lessons you have learned? For community stakeholders: Is there anything else that you think is important for us to understand about how the child welfare agency has changed over the past several years?						

Thank you for sharing your experiences with us today.

DRAFT 08/13/21 Mathematica 1