Site Visit Discussion Guide for Staff from Lead and Partner Organizations – Interview #1 (Instrument 5)

O MB Control No. 0970-0541
Expiration Date: 02/28/2023

Instrument 5a:

COVID Site Visit Discussion Guide for

Staff from Lead and Partner Organizations – Interview #1

Introduction to Discussion Guide 3

Discussion Guide for Staff from Lead and Partner Organizations – Interview #1 4

Introduction to Discussion Guide

Introduction: We are from the research firms Abt Associates and Child Trends and we are conducting a study of the Child Welfare Community Collaborations to Strengthen and Preserve Families (CWCC) grant initiative. The study is overseen by the Office of Planning, Research, and Evaluation (OPRE), in collaboration with the Children’s Bureau, in the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS). We are conducting a cross-site process evaluation study to learn how grantees receiving funding under the CWCC initiative are approaching their collaborative work.

As part of this study, we are talking with leaders and staff at the lead and partner organizations involved in [insert name of local CWCC-funded initiative]¹ to learn more about how grantees and their partners are collaborating to prevent child abuse and neglect in their communities. Discussion topics include: partnership approaches; data use; implementation of CWCC-funded activities or strategies; factors influencing implementation of your CWCC grant; and sustainability plans beyond the CWCC grant.

Before beginning our discussion, we want to thank you for agreeing to talk with us today. We want to acknowledge that we know that the COVID-19 pandemic has upended so much of everyone’s personal and professional lives and all of our communities. We are conducting our interviews, in part, to also learn how you and your organization are changing/adapting to this new context. At the same time, we know you are extremely busy and we appreciate the valuable time you are spending with us today. Is this still a convenient time to talk? The interview will take about 60 minutes. Your participation is voluntary, and there are no penalties for choosing not to take part in the interview. You can refuse to answer any questions or stop the interview at any time. Our aim is to learn from your insights and experience. There are no right or wrong answers. Your answers will be kept private to the extent permitted by law. Information you provide will not be shared with other staff at your agency or organization. Only the study team will have access to the information you provide through this interview. Your name will not be listed in any published reports, and comments will not be attributed to you. Instead, your information will be combined with information provided by others. However, because of the relatively small number of agencies/organizations participating in the study, there is a possibility that a response could be correctly attributed to you. We may be required to share your de-identified responses with ACF and authorized researchers. Your data will only be used for research purposes.

We would like to record this discussion with a digital recorder so we can listen to it later when we write up our notes. No one besides our research team and the transcription vendor will listen to the recording. If you want to say anything that you don’t want recorded, please let us know and we will be glad to pause the recorder. Do you have any objections to being part of this interview or to us recording our discussion?

The Paperwork Reduction Act Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0541, which expires 02/2//2023. Please send comments about this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Abt Associates, 6130 Executive Blvd., Rockville, MD, 20852, Attn: Allison Hyra OMB-PRA ______. If you have questions or concerns about your rights as a participant in this evaluation, please call Abt’s Institutional Review Board at 877-520-6835.

Do you have any questions before we begin?

Discussion Guide for Staff from Lead and Partner Organizations – Interview #1

First, I’d like to get an idea of the background of your organization and your work in it.

Background

1. Please provide an overview of the type of work your organization performs.

2. What is your title and role at [name of agency/organization]?

3. How long have you been at [name of agency/organization]?

4. How long have you been in your current role?

5. How many years of experience do you have working in this area?

6. What role does [name of agency/organization] play in [CWCC grant]?

7. Please describe your level of involvement within the [CWCC grant], including the specific roles or responsibilities that you perform.

Now, we have some questions about partnerships in [CWCC grant]. Throughout this discussion, please feel free to answer in ways that reflect how the [initiative name] has been functioning AND how it has shifted in light of the pandemic. We want to hear both about challenges you’ve been facing and successes you’ve achieved in addressing issues caused by the pandemic.

Partnership Approaches

8. Thinking about how the partnerships came to be formed: Who are the key partners in the [collaborative]? Are you familiar with what informed the partnerships?

1. Are you familiar with the extent to which data, if any, informed [lead organization name’s] approach to developing CWCC partnerships?

2. Were the voices or perspectives of frontline staff (from the lead and partner organizations) considered when identifying and creating new partnerships? What about the voices of families?

9. Are there any types of organizations that don’t exist in your community but you see a need for in order to prevent child maltreatment? (Probe: For example, is there a need for more advocacy organizations versus service providers? Or providers of different types of services? Are there organizations that focus on preventing child maltreatment but have waiting lists?)

10. How would you describe your experience partnering with other organizations on the [CWCC grant]?

11. In your opinion, how effectively has the [CWCC grant] been organized? (This is focused on organization either by the lead organization or by the partner organization.)

1. What structures or processes, if any, has the [CWCC grant] put in place that you think have been the most helpful in serving families at risk of child maltreatment? (Examples, if needed: regular meetings, shared network or website, shared planning tool, trainings, workshops. It should be structures and processes related to CWCC as opposed to other initiatives or projects.)

2. In what ways, if any, have you and your colleagues been supported to implement the new structures or processes you just described? (Probe for training on collaboration, working with others who may have different definitions for similar activities)

3. (if not answered in 8b) To what extent, if at all, do you believe the voices of frontline staff have been heard or represented in the work of the collaborative?

12. Are there clear expectations about what frontline staff and data managers at each partner organization are expected to do?

1. Do you think you have you received adequate training (and cross-training, if applicable) on your role and tasks within the [collaborative]?

2. How, if at all, have these expectations changed as a result of COVID-19?

13. Which partners (or types of partners) do you consider to be most essential to the work of [CWCC grant], and why? Are there any partners who seem less essential to this work? (If needed, clarify: Essential could mean that these partners provide many services to families or provide direct services to client families as opposed to partners outside of this core set.)

The next few questions are about data use and data management.

Data Use

14. Please describe the data sources, assessments, and screening tools, if any, that are used by your organization to determine family and community needs.

1. Who are you screening?

2. When are screenings and assessments conducted?

3. Do you know whether screening is conducted systematically across partners? (Probe: If not, are there any efforts to make screening more systematic across partners?)

4. How, if at all, are assessment or screening results used? (Probe: Are these data shared across partners? If so, in what ways are the shared data used?)?

15. When, if at all, does your organization enter screening and assessment data collected from participants?

1. Who is responsible for data collection and entry?

2. What are some of the challenges your organization faces, if any, in collecting and entering data in a timely manner?

16. To what extent, if any, does your organization use needs assessment processes to prioritize which services participants receive?

1. Are these needs assessment processes used to determine the order in which participants receive services? (Probe: Why or why not?)

17. To what extent, if any, does [CWCC grant] use data to inform its work?

1. Does data inform [collaborative] strategies and activities? (Probe: Why or why not?)

2. Are some strategies and activities more data-informed than others?

3. In what ways, if any, has the [CWCC grant]’s use of data changed since the [CWCC grant] began? (Probe for explanations and examples)

4. To what extent, if any, does your organization use data to inform its work within the [CWCC grant]? Has this changed over time?

18. Data Manager Only: Prior to [CWCC grant], what kind of data infrastructure did your organization have? What kind of infrastructure did your partner organizations have? Were the data shared amongst partners? Linked across partners?

19. Data Manager Only: Is there an expectation that partners participating in [CWCC grant] share data with [name of lead agency’s organization] as the lead organization?

1. If so, what are the purposes or goals of partners sharing data with the lead organization?

2. Are partners also expected to share data with other partner organizations? (Probe: Why or why not?)

3. Have expectations around data sharing changed since the [CWCC grant] began? (Probe: In what ways and why?)

20. Data Manager Only: What is the process, if any, for partner organizations to share data (either with the [name of lead agency’s organization] as lead organization or with other partner organizations)?

1. Do partners share data summaries or raw data?

2. Are partners entering data into an integrated system (available across multiple partner organizations)?

• If so, who is responsible for hosting/maintaining the integrated data system?

• Are data shared at the individual participant level (e.g., the parent/caregiver or child level) or in the aggregate (e.g., all caregivers, all children)?

• How frequently are data shared?

• What kind of training, if any, did staff receive related to data collection or entry?

• What kinds of security measures, if any, are in place to safeguard the data?

3. Have data sharing processes changed since the [CWCC grant] began? (Probe: In what ways and why?)

21. Data Manager Only: Which, if any, data from partner organizations are linked?

1. In what ways are data linked?

2. In what ways are data used to do the following:

• identify families in need of child abuse and neglect (CAN) prevention services

• identify the specific needs of families

• make informed decisions about service provision

• inform continuous quality improvement efforts

• track outcomes

Next, we have some questions about implementation of the [CWCC grant]-funded activities.

Implementation of CWCC-funded Activities

22. Thinking about [CWCC grant] as a whole, which services or activities, if any, are provided through collaborative efforts? (For tribal grantees, probe for information about cultural or spiritual activities and interventions included as part of their programming.)

1. Does this way of delivering services or activities differ from how services were delivered prior to the [CWCC grant]? (Probe: If so, how and why?)

2. Are there any services or activities that your organization started delivering collaboratively as a result of the [CWCC grant]?

   • If so, what are some of the pros and cons of working collaboratively to deliver these services? (Probe for issues related to organizational identity, changes in mission and/or scope of work)

   • Have these new approaches required changes in terms of activities, staffing, reallocation of resources, etc.?

3. Are staff on board with these changes?

23. Which services or activities offered by [CWCC grant], if any, are delivered by a single agency (as opposed to multiple agencies)?

1. Does this way of delivering services or activities differ from how services were delivered prior to the [CWCC grant]? (Probe: If so, how and why?)

2. Are there any services or activities that your organization delivers as a single agency?

3. If so, what are some of the pros and cons of being the sole partner responsible for delivering these services?

24. Do you believe there are services or types of services that are more efficiently and effectively provided through a community collaboration effort?

1. If so, which services and why?

2. Are there any circumstances in which collaborative service delivery is not feasible?

25. Do you believe there are services or types of services that are more efficiently and effectively provided through a single-agency approach (as opposed to by multiple agencies)?

1. If so, which services and why?

2. Are there any circumstances in which single-agency service delivery is not feasible?

26. To date, are there any promising practices that have emerged from implementation of [CWCC grant], including promising practices for addressing challenges emerging from COVID-19?

1. What implications, if any, might these practices have for the future of this particular [CWCC grant]?

2. What implications, if any, might they have for your organization?

3. What implications, if any, might they have for other grantees or sites interested in developing a CWCC initiative?

27. To date, have any new or modified policies (legislation or internal organizational procedures/policies) emerged from implementation of [CWCC grant]?

1. If so, which policies?

2. Have these policies been created or changed within particular agencies or organizations (state, local, or tribal)?

3. To what extent, if any, have these policies made their way into practice?

4. In what ways, if any, do these policies impact your organization and your partner organizations?

5. In what ways, if any, do these policies impact the families and children served by your organization and your partner organizations?

Now I’d like to ask a few questions about some of the factors that may be influencing the implementation of [CWCC grant].

Factors Influencing Implementation of the CWCC Initiative

28. Since the [CWCC grant] was awarded, have you noticed any shifts in the array of social service organizations available to support families in your community?

1. If so, please describe the changes. (Probe for changes in the number of service providers, types of services offered, target populations.)

2. In what ways, if any, have these changes impacted the work of your organization?

3. How have changes related to COVID-19 affected the work of your organization?

In these last few questions, we would like to know more about your perceptions of the progress your team is making towards achieving the project goals and also lessons learned thus far.

Perceptions of Progress Towards Goals

29. What are the ultimate goals of the [CWCC grant]? (Probe for intermediate system- and participant-level outcomes, as well as long-term outcomes)

1. What role did partners have in creating the goals?

2. How does the [CWCC grant] track or measure progress towards meeting its goals?

30. In what ways, if any, do you see your day-to-day work contributing to the goals and outcomes of the [CWCC grant]?

31. To what extent do you believe the [CWCC grant] is meeting its goals? (Probe for specific examples)

1. Can you describe some of the early “wins” or successes from the [collaborative]?

2. Are there any goals or outcomes that the [CWCC grant] is falling short in at this time? (Probe for specific examples)

32. In your opinion, is there anything that has happened since the [CWCC grant] started that would not have happened without the grant?

Lessons Learned

33. To date, what lessons, if any, have you learned about cross-agency collaboration (before and during COVID-19)?

1. In what ways, if any, have these lessons confirmed that the approach to collaboration was on-track?

2. In what ways, if any, have these lessons led you to change how you and your partners are approaching the CWCC work?

1 Throughout this document, wherever terms are in brackets, the grantee’s own term will replace the bracketed terms in order to facilitate comprehension. Potential terms include: partnership, collaboration, coalition, collective impact, collaborative, collective, program, or project.

Abt Associates  ▌pg. 10