Site Visit Discussion Guide for Project Directors and Leaders from Partner Organizations – Follow-Up Interviews (Instrument 6)

OMB Control No. ________
Expiration Date: ________

Instrument 6:

Site Visit Discussion Guide for Project Directors and Leaders from Partner Organizations – Follow-up Interviews

Introduction to Discussion Guide 3

Discussion Guide for Project Directors and Leaders of Partner Organizations – Follow-up Interviews 4

Introduction to Discussion Guide

Introduction: We are from the research firms Abt Associates and Child Trends and we are conducting a study of the Child Welfare Community Collaborations to Strengthen and Preserve Families (CWCC) grant initiative. The study is overseen by the Office of Planning, Research, and Evaluation (OPRE), in collaboration with the Children’s Bureau, in the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS). We are conducting a cross-site process evaluation study to learn how grantees receiving funding under the CWCC initiative are approaching their collaborative work.

As part of this study, we are talking with leaders and staff at the lead and partner organizations involved in [insert name of local CWCC-funded initiative]¹ to learn more about how grantees and their partners are collaborating to prevent child abuse and neglect in their communities. Discussion topics include: partnership approaches; data use; implementation of CWCC-funded activities or strategies; factors influencing implementation of your CWCC grant; and sustainability plans beyond the CWCC grant.

Before beginning our discussion, we want to thank you for agreeing to talk with us today. We know you are busy and we appreciate the valuable time you are spending with us today. The interview will take about 90 minutes. Your participation is voluntary, and there are no penalties for choosing not to take part in the interview. You can refuse to answer any questions or stop the interview at any time. Our aim is to learn from your insights and experience. There are no right or wrong answers. Your answers will be kept private to the extent permitted by law. Information you provide will not be shared with other staff at your agency or organization. Only the study team will have access to the information you provide through this interview. Your name will not be listed in any published reports, and comments will not be attributed to you. Instead, your information will be combined with information provided by others. However, because of the relatively small number of agencies/organizations participating in the study, there is a possibility that a response could be correctly attributed to you. We may archive the data collected by this discussion and allow access to the de-identified dataset to qualified researchers. Your data will only be used for research purposes.

We would like to record this discussion with a digital recorder so we can listen to it later when we write up our notes. No one besides our research team and the transcription vendor will listen to the recording. If you want to say anything that you don’t want recorded, please let us know and we will be glad to pause the recorder. Do you have any objections to being part of this interview or to us recording our discussion?

The Paperwork Reduction Act Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is _____, which expires ______. Please send comments about this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Abt Associates, 6130 Executive Blvd., Rockville, MD, 20852, Attn: Allison Hyra OMB-PRA ______. If you have questions or concerns about your rights as a participant in this evaluation, please call Abt’s Institutional Review Board at 877-520-6835.

Do you have any questions before we begin?

Discussion Guide for Project Directors and Leaders of Partner Organizations – Follow-up Interviews

First, I’d like to get an idea of the background of your organization and your work in it.

Background

1. Have there been any changes to the role [name of agency/organization] plays in [CWCC grant] since we last spoke in [Month, Year]?

2. Have there been any changes to your level of involvement within the [CWCC grant], including the specific roles or responsibilities that you perform, since we last spoke in [Month, Year]?

Now, we have some questions about partnerships in [CWCC grant].

Partnership Approaches

3. Have there been any changes to the composition of partnering organizations since we last spoke in [Month, Year]? Why did those changes happen?

4. Are there any types of organizations that don’t exist in your community but you see a need for in order to prevent child maltreatment? (Probe: For example, is there a need for more advocacy organizations versus service providers? Or providers of different types of services? Are there any organizations that focus on preventing child maltreatment but have waiting lists?)

For Project Directors from Lead Grantee Organization:

Have any of the initial challenges of setting up the partnership carried over to your efforts to maintain the CWCC partnerships?

1. Have any new challenges arisen amongst the partnership since partners started to implement the planned strategies, activities, and/or services of the [CWCC grant]?

2. What steps, if any, have been taken to resolve the challenges?

For Leaders from Partner Organizations:

Have you noticed or experienced whether any of the initial challenges of setting up the partnership carried over to the efforts to maintain the CWCC partnerships?

1. Have any new challenges arisen amongst the partnership since partners started to implement the planned strategies, activities, and/or services of the [CWCC grant]?

2. Have any new challenges arisen within your organization since partners started to implement the planned strategies, activities, and/or services of the [CWCC grant]? Are you aware of steps that have been taken to resolve the challenges?

For Project Directors from Lead Grantee Organization:

How would you describe your experience partnering with other organizations on the [CWCC grant]?

For Leaders from Partner Organizations:

How would you describe your experience partnering with the lead organization and other partner organizations on the [CWCC grant]?

7. In what ways, if any, have group processes and implementation approaches changed since we last spoke in [Month, Year]?

1. What are some of the structures or processes that have been the most helpful in supporting partners to implement CWCC activities or strategies?

2. Are all voices heard? Is there a shared sense of responsibility?

3. Are there still areas for improvement?

8. What are the various functions or roles that partners play in the work of the [CWCC grant]? (Probe for role/function of the lead organization, CW agency, direct service providers, courts, community-based organizations, faith-based organizations, parent/community members)

1. Are there clear expectations about what each partner organization is expected to do?

2. If so, to what extent are they contributing or performing as expected?

9. Which partners (or types of partners) do you consider to be most essential to the work of [CWCC grant], and why? Are there any partners who seem less essential to this work? (If needed to clarify: Essential could mean that these partners provide many services to families or provide direct services to client families as opposed to partners outside of this core set.)

The next few questions are about data use and data management.

Data Use

For Project Directors from Lead Grantee Organization:

Since we last spoke in [Month, Year], has your organization or any of your partner organizations made any changes to the data sources, assessments, and screening tools that are used to determine family and community needs? (Probe: Why or why not?)

1. Who are organizations screening?

2. When are screenings and assessments conducted?

3. Is screening conducted systematically across partners? (Probe: If not, are there any efforts to make screening more systematic across partners?)

4. How, if at all, are assessment or screening results used? (Probe: Are these data shared across partners? If so, in what ways are the shared data used?)?

For Leaders from Partner Organizations:

Since we last spoke in [Month, Year], has your organization made any changes to the data sources, assessments, and screening tools that are used to determine family and community needs? (Probe: Why or why not?)

1. Who is your organization screening?

2. When are screenings and assessments conducted?

3. Do you know whether screening is conducted systematically across partners? (Probe: If not, are there any efforts to make screening more systematic across partners?)

4. How, if at all, does your organization use assessment or screening results? (Probe: Do you share these data with other partners? If so, in what ways are the shared data used?)

5. Are you aware of how other partner organizations use assessment or screening results? (Probe: Do you receive these data from other partners?)

For Project Directors from Lead Grantee Organization:

To what extent, if any, do your organization and your partners’ organizations use needs assessment processes to prioritize which services participants receive?

1. Are these needs assessment processes used to determine the order in which participants receive services? (Probe: Why or why not?)

For Leaders from Partner Organizations:

To what extent, if any, does your organization use needs assessment processes to prioritize which services participants receive?

1. Are these needs assessment processes used to determine the order in which participants receive services? (Probe: Why or why not?)

For Project Directors from Lead Grantee Organization:

When, if at all, are your organization and your partners’ organizations entering screening and assessment data collected from participants?

1. What are some of the challenges, if any, partner organizations face in collecting and entering data in a timely manner?

For Leaders from Partner Organizations:

When, if at all, does your organization enter screening and assessment data collected from participants?

1. Who is responsible for data collection and entry?

2. What are some of the challenges your organization faces, if any, in collecting and entering data in a timely manner?

13. Have you noticed any changes in your organization’s data usage since the [CWCC grant] began?

14. To what extent, if any, does [CWCC grant] use data to inform its work?

1. Does data inform collaborative strategies and activities? (Probe: Why or why not?)

2. Are some strategies and activities more data-informed than others?

3. In what ways, if any, has the [CWCC grant]’s use of data changed since the grant began? (Probe for explanations and examples)

4. For Leaders from Partner Organizations Only: To what extent, if any, does your organization use data to inform its work within the [CWCC grant]? Has this changed over time?

5. For Leaders from Partner Organizations Only: Have you noticed any changes in data usage by other partners since the [CWCC grant] began?

15. Is there an expectation that partners participating in [CWCC grant] share data with [name of lead agency’s organization] as the lead organization?

1. If so, what are the purposes or goals of partners sharing data with the lead organization?

2. Are partners also expected to share data with other partner organizations? (Probe: Why or why not?)

3. Have expectations around data sharing changed since the [CWCC grant] began? (Probe: In what ways and why?)

16. What is the process, if any, for partner organizations to share data (either with the [name of lead agency’s organization] as lead organization or with other partner organizations)?

1. Do partners share data summaries or raw data?

2. Are partners entering data into an integrated system (available across multiple partner organizations)?

• If so, who is responsible for hosting/maintaining the integrated data system?

• Are data shared at the individual participant level (e.g., the parent/caregiver or child level) or in the aggregate (e.g., all caregivers, all children)?

• How frequently are data shared?

• What kind of training, if any, did staff receive related to data collection or entry?

• What kinds of security measures, if any, are in place to safeguard the data?

3. Have data sharing processes changed since the [CWCC grant] began? (Probe: In what ways and why?)

17. Since we last spoke in [Month, Year], have there been any changes to the way data are linked, collected, shared, and used by you or your partner organizations? (Probe: What changes were made and why?)

1. In what ways are data linked?

2. In what ways are data and used to do the following:

• identify families in need of child abuse and neglect (CAN) prevention services

• identify the specific needs of families

• make informed decisions about service provision

• inform continuous quality improvement efforts

• track outcomes

18. To date, how successful have the organizations partnering in [CWCC grant] been at sharing and using client data? Probes: Specifically, probe for whether they are doing the following (and ask why not, if applicable):

1. Using data to identify target populations?

2. Using data to identify client needs at intake?

3. Using data to decide on services to provide/refer to?

4. Using data to assess client service receipt and program participation?

5. Using data to assess client outcomes?

6. Using data to self-assess performance on above activities?

19. Are there particular processes or tools that have helped facilitate data sharing and use? (Listen for information about data systems, training/support to use those systems, templates, clear expectations about data sharing and use, regularly scheduled opportunities to share and discuss data)

For Project Directors from Lead Grantee Organization: Have partners received guidance on how to interpret data and apply it to their work?

For Leaders from Partner Organizations Only: Has your organization received guidance on how to interpret data and apply it to your work?

20. What are some of the barriers, if any, that hinder efforts for partners to share and use data? (Probe for barriers related to cost, man power, data security, legislation on data sharing/use, ; barriers to sharing across programs, across systems, etc.)

For Project Directors from Lead Grantee Organization:

How are these barriers being addressed?

For Leaders from Partner Organizations:

Are there any barriers that your organization has faced? How are the barriers faced by you and other partner organizations being addressed?

Next, we have some questions about implementation of the [CWCC grant]-funded activities.

Implementation of CWCC-funded Activities

21. Have any of the [CWCC grant] priorities shifted since we last spoke in [Month, Year]? (Probe: If so, in what ways and why?)

22. Have there been any changes to the continuum of services offered through the [CWCC grant] since the planning phase? Since the implementation phase?

1. What, if any, changes have been made to the eligibility requirements for receiving the continuum of services offered through the [CWCC initiative]?

23. Thinking about [CWCC grant] as a whole, which services or activities, if any, are provided through collaborative efforts? (For tribal grantees, probe for information about cultural or spiritual activities and interventions included as part of their programming.)

1. Does this way of delivering services or activities differ from how services were delivered prior to the [CWCC grant]? (Probe: If so, how and why?)

2. For Leaders from Partner Organizations Only: Are there any services or activities that your organization started delivering collaboratively as a result of the [CWCC grant]?

   • If so, what are some of the pros and cons of working collaboratively to deliver these services? (Probe for issues related to organizational identity, changes in mission and/or scope of work)

   • Have these new approaches required changes in terms of activities, staffing, reallocation of resources, etc.?

3. For Leaders from Partner Organizations Only: Are staff on board with these changes?

24. Which services or activities offered by [CWCC grant], if any, are delivered by a single agency (as opposed to multiple agencies)?

1. Does this way of delivering services or activities differ from how services were delivered prior to the [CWCC grant]? (Probe: If so, how and why?)

2. For Leaders from Partner Organizations Only: Are there any services or activities that your organization delivers as a single agency?

3. For Leaders from Partner Organizations Only: If so, what are some of the pros and cons of being the sole partner responsible for delivering these services?

25. Do you believe there are services or types of services that are more efficiently and effectively provided through a community collaboration effort?

1. If so, which services and why?

2. Are there any circumstances in which collaborative service delivery is not feasible?

26. Do you believe there are services or types of services that are more efficiently and effectively provided through a single-agency approach (as opposed to by multiple agencies)?

1. If so, which services and why?

2. Are there any circumstances in which single-agency service delivery is not feasible?

27. To date, are there any promising practices that have emerged from implementation of [CWCC grant]?

1. What implications, if any, might these practices have for the future of this particular initiative?

2. For Leaders from Partner Organizations Only: What implications, if any, might they have for your organization?

3. What implications, if any, might they have for other grantees or sites interested in developing a CWCC initiative?

28. Are there any lessons you would like to share regarding challenges or pitfalls that future CWCC initiatives should avoid?

1. What advice, if any, do you have for overcoming those challenges and pitfalls?

29. To date, have any new or modified policies (legislation or internal organizational procedures/policies) emerged from implementation of [CWCC grant]?

1. If so, which policies?

2. Have these policies been created or changed within particular agencies or organizations (state, local, or tribal)?

3. To what extent, if any, have these policies made their way into practice?

4. In what ways, if any, do these policies impact your organization and your partner organizations?

5. In what ways, if any, do these policies impact the families and children served by your organization and your partner organizations?

30. Do you expect any other policies to be created or modified as a result of implementation of [CWCC grant]?

Now I’d like to ask a few questions about some of the factors that may be influencing the implementation of [CWCC grant].

Factors Influencing Implementation of the CWCC Initiative

31. We know that often contextual factors (such as state, local, and tribal policies; resources; staff and organizational capacity, etc.) may facilitate or make your work easier Have any of the following contextual factors done this since we last spoke in [Month, Year]? If so, in what ways did the changes have an impact on your implementation?

32. Do you have any suggestions for how to leverage contextual factors to support implementation of CWCC initiatives?

1. Do you have any suggestions for how to handle negative contextual factors?

33. For Leaders from Partner Organizations Only: Since we last spoke in [Month, Year], have there been any changes to the target population(s) typically served by your organization?

1. If so, what informed or led to these changes?

2. Do you agree with those changes? Why or why not?

3. When were the changes made?

4. Do you anticipate any additional changes to target population? Why or why not?

For Project Directors from Lead Grantee Organization:

Since we last spoke in [Month, Year], have there been any changes to the target population(s) typically served by the partner organizations?

1. If so, what informed or led to these changes?

2. Do you agree with those changes? Why or why not?

3. When were the changes made?

4. Were these changes adopted by all partner organizations, or just some of them?

5. Do you anticipate any additional changes to target population? Why or why not?

For Leaders from Partner Organizations:

Since we last spoke in [Month, Year], have there been any changes to the target population(s) served by other partner organizations?

1. If so, what informed or led to these changes?

2. When were the changes made?

3. Were these changes adopted by all partner organizations, or just some of them?

4. Do you anticipate any additional changes to the [CWCC grant]’s target population? Why or why not?

For Project Directors from Lead Grantee Organization:

Since we last spoke in [Month, Year], have there been changes to the specific services and/or preventative interventions offered by partner organizations?

1. What informed or led to these changes?

2. When were these changes made?

3. Were these changes adopted by all partner organizations, or just some of them?

4. Do you anticipate any additional changes to specific services and/or preventative interventions? Why or why not?

For Leaders from Partner Organizations:

Since we last spoke in [Month, Year], have there been changes to the specific services and/or preventative interventions offered by partner organizations?

1. What informed or led to these changes?

2. When were these changes made?

3. Did your organization adopt these changes? Why or why not?

4. Were these changes adopted by all partner organizations, or just some of them?

5. Do you anticipate any additional changes to specific services and/or preventative interventions? Why or why not?

36. Since we last spoke in [Month, Year], have there been any changes to the CPS agency or any other major agency related to preventing child abuse and neglect? Any changes to state, local, or tribal policies?

1. Have there been changes in leadership at any of these agencies functioning at the state, local, or tribal level? If so, have these changes affected the implementation of the [CWCC grant]?

2. Have you noticed a difference in the guidance and/or policies coming out of state, local, or tribal agencies? (Probe: If so, is this impacting the work of the [CWCC grant]?)

3. Have there been any changes as a result of the [collaborative]’s work (e.g., review/removal of unnecessary service delivery or funding barriers)?

37. Since we last spoke in [Month, Year], have you noticed any shifts in the array of social service organizations available to support families in your community?

1. If so, please describe the changes. (Probe for changes in the number of service providers, types of services offered, target populations.)

2. For Leaders from Partner Organizations Only: In what ways, if any, have these changes impacted the work of your organization?

3. Do you anticipate any additional shifts in the array of social service organizations available to support families in your community? Why or why not?

In these last few questions, we would like to know more about your perceptions of the progress your team is making towards achieving the project goals and also lessons learned thus far.

Perceptions of Progress Towards Goals

38. Since we last spoke in [Month, Year], have there been any changes to the ultimate goals of the [CWCC grant]?

1. Have there been any changes in how the [initiative] tracks or measures progress towards meeting its goals?

39. To what extent do you believe the [CWCC grant] is meeting its goals? (Probe for specific examples)

1. Can you describe some of the early “wins” or successes from the [collaborative]?

2. Are there any goals that the initiative is falling short in at this time? (Probe for specific examples)

3. Are there any outcomes that the [CWCC grant] is falling short in at this time? (Probe for specific examples)

40. In your opinion, is there anything that has happened since our last site visit in [Month, Year] that would not have happened without the grant?

Lessons Learned

41. To date, what lessons, if any, have you learned about cross-agency collaboration?

1. In what ways, if any, have these lessons confirmed that the approach to collaboration is on-track?

2. In what ways, if any, have these lessons led you to change how you and your partners are approaching the CWCC work?

3. What advice would you give to other organizations preparing to launch or participate in a CWCC effort?

Finally, we are interested in your planning for sustaining [the initiative].

Sustainability Plans and Influential Factors

42. Knowing what you know about and have experienced with the work of [CWCC grant] so far, which aspects of the work do you expect to continue after the CWCC grant ends?

1. Which strategies, activities, and/or services, if any, should be sustained? Why?

2. For Leaders from Partner Organizations Only: Which strategies, activities, and/or services, if any, do you think your organization will continue? Why?

3. For Leaders from Partner Organizations Only: Which strategies, activities, and/or services, if any, do you think your organization will discontinue? Why?

4. Which partner organization(s), if any, would you expect to continue these strategies, activities, and/or services? Why? Which, if any, might not continue them? Why not, do you suppose?

5. Which strategies, activities, and/or services, if any, should not be sustained? Why?

43. How many [CWCC grant] strategies, activities, and/or services, if any, would need resources to be sustained?

1. What kind of resources would be necessary?

2. To what extent are these resources available?

44. To what extent might state, local, and tribal policies influence the extent to which [CWCC grant] strategies, activities, and/or services are continued after the CWCC grant ends?

1. For Leaders from Partner Organizations Only: Would any of these policies influence whether or how your organization would continue the [CWCC grant] strategies, activities, and/or services?

For Project Directors from Lead Grantee Organization:

Are the necessary resources (such as staff and organizational capacity and training) available to continue to support [CWCC grant] strategies, activities, and/or services after the grant ends?

For Leaders from Partner Organizations:

Does your organization have the necessary resources (such as staff and organizational capacity and training) available to continue to support [CWCC grant] strategies, activities, and/or services after the grant ends?

46. Are the existing partnership coordination processes and relationships strong enough to continue to support the [CWCC grant] strategies, activities, and/or services after the grant ends?

1. If no, is there anything that could strengthen partnership processes and relationships?

47. Are there any other factors that you think will promote or hinder efforts to continue the [CWCC grant] strategies, activities, and/or services after the CWCC grant ends?

1 Throughout this document, wherever terms are in brackets, the grantee’s own term will replace the bracketed terms in order to facilitate comprehension. Potential terms include: partnership, collaboration, coalition, collective impact, collaborative, collective, program, or project.

Abt Associates  ▌pg. 17