Cross-Center Evaluation: Leadership Interview – Tribes

PURPOSE: The purpose of the interview is to obtain information from Tribal Child Welfare Directors regarding factors that influence their decisions to engage in services with the CB-supported Capacity Building Collaborative (specifically the Center for Tribes); perceptions of the capacity building services received; and satisfaction with the Center’s services.

OMB #: 0970-0XXXX

Expiration Date: XX/XX/XXXX

PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN: This collection of information will be used to obtain information from tribal child welfare program directors regarding services received from the Capacity Building Collaborative. Public reporting burden for this collection of information is estimated to average 75 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. This is a voluntary collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: 0970-XXXX, Exp: XX/XX/XXXX. If you have any comments on this collection of information, please contact Chereese Phillips, ACF, Administration on Children, Youth and Families by e-mail at Chereese.Phillips@acf.hhs.gov.

Evaluation of the Capacity Building Collaborative

Tribal Child Welfare Leadership Interview

The purpose of the interview is to obtain information from Tribal Child Welfare Directors regarding factors that influence their decisions to engage in services with the CB-supported Capacity Building Collaborative (specifically the Center for Tribes); perceptions of the capacity building services received; and satisfaction with the Center’s services.

Interview Code: T___

Introduction:

Good morning/afternoon. My name is ______________. I am calling from James Bell Associates, an evaluation firm based in Arlington, VA. As you may know, James Bell Associates and ICF have been contracted by the Children’s Bureau to conduct an independent evaluation of the Children’s Bureau’s Capacity Building Collaborative, which includes of the Center for States, Center for Tribes, and Center for Courts.

Directions to interviewer – please inform the interviewee:

This evaluation addresses the role of the Capacity Building Collaborative in helping States, Tribes, and Court Improvement Programs (CIPs) to support and enhance the capacity of their child welfare systems. The Leadership Interview is an important component of the evaluation. We are conducting these telephone interviews with all State child welfare directors as well as Tribal directors and CIP directors who receive capacity building services from the Centers.

This interview will include two parts. I will let you know when we get to the end of the first part of this interview. The later part II of this interview includes a separate set of discussion topics that we would like to cover on behalf of the Center for Tribes. Your responses on part II will be de-identified and shared with the Center for Tribes evaluators.

I am going to ask you about the factors that may influence your decisions to use the services of the Capacity Building Collaborative and your experiences with the services you have received from the Center for Tribes.

Your participation in this interview is voluntary but important to this evaluation. We anticipate that our discussion with you today will take approximately 75 minutes. Your privacy is important to us. As we have noted in scheduling this interview with you, in order for us to ensure that we accurately capture everything, we would like to audio record this interview. The recording will be destroyed as soon as we transcribe it and we will remove all identifying information from the transcript. Responses that you provide will be combined with answers from other respondents and will not be associated with your program/your Tribe or you as an individual respondent. Do we have your permission to audio record this conversation? [Interviewer: Wait for the response] Do you have any questions or concerns before we begin?

I will now start the recording. [Interviewer: Begin audio recording]

PART I

1. RESPONDENT BACKGROUND

1. What is your official position title?

2. How long have you been in this position?

_______years or _____months

3. How long have you been with the child welfare program?

_______years or _____ months

B. THE CAPACITY BUILDING CENTER

First a question about the Capacity Building Center for Tribes.

1. How familiar are you with the Capacity Building Center for Tribes and its services? Please describe.

C. ASSESSMENT AND WORK PLANNING

Interviewer: CBCT uses NAFET 1. If the Tribe did not participate in the assessment, move to Section D.

1. During the assessment, did the Capacity Building Center staff have sufficient expertise to facilitate a meaningful assessment of your Tribal child welfare system? Please describe.

2. How useful, or not, was the assessment to your program?

   1. Did your child welfare program’s understanding/awareness of the strengths and gaps in your child welfare system increase through the assessment process? Please briefly explain.

   2. Did your child welfare program’s capacity to self-assess its capacity building needs increase through the assessment process? Please briefly explain.

   3. How accurately did the assessment summary identify the Tribal CW system’s capacity building needs?

3. Was a work plan developed as a result of the assessment?

   1. If yes  Did the work plan detail the planned delivery of services by the Center for Tribes?

   2. If no, skip to C5

4. In your opinion, how useful, or not, was the work plan?

   1. Did the work planning process help you to achieve things that you would not have achieved without the Center?

5. Do you have additional thoughts that you would like to share regarding your child welfare program’s experiences with the assessment process or the work plan that was developed with the Center?

D. WORKING WITH THE CAPACITY BUILDING CENTER

Now, let’s talk about the services that your child welfare program received from the Capacity Building Center for Tribes.

D1. According to the Children’s Bureau’s records, the services that your child welfare program received or participated in are listed in the document I sent you. Is that list of services accurate or is something missing?

Prefill from CapTRACK Tailored Services Records 

Assessment and workplans
Assessment service start	Assessment service end	Assessment hours	Workplan service start	Workplan service end	Workplan hours
Projects
Service start date	Date closed	Service hours	Project title		Project related to CFSR/PIP? (Y/N)
Other tailored services
Year	PIC hours	CFSP hours	FFPSA hours
CY20
CY19

D2. What were the key factors that influenced your decision to engage with the Center for Tribes to utilize the Center’s services? Please describe.

3. Let’s take a look at the list of potential factors that is included as item D3 in the interview questions summary document I sent you. Were there any additional key factors that also influenced your child welfare program’s decision to use (or not to use) the Center’s services?.

Affected decision to use services (+ or -)	Factors	Comments
	CFSR findings or development of the PIP
	Regional Office recommendation
	Federal law or policy change or requirements
	Center’s outreach efforts
	Prior relationship with the Center
	Timeliness in which services can be received
	Center staff’s level of knowledge and skills
	Cultural competency of the Center staff
	Usefulness of the Center’s products/materials
	Child welfare incident in the State or Tribe (such as a child fatality)
	Availability of free resources from the Center
	Availability of child welfare program staff time
	Tribe’s child welfare program leadership
	State/local law or policy change
	Lawsuit/legal settlement
	COVID-19 pandemic

E. SERVICES RECEIVED AND PROGRESS TOWARD OUTCOMES

The next questions (section E in the interview guide that we sent you) pertain to: [Prefill from item D1 brief description(s) of the tailored services]

1. First, could you describe the current status of your Tribe’s work with the Capacity Building Center for Tribes?

2. In your opinion, what are the [up to three] most important outcomes that your Tribe expected to achieve from the work with the Center?

Interviewer, list up to 3 key outcomes that the respondent identifies:

Now, let’s talk about the progress that has been made toward the outcomes.

[Interviewer: For the expected outcomes that were identified, ask the following questions, focusing on one outcome at a time]

3. Outcome 1:

Interviewer: Fill in 1^st outcome identified by the respondent: _______________________________

1. Please describe the progress that has been made toward [outcome 1].

2. To what extent did the services provided by the Center contribute toward this outcome?

3. Were there other key factors that also contributed toward this outcome?

4. Outcome 2:

Interviewer: Fill in 2^nd outcome identified by the respondent: _______________________________

1. Please describe the progress that has been made toward [outcome 2].

2. To what extent did the services provided by the Center contribute toward this outcome?

3. Were there other key factors that also contributed toward this outcome?

5. Outcome 3:

Interviewer: Fill in 3^rd outcome identified by the respondent: _______________________________

1. Please describe the progress that has been made toward [outcome 3].

2. To what extent did the services provided by the Center contribute toward this outcome?

3. Were there other key factors that also contributed toward this outcome?

6. Was your work with the Center intended to result in any practice changes in your child Tribal welfare system?

   1. If yes  In your opinion, are your child welfare program’s practices changing or improving in any way? Please describe.

   2. What do you attribute these changes to?

   3. And are there any changes in outcomes for the children and families served by your child welfare system?

   4. If yes  Please describe. (Interviewer: Probe also for what evidence they have of these changes)

      1. In your opinion, what contributed to these changes?

7. Were there any key factors that may have hindered your child welfare program from making progress toward achieving the intended outcomes? If so, please describe. [e.g., organizational or program factors, community factors, other environmental/contextual factors]

   1. Covid-19 question for Tribes is included in Part II.

8. Overall, how would you describe the quality of the services/materials your program received from the Center?

9. Based on the services your child welfare program received, how would you describe the expertise of the Center liaisons/consultants?

10. In your opinion, what have been the key benefits of working with the Center?

11. Based on your work with the Center, do you have any suggestions for improving the Center’s services?

F. TRIBE’S EXPERIENCE WITH THE CENTER FOR TRIBES

Please provide your level of agreement with the following statements, using the response categories, from 1 to 6: Strongly disagree; Disagree; Somewhat disagree; Somewhat agree; Agree; Strongly agree. These response categories are also listed in your copy of the interview questions document.		1 Strongly disagree	2 Disagree	3 Somewhat disagree	4 Somewhat agree	5 Agree	6 Strongly agree	Do not know
F1.	My child welfare program is satisfied with the level of accessibility of the Center for Tribes
If “disagree somewhat”, “disagree”, or “strongly disagree”: How could accessibility be improved?
F2.	When working with the Center for Tribes on issues related to my child welfare system, my program has been satisfied with the frequency of communication
If “disagree somewhat”, “disagree”, or “strongly disagree”: How could this be improved?
F3.	My program is comfortable disclosing areas of concern or weaknesses of our program to the Center for Tribes’ staff
If “disagree somewhat”, “disagree”, or “strongly disagree”: How could this be improved?
F4.	When working with the Center for Tribes, my program’s experience is that our program plays an active part in decision-making
If “disagree somewhat”, “disagree”, or “strongly disagree”: How could this be improved?
F5.	My program is satisfied with the level of follow-through from the Center for Tribes
If “disagree somewhat”, “disagree”, or “strongly disagree”: How could their follow-through be improved?
F6.	Overall, my program is satisfied with our relationship with the Center for Tribes
If “disagree somewhat”, “disagree”, or “strongly disagree”: How could the relationship be improved?
F7.	Overall, the services that the Center has provided to my program reflect an understanding of how my child welfare system operates
If “disagree somewhat”, “disagree”, or “strongly disagree”: How could this be improved?
F8.	The services provided by the Center were appropriate and aligned with my child welfare system’s needs
If “disagree somewhat”, “disagree”, or “strongly disagree”: How could this be improved?
F9.	The services provided by the Center took into account the characteristics and unique context of my Tribe
If “disagree somewhat”, “disagree”, or “strongly disagree”: How could this be improved?
F10.	I would recommend the Center for Tribes’ services to other Tribal child welfare programs

G. IN CONCLUSION

G1. [Conclusion question is at the end of Part II (item 36) for Tribal respondents]

This concludes the Part I of the interview questions.

***

The next section of the interview (part II) includes topics that we will cover on behalf of the Center for Tribes. Do you have any questions before we start?

PART II

Capacity Building Center for Tribes – Jurisdiction Leadership Interview

(Questions in red should only be asked of tailored services projects that have been completed)

Section 1: Service Provision for Your Project

Please rate how much you agree with the following statements about the services your child welfare program received from the Center for Tribes.	1 Strongly disagree	2 Disagree	3 Agree	4 Strongly agree
The process to request services from the Center for Tribes went smoothly.
The Center for Tribes engaged our program staff in the project.
The services provided gave our program useful tools for our work.
The services received will help our program make needed changes.
In general, services were conducted in a way that reflected how people from our tribe work together to accomplish something.
Our tribal community’s values and practices were respected during the provision of services.
The Center for Tribes linked our program to helpful resources (e.g., other providers, on-line resources through the Tribal Information Exchange, webinars, peer networks, publications, training opportunities).
Our program and the Center for Tribes collaboratively identified the goals of the project.
Program staff and key stakeholders involved in the project planning and implementation understand what they have to do to move the work forward.
The objectives of our project were met.

11. What have you learned or gained from the services provided by the Center for Tribes?

12. What did you find most useful about the services provided by the Center for Tribes?

13. What did you find least useful about the services provided by the Center for Tribes?

Section 2: Center for Tribes Team

Please rate how much you agree with the following statements about the Center for Tribes team.	1 Strongly disagree	2 Disagree	3 Agree	4 Strongly agree
The Center for Tribes effectively assessed our program’s strengths.
The Center for Tribes effectively assessed our program’s challenges.
The Center for Tribes worked in partnership with the tribal child welfare team through all stages of the project.
The Center for Tribes was knowledgeable about the topic(s)/content area(s) of our project.
The Center for Tribes communicated with us in a way that fit our tribal community’s values.
The Center for Tribes was a good match for helping us address our needs/project goals.
Our ideas and desires regarding this project were solicited and respected by the Center for Tribes team.

21. Please provide any additional feedback about the Center for Tribes team.

Section 3: Overall Experience

Please rate how much you agree with the following statements about the services your program received from the Center for Tribes.	1 Strongly disagree	2 Disagree	3 Agree	4 Strongly agree
Skip (see Part I satisfaction items)
Skip (see Part I, item F10)

If tailored services project has been completed, skip to question 26.

If tailored services project has not yet been completed, continue with questions 24 and 25 

24. Has progress towards completing your project work plan been impacted by the COVID-19 pandemic?

If yes, please describe.

25. Has progress toward completing your project work plan activities been impacted [slowed or impeded] by any other internal event or other unforeseen event or circumstance in your program, tribe, or community?

If yes, please describe.

If tailored services project has not been completed, skip to questions 35 and 36.

26. Was progress toward completing your project work plan impacted by the COVID-19 pandemic?

If yes, please describe.

27. Was progress toward completing your project work plan impacted [slowed or impeded] by any internal event or other unforeseen circumstance in your program, tribe, or community?

If yes, please describe.

Section 4: Next Steps [this section to be used for tailored services projects that have been completed]

Please rate how much you agree with the following statements about the services your program received from the Center for Tribes.	1 Strongly disagree	2 Disagree	3 Agree	4 Strongly agree
The services provided by the Center for Tribes have helped to strengthen our child welfare program.
I understand what the “next steps” are regarding the work related to this project.
Program staff and key stakeholders involved in the project planning and implementation understand what they have to do to move the work forward.
Our program has the resources to support continued efforts related to this project.
Our program intends to continue to build upon the work done through this project.

33. Please provide any additional feedback or comments about your program’s next steps.

34. What specific “next steps” were taken since the project was completed? How will this work be sustained in the future?

35. How could the Center for Tribes have improved your child welfare program’s experience with this project?

36. Do you have any additional comments you would like to share regarding CB’s Capacity Building Collaborative or your experiences with the services they provide?

THANK YOU.

WE GREATLY APPRECIATE YOUR PARTICIPATION!