Leadership Interview - States and Territories

Evaluation of the Child Welfare Capacity Building Collaborative

Leadership Interview—States

The purpose of the interview is to obtain information from State Child Welfare (CW) Directors regarding factors that influence their decisions to engage in services with the CB-supported Capacity Building Centers (specifically the Center for States); contextual factors that influence the jurisdictions; and perceptions of the capacity building services received and outcomes achieved.

Name of Agency:

Respondent Name:

Position Title:

Interviewer:

Date/time of interview:

Introduction:

Good morning/afternoon. My name is ______________. I am calling from James Bell Associates (or ICF International), an evaluation firm based in Arlington, VA. As you may know, James Bell Associates and ICF International have been contracted by the Children’s Bureau to conduct an independent evaluation of the Children’s Bureau’s Capacity Building Centers.

Directions to interviewer – please inform the interviewee:

This evaluation addresses the role of the Children’s Bureau Capacity Building Centers in helping States, Tribes, and Court Improvement Programs (CIPs) to enhance the capacity of their child welfare systems. The Leadership Interview is an important component of the evaluation. We are conducting this survey with all State CW directors as well as Tribal directors and CIP directors who receive capacity building services from the Centers.

Your participation in this interview is voluntary but critical to this evaluation. We anticipate that our discussion with you today will require at most 60 minutes. Your privacy is important to us. In order for us to have an accurate account of your responses, we would like to record this interview. The recording will be destroyed as soon as we transcribe it. We will remove all identifying information from the transcript. Responses that you provide will be combined with answers from other respondents and will not be associated with your agency/CIP or you as an individual respondent. Do we have your permission to record this conversation?

We are going to ask you about the factors that influence your decisions to use the services of the CB’s Capacity Building Center, contextual factors that influence your CW system, and your experiences with the services you have received from the Capacity Building Center for States.

Do you have any questions or concerns before we begin?

We would like to first start out by asking you some basic information on your background and history with the agency.

1. RESPONDENT BACKGROUND

1. What is your official position title?

2. How long have you been in this position?

_______years or _____months

3. How long have you been with the agency?

_______years or _____ months

2. CHARACTERISTICS OF THE CHILD WELFARE SYSTEM

1. Your child welfare system is:

Prefill 		State Administered
Prefill 		County Administered, State Supervised

Is this correct?

2. Your child welfare system utilizes private providers for [Prefill  All, Most, Some, None] placement and care responsibility on behalf of your child welfare agency. Is this correct?

3. [Prefill if information is available. Otherwise, ask:]

Is your child welfare system operating under a consent decree or settlement agreement as a result of a class action lawsuit? If yes  Please explain

4. Your State does (does not) have Title IV-E Waiver

Prefill 	Yes	As of [year]
Prefill 	No

Is this correct?

3. CAPACITY BUILDING SERVICES RECEIVED

Interviewer: if an agency has received services from more than one Center, please focus the interview on the primary Center that is assigned to the agency.

Now, let’s talk about the services that your agency has received from the Capacity Building Center for States to date. Please note, that this interview focuses on topics related to your experiences with the Capacity Building Center for States, and does not include topics related to your past experiences with NRCs or ICs.

1. According to the Children’s Bureau’s records of capacity building activities in the past year, the services that your agency received or participated in are listed in the document I sent you. Is that list of services accurate?

Prefill (from CapTrack) Universal/Constituency Services Received

Center Name	Month/year	Brief description

Prefill (from CapTrack) Tailored Services Received

Center Name	Month/year	Brief description of services (include brief vs intensive)

2. Have you noticed any changes in how technical assistance is now delivered by the Children’s Bureau’s new Capacity Building Centers? (In the past, the National Resource Centers (NRCs) and Implementation Centers (ICs) were the CB-supported T/TA providers.)

   1. If No  move to C3

   2. If Yes  What is different now, and what is your opinion of the new approach?

3. What factors influenced your decision to utilize these services from the Center for States? Please describe. [e.g., CFSR findings, etc.]

4. Have there been particular types of services and/or assistance that your agency needed that were not available from the Center?

5. Which types of services and/or assistance from the Center would you say have been most successful at meeting your agency’s needs?

6. Are there other TA providers (outside the Children’s Bureau supported Capacity Building Centers – such as local universities, Casey Family Programs, or other private TA providers) that are contributing to your agency’s capacity building efforts? Please describe.

Now we have some specific questions related to the types of assistance that your State has received from the Capacity Building Center:

4. ASSESSMENT AND WORK PLANNING

Interviewer: In round I (year 2) ask these (section D) questions of all agencies that are receiving tailored services. In round 2 (year 4), ask these questions of agencies that have newly (since round I) begun to receive tailored services.

First, a few questions about the assessment process:

1. Did your agency participate in the Center’s assessment process?

   1. If Yes  Why? [then move to D2]

   2. If No  Why not? [then move to D8]

2. During the assessment, did the Capacity Building Center staff have sufficient expertise to conduct a meaningful assessment of your CW system? Please describe.

3. Was the assessment comprehensive, taking into account all relevant aspects of your CW system? Please explain.

4. Did your agency’s capacity to self-assess its capacity building needs increase through the assessment process? Please briefly explain.

5. Did your agency’s understanding/awareness of the strengths and gaps in your CW system increase through the assessment process? Please briefly explain.

6. Did the assessment prioritize the areas of “need” for building capacity in your CW system? Please briefly explain.

7. How beneficial, or not, was the assessment process to your agency?

Now a few questions about the integrated capacity building plan that was developed as the result of the assessment process and about a work plan with the Center for States that may have also been developed.

8. Did the Capacity Building Plan identify appropriate resources and partners (e.g., consultants, agencies, or organizations – such as Casey Family Programs) to address your State’s capacity building needs? Please briefly describe.

9. Did the Capacity Building Plan include a work plan that details the planned delivery of services by the Center for States?

   1. If Yes  Please briefly describe the work outlined in the work plan. [then move to D10]

   2. If No  Why not? [then move to D11]

10. Please describe whether or not the work plan clearly outlined tasks, responsibilities, milestones, timelines, and desired short and long-term outcomes?

11. Do you have additional thoughts that you would like to share regarding your agency’s experiences with the assessment process or the Capacity Building Plan [if the State has a work plan with the Center ] or the work plan that was developed with the Center?

[Interviewer: Continue to section E for States that participate in tailored services OR skip to section F if the State has only participated in universal and/or constituency services.]

5. TAILORED SERVICES PROVIDED BY THE CENTERS

The next questions (section E in the interview guide that we sent you) pertain to: [Prefill from item C1 a brief description of the Tailored services]

1. In your opinion, was your agency ready to receive the services that the Capacity Building Center provided? [e.g., sufficient resources including staff resources, equipment and materials, evaluation resources]

   1. Please explain.

2. Did your agency revise existing policies and procedures or develop new ones with the support of the Center for States in order to address [Prefill from C1 a few words to describe the focus of the capacity building work]?

1. Please describe.

Now, let’s talk about the outcomes that were expected:

Along with the interview questions, we also sent you a brief description of the outcomes that were expected from the project supported by the Center’s services. Let’s first talk about which of these outcomes have been achieved to date:

[Interviewer: please go over each expected outcome and ask if it has been achieved. Please make a note of what was achieved and those outcomes that were not (or not yet) achieved.]

[Prefill from the work plan brief descriptions regarding intended outcomes/results]

Outcomes/results related to:	Brief description:

[Interviewer: For each outcome that was achieved, ask the next questions, focusing on one outcome at a time]

3. Outcome 1:

1. Please describe how this outcome was achieved?

2. To what extent did the services provided by the Center contribute to achieving this outcome?

2. How would you describe the quality of the Center’s services/materials?

3. Please describe what other factors contributed to this outcome?

4. Have these changes influenced your agency’s practice(s)? Please describe.

5. How will the changes that were achieved be sustained?

[If another outcome was achieved, ask E4]

4. Outcome 2:

1. Please describe how this outcome was achieved?

2. To what extent did the services provided by the Center contribute to achieving this outcome?

2. How would you describe the quality of the Center’s services/materials?

3. Please describe what other factors contributed to this outcome?

4. Have these changes influenced your agency’s practice(s)? Please describe.

5. How will the changes that were achieved be sustained?

[If a third outcome was achieved, ask E5]

5. Outcome 3:

1. Please describe how this outcome was achieved?

2. To what extent did the services provided by the Center contribute to achieving this outcome?

2. How would you describe the quality of the Center’s services/materials?

3. Please describe what other factors contributed to this outcome?

4. Have these changes influenced your agency’s practice(s)? Please describe.

5. How will the changes that were achieved be sustained?

[Interviewer: If more than three key outcomes were achieved, please add the items as additional sets of the questions in E5]

Many factors can either help or hinder your State’s ability to achieve the intended changes that the Center is supporting. These factors could be internal to the organization as well as external – and might include organizational or agency factors, community factors, or other environmental/contextual factors.

6. In your opinion, what were the key factors that helped your CW system to achieve the outcomes we discussed?

7. In your opinion, what key factors hindered your CW system from achieving the intended outcomes?

8. Have there been any unintended consequences or unforeseeable outcomes as a result of working with the Center? If yes  Please describe.

9. In your opinion, what have been some of the key benefits of working with the Center?

10. Based on your work with the Center, do you have any suggestions that you would like to share for improving the Center’s services? [Probes: e.g., regarding requesting services, service delivery, extent to which the services addressed the needs of the agency, etc.]

Now we have a few questions regarding more general services that the Capacity Building Centers (Center for States, Center for Courts, and Center for Tribes) provide, such as products and materials, webinars, presentations, etc.

6. UNIVERSAL/CONSTITUENCY SERVICES

1. Has your State used products/materials from the Capacity Building Centers or accessed content from the Centers’ websites?

   1. What was useful?

   2. What was not useful?

   3. Overall, how effective has the content of the Centers’ products and materials been in addressing the needs of your agency?

2. Has your State viewed or listened to any webinars, podcasts, or presentations offered by any of the Centers? If yes  which ones?

   1. What was useful?

   2. What was not useful?

   3. Overall, how effective has the content of the Centers’ webinars, podcasts, and presentations been in addressing the needs of your agency?

3. Could you talk about whether or not the strategies that the Centers use to disseminate information have been useful to your agency?

   1. Could you provide a specific example of what has been particularly useful and how?

   2. Could you provide an example of what has not been useful?

4. Has your agency participated in peer-to-peer networking activities provided by the Centers?

   1. What was useful?

   2. What was not useful?

5. Do you have any suggestions that you would like to share for improving the more general services that the Centers provide (e.g., peer-networking activities, webinars, materials)?

7. STATE’S RELATIONSHIP WITH THE CENTER

Now a few questions regarding your State’s opinion of working with the Center. Please provide the degree of your agreement with the following statements, using the seven response categories, from 7 to 1: Strongly agree; Agree; Agree somewhat; Undecided; Disagree somewhat; Disagree; Strongly disagree. These response categories are also listed in your copy of the interview questions document.

1. My agency is satisfied with the level of accessibility of the Center for States:

7-Strongly Agree

6-Agree

5-Agree Somewhat

4-Undecided

3-Disagree somewhat

2-Disagree

1-Strongly disagree

If “disagree somewhat”, “disagree”, or “strongly disagree”: How could accessibility be improved?

2. When working with the Center for States on issues related to my child welfare system, my agency has been satisfied with the frequency of communication:

7-Strongly Agree

6-Agree

5-Agree Somewhat

4-Undecided

3-Disagree somewhat

2-Disagree

1-Strongly disagree

If “disagree somewhat”, “disagree”, or “strongly disagree”: How could this be improved?

3. My agency is comfortable disclosing areas of concern or weaknesses of our agency to the Center for States’ staff:

7-Strongly Agree

6-Agree

5-Agree Somewhat

4-Undecided

3-Disagree somewhat

2-Disagree

1-Strongly disagree

If “disagree somewhat”, “disagree”, or “strongly disagree”: How could this be improved?

4. When working with the Center for States, my agency’s experience is that our agency plays an active part in decision-making:

7-Strongly Agree

6-Agree

5-Agree Somewhat

4-Undecided

3-Disagree somewhat

2-Disagree

1-Strongly disagree

If “disagree somewhat”, “disagree”, or “strongly disagree”: How could this be improved?

5. My agency is satisfied with the level of follow-through from the Center for States:

7-Strongly Agree

6-Agree

5-Agree Somewhat

4-Undecided

3-Disagree somewhat

2-Disagree

1-Strongly disagree

If “disagree somewhat”, “disagree”, or “strongly disagree”: How could their follow-through be improved?

6. Overall, my agency is satisfied with our relationship with the Center for States:

7-Strongly Agree

6-Agree

5-Agree Somewhat

4-Undecided

3-Disagree somewhat

2-Disagree

1-Strongly disagree

If “disagree somewhat”, “disagree”, or “strongly disagree”: How could the relationship be improved?

7. Overall, the services that the Center provided to my agency reflect an understanding of how my child welfare system operates.

7-Strongly Agree

6-Agree

5-Agree Somewhat

4-Undecided

3-Disagree somewhat

2-Disagree

1-Strongly disagree

If “disagree somewhat”, “disagree”, or “strongly disagree”: How could this be improved?

8. The services provided by the Center were appropriate and aligned with my child welfare system’s needs.

7-Strongly Agree

6-Agree

5-Agree Somewhat

4-Undecided

3-Disagree somewhat

2-Disagree

1-Strongly disagree

If “disagree somewhat”, “disagree”, or “strongly disagree”: How could this be improved?

9. The services provided by the Center took into account the characteristics and unique context of my State.

7-Strongly Agree

6-Agree

5-Agree Somewhat

4-Undecided

3-Disagree somewhat

2-Disagree

1-Strongly disagree

If “disagree somewhat”, “disagree”, or “strongly disagree”: How could this be improved?

7. IN CONCLUSION:

Do you have any additional comments you would like to share regarding CB’s Capacity Building Centers or your experiences with the services, products, and assistance they provide?

THANK YOU

WE GREATLY APPRECIATE YOUR PARTICIPATION!

12