Appendix A_CIP Self-Assessment

OMB Control No: 0970-0307

Expiration Date: 09/30/2019

State Court Improvement Program 2019 Annual Self-Assessment Report

This self-assessment is intended as an opportunity for Court Improvement Programs (CIPs) to review progress on required CIP projects, joint program planning and improvement efforts with the child welfare agency, and ability to integrate CQI successfully into practice. Questions are designed to solicit candid responses that help CIPs apply CQI and identify support that may be helpful.

1. CQI Analyses of Required CIP Projects (Joint Project with Agency and Hearing Quality Project) It is ok to cut and paste responses from last year, but please update according to where you currently are in the process.

Joint Project with the Child Welfare Agency:

Provide a concise description of the joint project selected in your jurisdiction.

Identify the specific safety, permanency, or well-being outcome this project is intended to address.

Approximate date that the project began:

Which stage of the CQI process best describes the current status of project work?

How was the need for this project identified? (Phase I)

What is the theory of change for the project? (Phase II) If you do not yet have a theory of change and/or would like assistance, please indicate such in the space below.

Have you identified a solution/intervention that you will implement? If yes, what is it? (Phase III)

What has been done to implement the project? (Phase IV)

What is being done or how do you intend to monitor the progress of the project? (Phase V). Be specific in terms of what type of evaluation (e.g., fidelity or outcome, comparison group, etc) or data efforts you have in place or plan to have in place to assess your efforts. If you have already evaluated your effort, how did you use this data to modify or expand the project?

What assistance or support would be helpful from the CBCC or Children’s Bureau to help move the project forward?

Hearing Quality Project:

Provide a concise description of the joint project selected in your jurisdiction.

Approximate date that the project began:

Which stage of the CQI process best describes the current status of project work?

How was the need for this project identified? (Phase I)

What is the theory of change for the project? (Phase II) If you do not yet have a theory of change and/or would like assistance, please indicate such in the space below.

Have you identified a solution/intervention that you will implement? If yes, what is it? (Phase III)

What has been done to implement the project? (Phase IV)

What is being done or how do you intend to monitor the progress of the project? (Phase V) Be specific in terms of what type of evaluation (e.g., fidelity or outcome, comparison group, etc) or data efforts you have in place or plan to have in place to assess your efforts. If you have already evaluated your effort, how did you use this data to modify or expand the project?

What assistance or support would be helpful from the CBCC or Children’s Bureau to help move the project forward?

2. Trainings, Projects, and Activities

For questions 1-9, provide a concise description of work completed or underway to date in FY 2019 (October 2018-June 2019) in the below topical subcategories.

For question 1, focus on significant training events or initiatives held or developed in FY 2019 and answer the corresponding questions.

1. Trainings

Topical Area	Did you hold or develop a training on this topic?	Who was the target audience?	How many persons attended?	What type of training is it? (e.g., conference, training curriculum/program, webinar)	What were the intended training outcomes?	What type of training evaluation did you do? S=Satisfaction, L=Learning, B=Behavior, O=Outcomes
Data	☐Yes ☐No					☐S ☐L ☐B ☐O ☐N/A
Hearing quality	☐Yes ☐No					☐S ☐L ☐B ☐O ☐N/A
Improving timeliness/ permanency	☐Yes ☐No					☐S ☐L ☐B ☐O ☐N/A
Quality legal representation	☐Yes ☐No					☐S ☐L ☐B ☐O ☐N/A
Engagement & participation of parties	☐Yes ☐No					☐S ☐L ☐B ☐O ☐N/A
Well-being	☐Yes ☐No					☐S ☐L ☐B ☐O ☐N/A
ICWA	☐Yes ☐No					☐S ☐L ☐B ☐O ☐N/A
Sex Trafficking	☐Yes ☐No					☐S ☐L ☐B ☐O ☐N/A
Other:	☐Yes ☐No					☐S ☐L ☐B ☐O ☐N/A

On average, with ordinary funding levels, how many training events do you hold per year?

What is your best prediction for the number of attorneys and judges that attend a training annually?

The Family First Prevention Services Act amends the Social Security Act adding an eligibility criterion for the training of judges and attorneys on the congregate care provisions of the Act. See the highlighted portion below.

(1)¹ IN GENERAL.–– In order to be eligible to receive a grant under this section, a highest State court shall have in effect a rule requiring State courts to ensure that foster parents, pre- adoptive parents, and relative caregivers of a child in foster care under the responsibility of the State are notified of any proceeding to be held with respect to the child, shall provide for the training of judges, attorneys, and other legal personnel in child welfare cases on Federal child welfare policies and payment limitations with respect to children in foster care who are placed in settings that are not a foster family home, and shall submit to the Secretary an application at such time, in such form, and including such information and assurances as the Secretary may require, including–

States have an option to delay implementation of the congregate care provisions by two years. The decision will have a direct impact on when judicial determinations and CIP training requirements must begin.

Do you know when your state plans to implement Family First? ☐ Yes ☐ No

If yes, when?

Have you been involved in planning with the agency on implementing Family First? ☐ Yes ☐ No

If yes, please describe how the CIP has been involved.

Have you been developing your Family First judicial training plan? ☐ Yes ☐ No

If yes, please describe what you have done.

2. Data Projects. Data projects include any work with administrative data sets (e.g, AFCARS, SACWIS), data dashboards, data reports, fostering court improvement data, case management systems, and data sharing efforts.

Do you have a data project/activity? ☐ Yes ☐ No (skip to #3)

Project Description	How would you categorize this project?	Work Stage (if applicable)
(add narrative here)
Do you have data reports that you consistently view? ☐ Yes ☐ No

2. How are these reports used to support your work?

3. Hearing Quality. Hearing quality projects include any efforts you have made to improve the quality of dependency hearings, including court observation/assessment projects, process improvements, specialty/pilot court projects, projects related to court orders or title IV-E determinations, mediation, or appeals.

Do you have a hearing quality project/activity? ☐ Yes ☐ No (skip to #4)

Project Description	How would you categorize this project?	Work Stage (if applicable)

4. Improving Timeliness of Hearings or Permanency Outcomes. Timeliness and permanency projects include any activities or projects meant to improve the timeliness of case processing or achievement of timely permanency. This could include general timeliness, focus on continuances or appeals, working on permanency goals other than APPLA, or focus on APPLA and older youth.

Do you have a Timeliness or permanency project/activity? ☐ Yes ☐ No (skip to #5)

Project Description	How would you categorize this project?	Work Stage (if applicable)

5. Quality of Legal Representation. Quality of legal representation projects may include any activities/efforts related to improvement of representation for parents, youth, or the agency. This might include assessments or analyzing current practice, implementing new practice models, working with law school clinics, or other activities in this area.

Do you have a quality legal representation project/activity? ☐ Yes ☐ No (skip to #6)

Project Description	How would you categorize this project?	Work Stage (if applicable)

6. Engagement & Participation of Parties. Engagement and participation of parties includes any efforts centered around youth, parent, foster family, or caregiver engagement, as well as projects related to notice to relatives, limited English proficiency, or other efforts to increase presence and engagement at the hearing.

Do you have an engagement or participation of parties project/activity? ☐ Yes ☐ No

Project Description	How would you categorize this project?	Work Stage (if applicable)

7. Well-Being. Well-being projects include any efforts related to improving the well-being of youth. Projects could focus on education, early childhood development, psychotropic medication, LGBTQ youth, trauma, racial disproportionality/disparity, immigration, or other well-being related topics.

Do you have any projects/activities focused on well-being? ☐ Yes ☐ No (skip to #8)

Project Description	How would you categorize this project?	Work Stage (if applicable)

8. ICWA. ICWA projects could include any efforts to enhance state and tribal collaboration, state and tribal court agreements, data collection and analysis of ICWA compliance, or ICWA notice projects.

Do you have any projects/activities focused on ICWA? ☐ Yes ☐ No (skip to #9)

Project Description	How would you categorize this project?	Work Stage (if applicable)

9. Preventing Sex Trafficking and Strengthening Families Act (PSTFSA). PSTFSA projects could include any work around domestic child sex trafficking, the reasonable and prudent parent standard, a focus on runaway youth, focus on normalcy, collaboration with other agencies around this topic, data collection and analysis, data sharing, or other efforts to fully implement the act into practice.

Do you have any projects/activities focused on PSTSFA? ☐ Yes ☐ No

Project Description	How would you categorize this project?	Work Stage (if applicable)

III. CIP Collaboration in Child Welfare Program Planning and Improvement Efforts

1. Please describe how the CIP was involved with the state’s CFSP due June 30, 2019.

   1. Does the CFSP include any of the following:

☐ legal/judicial strategies ☐ the CIP/Agency Joint Project ☐ the CIP Hearing Quality Project

If yes, please describe.

2. Please describe how the CIP was or will be involved in the most recent/upcoming title IV-E Foster Care Eligibility Review in your state.

3. Please describe how the CIP is or was involved in preparing and completing round 3 of the CFSR and PIP, if required, in your state. Please check all the ways that the CIP or Court Personnel were involved (or plan to be involved) in the CFSR and PIP Process. Feel free to add additional narrative to explain your involvement in the process.

☐ were not involved at all

☐ were involved in planning the statewide assessment

☐were CFSR reviewers

☐ were interviewed for CFSR

☐were invited to the exit conference at the close of the CFSR review

☐ were invited to the final CFSR results session at the conclusion of the report

☐Final CFSR report was shared with you

☐Final CFSR report shared with courts broadly across the state 

☐ were a part of a large group of stakeholders engaged to assist in design of the PIP 

☐ high level of inclusion during the entire PIP process

☐ made suggestions for inclusion in the PIP 

☐suggestions made by CIP for inclusion in the PIP were put forward by the child welfare agency

☐ had an opportunity to review and provide feedback on the PIP before it was submitted

☐meet (or plant to meet) ongoing with the child welfare agency to monitor PIP Implementation

The current version of the PIP includes (check all that apply):

☐court strategies ☐court/agency shared strategies

☐ the court/agency joint project described above ☐ the CIP hearing quality project

☐ specific practice changes that judges will make

☐ specific practice changes that attorneys will make

4. What strategies or processes are in place in your state that you feel are particularly effective in supporting joint child welfare program planning and improvement?

5. What barriers exist in your state that make effective joint child welfare program planning and improvement challenging?

6. Does the state child welfare agency currently offer professional partner training to judges, attorneys, and court personnel as part of its Title IV-E Training Plan?

If yes, please provide a brief description of what is provided and how.

If no, have you met with child welfare agency leadership to discuss and explore utilizing professional partner training for judges, attorneys and court personnel?

7. Have you talked with your agency about accessing Title IV-E funding for legal representation for parents or for children? Is the agency planning to seek reimbursement? If yes, describe any plans, approaches, or models that are under consideration or underway.

IV. CQI Current Capacity Assessment

1. Has your ability to integrate CQI into practice changed this year? If yes, what do you attribute the increase in ability to?

2. Which of the following CBCC Events/Services have you/your staff engaged in in the 2019 Fiscal Year?

☐ Designing & Evaluating Effective Trainings Workshop

☐ CQI Consult (Topic:_______________________________)

☐ Constituency Group- Hearing Quality ☐ Constituency Group- Safety Decision Making

☐ Constituency Group- CFSR ☐ Constituency Group- Quality Legal Rep

☐ Constituency Group – ICWA ☐ Constituency Group – Anti-Trafficking

☐ Constituency Group – New Directors ☐ Constituency Group – APPLA/Older Youth

☐ CIP All Call –- What % of All Calls does your CIP participate in? ____%

2. Do you have any of the following resources to help you integrate CQI into practice?

☐CIP staff with CQI (e.g., data, evaluation) expertise ☐Consultants with CQI expertise

☐a University partnership ☐ A statewide court case management system

☐Contracts with external individuals or organizations to assist with CQI efforts

☐Other resources:_________________________________________

3a. Do you record you child welfare court hearings? ☐ Yes ☐ No

If yes, are they ☐ audio ☐ video

3b. Can you remotely access your court case management system? For example, Odyssey systems often allow remote access to case files. ☐ Yes ☐ No

4. Consider the phases of change management and how you integrate these into practice. Are there phases of the process (e.g., Phase I-need assessment, Phase II-theory of change) that you struggle with integrating more than others?

5. Is there a topic or practice area that you would find useful from the Capacity Building Center for Courts? Be as specific as possible (e.g., data analysis, how to evaluate trainings, more information on research about quality legal representation, how to facilitate group meetings, etc.)

Self-Assessment – Capacity Continued

We would like you to assess your current capacities related to knowledge, skills, resources, and collaboration by responding to the following 2 sets of questions. In questions 6 and 7, we ask about CQI. When we say CQI we mean the entire change management process including root cause analysis, theory of change, strategy selection, implementation and evaluation.

6. Please indicate your level of agreement to the following statements.

   	Strongly Disagree	Disagree	Somewhat Disagree	Neither Agree nor Disagree	Somewhat Agree	Agree	Strongly Agree
   I have a good understanding of CQI.	☐	☐	☐	☐	☐	☐	☐
   I understand how to integrate CQI into all our work.	☐	☐	☐	☐	☐	☐	☐
   I am familiar with the available data relevant to our work.	☐	☐	☐	☐	☐	☐	☐
   I understand how to interpret and apply the available data.	☐	☐	☐	☐	☐	☐	☐
   The CIP and the state child welfare agency have shared goals.	☐	☐	☐	☐	☐	☐	☐
   The CIP and the state child welfare agency collaborate around program planning and improvement efforts.	☐	☐	☐	☐	☐	☐	☐
   We have the resources we need to fully integrate CQI into practice.	☐	☐	☐	☐	☐	☐	☐
   I have staff, consultants, or partners who can answer my CQI questions.	☐	☐	☐	☐	☐	☐	☐

7. How frequently do you engage in the following activities?

	Never	Rarely	Sometimes	Often	Always
We use data to make decisions about where to focus our efforts.	☐	☐	☐	☐	☐
We meet with representatives of the child welfare agency to engage in collaborative systems change efforts	☐	☐	☐	☐	☐
We create theories of change around systems change projects.	☐	☐	☐	☐	☐
We use evaluation/assessment findings to make changes to programs/practices.	☐	☐	☐	☐	☐
We evaluate (beyond monitoring outputs) our efforts.	☐	☐	☐	☐	☐

Appendix A: Definitions

Definitions of Evidence

Evidence-based practice – evidence-based practices are practice that have been empirically tested in a rigorous way (involving random assignment to groups), have demonstrated effectiveness related to specific outcomes, have been replicated in practice at least one, and have findings published in peer reviewed journal articles.

Empirically-supported- less rigorous than evidence-based practices are empirically-supported practices. To be empirically supported, a program must have been evaluated in some way and have demonstrated some relationship to a positive outcome. This may not meet the rigor of evidence-base, but still has some support for effectiveness.

Best-practices – best practices are often those widely accepted in the field as good practice. They may or may not have empirical support as to effectiveness, but are often derived from teams of experts in the field.

Definitions for Work Stages

Identifying and Assessing Needs – This phase is the earliest phase in the process, where you are identifying a need to be addressed. The assessing needs phase includes identifying the need, determining if there is available data demonstrating that this a problem, forming teams to address the issue.

Develop theory of change—This phase focuses on the theorizing the causes of a problem. In this phase you would identify what you think might be causing the problem and develop a “theory of change”. The theory of change is essentially how you think your activities (or intervention) will improve outcomes.

Develop/select solution—This phase includes developing or selecting a solution. In this phase, you might be exploring potential best-practices or evidence-based practices that you may want to implement as a solution to the identified need. You might also be developing a specific training, program, or practice that you want to implement.

Implementation ­– the implementation phase of work is when an intervention is being piloted or tested. This includes adapting programs or practices to meet your needs, and developing implementation supports.

Evaluation/assessment – the evaluation and assessment phase includes any efforts to collect data about the fidelity (process measures: was it implemented as planned?) or effectiveness (outcome measures: is the intervention making a difference?) of the project. The evaluation assessment phase also includes post-evaluation efforts to apply findings, such as making changes to the program/practice and using the data to inform next steps.

1 Sec. 50741(c) of P.L. 115-123 revised sec. 438(b)(1) to add language regarding training. Effective as if enacted on 1/1/18 (sec. 50746(a)(1) of P.L. 115-123).

3