Appendix B: Cross-Site and Center Evaluation Research Questions
Cross-Center Evaluation Questions
How and to what extent are key partners across and within the Centers collaborating?
What are the capacity building service interventions? To what degree are they evaluable? (Have they been sufficiently defined? Do we know what the defining characteristics, critical elements of the interventions are? Can we assess the degree to which these interventions are being delivered as intended and received? What assistance needs to be provided to help further specify these interventions?)
Universal service/dissemination strategies
Constituency-focused services
Tailored services
To what degree are Centers following common protocols for:
Product design?
Selecting dissemination strategies?
Learning experience choices and development?
Assessment and work planning?
Designing Intensive projects?
To what degree are service interventions being delivered/performed as designed? (What is being received? (i.e., intensity, frequency, etc.)
Dissemination strategies
Learning experiences
Peer networking activities
Assessment and work planning
Brief services
Intensive projects
How satisfied are recipients with the services received?
Universal services
Website content
Products and dissemination strategies (that we may be able to further categorize)
Constituency-focused services
Learning experiences
Peer networking activities
Tailored services
Assessment & work planning
Brief tailored services
Intensive projects (12-month work plans, MOUs, milestone schedules, etc.)
How effective were the service interventions?
Dissemination strategies
Learning experiences
Peer networking activities
Brief tailored services
Intensive tailored services
Which service approaches were most effective and under what conditions? (How did other factors influence the degree to which services were effective? Can this inform our decisions about the service strategies we choose in the future?)
Dissemination strategies
Learning experiences
Peer networking activities
Brief tailored services
Intensive tailored services
What are the costs of Universal, Constituent-focused, and Tailored products, activities, and services?
The data collection plan to address each of the evaluation questions is discussed in the sections below.
How do Centers operationalize the CB evidence-informed steps? What practices have the Centers identified as key to helping States to be successful in implementing these steps? (i.e. what is the Centers’ understanding of the steps? How consistent are they in applying the evidence-informed steps in their work?)
Capacity Building Center for States’ Evaluation Questions
Are project activities progressing on time and within expectations?
What are the services provided by the Center? Are deliverables being met?
What are Center staff perceptions regarding project structures, processes, and collaboration and their impact on the effectiveness of service provision? How do they evolve over the life of the Center?
To what extent are universal, constituency, and tailored services implemented with fidelity to coordinated and established processes? How do variations in processes and procedures impact service quality and effectiveness?
What are the structures in place to support continuous quality improvement? How does review of data impact service improvement and modification (e.g., changing work plans)?
As services become more solidified and refined, what are the lessons learned from Center development that can be shared with the field?
How do recipients perceive services provided by the Center?
Are Center services timely, accessible and in the right format for meeting needs?
Are Center services considered useful and relevant?
Are Subject Matter Experts of high quality and appropriate for assigned tasks?
How does State agency perception of engagement in the process of assessment, planning, and service delivery relate to the perceptions of service quality and effectiveness?
How and to what extent does each type of service (Universal, Constituency, Tailored) increase capacity?
Universal and Constituency:
How are priorities for universal and constituency services defined and determined?
What short-term outcomes in knowledge, skills, and attitudes result from universal and constituency services?
What changes in policy, practice, programs, and processes result from constituency services?
How are collaborative constituency services (e.g., peer networking) improving relationships across agencies and increasing knowledge, capacity, and efficient use of resources?
Tailored Services:
Does the collaborative and evidence-based assessment process lead to informed, realistic, and high quality work plans?
What short-term outcomes in knowledge, skills, and attitudes result from tailored services?
Which tailored services strategies were associated with achieving work plan goals?
What contextual factors influence successful accomplishment of work plans goals?
What changes in policy, practice, programs, and processes result from tailored services?
Capacity Building Center for Tribes’ (CBCT) Evaluation Questions
What were the strengths and challenges that affected the implementation of the CBCT?
How successful has the CBCT been in sustaining a collaborative governance structure that supports effective provision of capacity building services for tribal child welfare programs?
Universal Capacity Building Services
How successful is the CBCT in reaching a broad range of child welfare stakeholders with resources, information, and tools for serving AI/AN families?
Which stakeholders utilize the CBCT tools and resources, and how do these stakeholders perceive the accessibility, relevance, and usability of these resources?
What elements support child welfare stakeholders’ access to and utilization of information, tools, and resources related to serving AI/AN families? What challenges or barriers exist to access and utilization among these stakeholders?
Targeted Capacity Building Services
In what ways have the CBCT resources, strategies, and Targeted Capacity Building Services increased the knowledge, skills, and application of these interventions for recipients of tribal capacity building services?
How effective are P2P networks in fostering learning, support, collaboration, and innovation?
To what degree does the blended learning professional development series training advance the preparation of a culturally responsive and skilled child welfare workforce?
Tailored Capacity Building Services
In what ways have Tailored Capacity Building Services improved tribal child welfare program administration and practice?
How have Tailored Capacity Building Services improved tribal child welfare organizational processes, policies, and infrastructure?
How have Tailored Capacity Building Services helped to create a positive work environment and address workforce issues?
How have Tailored Capacity Building Services improved tribal child welfare programs’ interagency and cross-system collaboration?
Permanency Projects
In what ways have Permanency Projects built capacity in tribal child welfare programs to improve permanency outcomes for children in out-of-home placement?
What strategies or programs have tribes implemented to address the needs of children and families in adoption and guardianship situations?
Title IV-E Tribes
How has support for title IV-E tribes helped improve tribal–state agreements/collaboration?
In what ways have CBCT supports assisted tribes to build capacity to apply for a planning grant? To prepare a direct title IV-E application? To develop and implement a title IV-E program?
What supports do tribes need to successfully implement a direct title IV-E program?
Capacity Building Center for Courts’ (CBCC) Evaluation Questions
What services does CBCC provide? (Process Measures)
How many interactions (e.g., consultation, coaching, training, strategic planning, CQI efforts) do CBCC staff have with CIPS?
How many universal capacity building resources developed and distributed?
How many resources are accessed or downloaded online?
How many CIPs are pursuing development of evidence-informed/based practices?
How many subject specific materials are developed and disseminated?
How many targeted TA resources are provided around a specific topic?
How many locations are served with on-site targeted technical assistance?
How many regional and national events are held?
How many peer-to-peer learning and sharing opportunities are facilitated?
How many CIP collaborative events are held?
How many online training opportunities are developed and disseminated?
How many hours of TA are provided in collaboration with other T/TA capacity building providers?
How do CIPs perceive the services provided by CBCC? (Process/Satisfaction)
Do CIPs find the online resources easy to navigate?
Are tools developed by CBCC used by CIPs? In what way?
Are services provided timely?
Are tools, resources, and services of high quality and useful to CIPs?
Do CBCC staff and consultants have the appropriate skills and knowledge to assist the CIPs?
Do CIPs find CBCC staff helpful? In what ways?
How likely are CIPs to change or implement a new practice as a result of services provided by CBCC?
How does the focus on continuous quality improvement help CIPs to better work with their courts?
Are CBCC efforts related to specific changes in knowledge, behavior, or practice, including: (Outcome)
Increasing the CIPs understanding and application of continuous quality improvement to projects and activities;
Improving the CIPs ability to conduct ongoing assessments;
Improving the CIPs ability to collect, analyze and use data;
Increasing the CIPs collaboration with Agencies, Tribes, and stakeholders;
Increasing the CIPs meaningful participation in federal monitoring processes;
Increasing the number of shared goals and strategies between CIPs and CW agencies;
Increasing the capacity of CIPs to meaningfully participate in joint CQI efforts with the child welfare agency;
Increasing CIPs awareness and understanding of research on best practices;
Improving the CIPs ability to share data and supporting interoperability?
How effective are specific CBCC efforts in terms of building capacity? (Outcome)
Do CIPs report and increased understanding of how to improve or assess practice related to a specific topic area following participation in a constituency group?
Do CIPs report increases in collaborative practices following participation in a constituency group?
Do CIPs report a better understanding of continuous quality improvement in general or specific to a topic of interest after accessing resources available on the CBCC website?
Do CIPs initiate more robust assessments, evaluations, or research related to specific practices following engaging in specific CBCC activities (e.g., CQI workshops, CQI consults)?
Do CIPs demonstrate an increased awareness of resources and research after visiting the CBCC website?
Do CIPs show an increased interest in and understanding of federal monitoring?
Do CIPs demonstrate increased knowledge related to specific topical areas after viewing online training materials?
Do CIPs demonstrate increased knowledge related to specific topical areas after participation in a CQI workshop focused on that topical area?
How is CBCC contributing to the identification and dissemination of empirically supported best practices in dependency court?
Are CBCC efforts increasing the capacity of CIPs to improve safety, permanency, and well-being for children in foster care?
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | James DeSantis |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |