SUPPORTING STATEMENT FOR
Evaluation of the Child Welfare Capacity Building Collaborative: Part Two
Section B
Department of Health & Human Services
Children’s Bureau
Washington, DC
Contact person:
Brian Deakins
Children’s Bureau
Administration on Children, Youth and Families
Mary E. Switzer Building
3rd Floor, Mailstop 3602
330 C Street, SW
Washington, DC 20201
202-205-8769
Section B: Collections of Information Employing Statistical Methods
Capacity Survey: This survey will be administered to State, territory, and Tribal public child welfare agency staff who participate in intensive “tailored” services engagement when services last for 9 months or longer. Center for States may provide more than one “tailored” service engagement to a State or territory agency at one time; in this case the Center for States will select the service which is anticipated to be most intensive (most service hours delivered and/or longest duration of service delivery).
Tailored Services Satisfaction Survey: These items will be administered in conjunction with the Capacity Survey. The satisfaction items for tailored services will be included with the Capacity Survey on the annual administration of the survey for projects lasting more than one year and on the final administration of the Capacity Survey at completion of the tailored service. For tailored services that last less than one year, the satisfaction items will be included only on the final administration of the Capacity Survey at completion of the tailored service.
Foundational Assessment Survey: This survey will be administered annually to all staff identified by the Center as a participant in the assessment process.
CQI Workshop Follow-Up Survey: This survey will be administered to members of CIP teams that attend a CQI Workshop. The survey will be administered 6 months after the Workshop and again at 12 months after the Workshop. The second administration will be conducted as part of the Capacity Survey.
P.L. 113-183 Key Informant Survey: This survey will be administered to a child welfare agency staff person in each State, the District of Columbia, and Puerto Rico who has been identified as the key point of contact regarding P.L. 113-183. Pending approval, the survey will be administered three times: once in early 2017, once in end October 2017, and once in end 2018.
P.L. 113-183 Key Informant Interview: This interview will be conducted with a tribal child welfare agency staff person in any title IV-E tribes that agree to participate in data collection. Pending approval, the interview will be conducted three times: once in 2016, once in October 2017, and once in October 2018.
Center for States Tailored Services Interview: These interviews, or focus groups, will be conducted with a sample of up to 12 State and territory public child welfare agency staff upon completion of brief or intensive projects.
Center for States Assessment and Work Planning Survey: This survey will be administered every other year to half of the States, alternating with the Cross-Center Satisfaction with Assessment and Work Planning Survey.
Webinar Registration: This collection will occur for all State and territory public child welfare agency staff, federal staff, and other stakeholders who register to participate in Center for States webinars.
CapLEARN Registration: This form will be completed once by all individuals who create an account to access the Collaborative resource library and on-line trainings.
Center for Courts Universal and Constituency Services Survey: The target population for the Center for Courts Universal and Constituency Services survey is all Court Improvement Program Directors and Coordinators. The survey will be administered annually via a web link.
No statistical methodology for stratification and sample selection will be used for any of the data collection instruments aside from the Center for States Tailored Services Interview. For this instrument, a purposive sample of up to 12 States will be selected based upon geographic and services content specific variables such as regional location, length of service, topic of service in relation to Center priorities, and type of outcomes achieved.
Maximizing response rates is critical to the administration of the aforementioned surveys.1 The content and format of the instruments were developed in close consultation with key stakeholders, including the Collaborative Centers and were informed by previously developed measures involving technical assistance provision.
Strategies that emphasize flexibility, confidentiality, and a respect for the respondent’s time facilitate timely participation. The following strategies will be implemented to maximize participation in the data collection and achieve the desired response rates: 2
Introduction and notification: Strategies to introduce and notify respondents about data collection are used for several instruments. For the Center for Courts, all recipients of brief and intensive services, learning experiences, peer networking events, and constituency groups, will be notified at the beginning of service delivery of the Center’s evaluation efforts and the desire for their participation. For Center for States Tailored Services Interviews, potential participants will be informed of the administration and participation requested at least one month prior to administration. They will have the opportunity to ask any questions and will be informed that their feedback is critical to planning for future service provision and understanding service effectiveness. For the Center for States Assessment and Work Planning Survey, participants will be informed during initial messaging regarding the assessment and work planning process about the survey and how the information will be used to inform service planning and development along with understanding service effectiveness. An additional description about the survey and participation will be included in the message distributed with the survey. For the Webinar Registration Form, participants are notified of the opportunity to register for each webinar with the initial invitation and through webinar marketing efforts. Upon entry into the system, details regarding data to be collected are provided in the form. Participants encounter the Collaborative’s CapLEARN registration form on the Collaborative’s website as they set up their account to track their progress.
The Foundational Assessment Survey will be introduced to States, Tribes, and Court Improvement Programs first through discussion and a memorandum of understanding that indicates participation in the Cross-Center Evaluation by each jurisdiction receiving training and technical assistance from a collaborative Center is anticipated. Each Center will then introduce the Foundational Assessment Survey as part of its assessment process with the jurisdiction. An introductory email will then be sent from the Cross-Center Evaluation team to assessment participants from each jurisdiction identified by the Center that will inform all State, CIP, and Tribal respondents about the administration of the Foundational Assessment Survey. A brief description of the Cross-Center Evaluation will be included in this email. For prospective Tribal respondents, this email will also be sent to the Tribal Leader or Chairperson. Follow-up meetings and introductory calls will be made with prospective respondents through the Center assessment process to introduce the Cross-Center Evaluation team and to address any questions about the data collection. In the subsequent years, reminder emails will be sent to all State, CIP, and Tribal respondents. We recognize that there may be some turnover in the assessment participants over time and that we may have to re-introduce the survey. The CQI Workshop Follow-Up Survey will be introduced and described by Center for Courts’ staff to attendees of the CQI Workshops. Potential participants in the P.L. 113-183 Key Informant Survey and Interview will be informed of the administration via email and participation will be requested at least one month prior to administration.
Timing of data collection: Discussions were held with stakeholders to determine optimal periods for data collection in order to minimize respondent burden and to facilitate recall. The Foundational Assessment Survey will be completed in alignment with the assessment process, approximately one month prior to the start of the annual assessment for each jurisdiction receiving training and technical assistance. The Capacity Survey and Tailored Satisfaction Services Survey will be administered together at the end of “intensive tailored service” delivery for services lasting 9 to 12 months; for intensive tailored services of one year or more, the survey will be administered annually and at the end of services. The CQI Workshop Follow-Up Survey will be administered 6 months after respondents attend a CQI Workshop and again at 12 months after the Workshop in conjunction with the Capacity Survey and Tailored Services Satisfaction Survey. The P.L. 113-183 Key Informant Survey and Interview will be administered once in early 2017, and again in end 2017 and 2018. The Center for Courts Universal and Constituency Services Survey will be completed annually at a scheduled time to follow within six months of CIP annual reporting efforts.
The Webinar Registration and CapLEARN Registration will be administered throughout the project period following the administration guidelines described in section B-1. CapLEARN Registration is available online at all times for an individual to make an account to access the resource and training library. The Center for States interviews will be conducted annually, if needed, and the Assessment and Working Planning Survey will be administered at the end of assessment and work planning service delivery in in each year.
Pre-interview preparation: A copy of the Center for States Tailored Interviews and P.L. 113-183 Key Informant Interview will be sent to respondents in advance of the telephone interview.
Administration: An email invitation will be sent to the appropriate participants of the Foundational Assessment Survey, Capacity Survey and Tailored Satisfaction Services Survey, CQI Workshop Follow-Up Survey, and P.L. 113-183 Key Informant Survey with a request to complete the survey (i.e., by accessing a web-link to an online version of the survey or accessing an attached survey to complete and return via email, mail or secure fax). Weekly reminder emails will be sent for two weeks to respondents who have not yet completed the surveys. For the Center for Courts Universal and Constituency Services survey, all CIP Coordinators and Directors will receive an email from the Center for Courts evaluation team asking them to please participate in the survey. They will be provided an email link to take the survey electronically. A follow-up email reminder will be sent out after one week to increase response rates. For the Webinar Registration, announcements will be made via listservs, social media, and email communication about upcoming webinars and include registration links. The CapLEARN registration is open at all times to any individual to create an account. The Center for States Tailored Services Interview will be scheduled through tailored services liaisons when needed. The Assessment and Work Planning survey will be distributed via email to complete the survey. Reminder messages will be provided, when necessary.
Alternate response methods: Respondents will be given the option to use an alternate method for the Foundational Assessment Survey and Tailored Services Interviews. For the Tailored Services Interviews if a respondent prefers to submit written responses to a survey in lieu of participating in a telephone interview, we will provide him/her with a paper version to complete via fax, email, or mail. Similarly, paper versions of any of the electronic surveys will be sent to respondents upon request or will be administered through a telephone interview if requested to accommodate any special needs.
Assurances of data confidentiality: Respondents to all surveys will be assured that reported data are aggregated and not attributable to individuals or organizational entities.
There are no incentives provided for participation in any of the surveys.
The Cross-Center evaluation instruments contained herein were subject to review and feedback by key stakeholders, including the CB, and staff from all three Centers. The Center for States and Courts’ instruments were reviewed by key stakeholders, the CB, and Center staff.
Each of the instruments was pilot tested to confirm survey item validity and to identify possible procedural or methodological challenges in need of attention or improvement. Pilot tests were conducted for each instrument using a sample of no more than nine respondents (i.e., former State and Tribal child welfare Directors, former CIP directors, former child welfare agency personnel, and current Center staff). Following the pilot tests, the instruments were refined to minimize burden and improve utility. The pilot tests were instrumental in determining the amount of time required to complete the surveys and forms and develop the burden estimates.
National Cross-Center Evaluation Contractor |
|
James Bell Associates 1001 19th Street North, Suite 1500 Arlington, VA (703) 528-3230 |
ICF International 9300 Lee Highway Fairfax, VA 22030 |
Capacity Building Center for States |
|
ICF International 9300 Lee Highway Fairfax, VA 22030 |
|
Capacity Building Center for Courts |
|
National Council of Juvenile and Family Court Judges PO Box 8970 Reno, NV 89507 (775)-784-4883 |
|
1 Strategies that pertain to two or more data collections are discussed together.
2 Strategies that pertain to two or more data collections are discussed together.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting statement Part B |
| Author | Jan Rothstein |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |