CAP-B SUPPORTING STATEMENT Part II Sec A JR

SUPPORTING STATEMENT FOR

Evaluation of the Child Welfare Capacity Building Collaborative: Part Two

Section A

Submitted by

Department of Health & Human Services

Children’s Bureau

Washington, DC

Contact person:

Brian Deakins

Children’s Bureau

Administration on Children, Youth and Families

Mary E. Switzer Building

3rd Floor, Mailstop 3602

330 C Street, SW

Washington, DC 20201

202-205-8769

Brian.Deakins@acf.hhs.gov

Section A: Justification

1. Circumstances making the collection of information necessary

The Evaluation of the Child Welfare Capacity Building Collaborative: Part Two is sponsored by the Children’s Bureau (CB) in the Administration on Children, Youth and Families (ACYF), Administration for Youth and Families (ACF), U.S. Department of Health and Human Services (DHHS) which seeks approval for the data collection instruments and procedures described herein. The proposed information collection is necessary to facilitate, track, monitor, and evaluate the activities of the Capacity Building Collaborative which includes three Federally funded centers (Center for States, Center for Tribes, and Center for Courts) that deliver national child welfare expertise and evidence-informed training and technical assistance services to State, Tribal, and Territorial public child welfare agencies and Court Improvement Programs (CIPs).

The CB-funded Centers’ collective goal is to build the capacities of State, local, Tribal child welfare systems to successfully undertake practice, organizational, and systemic reforms necessary to implement federal policies, meet federal standards, and achieve better outcomes for the children, youth and families they serve.

This new data collection is the second part of a larger data collection effort being conducted for the evaluation of the Child Welfare Capacity Building Collaborative. The first group of instruments has already been submitted for this evaluation. This supporting statement details the second group of instruments that will be used for data collection as part of this evaluation.

Legislative Background and Purpose

Agencies that receive formula funding through the Child Abuse Prevention and Treatment Act (CAPTA) and titles IV-B and IV-E of the Social Security Act are eligible for technical assistance from the Children’s Bureau to support implementation of these programs, compliance with federal requirements, and improvement of outcomes. The proposed information collection is necessary to perform routine evaluation of quality and effectiveness and to inform future planning and decision making about the provision and improvement of technical assistance services authorized under multiple sections of CAPTA and title IV of the Social Security Act. This information collection also complies with the statutory requirement for training projects authorized by Section 5106 of CAPTA to be evaluated for their effectiveness.

2. Purpose and use of the information collection

In 2014, the CB restructured its technical assistance delivery structure and invested in new approaches to service delivery. The Centers’ services have been organized into three major categories: (1) product development and information dissemination, including the creation and release of website content, publications, and other resources; (2) training and peer networking, including the delivery of online courses or “learning experiences,” virtual presentations, and facilitated peer discussions; and (3) jurisdiction-specific consultation and coaching, including workshops and onsite visits to States and tribes to provide customized support. Each service category has been designed to achieve specific outcomes that require different levels of engagement and interaction between the Center and its targeted service recipients.

Data collected through this proposed information collection will be used by the Centers and the CB to improve the development and design of services in each category and evaluation findings will be shared with other providers and service recipients to increase knowledge about technical assistance strategies and approaches. Evaluation findings will also inform future decision making about the service delivery structure and federal resource allocation.

Consistent with this approach, the CB recently released findings from its prior evaluation of 15 training and technical assistance providers to the public. Findings from the final report have been summarized in an executive summary and several briefs to make key information accessible to technical assistance providers, evaluators, and consumers of services. The prior evaluation design has been shared with other Federal agencies and departments that fund training and technical assistance systems and results have been presented at national conferences. The CB used data from the prior information collection and findings from its analysis as a basis for the changes to its current service delivery system.

The Centers’ services will be evaluated by both Center-specific evaluations and a Cross-Center Evaluation. The Center-specific evaluations are designed to collect data on Center-specific processes and outcomes. The Cross-Center Evaluation is designed to respond to a set of cross-cutting evaluation questions posed by the CB. Data collected will address several critical evaluation questions. Research questions for the Cross-Center Evaluation and Center-specific evaluations are provided in Appendix A.

Proposed Cross-Center Evaluation data sources for this effort include (1) a capacity survey to capture perceived changes in organizational capacity after receiving Center services; (2) a tailored services satisfaction survey administered in conjunction with the capacity survey to capture satisfaction with tailored services; (3) a foundational assessment to capture contextual data regarding the organizational health and functioning of child welfare agencies and courts; (4) a follow-up survey that will examine short-term and intermediate outcomes among CIPs that receive different levels of tailored services following continuous quality improvement (CQI) workshops; and (5) a key informant survey and interview to examine how capacity building services are incorporated into State and tribal activities to support implementation of P.L.113-183, the Preventing Sex Trafficking and Strengthening Families Act. Center-specific data sources for this effort include (1) registration forms for webinar registration and for CapLEARN, a learning management system; and (2) service-specific feedback forms and interviews, such as the Center for States Tailored Services interviews and the Center for Courts Universal and Constituency Services survey. This OMB package represents the second of two submissions for this project. These instruments represent the data collection that is necessary in the second phase of the project.

Personally Identifiable Information

The Capacity Building Collaborative Centers will collect personally identifiable information (PII) on several instruments: the Center for States’ CapLEARN registration form, Webinar Registration form and the Cross-Center Capacity Survey and Tailored Services Satisfaction Survey, Foundational Assessment Survey, P.L.113-183 Key Informant Survey and Interview, and the CQI Workshop Follow-Up Survey. Survey recipients are informed that participation in surveys is voluntary.

The CapLEARN registration form and Webinar registration form will include the following data elements: name of contact, job title, organization, and email address. This information will be used to create an account for CapLEARN that allows users to track their own learning progress, provide information regarding webinar access, schedule meetings and coordinate service delivery opportunities, and allow for understanding of whether service delivery is reaching the intended audience in an aggregate. A PIA is underway at the time of this submission for the items in the CapLEARN registration.

The Webinar Registration Data is held in third party software Adobe Connect. Adobe Connect data security protocols include all data traffic being SSL enabled (HTTPS by default; RTMPS is optional; both 128-bit and 256-bit are supported), isolated and restricted to respective users. The data is collected by the Adobe Connect server and remains in the data center where the account was provisioned until it is downloaded, moved, or deleted. Adobe Connect is TRUSTe and Joint Interoperability Test Command (JITC) certified. The Adobe Connect hosted deployment is also Service Organization Controls 2 (SOC 2) compliant. SOC 2 replaces SAS 70; portions of a SOC 2 Report can address some aspects of Federal Information Security Management Act (FISMA), ISO27001, and Health Insurance Portability and Accountability Act (HIPAA) certifications. Adobe Connect Managed Services (ACMS) is fully HIPAA compliant. Full information for Adobe Connect data security can be found here: https://wwwimages2.adobe.com/content/dam/Adobe/en/products/adobeconnect/pdfs/security/Adobe-Connect-hosted-security.pdf

The Cross-Center Capacity Survey and Tailored Services Satisfaction Survey, Foundational Assessment Survey, the P.L. 113-183 Key Informant Survey and Interview, and the CQI Workshop Follow-Up Survey will include the following data elements: jurisdiction (name of State/Territory/Tribe), name of contact, job title, and email address. This information will be collected for survey administration purposes, to compare responses from the same contacts over time, to provide aggregate-level descriptions of survey respondents, to link responses on the Capacity Survey and Tailored Services Satisfaction Survey to service delivery data saved in CapTRACK (the Capacity Building Collaborative’s service delivery database), and to allow for analyses of data by role and jurisdiction.

All PII collected by the three Centers and Cross-Center Evaluation team are confidential and only select data such as jurisdiction and professional role will be shared with evaluators from the respective Center. Only the Cross-Center evaluation team will have access to identifiers such as contact name and email address for those surveys administered by the Cross-Center evaluation team, and those items will be removed and replaced with identification numbers. The Cross-Center Evaluation team and Center evaluators will store all PII contact data in separate files on their respective servers in password protected, secure data systems in order to ensure privacy. Data collected on Cross-Center Evaluation instruments will be coded using identification numbers, and links between identification numbers and names will be stored in password protected secure files. Identifiers will not be used in any evaluation reporting.¹

3. Use of improved information technology and burden reduction

Wherever possible and appropriate, information technology will be used to capture information and reduce burden relative to alternative methods of data collection. Administration of the majority of evaluation surveys will be web-based, utilizing email notification and Internet-based survey technologies creating efficiencies for survey administrators, allowing flexibility and convenience for participants, and ideally resulting in a user-friendly experience for respondents. Based on the services provided, survey respondents will receive an email notification inviting them to complete the appropriate survey instrument by accessing a web-link to an online version of the survey.

Nearly all of the targeted respondents are expected to be able to access the web-link to the surveys.² The majority of questions in most surveys are closed-ended response items that can be completed quickly (within 10 – 18 minutes), allowing descriptive and comparative analyses. The Center for Tribes may facilitate the completion of the Foundational Assessment Surveys by distributing hard copies and completing them in small groups together, subsequently entering the data online. CapLEARN Registration form and the Center for States Webinar Registration forms are fully web accessible and can be completed in 5 minutes or less.

4. Efforts to identify duplication and use of similar information

The proposed instruments are intended to uniformly collect data that will allow for the evaluation of Center-specific processes and outcomes and to answer a set of cross-cutting evaluation questions posed by the CB. CB has required its Cross-Center and Center-specific evaluators to ensure data collection is necessary and complementary. The information collection and the Center-specific evaluation activities have been coordinated to avoid potential duplication of data collection and reduce burden to respondents. Each of the three Centers have met with the Cross-Center evaluation team and reviewed each of the Cross-Center data collection instruments. The instruments have been revised to address potential overlap and the timing of data collection activities is being closely coordinated to minimize burden. While Center-specific data will yield important and relevant information it will not be sufficient to meet the Cross-Center purposes for the proposed information collection.

5. Impact on small businesses or other small entities

The full range of information will be requested of all respondents. The size of the organization will not affect the relevance of particular questions. A number of efforts are in place to minimize respondent burden, regardless of organizational size, for each of the data collection strategies described herein. Skip patterns have been included in the survey instruments based on the types of services received, and the timing of data collection activities is being coordinated to minimize respondent burden. Information being requested has been held to the minimum necessary to respond to the intended evaluation questions.

6. Consequences of collecting the information less frequently

In order to improve the Centers’ services and collaborate effectively to provide coordinated support to State, Tribal, and Territorial public child welfare agencies and Court Improvement Programs, CB and its providers need timely data on the provision of services delivered by the Centers, the accessibility of services, the perceived effect and quality of the services received, and the interactions of service providers with one another. Less frequent data collection would inhibit the timely use of the information by CB and providers to improve service coordination and service quality and to potentially make decisions about service delivery.

7. Special circumstances relating to the guidelines of 5 CFR 1320.5

There are no special circumstances associated with this data collection.

8. Comments in response to the Federal Register Notice and efforts to consult outside the agency

Following publication of the notice that appeared in the Federal Register, Volume 81, Number 111, Thursday, June 9, 2016, page 37196-37198, no requests were received from the general public for copies of the proposed information collection instruments.

However, numerous opportunities were provided for direct stakeholders to review the proposed instruments and to contribute to their development throughout the design phase of this study. The Cross-Center and Center evaluation teams have made an effort to be responsive to stakeholders’ comments whenever possible and have used their feedback in revising the data collection instruments. Instruments were pilot-tested with individuals who were knowledgeable of the topics addressed and who had served in positions similar to the potential respondents (i.e., State/Tribal Child Welfare Directors, CIP Directors, State technical assistance liaisons, current and former Center staff members and consultants). Following stakeholder review and pilot testing, revisions were made to instruments based on comments to improve clarity of instructions and items for each Center. None of these revisions have added to the burden of completing the instruments and forms.

9. Explanation of any payment or gift to respondents

No payments or gifts are provided to respondents for completing this information request.

10. Assurance of confidentiality provided to respondents

No assurances of complete confidentiality are provided to respondents. However, all respondents are informed of the importance of maintaining their privacy and that reported data are aggregated; they are not attributed to individuals.

11. Justification for sensitive questions

No questions of a sensitive nature are included in this evaluation.

12. Estimates of annualized burden hours and costs

Having applied hourly wage estimates to burden hours in each respondent category, the current annual cost to the respondents is as follows: (1) $3,020.09 for the Capacity Survey; (2) $835.56 for the Tailored Services Satisfaction Survey; (3) $603.58 for the Foundational Assessment Survey (4) $2,124.53 for the Center for States Survey and Interview; (5) $251.02 for the CQI Workshop Follow-Up Survey; (6) $929.13 for the Center for Courts Universal and Constituency Services Survey; (7) $403.55 for the P.L. 113-183 Key Informant Survey and Interviews; and (8) $4,137.92 for Webinar Registration and CapLearn Registration. ³ The total annual cost to the respondents if all data collection instruments were employed in the same given year is $12,305.38.

Exhibit A-1. Annual Burden Estimates

Instrument	Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Total Annual Burden Hours
Capacity Survey – States, Courts, and Tribes	462	1	.3	138.60
Tailored Services Satisfaction Survey	462	1	.083	38.35
Foundational Assessment Survey – States, Courts, and Tribes	277	1	.1	27.7
CQI Workshop Follow-Up Survey	48	2	,12	11.52
P.L. 113-183 Key Informant Survey	52	1	.26	13.52
P.L. 113-183 Key Informant Interview4	5	1	1	5
Center for Courts: Universal and Constituency Services	104	1	.41	42.64
Webinar Registration	4650	1	.03	139.5
Center for States: Tailored Services Interview	60	1	1	60
Center for States: Assessment and Work Planning Survey	150	1	.25	37.5
CapLEARN Registration	600	1	.084	50.4
Total				564.73

13. Estimates of other total annual cost burden to respondents and record keepers

No additional cost burden will apply for respondents or record keepers.

14. Annualized cost to the Federal government

The associated costs for administering the surveys are outlined in Table A-2 below.  The annual cost to the Federal government for administration is (1) $12,190.46 for the Capacity Survey; (2) $6,279.94 for the Tailored Services Satisfaction Survey; (3) $10,389.60 for the Foundational Assessment Survey (4) $20,779.20 for the Center for States Survey and Interview; (5) $11,544.00 for the CQI Workshop Follow-Up Survey; (6) $230.88 for the Center for Courts Universal and Constituency Services Survey; (7) $1,587.30 for the P.L. 113-183 Key Informant Survey and Interviews; and (8) $10,216.44 for Webinar Registration and CapLearn Registration.⁵ The total annual cost to the federal government if all data collection instruments were employed in the same given year is $73,217.82.

Exhibit A-2. Annualized Costs for Survey Administration

Instrument	Administration Activities	Staff Time	Total Cost
Capacity Survey	Distribute electronic survey & follow up	211.2	$12,190.46
Tailored Services Satisfaction Survey	Distribute electronic survey & follow up	108.8	$6,279.94
Foundational Assessment Survey	Distribute electronic survey & follow up	180	$10,389.60
CQI Workshop Follow-Up Survey	Distribute electronic survey & follow up	200	$11,544.00
P.L. 113-183 Key Informant Survey	Distribute electronic survey & follow up	5	$288.60
P.L. 113-183 Key Informant Interview	Conduct interview & follow up	22.5	$1,298.70
Center for Courts: Universal and Constituency Services	Distribute electronic survey & follow up	4	$230.88
Webinar Registration	Online registration	57	$3,290.04
Center for States: Tailored Services Interview	Conduct interview	240	$13,852.80
Center for States: Assessment and Work Planning Survey	Distribute electronic survey & follow up	120	$6,926.40
CapLEARN Registration	Online registration	120	$6,926.40
Total			$73,217.82

15. Explanation for program changes or adjustments

Not applicable.

16. Plans for tabulation and publication and project time schedule

Tabulation: Frequency distributions will be calculated to generate summaries of survey items, as well as to examine variability in the data. Parameter estimates, such as variances and means, will be established for each quantitative item. Cross-tabulations and significance tests will be conducted as appropriate. Qualitative analysis will be conducted on open-ended survey items and interview transcripts, and will entail systematic coding, creation of a hierarchy of codes, and cross-case and cross-source thematic analysis. Analyses will be conducted to determine subgroup variation.

Publication: The findings from the annual information collections will be summarized and tabulated in a series of annual briefings and reports to the CB beginning in the first year of data collection in FY 2016. For Center for States, findings specific to Center for States service delivery will be reported on an ongoing basis to Center staff to inform planning and service delivery. Reports of research findings will include descriptive analyses, and the implications of the findings. For Center for Courts, findings specific to Center for Courts service delivery will be reported on an ongoing basis to Center staff to inform planning and service delivery. Reports of research findings will include descriptive analyses, and the implications of the findings. In the last stage of analysis beginning in 2018, data will be merged from multiple sources to enable final summative analyses to address major questions on the cumulative, overall results of the three Centers. A final synthesis report of the project’s findings for all years will be submitted to the CB in FY 2018 for dissemination to Federal, state and tribal stakeholders.

Project Timetable: Pending approval, the Capacity Survey and Tailored Services Satisfaction Survey will be administered together beginning in January 2017 and will be administered by the Cross-Center Evaluation team on an ongoing basis as services are delivered to child welfare agencies throughout the project period. The Foundational Assessment survey will be administered by both the Cross-Center Evaluation team and individual Centers in alignment with their assessment process, prior to the work plan, as often as a new assessment is conducted, but no more frequently than annually. The CQI Workshop Follow-Up Survey will be administered by the Cross-Center Evaluation team at 6 and 12 months after the CQI Workshop. The second administration will be conducted as part of the Capacity Survey. The P.L. 113-183 Key Informant Survey and Interview will be administered three times: once in early 2017, once in end of 2017, and once in end 2018. The Center for States Webinar Registration will be utilized for registration prior to every webinar and the CapLEARN Registration form is required once to create an account to access the Collaborative resource library, including on-line trainings. The Center for States Tailored Services Interviews will be administered once for a sample of “tailored” services engagements during the year that the project is underway. The Intensive Projects Survey will be administered once, approximately 12 months into the project. If the intensive project continues for subsequent years, this survey will be administered again using the same time frame. The Center for States Assessment and Work Planning Survey will be administered to half of the States at the completion of the assessment process during each Fiscal Year. The Cross-Center team will administer the Assessment and Capacity Building Plan Satisfaction Survey (submitted in the first OMB package for this project) to the other half of States at the completion of the assessment process during each Fiscal Year. The Center for Courts Universal and Constituency Services Survey will be administered annually to CIP Coordinators and Directors. The timing of the survey will occur approximately 6 months after CIP self-assessment reports are submitted.

17. Reason(s) display of OMB expiration data is inappropriate

The OMB expiration date for the information collection will appear on the instruments.

17. Exceptions to Certification for Paperwork Reduction Act Submissions

No exception is requested.

1 JBA’s IT staff will dispose of PII data using a commercial product that meets the U.S. Department of Defense’s 5220.22-M (ECE) standards for sanitizing media. The timing of the disposal of the PII data will be in accordance with the guidelines outlined in the contract with CB.

2 A hard copy of the surveys will be provided to those who cannot access the surveys online.

3 The annual respondent burden and annualized cost varies by year and depends upon the data collection strategies employed.

4 The P.L. 113-183 Key Informant Interview asks the same questions that are on the P.L. 113-183 Key Informant Survey, just in an interview format suitable for any Tribes that agree to participate.

5 The annual respondent burden and annualized cost varies by year and depends upon the data collection strategies employed.

9