Section A Extension 2013 SUPPORTING STATEMENT1.29.13

SUPPORTING STATEMENT FOR

Cross-Site Evaluation of Children’s Bureau’s Child Welfare Technical Assistance Implementation Centers and National Child Welfare Resource Centers

Submitted by

Department of Health & Human Services

Children’s Bureau

Washington, DC

Contact person:

Brian Deakins

Children’s Bureau

Administration on Children, Youth and Families

1250 Maryland Avenue, SW

Portals Building, Floor 8150

Washington, DC 20024

202-205-8769

Brian.Deakins@acf.hhs.gov

January 10, 2013

Table of Contents

Section A: Justification 1

1. Circumstances making the collection of information necessary 1

2. Purpose and use of the information collection 1

3. Use of improved information technology and burden reduction 2

4. Efforts to identify duplication and use of similar information 2

5. Impact on small businesses or other small entities 2

6. Consequences of collecting the information less frequently 2

7. Special circumstances relating to the guidelines of 5 CFR 1320.5 3

8. Comments in response to the Federal Register Notice and efforts to consult outside the agency 3

9. Explanation of any payment or gift to respondents 3

10. Assurance of confidentiality provided to respondents 3

11. Justification for sensitive questions 3

12. Estimates of annualized burden hours and costs 3

13. Estimates of other total annual cost burden to respondents and record keepers 4

14. Annualized cost to the Federal government 4

15. Explanation for program changes or adjustments 5

16. Plans for tabulation and publication and project time schedule 8

17. Reason(s) display of OMB expiration data is inappropriate 8

18. Exceptions to Certification for Paperwork Reduction Act Submissions 8

Section B: Collections of Information Employing Statistical Methods 9

1. Respondent universe and sampling methods 9

a. Agency Results Survey 9

b. Training and Technical Assistance (T/TA) Activity Survey 10

c. Web-Based Network Survey 10

2. Procedures for the collection of information. 11

3. Methods to maximize response rates and deal with nonresponse 11

4. Tests of procedures or methods to be undertaken 13

5. Individuals consulted on statistical aspects and individuals collecting and/or analyzing data 14

Appendix A: Cross-site evaluation research questions 16

Appendix B: Stakeholder Feedback 19

Appendix C: Sample reports 22

Section A: Justification

1. Circumstances making the collection of information necessary

The Children’s Bureau (CB) in the Administration on Children, Youth and Families (ACYF), Administration for Youth and Families (ACF), U.S. Department of Health and Human Services (DHHS) seeks approval for the extension of currently approved data collection instruments and procedures with the changes described herein. The continued information collection is necessary to track, monitor, and evaluate the activities of Federally-funded training and technical assistance providers that deliver services to State, local, Tribal, and other publicly administered or publicly supported child welfare agencies and family and juvenile courts.

CB funds a network of training and technical assistance (T/TA) providers whose collective goal is to build the capacities of State, local, Tribal child welfare systems to successfully undertake practice, organizational, and systemic reforms necessary to achieve better outcomes for the children, youth and families they serve. The Child Welfare Training and Technical Assistance Network (T/TA Network) is comprised of providers that currently include 11 National Child Welfare Resource Centers (NRCs) and five regional Child Welfare Implementation Centers (ICs). CB intends for information collection to continue without interruption after the extension of OMB approval is received and to continue collecting data from T/TA providers during any subsequent funding cycle that may occur after September 2013.

Legislative Background and Purpose

Two sections of Federal legislation provide further support and justification for this data collection. The proposed information collection is necessary to support CB’s ability to satisfy the general purposes of Section 203 of Adoption Opportunities [42 USC 5113(a)] as amended. Data will be used by CB to inform administration, coordination, and planning related to departmental activities affecting adoption and foster care, and in particular, decision making about the provision and appropriateness of its technical assistance. In addition, many of the ICs and NRCs that are the focus of the cross-site evaluation are discretionary grant projects authorized by Section 105 of the Child Abuse Prevention and Treatment Act [42 USC 5106] as amended. All grants for projects funded under this section are required to be evaluated for their effectiveness (Section 105(c)).

2. Purpose and use of the information collection

The information collection described in this request will allow CB to monitor the quality of services provided by T/TA providers to child welfare agencies and courts, to track changes in utilization of services, and to assess the effectiveness of the services delivered. The data will also help CB evaluate the collaboration and operation of Network members. Findings based on the information collected will facilitate monitoring of services provided by CB-funded discretionary grant recipients, coordination of federal consultation with CB-funded T/TA to States and Tribes, and State and Tribal accessibility to services. Findings will also be used to improve service coordination and delivery, prevent duplication, appropriately allocate resources, and increase the likelihood that the child welfare agencies and courts perform better as a result of the services received.

Data collected will address several critical evaluation questions. The first set of questions pertains specifically to IC and NRC T/TA activities and their relationship to intended organizational and systems changes within State and Tribal child welfare. The second set of questions pertains to the identity, cohesion and functioning of the T/TA Network. Research questions are provided in Appendix A. There are four primary data collection instruments employed in the cross-site evaluation: the Agency Results Survey, the T/TA Activity Survey, the Web-Based Network Survey and the OneNet technical assistance tracking system.

3. Use of improved information technology and burden reduction

The OneNet technical assistance tracking system has been designed to minimize respondent reporting burden by including an efficient set of data fields, and incorporating data-entry time saving features, such as drop-down fields, check boxes, and auto-fill features. In addition the system has been structured so that only one entry is necessary when multiple T/TA providers (ICs and/or NRCs) are working together to provide T/TA. Users have 24-hour access to OneNet via secure internet connection. After using OneNet for more than 2 ½ years, users are more experienced and efficient when entering data compared to when the OneNet forms were initially approved for data collection. In addition, administration of the T/TA Activity Survey and the Web-Based Network Survey is primarily electronic, utilizing email notification and Internet-based survey technologies. Using email address information from OneNet and from the T/TA Network directory, the national cross-site evaluators send an email notification to respondents and invite them to complete the appropriate survey instrument by accessing a web-link to an online version of the survey. The majority of respondents that completed prior surveys had the capabilities to access the web-link to the surveys.¹

4. Efforts to identify duplication and use of similar information

The proposed instruments are intended to uniformly collect data that allows CB to evaluate the collective efforts and effectiveness of the NRCs and ICs on child welfare systems. In addition to the proposed cross-site information collection, each NRC and IC also designed and is conducting its own center-specific evaluation activities to address its unique legislative and programmatic priorities. CB has required its cross-site and center-specific evaluators to ensure data collection is necessary and complementary. The information collection and the center-specific evaluation activities of the NRCs and ICs have been coordinated to avoid potential duplication of data collection and reduce burden to respondents. The evaluators of each IC and NRC have met with the national cross-site evaluator and have reviewed each of the cross-site data collection instruments. The instruments have been revised to address potential overlap and the timing of data collection activities is being closely coordinated to minimize burden. As findings become available through the national cross-site evaluation, the evaluator shares the findings with the ICs and NRCs. While center-specific data will yield important and relevant information that will assist each NRC and IC to tailor and improve its services, center-specific data will not be sufficient to meet the cross-site purposes for the proposed information collection. There is no similar information available that addresses the coordination of T/TA activities and the functioning of the T/TA Network.

5. Impact on small businesses or other small entities

The full range of information will continue to be requested of all respondents. Organizational size will not affect the relevance of particular questions. A number of efforts are in place to minimize respondent burden, regardless of organizational size, for each of the data collection strategies described herein. Skip patterns have been included in the survey instruments based on the types of services received, and the timing of data collection activities is being coordinated to minimize respondent burden. Information being requested has been held to the absolute minimum and is sufficient enough to respond to the intended evaluation questions.

6. Consequences of collecting the information less frequently

In order to improve T/TA Network services and collaborate effectively to provide coordinated support to State and Tribal child welfare systems, CB and its providers need timely data on the provision of T/TA, the accessibility of services, the perceived effect and quality of the services received, and the interactions of service providers with one another. Less frequent data collection would inhibit the timely use of the information by the CB and providers to improve T/TA coordination and service quality.

7. Special circumstances relating to the guidelines of 5 CFR 1320.5

There are no special circumstances associated with this data collection.

8. Comments in response to the Federal Register Notice and efforts to consult outside the agency

Following publication of the notice that appeared in the Federal Register, Volume 77 Number 200, Tuesday, October 16, 2012, pages 63317-63318, no requests were received from the general public for copies of the proposed information collection instruments.

The direct stakeholders already have a close familiarity with the proposed instruments because they have been participating in data collection activities that utilize these instruments since 2010. Further, stakeholders originally contributed to the development of these instruments throughout the design phase of this study. Appendix B identifies individuals who originally reviewed and offered written comments on the data collection approach, survey instruments (including scales and open-ended questions), and required elements and data entry forms for OneNet as well as persons who reviewed subsequent additions and deletions of fields to the OneNet forms. The evaluation team has made an effort to be responsive to stakeholders’ comments whenever possible and has used their feedback in revising the data collection instruments.

The Agency Results Survey, the T/TA Activity Survey, and the Web-Based Network Survey were pilot-tested with individuals who were knowledgeable of the topics addressed and who had served in positions similar to the potential respondents (i.e, State/Tribal Child Welfare Directors, State technical assistance liaisons, current and former T/TA Network members). Since OMB’s approval of these instruments in 2010, two waves of data have been collected using the Agency Results Survey and four waves of data have been collected using the T/TA Activity Survey. In addition, two waves of data using the Web-based Network Survey have been collected. OneNet data collection forms were pilot-tested prior to their implementation with individuals that were experienced with T/TA service provision and who would be future users of the OneNet system. All ICs and NRCs have been reporting information using the OneNet instruments since July 2010. Further information on the pilot tests is noted in Section B.

9. Explanation of any payment or gift to respondents

No payments or gifts are provided to respondents for completing this information request.

10. Assurance of confidentiality provided to respondents

No assurances of complete confidentiality are provided to respondents. However, all respondents are informed of the importance of maintaining their privacy and that reported data are aggregated; they are not attributed to individuals.

11. Justification for sensitive questions

There are no questions or requirements of a sensitive nature contained in the survey instruments and data collection forms described herein.

12. Estimates of annualized burden hours and costs

Having applied hourly wage estimates to burden hours in each respondent category, the current annual cost to the respondents is as follows: (1) $5,707.62 for the Agency Results Survey; (2) $5,413.20 for the T/TA Activity Survey; (3) $194.66 for the Web-Based Network Survey; and (4) $85,774 for OneNet. The total annual cost to the respondents if all data collection instruments were employed in the same given year is $97,089.48.

Exhibit A-12. Annual burden estimates

Instrument	Number of respondents	Number of responses per respondent	Average burden hours per response	Total burden hours
Agency Results Survey	74 State/Tribal Child Welfare Directors	1	1	74.00
Training and Technical Assistance (T/TA) Activity Survey	160 State/Tribal T/TA Recipients	3	0.25	120.00
Web-Based Network Survey	15 T/TA Network members	1	0.25	3.75
OneNet Forms
Implementation Project Application Form	5	1.7	0.40	3.4
Implementation Project Assessment and Work Plan Form	5	4.6	0.28	6.44
Implementation Project T/TA Activity Form	5	600	0.30	900
Implementation Project Monthly Report Form	5	36	0.17	30.60
T/TA Request Form	13	12.31	0.40	64.00
T/TA Assessment and Work Plan Form	13	6.2	0.28	22.568
T/TA Activity Form	12	160	0.30	576.00
General T/TA Event Form	17	11.8	0.25	50.00
Total				1850.76

13. Estimates of other total annual cost burden to respondents and record keepers

No additional cost burden will apply for respondents or record keepers.

14. Annualized cost to the Federal government

The associated costs for administering the surveys and developing and maintaining OneNet are outlined in Table A-13 below.  The annual cost to the Federal government to administer the Agency Results Survey is $8,838.93; the annual cost to administer the T/TA Activity Survey is $2,287.32; and the annual cost to administer the Web-Based Network Survey is $255.56.  The total annualized cost to provide ongoing quality assurance for OneNet data, system enhancements (e.g., reporting and query features), training for new users, and maintaining the system (including labor and server costs) is $103,200.

Exhibit A-13. Annualized Costs for Survey and OneNet Administration

Instrument	Administration Activities	Staff Time	Total Cost
Agency Results Survey	74 State/Tribal CW Director Interviews	1.5	$8,838.93
T/TA Activity Survey	Sample TA events Distribute electronic survey & follow up	10.0 8.0	$1776.20 $511.12
Web-Based Network Survey	Distribute electronic survey & follow up	2.0	$255.56
OneNet	Maintenance, enhancements, and quality assurance	125.0	$103,200.00
Total			$114,581.81

15. Explanation for program changes or adjustments

Estimated burden hours have decreased by 284 hours from the original submission (the estimated total annual burden hours were 2,135.12).

Agency Results Survey. Program changes were made to the Agency Results Survey that are not anticipated to affect respondent time burden or cost. Questions that were determined by the Children’s Bureau to no longer be relevant were removed from the survey while others were enhanced to provide greater clarity. Several questions were reworded to better align with stage of the service delivery being experienced by respondents. Additional questions were added to the Agency Results Survey to capture information about perception of effectiveness and the sustainability of changes attributed to providers’ T/TA.

Agency Results Survey	Question #
Questions removed from survey	B3, E1, E3, E4, H1, H2
Questions added to survey	C1-C5, D2-D4, F8, H1, H4, H5
Questions reworded and/or enhanced for clarity (previous item # in previous instrument new item in current instrument)	B5  B3, C1C4, E5E2, F4F4, F6F6, F7F7, H3H3, H4H6

T/TA Activity Survey. No program changes or adjustments apply to the T/TA Activity Survey. Estimated respondent time and cost burdens remain the same. The survey has been reformatted for electronic administration but the items have not changed. Three attachments are included regarding this survey. The first (2a) provides introductory information regarding the T/TA Activity Survey, and the other two files (2b and 2c) are each versions of the same instrument which account for skip patterns based on the type of service received by the respondent.

Web-based Network Survey. Minor program changes to the administration of the Web-based Network Survey have resulted in reductions to the estimated respondent time and cost burdens. Due to changes in T/TA Network roles and coordination, the Children’s Bureau’s has decided to focus administration of the survey on a cohort of 15 respondents instead of all 30 Network member organizations originally surveyed. Web-based Network Survey questions have been refined so that respondents address only how they work with members of the smaller cohort, and some questions that are no longer applicable have been removed based on this narrowing of respondents.

Web-based Network Survey	Changes
Questions removed from survey	Section C – fifteen questions removed Section D – one question removed
Questions added to survey	none
Rewording of questions (previous language  new language)	“Network members”  “ICs and NRCs”

OneNet Forms. CB continues to attempt to make the OneNet data system as responsive to users as possible. Instruction and data fields in the automated system appear as a sequence of screens for the user (herein described as “forms”). The OneNet forms in this submission have been enhanced to meet these needs. A combination of program changes and adjustments have resulted in an overall reduction in the anticipated respondent and cost burden for respondents entering data into OneNet forms.

Several of the titles of OneNet forms have changed since approval of the initial information collection package to provide greater clarity for OneNet users.

OneNet Form title in previously approved collection	OneNet Form title in this package	Notes
	OneNet Form: T/TA Request	New form
OneNet Form: NRC T/TA Work Plan	OneNet Form: T/TA Assessment and Work Plan	Renamed
OneNet Form: NRC T/TA Close-Out		No longer a separate form - consolidated into OneNet Form: T/TA Assessment and Work Plan
OneNet Form: NRC T/TA Activity	OneNet Form: T/TA Activity	Renamed
	OneNet Form: Implementation Project Application	New form
OneNet Form: Implementation Project Description	OneNet Form: Implementation Project Assessment and Work Plan
OneNet Form: Implementation Project T/TA Activity	OneNet Form: Implementation Project T/TA Activity	No change
OneNet Form: Implementation Project Monthly Report	OneNet Form: Implementation Project Monthly Report	No change
OneNet Form: T/TA General Event	OneNet Form: T/TA General Event	No change

Adjustments due to increased system familiarity and data entry efficiency, changes in provider workload and capacity to solicit and initiate new T/TA projects, and more accurate estimates of service delivery have each contributed to reductions in time burden. This is most evident in the reduced burden estimates for the Implementation Project Assessment and Work Plan Form (formerly named the Implementation Project Information Form), Implementation Project Monthly Report Form, T/TA Assessment and Work Plan Form (formerly named the NRC T/TA Work Plan Form), T/TA Activity Form (formerly named the NRC T/TA Activity Form), and General T/TA Event Form.

An adjustment due to higher volumes of service delivery than originally anticipated has resulted in a greater anticipated respondent burden estimate for users of the Implementation Project T/TA Activity Form.

Program changes, including the agreement by T/TA providers that use the OneNet system to establish standard protocols for soliciting and processing requests for services has resulted in the inclusion of the Implementation Project Application Form and T/TA Request Form which have added burden.

Similarly, program changes have been made to several OneNet forms (including the deletion, modification, consolidation, and addition of fields) at the request of respondents to better align with recently agreed upon processes. Forms now better accommodate the assessment of service delivery needs, tracking of federal approvals for resource allocation, documentation of service delivery milestones, and recording of who completed the data entry. In addition, program changes have been made in response to respondents’ requests for the OneNet forms to capture additional types of T/TA services that had not been previously reported. Providers requested that OneNet more comprehensively capture information about both direct and indirect service provision and that the data better reflect the level of effort necessary to perform T/TA with jurisdictions. While some fields have been added based on these requests, increases in respondent burden have been largely offset by improvements in respondents’ data entry efficiency. Consolidation of the NRC T/TA Close Out Form (approved previously) into the T/TA Assessment and Work Plan Form has resulted in the elimination of one “instrument” from the previous information collection approval package but no change in burden.

The fields added to and fields omitted from OneNet forms are summarized below.

OneNet Form	Fields added	Fields removed
OneNet Form: IP assessment and work plan	 Are activities being carried out a as a part of authorized assessment  Information entered by  Work plan tab entered by  Is the work plan complete  Date of completion  Approval status of work plan  Name of the approver  Approval date  Approval entered by  Closeout entered by	 Abstract  IP Logic Model  Work Plan
OneNet Form: IP Activity form	 Activity entered by  Does the T/TA activity involves only indirect TA  Hours of direct TA effort  Hours of indirect TA effort  Type(s) of indirect TA  Description of indirect TA  Activity tab entered by
OneNet Form: T/TA Assessment and work plan form	 Are activities being carried out a as a part of authorized assessment  Will this Work Plan only involve indirect TA?  Work Plan entered by  Work plan tab entered by  Is the work plan complete  Date of completion  Approval status of work plan  Name of the approver  Approval date  Approval entered by  Closeout entered by	 T/TA Matrix  If estimate exceeds 10 onsite days, was FPO approval obtained?  T/A Plan
OneNet Form: T/TA Activity Form	 Activity entered by  Does the T/TA activity involves only indirect TA  Hours of direct TA effort  Hours of indirect TA effort  Type(s) of indirect TA  Description of indirect TA  Activity tab entered by

16. Plans for tabulation and publication and project time schedule

Tabulation: Frequency distributions will be calculated to generate summaries of survey items and OneNet outputs, as well as to examine variability in the data. Parameter estimates, such as variances and means, will be established for each quantitative item. Cross-tabulations and significance tests will be conducted as appropriate. Content analysis will be conducted on open-ended survey items, and will entail systematic coding, creation of a hierarchy of codes, and cross-case and cross-source thematic analysis. Sample pages reports for the surveys are provided in Appendix C. Analyses will be conducted to determine subgroup variation. Multivariate analyses and group comparisons will examine the cumulative results of T/TA. Mixed categorical and interval variables will represent a range of measurements (i.e., characteristics of T/TA models; types and levels of intensity of T/TA support delivered to States and Tribes; and associated child welfare practice-, organizational- and system-level changes). An evaluation goal is to develop predictive models that identify factors associated with stronger and weaker T/TA results among States and Tribes.

Publication: To date, there have been five reports and six presentations produced under the national cross-site evaluation based on data collected using the proposed instruments (under the previous OMB approval). These reports and presentations have been used for internal purposes and have not been shared outside of the CB’s T/TA Network. The findings from the annual information collections will continue to be summarized and tabulated in annual briefings and reports to the CB. Reports of research findings will continue to include descriptive analyses, identification of moderating and mediating variables, and the implications of the findings. In the last stage of analysis at the end of 2013, data will be merged from multiple sources to enable final summative analyses to address major questions on the cumulative, overall results of IC and NRC T/TA. A final synthesis report of the project’s findings for all years will be submitted to the CB at the end of FY 2013 for dissemination to federal, state and tribal stakeholders.

Project Timetable: The Agency Results Survey and the Web-Based Network Survey will be administered once per year during the approved time period. The T/TA Activity Survey will be administered twice per year during the approved period. T/TA provider staff will record information about their service delivery into OneNet in accordance with timeframes established by the CB and articulated in the business process for T/TA. Data entry will be ongoing throughout the providers’ respective project periods.

17. Reason(s) display of OMB expiration data is inappropriate

The OMB expiration date for the information collection will appear on the instruments.

17. Exceptions to Certification for Paperwork Reduction Act Submissions

No exception is requested to the certification statement identified in Item 19, “Certification for Paperwork Reduction Act Submissions,” of OMB Form 83-I.

1 Situations in which problems were encountered during prior waves of data collection include surveys being automatically returned to the evaluation team because of incorrect e-mail addresses; child welfare agency firewall settings that rejected the e-mail from the evaluation team, and mailboxes that were filled and no longer receiving new e-mail. In these situations, evaluation team members established personal contact with potential respondents to ensure they received a copy of the survey(s) for completion.