Part A-Cross Site Evaluation for 2012 - THE SUPPORTING STATEMENT

THE SUPPORTING STATEMENT

Cross-site Evaluation Survey 2012 Family Connection Grantees

A. Justification. Requests for approval shall:

1. Circumstances Making the Collection of Information Necessary

The legislative authority for the Family Connection Grant Program (See Attachment A – Legislative Authority) requires the conduct of a rigorous evaluation of the activities funded with grants under this authority. The Children’s Bureau has contracted with James Bell Associates, Inc., (heretofore known as “the contractor”) to conduct the Cross-site Evaluation for 2012-funded Family Connection Discretionary Grants. (See Attachment B – Contract Statement of Work Excerpt).

The cross-site evaluation examines the effectiveness of 17, three-year grants awarded in September 2012 which support demonstration projects to test the effectiveness of Kinship Navigator-Child Welfare/TANF, combination Family-finding/Family Group Decision-Making, and comprehensive Residential Family Treatment projects. (See Attachment C – Description of Cross-Site Evaluation). Attachment C – Description of Cross-Site Evaluation has been enhanced with additional information that describes each grantee cluster.

In the interest of providing as complete an evaluation report as possible by the end of FY15, when the current contract for the cross-site evaluation ends, the Children’s Bureau has directed the contractor conducting the Cross-site Evaluation to adopt the most efficient means possible to collect process evaluation data from grantees.

A web-based electronic survey of a cross-section of grantee and project partner representatives will contribute critical data to the process evaluation, enabling the Children’s Bureau to more thoroughly understand the stages of project maturation from design and implementation to maintenance and sustainability. A web-based survey is the most efficient and timely means possible to collect process evaluation from grantees. Use of this survey will enable the contractor to have adequate time to prepare and submit a full and final report to the Children’s Bureau by the end of FY15.

Given the proximity to the end of FY15, the Children’s Bureau is exploring the possibility of a no-cost extension so that the contractor may analyze quantitative and qualitative data. Should a no-cost extension be unavailable, the Children’s Bureau and the contractor will prioritize analysis and reporting needs in order to best meet contract requirements and produce a deliverable useful to the Children’s Bureau, grantees, and other audiences. Potential new timelines have been developed based on OMB approval prior to July 1, 2015 and a three-month, no-cost extension.

2. Purpose and Use of the Information Collection

The proposed electronic survey will be administered by the contractor. It will enable the Children’s Bureau to collect key information on project design, implementation, maintenance, and sustainability from key grantee representatives in a minimally burdensome way. The quantitative nature of the surveys will enable rapid data analysis and reporting.

The Children’s Bureau will benefit from the timely receipt of new knowledge developed by grant projects and produced from analysis of survey results. The new knowledge will be disseminated to the field in a timely manner and will inform technical assistance activities planned by the Bureau for grantees as well as Bureau recommendations for future research and grant funding.

3. Use of Improved Information Technology and Burden Reduction

Surveys will be one source of information to be used in the cross-site evaluation to address several process evaluation questions. Survey questions will be targeted to address information not provided via already-existing sources.

The majority of response options to each survey question will be quantitative (e.g., choose one, choose the top three, rate the options, etc.) and the opportunity for open-ended responses will be provided as appropriate.

In regard to coding and analyzing qualitative data in time to complete a report by the end of FY15, the Children’s Bureau is exploring the possibility of a no-cost extension so that the contractor may analyze quantitative and qualitative data. Should a no-cost extension be unavailable, the Children’s Bureau and the contractor will prioritize analysis and reporting needs in order to best meet contract requirements and produce a deliverable useful to the Children’s Bureau, grantees, and other audiences.

Items that may be obtained from other sources will be omitted in order to minimize the time survey respondents spend on the activity. Skip patterns will be used, so respondents will spend time only on the questions most applicable to them. The survey will be administered electronically so participants can respond easily and rapidly.

In regard to whether all respondents have internet access, the contractor has worked with grantee organization project leaders to develop dissemination lists and test the accuracy of e-mail addresses from dissemination lists. In doing so, the contractor identified a small group of Kinship Navigator-TANF service providers who do not have e-mail addresses. The contractor developed a process to administer paper-based surveys to these respondents.

4. Efforts to Identify Duplication and Use of Similar Information

N/A. Similar information is not available.

5. Impact on Small Businesses or Other Small Entities

N/A. The burden is minimal. There is no need to make any adjustments to accommodate small businesses or other small entities.

6. Consequences of Collecting the Information Less Frequently

This is a one-time project. If the collection is not conducted, the Children’s Bureau will not have access to information from the survey, which will be detrimental to any efforts to plan or conduct future Family Connection work or other research.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

N/A. The special circumstances do not apply.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

The first notice published in the Federal Register on 11/12/2014 (Vol. 79, No. 218, p. 67167). No comments were received.

The contractor was consulted to obtain their views on the availability of data, collection frequency, clarity of instructions and recordkeeping, disclosure, reporting format, and on data elements to be recorded, disclosed, or reported.

Representatives from Family Connection grantee organizations were consulted on survey content and clarity of instructions. (See Attachment D - Consultants).

9. Explanation of Any Payment or Gift to Respondents

N/A. No payment or gift will be provided to respondents.

10. Assurance of Confidentiality Provided to Respondents

The contractor will not disclose any individual-level survey information to grantee organizations or the Children’s Bureau. Information will not be published that could be used to identify individual respondents. All survey results will be analyzed and reported in aggregate for all reports, presentations, and publications. The contractor will carefully evaluate all survey analyses to ensure that no individual may be identified in these documents. All contractor staff members sign an employee agreement on ideas, inventions, and confidential information as a condition of employment, and all survey information collected will be limited to immediate project team members. The Children’s Bureau does not require access to any personally identifiable data.

The contractor submitted an exemption determination form to Western IRB on January 27, 2015 to request that the process for collecting web-based, electronic survey data from grantee project and evaluation staff members and project partners be exempt from an IRB review. Along with the form, the contractor included a description of research and protocols for the intended cross-section of grantee representatives (project leadership, service provider, child welfare partner, community partner, and evaluator/evaluation team). The contractor received a regulatory opinion approving the exemption from Western IRB on February 3, 2015. (See Attachment E – WIRB Exemption Determination Letter 2-3-15).

The contractor will protect respondent’s information to the extent allowed by Federal law. This protection is offered under Title V of the E-Government Act, Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA).

11. Justification for Sensitive Questions

N/A. No questions of a sensitive nature will be asked of survey respondents.

12. Estimates of Annualized Burden Hours and Costs

ANNUAL BURDEN ESTIMATES Instrument	Number of Respondents	Number of Responses per Respondent	Average Burden Hours per Response	Total Burden Hours
Project Leadership Protocol	79	1	.75	59.25
Service Provider Protocol	153	1	.5	76.5
Public Child Welfare Partner Protocol	34	1	.25	8.5
Community Partner Protocol	34	1	.25	8.5
Evaluator Protocol	51	1	.75	38.25

Total burden hours is 191 hours.

13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

There are no additional costs resulting from the collection of this information.

14. Annualized Cost to the Federal Government

The evaluation contractor’s annual costs to conduct this activity are estimated at $105,000. This includes survey development and administration (communication with grantee organizations, survey dissemination and collection, data analysis). Evaluation contractor costs consist primarily of labor hours, with the number of hours for each staff member multiplied by hourly rates.

Survey costs can be broken down into four key categories:

• Survey Development – $11,500

• Survey Administration – $38,900

• IRB and OMB Package Submission – $4500

• Data Analysis and Reporting – $50,100

Survey development includes: Developing five surveys; coordinating survey review by grantee representatives; survey revisions; and survey formatting for paper surveys. Survey administration includes: Developing survey respondent e-mail list; grantee communications; Qualtrics programming, review, testing, and revisions; initial dissemination and reminders. IRB and OMB package submission includes: Developing IRB exemption determination submission form and supporting documents; and developing 60-day and 30-day OMB packages. Data analysis and reporting includes: Developing plans for quantitative and qualitative analysis; data cleaning; actual quantitative and qualitative analysis; report drafts; report review by internal and external audiences; and report revisions.

Please note that the contractor has expended funds in regard to survey development, IRB and OMB package submission, and selected tasks in survey administration and data analysis and reporting.

15. Explanation for Program Changes or Adjustments

This is a new project.

16. Plans for Tabulation and Publication and Project Time Schedule

The Cross-Site Evaluation Report, based partially on the results of this survey, will be delivered to the Children’s Bureau, and shared with 2012-funded grantees described in the report. As a means of disseminating the knowledge developed in these grant projects, the report may be posted in the fall of 2015 on websites of the Children’s Bureau and its services and on those of grantees included in the report.

The timeline for this project is September 30, 2014 through September 29, 2015. Specific project activities were originally projected to take place during the following times:

• September 30, 2014 to March 31, 2015: Planning for data collection. This includes developing data collection instruments, developing the list of survey respondents, and developing processes to administer data collection instruments. This also includes time to prepare OMB submissions and obtain approval.

• April 1, 2015 to May 31, 2015: Data collection, including initial survey dissemination and two reminders.

• June 1–30, 2015: Data analysis, per plans detailed below.

• July 1-31, 2015: Draft data report.

• August 1-31, 2015: Report review by Children’s Bureau.

• September 1-29, 2015: Report revisions/finalizing report.

As noted earlier, the Children’s Bureau is exploring the possibility of a no-cost extension so that the contractor may analyze quantitative and qualitative data. Should a no-cost extension be unavailable, the Children’s Bureau and the contractor will prioritize analysis and reporting needs in order to best meet contract requirements and produce a deliverable useful to the Children’s Bureau, grantees, and other audiences. The following is a potential revised timeline based on OMB approval prior to July 1, 2015:

• July 1-31, 2015: Data collection, including initial survey dissemination and two reminders

• August 1-31, 2015: Data cleaning and analysis.

• September 1-15, 2015: Include detailed appendices of survey data in each cluster report with appropriate references.

• September 16-21, 2015: Limited review of survey data by Children’s Bureau.

• September 22-29, 2015: Report revisions/finalize report.

A potential timeline with a three-month, no-cost extension is below:

• July 1 to August 31, 2015: Data collection, including initial survey dissemination and two reminders.

• September 1-29, 2015: Data cleaning and analysis.

• October 1-31, 2015: Draft data report. Revisions based on internal review.

• November 1-15, 2015: Report review by Children’s Bureau.

• November 16-30, 2015: Report revisions

• December 1-15, 2015: Finalize and formal report.

Data collection consists of a one-time distribution of multiple versions of a survey to up to 351 respondents from 17 grantee organizations. The grantee organizations have been funded as demonstration sites to develop and implement three types of child welfare projects, with seven grantees operating Family-finding and Family Group Decision-Making projects, five grantees operating Kinship Navigator-TANF projects, and five grantees operating Residential Family Treatment projects. Survey respondents are a cross-section of project leaders, service providers, local evaluators, and public child welfare and community partners. Surveys have been developed for each of the five respondent groups. While there are many questions in common across surveys, each survey has been customized to ask appropriate questions of most importance from each of these key groups. (See APPENDIX II - SURVEYS).

Complex analytical techniques will be used to analyze the data. (See Attachment F – Data Analysis). Additional details regarding qualitative analysis techniques have been added to Attachment F – Data Analysis.

17. Reason(s) Display of OMB Expiration Date is Inappropriate

N/A. This approval is not being sought.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

N/A. There are no exceptions.