Supporting Statement A - Formative Generic - FaMLE Cross-Site

Fatherhood and Marriage Local Evaluation and Cross-Site Services Components

OMB Information Collection Request

0970 - 0356

Supporting Statement

Part A

May 2019

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Seth Chamberlain

Senior Social Science Research Analyst

Administration for Children and Families

U.S. Department of Health and Human Services

A1. Necessity for the Data Collection

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval to obtain analysis plans from Healthy Marriage and Responsible Fatherhood (HMRF) grantees, funded by the Office of Family Assistance (OFA) in ACF. The HMRF grantees are conducting their own local impact and descriptive evaluations. These grants are described in the background section below. The grantees’ local evaluations are intended to address questions they have about their own programs and to contribute to the broader evidence base on impacts of HMRF programs. Grantees conducting local evaluations have nearly completed their evaluation implementation and data collection and are ready to plan their analysis.

As part of the ACF Office of Planning, Research, and Evaluation’s (OPRE) Fatherhood and Marriage Local Evaluation and Cross-Site (FaMLE Cross-Site) project, we are submitting this information collection request (ICR) for the following:

1. To ask HMRF grantees conducting local impact evaluations to complete two standardized analysis plan templates with detailed information about their impact evaluation data analysis plans and their implementation data analysis plans. ICs include:

   1. Impact Evaluation Analysis Plan Template for HMRF Grantees (Appendix A)

   2. Implementation Analysis Plan Template for HMRF Grantees (Appendix B)

2. To ask HMRF grantees conducting local descriptive evaluations to complete one standardized analysis plan template with detailed information about their descriptive evaluation data analysis plans. IC includes:

1. Descriptive Evaluation Analysis Plan Template for HMRF Grantees (Appendix C)

Please note that each template has accompanying instructions to guide the grantees in completing the template. We have provided these instructions as Appendices D, E, and F.

OPRE and the FaMLE Cross-Site staff will use the templates to offer TA to grantees and their evaluators to support the development of rigorous and relevant analysis plans, which accurately reflect the grantees’ evaluation designs and research objectives. The information collected will be used for internal purposes only – specifically to inform the provision of technical assistance (TA), one of the key goals cited as relevant for formative generic clearance.

Study Background

Although family life has changed rapidly in the past few decades, the valuable role that families play in the lives of children and adults remains. In the past twenty-five years, marriage rates have declined, whereas cohabitation and nonmarital childbearing have increased (Lamidi 2015, 2016; Manning et al. 2014). Yet, research continues to show that children tend to have better outcomes when raised by married, biological parents in stable relationships (Acs & Nelson 2004; Amato 2005; McLanahan & Sandefur 1994; Waldfogel et al. 2010). A healthy, stable relationship also may have advantages, both economic and emotional, for adults (Waite & Gallagher 2000; Williams & Dunne-Bryant 2006).

To support families, OFA has provided grants for services designed to strengthen family relationships. In fiscal year 2015, OFA awarded 90 grants: 46 grants to support healthy marriage and relationships (HM), 39 for fathers and families, and 5 for incarcerated fathers (we refer to the fatherhood grants as responsible fatherhood, or RF). HM services are designed to promote healthy relationships and marriage, and may also address other issues affecting families, such as parenting and economic stability. RF services are designed to support father involvement and parenting skills, improve economic well-being, and support healthy relationships.

As a stipulation of the funding, all grantees are required to collect, store, and report on a set of performance measures (OMB control number 0970-0460). Select grantees are also conducting their own local evaluations of program services to address questions grantees have about their own programs and to contribute to the broader evidence base on HMRF programming.

A key objective of the FaMLE Cross-Site project is to strengthen the capacity of grantees, working with their own local evaluators, to conduct rigorous evaluations that add to the body of evidence on program effectiveness, operations, and outcomes. OPRE awarded the FaMLE Cross-Site project to Mathematica Policy Research. While grantees developed their information collections independently to address their questions, Mathematica staff on the FaMLE Cross-Site project are providing technical assistance to 33 grantees (one grantee is conducting both an impact and a descriptive evaluation) to help them carry out their local evaluations (18 impact evaluations and 16 descriptive evaluations).

Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

One of the objectives of the FaMLE Cross-Site project is to strengthen the capacity of grantees, working with their own local evaluators, to conduct evaluations that add to the body of evidence on HMRF program effectiveness, operations, and outcomes. The analysis plan templates (the information collection instruments submitted through this request) will help the federal government and contractor staff provide evaluation technical assistance to grantees. Grantees will complete the templates using the instructions as guidance, and then the government and contractor will review them and provide detailed feedback. This review and feedback on the analyses will support grantees in conducting high-quality analysis of data they have collected through their local evaluations to ensure the data analysis meets ACF’s research and evaluation standards of rigor and relevance.

Universe of Data Collection Efforts

The FaMLE Cross-Site team will provide support for 18 local evaluations with an impact design and 16 local evaluations with a descriptive design. To achieve this, OFA proposes to collect the proposed analysis plans from those 33 grantees, which will differ by their study design. Table A2.1 shows which grantees will complete each instrument, based on the evaluation design.

Table A2.1. Data Collection Instruments, by Grantee’ Proposed Evaluation Design

Instrument	Grantees conducting local evaluations with an impact design	Grantees conducting local evaluations with a descriptive design
Appendix A: Impact Evaluation Analysis Plan Template for HMRF Grantees	18 grantees
Appendix B: Implementation Analysis Plan Template for HMRF Grantees	18 grantees
Appendix C: Descriptive Evaluation Analysis Plan Template for HMRF Grantees		16 grantees

Impact Analysis Plan (Appendix A). Only grantees conducting local evaluations with an impact evaluation design (that is, those with a comparison group) will complete the impact analysis plan template. This template asks grantees to state their primary research questions and then the secondary research questions. The second section asks for information about the design of the study, including a description of the intended intervention and the counterfactual condition; sample formation, random assignment, consent, and data collection. This will provide context for providing feedback on the data analysis plan. The next section asks for plans for analyzing the data, including specifying outcome measures, the analytic sample available for analyzing impacts, plans to assess baseline equivalence of the analytic sample, measures of crossover between intervention and comparison groups, and impact analysis model specifications. The analysis plan template also includes a CONSORT diagram, which collects data on the number of individuals, couples, or clusters, if applicable, enrolled in the evaluation and retained through data collection.

Implementation Analysis Plan (Appendix B). Only grantees conducting local evaluations with an impact evaluation design will complete the implementation analysis plan template. This template asks grantees to describe the research questions they will examine about implementation of the intervention, the data they will use to answer the questions, and the methods to be used to analyze the implementation data and describe the findings. Information about research questions and data to answer the questions are asked to provide context for providing feedback on the data analysis plan.

Descriptive Analysis Plan (Appendix C). Only grantees conducting local evaluations with a descriptive evaluation design will complete the descriptive analysis plan template. The descriptive analysis plan template first asks for information on the program being evaluated, including program components, content, dosage, and target population. Next, the template has separate sections for an outcomes study and a process/implementation study. Grantees conducting an outcomes study would complete the outcomes study section of the template, while grantees conducting a process/implementation study would complete the process/implementation study section of the template. Each of these sections includes instructions for stating the research questions, describing sample formation and data collection, specifying outcome measures, and discussing the plan for data analyses. Information about research questions and data to answer the questions are asked to provide context for providing feedback on the data analysis plan.

A3. Improved Information Technology to Reduce Burden

ACF and its contractors will employ information technology as appropriate to reduce the burden of respondents who agree to participate. Grantees can complete and submit the templates electronically.

A4. Efforts to Identify Duplication

No other sources of information will allow ACF and the FaMLE Cross-Site evaluation technical assistance team to assess the quality of ACF-funded HMRF grantees’ plans for data analyses to meet their descriptive or impact evaluation objectives. No unnecessary information is being requested of program staff or grantees. None of the instruments will ask for information that can be reliably obtained through other sources.

A5. Involvement of Small Organizations

The potential exists for data collection activities to affect small entities associated with the grantees. HMRF grantees may conduct evaluations led by local evaluators affiliated with small organizations. Grantees may task the local evaluator with the collection of some or all of the information requested. The proposed analysis plan templates are designed to minimize the burden on all organizations involved, including small businesses and entities, by collecting only critical information using the standardized templates.

A6. Consequences of Less Frequent Data Collection

The purpose of each information collection instrument included in this submission is described in Item A2, above. Not collecting the information using the analysis plan templates would limit the government’s ability to understand the quality of grantees’ evaluation plans and provide TA to facilitate high quality evaluation results.

A7. Special Circumstances

There are no special circumstances associated with this information collection.

A8. Federal Register Notice and Consultation

Federal Register Notice and Comments

Formative Generic

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for formative information collection. This notice was published on October 11, 2017, Volume 82, Number 195, page 47212, and provided a sixty-day period for public comment. During the notice and comment period, no substantive comments were received.

Consultation with Experts Outside of the Study

ACF consulted staff from the project contractor, Mathematica Policy Research, when preparing the templates.

A9. Incentives for Respondents

No incentives are proposed for respondents completing instruments for the local evaluation information collection.

A10. Privacy of Respondents

As specified in the contract, Mathematica (the Contractor) shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor shall ensure that all of its employees, who perform work under this contract are trained on data privacy issues and comply with the above requirements. All Mathematica staff are required to sign the Mathematica Staff Confidentiality Agreement and participate in annual security awareness training. Respondents will be informed about the planned uses of data, and that their information will be kept private to the extent permitted by law.

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individual’s personal identifier.

A11. Sensitive Questions

There are no sensitive questions for the proposed instruments.

A12. Estimation of Information Collection Burden

Table A12.1 provides the estimated annual reporting burden calculations for the three instruments included in this request. The total annual burden is estimated to be 312 hours. Assumptions by instrument follow.

• Impact analysis plan template (Appendix A). At most, 18 grantees will complete the impact analysis plan template (grantees with impact evaluations). On average, it will take 8 hours to complete this template. The estimated total annual burden for this effort is 144 hours.

• Implementation analysis plan template (Appendix B). At most, 18 grantees will complete the impact analysis plan template (grantees with impact evaluations). On average, it will take 4 hours to complete this template. The estimated total annual burden for this effort is 72 hours.

• Descriptive analysis plan template (Appendix C). At most, 16 grantees will complete the descriptive analysis plan template (grantees with descriptive evaluations). On average, it will take 6 hours to complete this template. The estimated total annual burden for this effort is 96 hours.

Table A12.1. Total Burden Requested Under this Information Collection

Instrument	Total Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Total Annual Cost
Appendix A Impact Analysis Plan Template	18	1	8	144	$30.82	$4,438.08
Appendix B Implementation Analysis Plan Template	18	1	4	72	$30.82	$2,219.04
Appendix C Descriptive Analysis Plan Template	16	1	6	96	$30.82	$2,958.72
Estimated Annual Burden Total				312	$30.82	$9,615.84

Total Annual Cost

The estimated annualized cost to respondents is $9,615.84. For cost calculations for the labor associated with completing the analysis plans, we estimate the average hourly wage for program directors and managers to be the average hourly wage for “Social and Community Services Manager” ($30.82), taken from the U.S. Bureau of Labor Statistics, Occupational Employment Statistics, 2017.¹

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The estimated total cost to the federal government for data collection associated with this Information Collection Request (ICR) is $29,359.

For cost calculations, we estimated 92 hours of time for a GS-12, 92 hours for a GS-13, and 92 hours for a GS-14. These hours will be used by multiple ACF staff to review the templates, and provide feedback on grantees’ analysis plans.

A15. Change in Burden

This is a request for a generic information collection under OMB #0970-0356.

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

The schedule for data collection is shown below in Table A16.1. Data collection will begin upon OMB approval.

Table A16.1. Schedule for HMRF Local Evaluation Analysis Plan Data Collection

Activity	Time Frame
Impact analysis plan	5 months
Implementation analysis plan	5 months
Descriptive analysis plan	5 months

Grantees conducting local impact evaluations will complete the impact analysis plan template and the implementation analysis plan template. No publications are expected from this data collection.

Grantees conducting local descriptive evaluations will complete the descriptive analysis plan template. No publications are expected from this data collection.

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1 Bureau of Labor Statistics, U.S. Department of Labor, Occupational Outlook Handbook, Social and Community Service Managers, on the Internet at https://www.bls.gov/ooh/management/social-and-community-service-managers.htm (visited December 26, 2018).

9