Program Support GenCI SSA: SIMR - Analysis Plan Template

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Strengthening the Implementation of Marriage and Relationship Programs (SIMR) - Analysis Plan Template

Formative Data Collections for Program Support

0970 – 0531

Supporting Statement

Part A

November 2023

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Samantha Illangasekare

Rebecca Hjelm

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Formative Data Collections for Program Support (0970-0531).

• Description of Request:

We are seeking clearance to collect information from Healthy Marriage and Relationship Education (HMRE) award recipients and local evaluators about the analysis plan for their local evaluations using an analysis plan template. The purpose of collecting this information on local evaluation analysis plans is to facilitate ACF’s provision of evaluation technical assistance with the goal of strengthening the capacity of HMRE award recipients and local evaluators to conduct high quality, rigorous evaluations. We do not intend for this information to be used as the principal basis for public policy decisions.

• Time Sensitivity: We would like to distribute the analysis plan template and instructions to HMRE award recipients and local evaluators as soon as possible – ideally beginning in November 2023 – to allow them sufficient time to set up planned analysis ahead of the end of data collection. These templates will also serve as tools for initiating evaluation technical assistance that will refine and strengthen award recipient analysis plans.

A1. Necessity for Collection

The Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services requests permission to collect information from Healthy Marriage and Relationship Education (HMRE) award recipients funded by the Office of Family Assistance (OFA) about their analysis plans for their local evaluations. HMRE award recipients designed their own local evaluations – either descriptive or impact. To inform ACF’s provision of technical assistance (TA), ACF is seeking clearance for HMRE award recipients to use standardized templates to document their analysis plans. The completed plans will be reviewed by the HMRE Evaluation TA team for the purpose of developing recommendations for improvement and informing subsequent TA. The HMRE Evaluation TA team will use these completed plans to design TA to strengthen the capacity of award recipients and local evaluators to conduct high quality, rigorous evaluations.

Study background

Since 2005, Congress has authorized dedicated funding for discretionary awards from ACF’s Office of Family Assistance (OFA) to support HMRE programs. A subset of these HMRE award recipients proposed and are currently conducting independent local evaluations of program services to contribute to the field’s understanding of varying HMRE program approaches, contexts, and target populations, while answering questions important to local communities.

In 2019, OPRE contracted with Mathematica to implement the Strengthening the Implementation of Marriage and Relationship Programs (SIMR) project to 1) understand implementation challenges faced by HMRE programs and test strategies to address these challenges, and 2) support HMRE programs’ local evaluations through TA. In 2020, ACF received OMB approval to collect information from HMRE award recipients conducting local evaluations about the design of their local evaluations using an evaluation plan template (OMB #0970-0356). The information collected through the evaluation plan templates has allowed the HMRE Evaluation TA Team to support award recipients’ success in launching and collecting data for their evaluations by providing TA tailored to each award recipient’s needs, based on their evaluation design and plan.

In order to provide tailored TA to award recipients in analyzing their local evaluation data, the TA team must now understand their current plans for analysis. The current generic information collection (GenIC) request involves asking HMRE award recipients conducting a local evaluation to complete an analysis plan template that documents the research questions, measures, study design, planned and actual implementation of the program, and planned analytic methods for their evaluation.

Legal or administrative requirements that necessitate the collection

There are no requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

The purpose of this information collection is to ask award recipients and their local evaluators to document their analysis plans using standardized templates and accompanying instructions, tailored to their evaluation design (descriptive or impact). Developing a structured study analysis plan before examining the data will allow the award recipients to foster an efficient and effective approach for analyzing the data and reporting the findings. This analysis template is critical for the HMRE Evaluation TA team to identify their strengths, determine areas where the plans do not meet standards of rigor, develop recommendations for improvement, and inform subsequent TA.

This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections for program support (0970-0531):

• Delivery of TA.

• Planning for provision of evaluation-related TA.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker, and is not expected to meet the threshold of influential or highly influential scientific information.

Guiding Questions

The HMRE Evaluation TA team will use the completed analysis plans to answer three questions:

1. To what extent does each analysis plan propose appropriate analytic methods and describe them in sufficient detail to demonstrate the award recipient’s capacity to carry out the analysis?

2. What are the analysis plans’ strengths and areas for improvement?

3. What are additional topics that could need evaluation TA as award recipients advance to the analysis and reporting phase?

Study Design

The templates and instructions will be disseminated by the HMRE Evaluation TA team using the email templates for each evaluation type (Appendix G, for descriptive studies, and Appendix H for impact studies). The HMRE Evaluation TA team will provide award recipients with a standardized template and instructions for completing it for each type of evaluation: descriptive (Appendices A and B), impact (Appendices C and D), and implementation (Appendices E and F). HMRE award recipients executing descriptive studies will complete the descriptive analysis plan template (Appendix A); award recipients executing impact studies will complete both the impact analysis plan template (Appendix C) and the implementation analysis plan template (Appendix E). Table A2.1 shows the data collection activities, instruments, respondents, content, purpose of collection, as well as the mode and expected duration to complete the instruments.

Table A2.1 Data Collection Activities

Data Collection Activity	Instrument and instructions	Respondent, Content, Purpose of Collection	Mode and Duration
Descriptive Analysis Plan	Descriptive Analysis Plan Template (Appendix A) Descriptive Analysis Plan Instructions (Appendix B)	Respondents: award recipients conducting local evaluations with a descriptive design (20 recipients). Content: The descriptive analysis plan describes the program design; the research questions; and the analytic approach for the outcome study, the implementation study, or both (depending on the studies the award recipient is conducting). In addition, for the outcome study, the document includes a description of the evaluation enrollment process, data collection procedures, and outcome measures; for the implementation study, the plan also discusses the research questions and the data used to answer the research questions. Purpose: This document is designed to inform assessment of the award recipient approach for analyzing data and reporting findings to identify strengths, determine whether the plans meet standards of rigor, develop recommendations for improvement, and inform subsequent TA.	Mode: Email Duration: 6 hours
Impact Analysis Plan	Impact Analysis Plan Template (Appendix C) Impact Analysis Plan Instructions (Appendix D)	Respondents: award recipients conducting local evaluations with an impact design (20 recipients). Content: The impact analysis plan is a document that describes the proposed research questions, the selected outcome measures, the program design and counterfactual conditions, the evaluation study design, and the proposed analytic approaches to gauge the effect or impact of the intervention. Purpose: This document is designed to inform assessment of the award recipient approach for analyzing data and reporting findings to identify their strengths, determine whether the plans meet standards of rigor, develop recommendations for improvement, and inform subsequent TA.	Mode: Email Duration: 8 hours
Implementation Analysis Plan	Implementation Analysis Plan Template (Appendix E) Implementation Analysis Plan Instructions (Appendix F)	Respondents: award recipients conducting local evaluations with an impact design (20 recipients). Content: The implementation study analysis plan is a document that describes the research questions, the data that will be used to answer those research questions, and the methods employed to analyze the implementation data and describe the findings. Purpose: This document is designed to inform assessment of the award recipient approach for analyzing data and reporting findings to identify their strengths, determine whether the plans meet standards of rigor, develop recommendations for improvement, and inform subsequent TA.	Mode: Email Duration: 4 hours

The HMRE Evaluation TA team will collect and review completed analysis plans from all award recipients that are currently conducting an evaluation (40 of the 55 funded HMRE awards). Upon OMB approval, the HMRE Evaluation TA team will send an email to the project directors and local evaluators for each HMRE award. The emails (Appendices G and H) will introduce the templates (Appendices A, C, and E for descriptive, impact, and implementation, respectively), the instructions (Appendices B, D, and F, respectively), and the timeline for completing the plan. Because the analysis plans must be developed and refined as a condition of award recipients’ funding, we expect a response rate of 100 percent among those responding to the data collection activities outlined in this package.

Local evaluators will use the instructions for completing the plan to understand how to populate the analysis plan template. The instructions will give them guidance about the type of information to include in each section of the analysis plan along with any population-specific details they should consider in their analysis. Local evaluators will complete the analysis plans, with input from their program directors and program staff as appropriate.

The HMRE Evaluation TA team will use three methods to provide support through TA while award recipients complete the plans to minimize overall burden to respondents in completing the template(s): (1) group TA, including webinars and Q&A sessions; (2) one-on-one TA to award recipients on an as-needed basis only, and (3) an evaluation TA help desk. For all webinars and calls, the HMRE Evaluation TA team will invite the award recipient’s project director and key staff, the local evaluator, and the family assistance program specialist from OFA.

In-person training. The HMRE Evaluation TA team will offer an optional in-person training, with a virtual option for participation, on the analysis plan templates, to provide support to evaluators to plan analyses to meet ACF standards.

Group TA. The HMRE Evaluation TA team will lead a webinar during the analysis planning period on topics covered in the analysis plan templates. The HMRE Evaluation TA team will maintain some flexibility in the planned approach to the webinar and tailor content to specific needs of award recipients.

One-on-one TA. The evaluation technical assistance liaisons will provide support during TA calls held at regular intervals. Additional HMRE Evaluation TA team members may also meet with award recipients on a case-by-case basis.

Evaluation TA help desk. The HMRE Evaluation TA team will provide a help desk in the form of a monitored email address that award recipients can reach out to with clarifying questions or requests for resources. This email address will be intended for simpler award recipient queries that do not require a more in-depth conversation.

Award recipient teams will submit a draft of the plan by March 2024 and revise and finalize the plans by October 2024. Each of the analysis plans will undergo a review process conducted by the HMRE Evaluation TA team. Using a standardized process, the HMRE Evaluation TA team will review the award recipients’ analysis plans to succinctly document key strengths and weaknesses in the proposed analysis plans and identify opportunities for additional TA.

There are no quantitative components for the current request.

A3. Use of Information Technology to Reduce Burden

The SIMR HMRE Evaluation TA team plans to use improved information technology wherever possible. The HMRE award recipients can use their existing local evaluation plans (i.e., existing written plans detailing evaluation designs) to inform and build many parts of the analysis plan. Additionally, the development of the plans will be helpful to HMRE award recipients when they develop their final reports later in the project as they will be able to draw heavily on the analysis plans. Analysis plans will be collected virtually and will not require any in-person follow-up.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The HMRE Evaluation TA team will encourage award recipients to build on their local evaluation plans to complete the analysis plans. The analysis plan template will also encourage the award recipients to provide details about the plans that they did not include in their applications.

A5. Impact on Small Businesses

The award recipients in the study are small, nonprofit organizations. The proposed analysis plan templates are designed to minimize the burden on all organizations involved, including small businesses and entities, by collecting only critical information using the standardized templates. Additionally, the TA provided to award recipients to help with completing the analysis plan template(s) will also minimize the burden on all organizations involved.

A6. Consequences of Less Frequent Collection

Without the collection of these standardized analysis plans, the HMRE Evaluation TA team would be unable to complete a timely and efficient review of award recipient analysis plans or identify additional TA opportunities, which affect the overall quality and rigor of the evaluations.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection request to extend approval of the umbrella generic with minor changes. The notice was published on January 28, 2022, (87 FR 4603), and provided a sixty-day period for public comment. ACF did not receive any comments on the first notice. A second notice was published, allowing a thirty-day period for public comment, in conjunction with submission of the request to OMB. ACF did not receive any comments on the second notice.

Consultation with Experts

The larger SIMR study has engaged several experts in HMRE programming and research for consultation to the SIMR study team and ACF on multiple occasions. However, for this portion of the study, we do not expect we will have to consult experts outside the study.

A9. Tokens of Appreciation

No tokens of appreciation for respondents are proposed for this information collection.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

This data collection effort does not include collecting personally identifiable information.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents are not considered human subjects, but they will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.

Data Security and Monitoring

No information will be given to anyone outside of the SIMR HMRE Evaluation TA team and ACF. The analysis plans submitted by award recipients will be stored on Mathematica’s network, which is accessible only to the HMRE Evaluation TA team.

A11. Sensitive Information ¹

There are no sensitive questions in this data collection.

A12. Burden

Explanation of Burden Estimates

Table A12.1 provides the estimated annual reporting burden calculations for the three instruments included in this request. The total annual burden is estimated to be 360 hours. Assumptions by instrument follow.

Award recipients with descriptive evaluations will complete the descriptive analysis plan template (Appendix A). This will be at most 20 award recipients. On average, it will take 6 hours to complete both the draft and final versions of the template. The estimated total annual burden for this effort is 120 hours.

Award recipients with impact evaluations will complete the impact analysis plan template (Appendix D) and the implementation analysis plan template (Appendix G). This will be at most 20 award recipients.

• On average, it will take 8 hours to complete both the draft and final versions of the impact analysis plan template. The estimated total annual burden for this effort is 160 hours.

• On average, it will take 4 hours to complete both the draft and final versions of the implementation analysis plan template. The estimated total annual burden for this effort is 80 hours.

Estimated Annualized Cost to Respondents

We expect local evaluators for each award recipient will review instructions and complete the analysis plan template. To compute the total estimated annual cost, the total burden hours were multiplied by the estimated average hourly wage for evaluators (see table below). We estimate the average hourly wage for local evaluators to be the average hourly wage for “Economists” ($54.78), taken from the U.S. Bureau of Labor Statistics, Occupational Employment Statistics, 2022.²

Table A.12.1 Total Burden Requested Under this Information Collection

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total/ Annual Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Descriptive Analysis Plan Template	20	1	6	120	$54.78	$6,573.60
Impact Analysis Plan Template	20	1	8	160	$54.78	$4,382.40
Implementation Analysis Plan Template		1	4	80	$54.78	$8,764.80
Total	40			360		$19,720.80

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

The total estimated cost to the federal government for the data collection activities under this current request will be $140,675. This includes personnel effort plus other direct and indirect costs.

Cost Category	Estimated Costs
Review and TA support of analysis plans	$140,675
Total costs over the request period	$140,675

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella formative generic clearance for program support (0970-0531).

A16. Timeline

The information collection will take place upon OMB approval through October 2024, which will allow for award recipients to submit a draft version of the plan in March 2024 and revise and finalize the plan prior to analysis beginning in the final year of the award period. The information collected under this request will be used to strengthen the capacity of HMRE award recipients and local evaluators to conduct high quality, rigorous evaluations. There are no plans to publish the information collected under this request.

A17. Exceptions

All instruments will display the expiration date for OMB approval. No exceptions are necessary for this information collection.

Attachments

Appendix A: Descriptive Evaluation Analysis Plan Template for HMRE Award Recipients

Appendix B: Instructions for Descriptive Evaluation Analysis Plan Template for HMRE Award Recipients

Appendix C: Impact Evaluation Analysis Plan Template for HMRE Award Recipients

Appendix D: Instructions for Impact Evaluation Analysis Plan Template for HMRE Award Recipients

Appendix E: Implementation Study Analysis Plan Template for HMRE Award Recipients

Appendix F: Instructions for Implementation Study Analysis Plan Template for HMRE Award Recipients Conducting Impact Studies

Appendix G: Descriptive Evaluation Analysis Plan Email for HMRE Award Recipients

Appendix H: Impact and Implementation Evaluation Analysis Plan Email for HMRE Award Recipients

1 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

2 Bureau of Labor Statistics, U.S. Department of Labor, Occupational Outlook Handbook, Social and Community Service Managers, on the Internet at https://www.bls.gov/ooh/management/social-and-community-service-managers.htm (visited September, 2023).

10