Supporting Statement B: SIMR

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Strengthening the Implementation of Marriage and Relationship Programs (SIMR)

Formative Data Collections for ACF Research

0970-0356

Supporting Statement

Part B

October 2020

Submitted by:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4th Floor, Mary E. Switzer Building

330 C St. SW

Washington, D.C. 20201

Project Officers:

Samantha Illangasekare

Shirley Adelstein

Part B

B1. Objectives

Study objectives

For many years, the Administration for Children and Families (ACF) Office of Planning Research and Evaluation (OPRE) has led a sustained effort within the federal government to develop, document, and test healthy marriage and relationship education (HMRE) programs, particularly with low-income populations. The Strengthening the Implementation of Marriage and Relationship Programs (SIMR) study seeks to generate evidence about how to improve the effectiveness of HMRE programs through testing and refining strategies to overcome common implementation challenges related to recruitment, retention, and content engagement. The study team will partner with grantees to implement solutions tailored to the context and specific challenges of each grantee and conduct multiple learning-cycle assessments of the solutions. Early cycles will seek to understand the feasibility of implementing a strategy, while later cycles will examine the effectiveness of the solutions through more rigorous designs (for example, quasi-experimental).

The study team is requesting permission to contact programs from the 2020 round of Office of Family Assistance (OFA)-funded HMRE grantees to gather information about their program plans, experience, and the feasibility of implementing specific solutions to common implementation challenges and engaging in the rapid-learning activities. Once OMB approval is received, the information collection will take place from October 2020 through January 2021.

Generalizability of results

This study is intended to present an internally-valid description of HMRE grantees in chosen sites, not to promote statistical generalization to other sites or service populations.

Appropriateness of study design and methods for planned uses

These data collection efforts are important for understanding feasibility and program relevance of the SIMR study design and approaches. To do this, we are obtaining as much information as we can from HMRE grantee applications. The phone interview and virtual meetings will focus only on confirming information from our review of applications and obtaining information we cannot extract from applications, such as detailed information about anticipated challenges and feasibility of potential solutions. As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and design

Target population

The recruitment pool for this information gathering process is restricted to HMRE programs funded by ACF’s OFA in September 2020. Our process will primarily involve interacting with leadership (up to two leaders) at the HMRE programs, but up to four additional staff members with knowledge about the program will be asked to join in the final phase of data collection.

Sampling and site selection

This phase of the SIMR project seeks to gather information from HMRE grantees. The universe of programs includes up to 60 organizations that are awarded HMRE grants administered by OFA in fall 2020. In order to recruit respondents from up to 20 programs for phone interviews, the study team is conducting a review of grantee applications to identify the sites that best meet the criteria for inclusion in the study. Selection criteria include (1) whether the site’s plan is consistent with ACF priorities on target populations, program models, and other program characteristics; (2) the relevance of at least one of the priority approaches we have identified for SIMR to the site; and (3) the site’s ability to provide input and feedback in developing a rapid-cycle learning design. Study team members will fill out a standardized rubric assessing each criterion. Study leadership will consult with ACF to review the criteria for each grantee and select the 20 programs that are invited to participate in the phone interviews. Phone interviews serve to gain additional knowledge that cannot be gleaned from the applications (for example, clarifying program or service structure or assessing the site’s ability to provide input and feedback in developing a rapid-cycle learning design). From these 20 grantees, further information gathering in the form of a series of virtual meetings will be conducted with up to 12 sites using the same criteria noted above. The study team will update the rubrics with the information collected from the phone interviews and review the information with ACF to select the 12 grantees for virtual meetings.

The purpose of this collection is to gather information to understand the programs and services offered by HMRE grantees, explore the field’s interest in SIMR, and inform the design of the SIMR study. There is minimal burden involved with this collection. For this reason, we expect a high response rate (nearly 100 percent participation) among those who are contacted for the data collection activities outlined in this package.

B3. Design of data collection instruments

Development of data collection instruments

The SIMR telephone interview protocol was developed based on a similar protocol used to inform the Strengthening Relationship Education and Marriage Services (STREAMS) evaluation (OMB Control # 0970-0481). The STREAMS tool was effective in collecting preliminary information about the HMRE field in order to inform the design of the STREAMS evaluation. Therefore, we believe this adapted version of the protocol will be similarly effective for SIMR. The study team tailored questions based on the different goals and information needs to inform study design development for SIMR. The study team streamlined the protocol to ask only questions that will be helpful in making an assessment about the applicability and feasibility of implementation approaches to be tested and the field’s interest in SIMR. The SIMR study team aims to collect as much information as possible prior to the phone call via a review of HMRE grantee applications; thus, the interview will be tailored based on this review and used to confirm information and probe deeper into factors that are difficult to ascertain from the application or are not covered in it (for example, detailed information about anticipated challenges and feasibility of potential solutions).

The design of the virtual meetings was informed by recent successful virtual meetings, including a March 2020 meeting with experts for the SIMR project. As a result of COVID-19, we shifted an in-person meeting with experts to a virtual one. The virtual meeting used activities identical to the ones described in this package.

The data collection instruments will not be pretested. Previous large-scale evaluations, including STREAMS, have successfully used similar instruments during the information gathering processes.

B4. Collection of data and quality control

Two SIMR study team members (one senior and one junior) will conduct the phone interviews. The senior person will facilitate the interview and the junior staff member will take notes. Three SIMR study team members (one senior and two junior) will facilitate the virtual meetings. The senior staff member will facilitate the meeting with one junior staff member taking notes and the other operating the technology platform. Team members are experienced in the process of gathering information for purposes of informing study development and design. In addition, all team members will receive training to ensure that grantees are engaged in a consistent manner. All meetings will be recorded for note-taking purposes. See Supporting Statement Part A, Section A2 for information about the study design.

B5. Response rates and potential nonresponse bias

Site/respondent selection

Because there is minimal burden involved with the initial phone call, we expect nearly 100 percent participation. The purpose of this study is to build evidence for promising solutions to address common implementation challenges for HMRE programs, and we expect that most program leaders will be eager to contribute to this effort, provided they have the capacity to participate. It is possible that due to the COVID-19 public health emergency, some HMRE program staff will have reduced capacity to participate due to increased demands on their time. The selection of programs for participation in virtual meetings will be based in part on their willingness to have additional conversations. For this reason, we also expect high response rates for the virtual meetings.

The data collection activities are geared toward helping the study team gather information to understand the programs and services offered by HMRE grantees, explore the field’s interest in SIMR, and inform the design of the SIMR study; they are not designed to produce statistically generalizable findings, and participation is wholly at the respondent’s discretion. For these reasons, response rates will not be calculated or reported.

Nonresponse

We expect little to no nonresponse. If we do not receive a response to the initial email request for a meeting, we will not pursue a response from the grantee. To help ensure high response rates, the study team will accommodate respondents’ schedules and be flexible about setting dates and times for phone calls and meetings. As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented and reported in written materials associated with the data collection.

B6. Production of estimates and projections

Data collected for the site selection process is for internal use only. The data will not be used to generate population estimates, either for internal use or dissemination.

B7. Data handling and analysis

Data handling

No information will be given to anyone outside of the SIMR study team and ACF. Typed notes and audio recordings will be stored on Mathematica’s network, which is accessible only to the SIMR study team.

Data analysis

Data from the study will be used to inform study design and development of the SIMR project, including next steps such as site recruitment, development of learning cycle study designs, and survey development. We will synthesize the information collected via the phone interview and virtual meetings to inform these critical aspects of the SIMR project development.

B5. Contact person

• Robert Wood, Mathematica, project director, rwood@mathematica-mpr.com

• Samantha Illangasekare, OPRE, point of contact, Samantha.Illangasekare@acf.hhs.gov

Attachments

Appendix A, SIMR Phone Meeting Email Template

Appendix B, SIMR Project Description

Appendix C, Topics for SIMR Phone Meeting

Appendix D, SIMR Phone Interview Protocol

Appendix E, SIMR Virtual Meeting Series Agenda