SSA - MOMS GenIC

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Next Generation of Enhanced Employment Strategies Project

Formative Data Collections for Program Support

0970-0531

Supporting Statement

Part A

September 2021

Submitted by:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4th Floor, Mary E. Switzer Building

330 C St., SW

Washington, D.C. 20201

Project Officers:

Marie Lawrence

Gabrielle Newell

Sarita Barton

Megan Reid

Part A

Executive Summary

• Type of request: This information collection request is for a generic information collection under the umbrella generic, Formative Data Collections for Program Support (0970-0531).

• Description of request: This request includes formative evaluation activities to refine the implementation of a program being evaluated as part of the Next Generation of Enhanced Employment Strategies (0970-0545; approved April 2020). The program is the Mental Health Outreach to MotherS (MOMS) Partnership. Proposed data collection efforts include surveys of program participants and potential participants who attend information sessions about the program and focus groups of program staff and staff at referral agencies. We do not intend for this information to be used as the principal basis for public policy decisions.

• Time sensitivity: To begin the formative evaluation process when MOMS Partnership is first piloted in November 2021, we request approval by October 31, 2021.

A1. Necessity for the data collection

The Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services proposes to conduct formative evaluation activities for the purpose of refining the implementation of a program that is being evaluated as part of the Next Generation of Enhanced Employment Strategies (NextGen) Project (OMB #0970-0545). ACF has contracted with Mathematica to conduct the NextGen Project.

ACF has spent decades studying strategies to help low-income people find and advance in jobs. Findings from these studies have been mixed, revealing variation in what works for whom and the duration and magnitude of impacts. The NextGen Project is intended to build on the findings and lessons learned from these past and ongoing evaluations by identifying and rigorously evaluating the “next generation” of employment strategies for highly vulnerable populations with complex barriers to obtaining and retaining employment.

One program being evaluated under the NextGen Project is Mental Health Outreach to MotherS (MOMS) Partnership. MOMS was designed to serve female caregivers with depressive symptoms and low income. It has been implemented in Connecticut, Vermont, Kentucky, and Washington, DC. For the NextGen Project, it will be implemented and evaluated in Springfield and Holyoke, Massachusetts. A nonprofit organization, Viability, will administer the program. The core of the MOMS Partnership model is a stress management course delivering cognitive behavioral therapy (CBT). The course includes eight weekly 90-minute group classes teaching practical skills for managing life stress and improving mood. These classes will be co-facilitated by a clinician and a staff person with lived experiences similar to the program participants. In addition, the MOMS Partnership as implemented in Massachusetts will offer participants the opportunity to meet with an employment coach, individually and in groups, in sessions called MOMS Moving Forward. Participants for the programs will be recruited from the Massachusetts Department of Transitional Assistance (DTA) and other agencies that serve mothers with low-income (such as Head Start, American Job Centers, and nonprofit organizations serving low-income populations).

The current data collection request is necessary to ensure high-quality implementation of the program while it is being evaluated. While the main features of the MOMS Partnership will not change, we expect that the details of its implementation will be refined in response to the data collection. We will encourage Viability, the program administrator, to continue these types of data collection independently after the formative evaluation is complete as part of continuous quality improvement of the program.

There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and use

This proposed information collection meets the following goal of ACF’s generic clearance for formative data collections for program support (0970-0531):

• Use of rapid-cycle testing activities to strengthen programs in preparation for summative evaluation.

The purpose of this data collection is to ensure that the evaluation of MOMS Partnership conducted as part of the NextGen Project produces useful information for ACF, Viability, other organizations planning to implement the MOMS Partnership program, and other stakeholders. This information collection will be used to refine implementation of the MOMS Partnership to ensure high-quality implementation of the program while it is being evaluated. ACF will use the information collected through surveys of participants and potential participants and focus groups of program staff and staff at referral agencies to work with MOMs Partnership staff to improve the implementation of program components. The evaluation will be a fairer test of whether MOMS Partnership is effective if the program is well implemented.

This information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

Study objectives

The ultimate objective of this requested formative data collection is to ensure that MOMS Partnership is well implemented when the program is being evaluated under the NextGen Project. The formative data collection will also help in refining the procedures for recruiting and enrolling study participants. Four research questions will be addressed:

1. Are there ways to make the orientation to MOMS more informative, engaging, and easy for the participants to attend?

2. Are there ways to make the Stress Management Course more informative and increase the likelihood of MOMS participants engaging in the Course?

3. Are there ways to make the MOMS Moving Forward sessions more informative and increase the likelihood of MOMS participants engaging in the sessions?

4. Are there ways to increase the number of eligible study participants who are referred to MOMS Partnership, who sign up for the study enrollment session, and who agree to participate in the study?

Study design

Starting in November 2021, Viability will implement a pilot of the MOMS Partnership program. They will provide an orientation, the MOMS Stress Management Course, and MOMS Moving Forward sessions to about 20 mothers who are likely eligible for MOMS Partnership. These participants will receive the MOMS Partnership services but will not be part of the summative evaluation (0970-0545; approved April 2020). In late December 2021, Viability will begin recruiting participants for the summative evaluation. Services will be provided to the study participants assigned to the research group that is offered a slot in MOMS Partnership beginning in January 2022.

The study design includes rapid learning cycles. Each cycle involves: collecting data on a program component, analyzing the data, and working with the program staff to refine implementation of the component. The cycles are repeated so that data is collected on the refined implementation of the component. We will examine four components: (1) the MOMS Partnership orientation; (2) Stress Management Course classes; (3) MOMS Moving Forward sessions; and (4) outreach, recruitment, and enrollment into the study.

The first data collection for the orientation, Stress Management Course, and MOMS Moving Forward sessions will occur within the first four weeks of the MOMS Partnership pilot in November 2021. A second cycle will occur in December 2021. Further cycles of these components will occur once the summative evaluation has begun in late December 2021. The first cycle to collect data on the outreach, recruitment, and enrollment into the study will begin in January 2022. We will conduct up to four cycles of each component. Hence, there will be several months in which we are both collecting data for the formative evaluation (this information collection) and the summative evaluation (0970-0545).

We will collect data on the program implementation from both program staff and participants and on recruitment from staff at referral agencies and potential study participants. The data collection is designed to be low burden.

After a cycle is concluded, the study team will analyze data collected and work with program staff to identify lessons learned and opportunities for additional improvement. Together, the study team and program staff members will refine the program implementation or recruitment efforts and plan for the next iterative learning cycle if needed.

Universe of data collection efforts

The current request includes ten data collection efforts (summarized in the table below). The efforts fall into four main categories:

1. Short surveys of MOMS Partnership participants (Instruments A, C, and E). For each of the four learning cycles, we will conduct short surveys of about 20 participants at the end of selected orientations, Stress Management Course classes, and MOMS Moving Forward sessions. For each learning cycle, we expect to administer the surveys at four or five orientations, two Stress Management Course classes (such as the first and the fourth in the series), and four or five MOMS Moving Forward sessions (including both virtual and in-person sessions). These surveys will be short and ask for assessments of the facilitation and logistics (such as place, time, day, and childcare) of the sessions and classes. The surveys will ask about the content of the orientation and MOMS Moving Forward sessions; the surveys will not ask about the content of the Stress Management Course classes as the curriculum cannot be changed before this evaluation.

2. Focus groups of MOMS Partnership staff (Instruments B, D, F, and J). For each of the four learning cycles, we will conduct focus groups with the staff facilitating the orientation, staff facilitating the Stress Management Course classes, MOMS Moving Forward coaches (who facilitate the MOMS Moving Forward sessions), and intake specialists who conduct recruitment, outreach, and the study enrollment interviews. We expect about eight participants to attend a focus group during each of the four cycles. At these groups, we will gather feedback on challenges staff see with the implementation of a specific program component and how it could be improved.

3. Surveys and focus groups of staff at referral agencies (Instruments G and I). For each of the four learning cycles, we will conduct short surveys of staff at referral agencies. For example, we expect to conduct periodic meetings of DTA staff and staff at other referral agencies in which we describe the study and the program. At the end of selected meetings, we will ask staff to respond to a quick survey about the content of the meeting. We expect about 60 survey respondents who will respond during each cycle. To understand more fully how we can improve our messaging, we will also conduct focus groups with about eight staff at DTA and eight staff at other referral agencies. These staff will attend a focus group during each of the four cycles.

4. Surveys of potential study participants (Instrument H). At regular intervals, MOMS Partnership intake specialists will present to groups of potential program participants who may be eligible for the program and study. For each of the four learning cycles, we expect to administer a short survey to these people to assess the clarity and efficacy of the messages. We expect to administer surveys to about 30 people in each cycle.

Data collection activity		Instruments		Respondent, content, purpose of collection		Mode and duration
Orientation
Survey of orientation attendees		Instrument A		Respondents: Attendees at orientation Content: Attendees perceptions of the orientation Purpose: To learn how to improve the orientation		Mode: Web Duration: 5 minutes
Focus group of orientation facilitators		Instrument B		Respondents: Orientation facilitators Content: Challenges faced and opinions of how it could be improved Purpose: To learn how to improve the orientation		Mode: Video conference Duration: 60 minutes
MOMS Partnership Stress Management Course
Survey of Stress Management Course attendees		Instrument C		Respondents: People who attended a Stress Management class Content: Satisfaction with facilitation and logistics of the classes Purpose: To learn how to improve the facilitation and logistics of the classes		Mode: Web Duration: 5 minutes
Focus Groups of Stress Management Course Facilitators		Instrument D		Respondents: Facilitators of the Stress Management Course classes Content: Satisfaction with facilitation and logistics of the classes and how they could be improved Purpose: To learn how to improve the facilitation and logistics of the classes		Mode: Video conference Duration: 60 minutes
MOMS Moving Forward Sessions
Survey of MOMS Moving Forward attendees	Instrument E		Respondents: Attendees at MOMS Moving Forward sessions Content: Satisfaction with content, facilitation, and logistics of the sessions Purpose: To learn how to improve the content, facilitation, and logistics of the sessions		Mode: Web Duration: 5 minutes
Focus Groups of MOMS Moving Forward coaches	Instrument F		Respondents: MOMS Moving Forward coaches and community mental health ambassadors Content: Satisfaction with content, facilitation, and logistics of the sessions and how they could be improved Purpose: To learn how to improve the content, facilitation, and logistics of the sessions		Mode: Video conference Duration: 60 minutes
Outreach, recruitment, and enrollment
Survey of staff at referral agencies	Instrument G		Respondents: Staff at DTA and other referral agencies Content: Response to messages and information provided about MOMS Partnership and the study Purpose: To improve the messages given to referral staff at presentations about MOMS Partnership and the study		Mode: Web Duration: 5 minutes
Survey of attendees at recruitment events	Instrument H		Respondents: Potential study participants Content: Potential study participants response to messages about MOMS Partnership and the study Purpose: To improve the messages given to potential study participants at presentations about MOMS Partnership and the study		Mode: Web Duration: 5 minutes
Focus groups of staff at referral agencies	Instrument I		Respondents: Staff at DTA and other referral sources Content: Perceptions of messages about MOMS Partnership and the study, experiences recruiting people, challenges faced when recruiting people and referring people to the study Purpose: To learn how to improve the process of recruiting study participants		Mode: Video conference Duration: 60 minutes
Focus groups of intake specialists	Instrument J		Respondents: Intake specialists Content: Process of recruiting, messages, reasons people do not show up to the enrollment interview, why people who show up to the enrollment interview do not enroll in the study Purpose: To learn how to improve the process of recruiting study participants and conducting enrollment interviews		Mode: Video conference Duration: 60 minutes

Other data sources and uses of information

The information collected by the surveys and focus groups will be supplemented with information we collect from the Random Assignment, Participant Tracking, Enrollment and Reporting system (RAPTER^®). As part of the NextGen study (0970-0545; approved April 2020), this system will collect information on how many study participants are enrolled in the study group that are able to participate in MOMS Partnership and whether participants attend an orientation, each Stress Management Course class, or a MOMS Moving Forward session as well as whether they meet one-on-one with a MOMS staff person.

A3. Use of information technology to reduce burden

The surveys will be available online. We will administer them in a way that is easy for the respondent. They will typically be administered via Menti or other presentation software at the end of meetings. This will just require that participants use their smartphone or be given access to a tablet. We may also email invitations to online surveys that can be completed using a smartphone, tablet, laptop, or desktop computer.

The focus groups will be conducted virtually using videoconferencing software. Conducting the focus groups virtually will reduce the need for participants to travel to participate in the groups.

A4. Use of existing data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

None of the data collection instruments ask for information that can be reliably obtained through other sources. As MOMS Partnership is a new program in Massachusetts, no other agency will collect this information. Other sources also could not provide the data in the time frame required to make refinements in the program and test again in preparation for a summative evaluation.

A5. Impact on small businesses

No small businesses will be involved in this data collection.

A6. Consequences of less frequent data collection

A key goal of the formative data collection is iterative testing of the implementation of MOMS Partnership. Under the rapid cycle testing design, we will collect data, refine the program’s implementation, and then test again. If we collected the data less frequently, the number of iterations and hence changes we could make to the program based on the data collected would be fewer. We will conduct no more than four iterations and we will stop iterating and collecting data if the data collected suggests that the component is working well.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register notice and comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for formative information collection. This notice was published on

October 13, 2020, Volume 85, Number 198, page 64480, and provided a sixty-day period for public comment. ACF did not receive any substantive comments.

Consultation with experts outside of the study

The project team did not consult with experts outside of the study for this information collection.

A9. Tokens of appreciation

There are no tokens of appreciation associated with this information collection.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally identifiable information

Little personally identifiable information will be collected on survey respondents. If the survey is completed using a presentation software such as Menti, the responses will not require that we obtain any contact information. If we need to email the online survey, we will ask for email addresses.

We will request names and email addresses of staff at Viability who administer the program as well as staff at referral agencies in order to organize the focus groups. Recordings of the focus groups will be stored on secure servers.

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.

Assurances of privacy

Prior to the start of any data collection, respondents will be informed about the purpose of the data collection, that their participation is voluntary, and that their information will be kept private. Mathematica will comply with all Federal and Departmental regulations for private information. Assurances will be made to survey and focus group participants before any data collection begins. We will obtain permission from all focus group participants before we record the session.

Data security and monitoring

Only staff from Mathematica will handle individual survey responses collected under this clearance. All Mathematica staff involved in the project receive annual training on (1) limitations of disclosure; (2) safeguarding the physical work environment; and (3) storing, transmitting, and destroying data securely. All Mathematica staff sign the Mathematica Confidentiality Agreement, complete online security awareness training when they are hired, and receive annual refresher training thereafter. Training addresses security policies and procedures found in the Mathematica Corporate Security Manual.

Only staff at Viability (the organization that administers MOMS Partnership) who are involved in the implementation of MOMS Partnership and staff at Mathematica involved in the study will view the focus group recordings.

Survey responses and recordings of focus groups will be downloaded to an encrypted project folder on Mathematica’s server. Mathematica uses access control lists to restrict access to the encrypted project folders where sensitive and confidential project data are stored. Access to the project folder is explicitly authorized by the Project Director on need-to-know and least privilege bases. Mathematica staff are required to change their password for computer and network access every thirty days, and passwords must adhere to strict composition standards. Staff access rights to the project folder are revoked when they leave the project. If a staff member leaves Mathematica, his or her access to computing assets, including network access, is terminated.

A11. Sensitive Information

There are no sensitive questions as part of these data collections.

A12. Burden

Estimated hours

The total number of burden hours is 255. The estimated hours for each instrument are provided in the table below. They are calculated as follows.

• Survey of orientation attendees. Twenty different orientation attendees in each of four cycles will respond to a 5-minute survey (20 x 5/60 x 4 = 7 hours).

• Focus group of orientation facilitators. Eight orientation facilitators will attend a one-hour focus group in each of four cycles (8 x 1 x 4 = 32 hours).

• Survey of Stress Management Course attendees. Twenty participants will respond twice to a 5-minute survey after different classes, repeated with different participants in each of four cycles (20 x 2 x 5/60 x 4 = 13 hours).

• Focus group of Stress Management Course facilitators. Eight Stress Management Course facilitators will attend a one-hour focus group in each of four cycle (8 x 1 x 4 = 32 hours).

• Survey of MOMS Moving Forward attendees. Twenty participants will respond twice to a 5-minute survey after different sessions, repeated with different participants in each of four cycles (20 x 2 x 5/60 x 4 = 13 hours).

• Focus group of MOMS Moving Forward coaches. Eight MOMS Moving Forward coaches will attend a one-hour focus group in each of four cycles (8 x 1 x 4 = 32 hours).

• Survey of staff at referral agencies. Sixty staff at referral agencies who attend meetings about MOMS Partnerships will respond to a 5-minute survey in each of four cycles (60 x 5/60 x 4 = 20 hours).

• Survey of attendees at recruitment events at referral agencies. Thirty different attendees at study recruitment events will respond to a 5-minute survey in each of four cycles (30 x 5/60 x 4 = 10 hours).

• Focus groups of staff at referral agencies. Eight DTA staff and eight staff from other referral agencies will attend a one-hour focus group in each of four cycles ( (8 + 8) x 1 x 4 = 64 hours).

• Focus groups of intake specialists. Eight intake specialists will attend a one-hour focus group in each of four cycles ( 8 x 1 x 4 = 32 hours).

Estimated annualized cost to respondents

To compute the total estimated annual cost, the total burden hours were multiplied by the estimated average hourly wage for program staff and participants. According to the Bureau of Labor Statistics’ Current Population Survey 2020, the median hourly wage for full-time community and social service specialists is $26.30. The hourly wage ($7.25) for participants and potential study participants is the federal minimum wage. The total annual burden cost is $5,887.35.

Instrument	Respondent	Total/ annual number of respondents	Number of responses per respondent	Average burden hours per response	Total/Annual burden hours	Average hourly wage	Total annual cost
Survey (A)	Orientation attendees	80	1	0.083	7	$7.25	$50.75
Focus group (B)	Orientation facilitators	8	4	1.000	32	$26.30	$841.60
Survey (C)	Stress Management Course attendees	80	2	0.083	13	$7.25	$94.25
Focus group (D)	Stress Management Course facilitators	8	4	1.000	32	$26.30	$841.60
Survey (E)	MOMS Moving Forward attendees	80	2	0.083	13	$7.25	$94.25
Focus group (F)	MOMS Moving Forward coaches	8	4	1.000	32	$26.30	$841.60
Survey (G)	Staff at referral agencies	60	4	0.083	20	$26.30	$526.00
Survey (H)	Attendees at recruitment events	120	1	0.083	10	$7.25	$72.50
Focus group (I)	Staff at referral agencies	16	4	1.000	64	$26.30	$1,683.20
Focus group (J)	Intake specialists	8	4	1.000	32	$26.30	$841.60
Estimated annual burden total					255		$5,887.35

A13. Costs

There are no additional costs to respondents.

A14. Estimated annualized costs to the federal government

The total estimated cost for the federal government for the data collection activities under this current request will be $25,400. This includes personnel effort plus other direct and indirect costs.

Cost category	Estimated costs
Instrument development and OMB clearance	$5,000.00
Data Collection	$15,400.00
Analysis and PowerPoint describing findings	$5,000.00
Total costs over the request period	$25,400.00

A15. Reasons for changes in burden

This is an individual information collection request under generic clearance 0970-0531.

A16. Timeline

Beginning November 1, 2021, pending OMB approval of this request, Viability will pilot the MOMS Partnership program with about 20 participants. These participants will not be part of the summative evaluation as part of the NexGen project (0970-0545). The first two learning cycles of the orientation, Stress Management Course, and MOMS Moving Forward will occur in November and December 2021, respectively. Subsequent cycles will occur in early 2022. The four learnings cycles for the outreach, recruitment, and enrollment will begin in January 2022.

A17. Exceptions

All instruments will display the expiration date for OMB approval. No exceptions are necessary for this information collection.

Attachments

Instrument A: Survey of orientation attendees

Instrument B: Protocol for focus group of orientation facilitators

Instrument C: Survey of Stress Management Course attendees

Instrument D: Protocol for focus Groups of Stress Management Course Facilitators

Instrument E: Survey of MOMS Moving Forward attendees

Instrument F: Protocol for focus groups of MOMS Moving Forward coaches

Instrument G: Survey of staff at referral agencies

Instrument H: Survey of attendees at recruitment events

Instrument I: Protocol for focus groups of staff at referral agencies

Instrument J: Protocol for focus groups of intake specialists