Strengthening the Implementation of Marriage and Relationship Programs (SIMR)

Appendix D:

SIMR Phone Interview Protocol

What is SIMR? What is the purpose of our call?

We’re contacting you on behalf of the Strengthening the Implementation of Marriage and Relationship Programs (SIMR) study. SIMR is a new evaluation sponsored by the federal government to identify and test strategies that could improve the delivery and effectiveness of healthy marriage and relationship education (HMRE) programs. As you may know, since 2005, there has been dedicated federal program funding for organizations that provide HMRE services to youth, adult individuals, and couples. Although there is a growing body of research on these efforts that we can learn from, the evidence about which program approaches are most effective is still limited. The Office of Planning, Research, and Evaluation of the Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services awarded Mathematica and its partner Public Strategies a contract to test new approaches.

We are calling today because we’re in a preliminary stage of learning about HMRE grantees to inform the SIMR study. We learned about your program through your HMRE grant application. We are interested in learning more about your program especially because [TAILORED BASED ON APPLICATION REVIEW]. Over the next 60 minutes, we would like to walk through the questions we provided before the meeting. The information you provide during this interview will help us determine if the approaches we are considering for the SIMR study fit with your program. Once we complete interviews with other programs, we will review the information with ACF and reach out to a subset of programs to learn even more in a next phase. The next phase involves a series of virtual meetings to identify potential solutions that may resonate with each grantee and get input on a preliminary research design. We will again review the information from these virtual meetings with ACF to help inform future study directions.

Description of SIMR

SIMR will use rapid-cycle learning to pilot and refine promising solutions to implementation challenges faced by HMRE programs. Rapid-cycle learning involves testing and refining solutions in a series of short cycles—to pilot the solution, collect and analyze data on the pilot, refine the solution based on what we learned, and try it again. Our team has already conducted an extensive process to identify common implementation challenges as well as promising solutions to address them. Our aim in this phase is to identify strong partners from the HMRE grantees that could benefit from one or more of the identified solutions (which we will go over later) and that have an interest in testing and learning about their effectiveness. As part of this study, the SIMR team will work closely with program partners to pilot and refine the solutions. Additionally, participation in SIMR might facilitate future participation in a large federal evaluation of HRME programs.

How will SIMR benefit HMRE programs?

The information gained from this study will likely help all HMRE programs address implementation challenges. This may lead to improved program delivery and better participant outcomes. For programs that are selected to participate, there will be additional benefits as well.

Being part of SIMR means being part of a partnership. If selected, we would work closely with the staff at the selected program to help make their HMRE program even better. Program staff would work alongside our project team members to try out and refine innovative solutions to address implementation challenges and strengthen your program. We would start with piloting these new solutions to see how they go. We would then see what staff think of them and whether we need to refine things a bit. We would then work with staff to refine the solutions. Later, we would start collecting data on how the solutions are working. Please note that SIMR participants are not randomly assigned to a group that will not receive services. However, they may be randomly assigned to receive certain aspects or services that we, as a team, agree to change as part of SIMR.

For programs selected to participate in SIMR, there are several benefits, including the opportunity to:

• Improve the program by focusing on a key implementation challenge and how to address it

• Create a tailored approach to the circumstances and contexts of the program, developed through a collaborative process where program staff provide input on strategies likely to work best

• Work with SIMR project staff who are highly knowledgeable about HMRE programming and receive technical assistance from them, including training on how to implement the solutions

• Receive financial support from the SIMR project for implementing the solutions

• Contribute to the HMRE field and help us better understand how to address common implementation challenges

What questions do you have about SIMR?

Next, we would like to ask some specific questions about your organization and HMRE program.

Organization structure

[Note: Before asking each question, determine if it is necessary by reviewing the grant application and seeing whether the information provided is missing or unclear.]

• Tell us about your organization. How long has your organization been operating? How many programs do you operate and how many participants do you serve across all programs each year?

• What other types of programs/services are provided within the organization?

• [For returning grantees] How long has your organization been providing HMRE programming?

• [For new grantees] Has your organization managed federal grants in the past? Have you ever received an ACF grant in the past? For what kind of programming?

Program features

[Note: Before asking each question, determine if it is necessary by reviewing the grant application and seeing whether the information provided is missing or unclear.]

• [Review the components of the HMRE program]. Have you made any changes from what you described in your grant application? Probes: Added any components? Changed the delivery format? Eliminated any components?

• How does a participant move through the program? How are the components sequenced?

• What is the delivery format for each component offered through the HMRE program (for example, workshop sessions or case management)? Is it cohort-based or rolling enrollment?

• How often do clients receive HMRE programming (for example, daily, weekly, monthly)? How long is the program (in number of weeks, sessions, months, etc.)?

• What curricula or program models are you using? What adaptations or changes have you made to them? Who did you work with to make these changes?

• In your grant application, you noted [TARGET POP] as your target population. Is this still accurate? Do you have formal criteria about who can participate (for example, must be 18–25, parenting status, relationship status, income level, special status, employment status)? What are those criteria?

• [Confirm projected enrollment numbers.] Describe your planned recruitment strategies and process. Probe: How will you identify potential participants? How will you advertise for the program? How will you screen and enroll the participants?

• How did you calculate the number you can serve? Based on experience? Community data? Info from another program?

• From your application, it sounds like you’re delivering the program in a [school/community/other] setting. Is that correct? Is the program delivered in multiple locations or organizations? Who are your key partners?

• Do similar kinds of programs exist in your community? In what ways is the HMRE program unique? How is it different from other relationship programming in the community?

• What additional funding sources do you have to support your HMRE services, beyond the ACF grant?

Program reflections

[Note: Before asking each question, determine if it is necessary by reviewing the grant application and seeing whether the information provided is missing or unclear.]

• [For returning grantees] What was the most challenging aspect of providing services in your past HMRE program? [If necessary, ask about the implementation areas of recruitment, retention, and content engagement.]

• [For new grantees] What was the most challenging aspect of providing services in a past program? [Ask them to consider a program that is most like an HMRE program, and/or one that served a similar population. If necessary, ask about the implementation areas of recruitment, retention, and content engagement]

• What do you usually do when you run into a problem with a program you’re running—say you’re not hitting enrollment targets, or you’re having issues with attendance, or your participants are not responding as you anticipated to some part of your program? What’s your typical approach to addressing those kinds of challenges? What are the key challenges are you anticipating with the HMRE program you are currently launching?

Data and evaluation

[Note: Before asking each question, determine if it is necessary by reviewing the grant application and seeing whether the information provided is missing or unclear.]

• Tell us about your processes for performance monitoring. Probe: What data do you collect? Who collects them? What do you do with the data? How do you use the findings?

• Do you have a process for program improvement or a plan for regularly using data to check how things are going with your program? In other words, do you have a way that you plan to use data to find potential implementation issues and help you address them? Probe: What data do you collect? Who collects them? What do you do with the data? How do you use the findings?

• Do you have a designated data person on staff? If not, who oversees data collection and management?

• Has your organization participated in an evaluation previously? Did you partner with an external evaluator? Tell me more about your experience working with your evaluator. Probes: Was it collaborative, or did the evaluator work independently of your program? Was it a positive experience for your program? What did you learn from the evaluation that was useful to your organization?

• Tell us about any plans for a local evaluation. Probes: What are the key research questions for your local evaluation? What is the timeline? What data are you collecting? What are your goals?

Approaches to Address Implementation Challenges

We have identified seven approaches to evaluate in SIMR. These priority approaches offer a general strategy to address common implementation challenges of HMRE programs. Each approach contains a set of more-specific solutions that programs can implement and tailor to their individual context. The approaches are:

1. Deliver services virtually: This approach is designed to develop and offer options for virtual service delivery. The solutions in this approach all leverage technology to assist with implementation challenges. Example solutions include:

1. Providing online intake and assessments

2. Offering virtual content to improve retention

3. Using cell phones to encourage engagement

1. Develop relationships with and between participants: This approach seeks to build relationships between staff and participants, among participants, and within the organization as a whole. Example solutions include:

1. Fostering peer relationships through positive group dynamics and peer learning

2. Establishing protocols for and providing training on how to build and nurture relationships

3. Establishing a trusting relationship with the organization through the establishment of a safe and welcoming environment

3. Improve recruitment effectiveness: The goal of this approach is to improve the effectiveness of recruitment strategies to increase enrollment. The solutions are designed to enhance recruiter skills and program messaging. Example solutions include:

1. Building staff knowledge of communities and participants through training and other supports

2. Training staff on marketing strategies and how to describe program services in ways that will increase take-up

3. Using current and former participants as part of recruitment efforts

4. Engaging recruitment partners

4. Provide facilitators with training and support: This approach seeks to enable facilitators to better relate to, connect with, and engage participants by improving facilitation quality. Example solutions include:

1. Training facilitators on group or discussion management techniques

2. Training staff on relevant service delivery approaches, such as trauma-informed service delivery

3. Fostering facilitator collaboration with and support from other program staff (such as case managers)

5. Remove barriers to enrollment and participation: This approach focuses on improving the identification of participants’ needs and challenges, enhancing the types of incentives and supports that programs provide to them (such as basic needs referrals or transportation vouchers), and increasing the flexibility of the program structure. Example solutions include:

1. Improving the identification of participants’ needs and challenges

2. Enhancing the types of incentives and supports that programs provide to them (for example, basic needs referrals or transportation vouchers)

3. Increasing the flexibility of the program structure (for example, scheduling of workshops)

6. Use data to drive decisions and make program improvements: This approach is designed to create tools and systems to collect and track data to drive program improvements and delivery. It seeks to enhance program improvement efforts by collecting and using quantitative and qualitative data. Example solutions include:

1. Leveraging the data programs already collect

2. Collecting additional data from participants, partner organizations, grantee staff, and the community

3. Using data to assess service and partnership quality

4. Using data to streamline intake and refine incentive policies

7. Use participant-centered strategies: This approach is designed to enhance the relevance of program content and services for participants. It focuses on obtaining information and guidance from participants and other stakeholders (for example, community leaders) to enhance the program. Example solutions include:

1. Obtaining information and guidance from participants and other stakeholders

2. Using feedback to inform program design

3. Training staff on service delivery approaches that promote a more participant-centered delivery (for example, trauma-informed delivery)

4. Providing clarity around allowable adaptations to and tailoring of curriculum content

Solutions range from developing protocols and procedures, training staff, and implementing strategies such as case management or facilitation techniques and tools. We can go over these in a future meeting if we decide to reach out to you to learn even more in the next phase or information gathering. At that point, we would work with you to think through more tailored potential solutions to common implementation challenges your program is facing.

Which of these approaches might be of interest to you, and why? Might your staff be interested in trying these solutions in your programs?

If selected to participate in SIMR, we would work with program staff to develop a tailored approach to strengthening program implementation. We would then continue working together to implement and test the approach.

Testing the approach would involve implementing the solutions on a small scale, then proceeding in rounds we call “learning cycles.” We would begin by working with a few staff members to test the feasibility of implementing the approach. This early work would rely heavily on staff input and on data collected through the nFORM system. If the results show promise, in the next cycle, we would gather additional data from staff and participants to get a better sense of how the solutions are working. If the additional data continue to suggest that the solution shows promise, in the next round we could conduct a small-scale random assignment study to confirm that the solution is improving implementation in the way it was intended. All participants would still receive services. We would use a computer to randomly place people into different versions of services, such as a version where they receive additional reminders about attendance. Throughout this process, we would work closely with program staff to plan each cycle and review the data from the previous cycle to plan the next one. The SIMR team would also conduct site visits to collect data and meet with staff.

Do you have any questions or thoughts about how this would work? What about this process would work well in your program? What might be challenging?

Next steps

We will review the information from this interview with ACF and determine which grantees will continue to the next phase of information gathering to inform the SIMR research design. The next phase involves a series of virtual meetings to identify potential solutions grantees may test as part of SIMR and get input on preliminary research designs. You will hear from us in the following weeks to schedule these meetings. We will again review the information from these virtual meetings with ACF and make a final selection and formal invite selected grantees to participate in SIMR.

Do you have any additional questions about the overall process or goals for SIMR?

NOTE: The Paperwork Reduction Act Statement: This collection of information is voluntary and will be used to gather preliminary information about the healthy marriage-relationship education (HMRE) field and explore with HMRE programs the research questions that are of interest and the design options that are feasible. Public reporting burden for this collection of information is estimated to average 60 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: 0970-0356, Exp: 06/30/2021. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Robert Wood; RWood@Mathematica-mpr.com