SIMR is a new evaluation sponsored by the federal government. It is designed to identify and test strategies for improving the delivery and effectiveness of healthy marriage and relationship education (HMRE) programs. As you may know, since 2005, dedicated federal program funding has been available for organizations that provide HMRE services to youth, adult individuals, and couples. Although there is a growing body of research on these efforts that we can learn from, evidence about which program approaches are most effective is still limited. The Office of Planning, Research, and Evaluation of the Administration for Children and Families in the U.S. Department of Health and Human Services awarded a contract to Mathematica and its partner Public Strategies to test new approaches. SIMR will use rapid-learning cycles to pilot and refine promising solutions. Rapid-learning cycles involve testing and refining solutions in a series of short cycles—to pilot the solution, collect and analyze data on the pilot, refine the solution based on what we learned, and try it again. Our team has already conducted an extensive process to identify common implementation challenges and promising solutions to address them.
Our aim in this phase is to conduct a series of virtual meetings to identify potential solutions that may resonate with each grantee and get input on a preliminary research design. We will review the information from these virtual meetings with ACF to help inform future study directions. Ultimately, the SIMR team will work closely with program partners to pilot and refine the solutions. Additionally, participating in SIMR might help programs participate in a future, large federal evaluation of HRME programs
By the conclusion of the webinars, Mathematica and individual grantees will:
Identify priority implementation challenges
Identify approaches that could address the challenges
Tailor the approach to the grantee and identify a rapid-learning design to evaluate it
Understand the capacity and feasibility to test potential solutions
Teams of three from Mathematica and Public Strategies will facilitate a series of site-specific virtual sessions with up to 12 HMRE grantees. Staff from the grantee will include the project and organizational leader and up to four frontline staff or managers who are knowledgeable about the participants, the program, and the services provided.
The series of meetings with each grantee team will take place virtually using WebEx, a web-based videoconferencing platform. We will use Mural, a virtual whiteboard, to facilitate the activities for each session and take notes.
Introductions/overview of SIMR (10 minutes)
Study overview and goals
Study design and timeline
Q & A
Identify challenges (70 minutes)
We will start with an activity designed to identify challenges and areas of opportunity for improvement. Grantee staff will generate a list related to their HMRE program of things that they are doing (or have done) well or are proud of (for example, services, practices, processes). They will then generate a list of opportunities for improvement or things they think are promising but need more thought. Finally, they will generate a list of challenges (or anticipated challenges). They will conduct this activity for each of the implementation areas: recruitment, retention, and content engagement.
For new grantees, who may struggle to anticipate challenges, we will prepare scenarios to help contextualize potential challenges. An example scenario might be, “An organization that is your primary referral source closes. In what ways does this affect you, and what challenges do you foresee in adapting to this closure?”
We will then focus on opportunities and challenges. We will conduct activities to cluster common ideas together and vote on those that are most important to the staff and likely to make an impact on program implementation.
Preview solutions (10 minutes)
Next, we will share the goal of the second webinar, which is to find solutions to the challenges identified during the current webinar. We will preview the approaches and send grantees a list of seven for their review prior to the second webinar. We will ask the grantee staff to send their top two choices before the second webinar.
In between sessions, the SIMR team will review identified challenges and find 2–3 approaches to address the challenges. We will compare our choices with those of the grantee.
Review top approaches (10 minutes)
We will review the top choices, ask grantee staff to explain the rationale behind their choices, and discuss the rationale behind ours. We will then reach consensus on the two approaches to focus on.
Prioritize solutions (60 minutes)
We will review the solutions in each approach and have attendees identify and rank the most promising solutions within each approach. We will aim to identify fewer than 12 solutions across the two approaches.
For each solution identified, we will engage in an interactive activity where attendees place the solution along a continuum, with potential impact the y-axis and perceived effort the x-axis. For example, the attendees will consider the potential impact the solution will have on their program and its implementation—with the top part of the axis being high impact and the bottom low impact. They will then assess the perceived effort necessary to implement that same solution—with the right portion being high effort and the left low effort. This activity will provide information from the grantees’ perspective about concerns with implementing solutions specific to their site/context, as well as resources they would need for successful implementation.
We will then focus on the more feasible solutions (for example, higher-impact solutions, with low-to-moderate effort compared to other solutions). We will ask grantees the following questions to explore their capacity to support implementation of solutions.
How would potential solutions fit with the overall program implementation plan?
How might the solutions need to be adapted to the target population, to subpopulations, or to various staff types (for example, case managers vs. facilitators vs. recruiters)?
What specific resources are needed to implement the solutions?
Discuss rapid-learning cycles (20 minutes)
We will briefly introduce the rapid-learning process and share what a potential cycle might look like. At the conclusion, we will ask, what challenges or barriers do you see in implementing this process with the solutions we identified?
In between sessions, the SIMR team will review the identified solutions and create a preliminary plan for implementation and testing.
Review solutions to implement and review rapid-learning design (90 minutes)
We will review a potential preliminary plan including the solutions to be tested and the proposed rapid-learning design.
We will then have a Q&A session with the grantee about implementing the solution and executing the rapid-learning cycles. The goal will be to get feedback and input on how to implement the solutions and identify any issues related to rapid-learning design.
We will also discuss how SIMR research may align with or disrupt any local evaluations.
Then we will discuss the resources that would be needed and expectations of grantees who participate in the study.
NOTE: The Paperwork Reduction Act Statement: This collection of information is voluntary and will be used to gather preliminary information about the healthy marriage-relationship education (HMRE) field and explore with HMRE programs the research questions that are of interest and the design options that are feasible. Public reporting burden for this collection of information is estimated to average 90 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: 0970-0356, Exp: 06/30/2021. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Robert Wood; RWood@Mathematica-mpr.com
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Mathematica Memo |
| Author | Daniel Friend |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |