Instrument 4. Focus Group Protocol for Model Developers_clean

Focus Group Protocol:

Model Developers

Introduction and Consent Script (5 minutes)

We are working with the Administration for Children and Families (ACF), Office of Planning, Research, and Evaluation, and the Health Resources and Services Administration (HRSA). Both these agencies sit within the U.S. Department of Health and Human Services. We are part of a team exploring the possibility of updating the HHS Criteria for Evidence-Based Early Childhood Home Visiting Models, the criteria HHS uses to determine if early childhood home visiting models are evidence-based.

As part of this effort, we are engaging experts in the field to gather input on how well the current criteria are working and priorities for improving them. Along with model developers, we intend to speak with MIECHV administrators, experts in evidence-based policy, and home visiting model developers, researchers, and advocates.

Before we begin the focus group, we’d like to do two things: first, we want to be sure that you are aware of your rights to participate – or not participate – in this focus group and the efforts that we will make to protect your privacy to the greatest extent possible. Second, we want to give a brief overview of the evidence criteria that we are considering updating to ensure we are all on the same page about the task at hand for the government.

First, a word about your rights. A Federal agency may not conduct or sponsor, and no individual or entity is required to respond to, nor shall an individual or entity be subject to a penalty for failure to comply with a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless that collection of information displays a currently valid OMB Control Number. The OMB # for this effort is 0970-0531 and the expiration date is 9/30/2025.

Second, it is important that you know how we intend to use the information we collect from you here. We do not intend to release a public report describing the findings of this or other focus groups or engagements with experts. The information gathered here will be used to inform ACF and HRSA. For internal reports and discussions, any results we share from this focus group will be summarized across what we hear from you and the others in this conversation. You will not be identified by name in any form of analysis or report, and data will be reported in a de-identified manner. In addition, when we need to use quotes to help illustrate the findings, we will only identify them by role of the participant (such as, “evidence-based policy expert”). In some cases, our team may follow up with you to request additional feedback about your comments. All data collected for the review will be stored in secure environments, and we will protect the privacy of the information you provide. Finally, your participation in this focus group is voluntary, and you may decline to respond at any time.

We would like to record this conversation to ensure we accurately capture your comments. If you agree, we will retain the recording only until we can validate our notes, at which point we will destroy the audio recording. Since we are in a group setting today, if you are not comfortable being recorded at this time, we ask that you exit the call, and we will follow up with you at a later time. At this time, I will pause to make sure everyone we have with us agrees to allow us to record the focus group. Please do so by typing “yes” in the Zoom chat or stating “yes” verbally.

At this time, I am going to turn on a recording to capture consent for participation in this focus group and recording.

[Turn on recording]

[Facilitators to make note of anyone leaving the call, confirm that all remaining have consented]

Today, our main topics for discussion will be about the HHS Criteria for Evidence-Based Early Childhood Home Visiting Models, which are the criteria that the U.S. Department of Health and Human Services have outlined to determine which home visiting models are considered evidence-based for the Maternal, Infant, and Early Childhood Home Visiting – or MIECHV -- program. These criteria have implications for how states and jurisdictions can spend their MIECHV program dollars. Our plan is to give you an overview of these criteria – including how they were developed and how they are operationalized in a systematic evidence review. We will then facilitate a dialogue with you to get your thoughts on the strengths and challenges with these criteria and their implications for model developers like yourselves. The criteria were developed more than a decade ago, and we want to be sure that they evolve as the broader fields of home visiting and evidence-based policy evolves.

We have scheduled 60 minutes for this focus group. We would like everyone to feel comfortable sharing open feedback, so we ask that everyone here hold our conversation in private. We will be using a tool called “PollEverywhere” to ask interactive, polling-style questions; responses on PollEverywhere cannot be tied to you and are only used to inform our conversation. We also welcome you to use the Zoom chat to type comments in addition to speaking. Before we get started, I would like to allow you all to ask any questions you have about our work or the plan for our time together today.

[Address any comments or questions]

Questions and Prompts

Section 1: Introductions (7 minutes)

To start, we’d like to spend a few minutes getting to know you all and learning about your familiarity with MIECHV and components related to the determination of evidence-based home visiting models in MIECHV.

Topic	Home Visiting Model Developers Questions & Prompts
Introductions	Could you please tell us your name and if HomVEE designates your model as “evidence-based” according to the HHS Criteria? If not, are you working towards receiving that HomVEE designation? [Facilitator to call name of individuals from the list]
Assessing familiarity with MIECHV evidence criteria, statutes, and evidence reviews	We’d like to use a poll to quickly get some information about how familiar you are with MIECHV, the HHS evidence criteria, and the systematic review used in MIECHV (called HomVEE). We have shared a link to a website called PollEverywhere in the chat. You can open this link; [PollEverywhere multiple choice style question:] How familiar are you with the MIECHV federal home visiting program? Not at all familiar A little familiar Somewhat familiar Very familiar How familiar are you with the HHS Criteria for evidence-based early childhood home visiting program? Not at all familiar A little familiar Somewhat familiar Very familiar How familiar are you with the Home Visiting Evidence of Effectiveness evidence review (HomVEE?) Not at all familiar A little familiar Somewhat familiar Very familiar

Section 2: Overview of HHS Criteria for Evidence-Based Early Childhood Home Visiting Models (10 minutes)

Before we begin our discussion, we want to be sure that everyone has some basic information about the content of the HHS Criteria.

There are four terms that we feel are important to clarify before we begin. (Show slide with this language.) These are:

• Maternal, Infant, and Early Childhood Home Visiting Program, called MIECHV for short. The MIECHV Program supports home visiting for expectant and new parents with children up to kindergarten entry age who live in communities that are at-risk for poor maternal and child health outcomes. By law, state, tribal, and territory awardees must spend the majority of their MIECHV Program grants to implement evidence-based home visiting models, with up to 25 percent of funding available to implement promising approaches that will undergo rigorous evaluation. MIECHV’s authorizing language includes some specific requirements for evidence-based home visiting models. This language serves as a foundation for HHS evidence criteria we will be discussing today.

• HHS Criteria for Evidence-Based Early Childhood Home Visiting Models, which we will refer to as the HHS Criteria for short – This is the core of our conversation today. It is the language that HHS has issued to indicate what evidence is required for home visiting models to be considered evidence-based. We will share the specific language of these criteria in the next slide.

(Show next slide) On this slide, we present the specific language of the HHS Criteria

“To meet HHS criteria for an “evidence-based early childhood home visiting service delivery model,” models must meet at least one of the following criteria:

• At least one high- or moderate-quality impact study of the model finds favorable, statistically significant impacts in two or more of the eight outcome domains.

• At least two high- or moderate-quality impact studies of the model using non-overlapping analytic study samples find one or more favorable, statistically significant impacts in the same domain.

• In both cases, the impacts must either (1) be found in the full sample or (2) if found for subgroups but not for the full sample, be replicated in the same domain in two or more studies using non-overlapping analytic study samples.

Additionally, following the MIECHV-authorizing statute, if the model meets the above criteria based on findings from randomized controlled trial(s) only, then two additional requirements apply. First, one or more favorable, statistically significant impacts must be sustained for at least one year after program enrollment. Second, one or more favorable, statistically significant impacts must be reported in a peer-reviewed journal.”

• Home Visiting Evidence of Effectiveness – Called HomVEE for short. This is the systematic review that HHS funds to review the evidence for individual home visiting models to see whether they align with the HHS Criteria. HomVEE has detailed standards – documented in a nearly 200-page handbook – that operationalize the HHS Criteria into far more specific details. For instance, the HHS Criteria say “high- to moderate-quality impact studies”, while the HomVEE review operationalize those in terms of features of the study design and execution required to meet high or moderate quality.

The reason we wanted to share this with you is so that you see the criteria themselves, but also understand the factors influencing the HHS Criteria – including most important, the MIECHV authorizing language – and also understand the distinction between the HHS Criteria and how they are operationalized through HomVEE. Since you do not all have deep expertise in MIECHV, the HHS Criteria or HomVEE, our conversation today will focus more generally on best practices in evidence-based policymaking.

Any questions about these terms or MIECHV before we begin?

[Answer questions]

Section 3: Perceptions of HHS Evidence Criteria (30 minutes)

As we noted, the HHS Criteria have significant implications for both how the evidence review – HomVEE – is structured and how states and jurisdictions can spend their MIECHV grant dollars. The purpose of this update is to understand how well the current standards align with the newest information about home visiting and the best practices in evidence-based policymaking and – if necessary – update the criteria to better align to those practices. Below, we’d like to ask you some questions about your general perspectives on the evidence requirements in MIECHV as well as a few specific areas of the HHS Criteria that we will be exploring.

Topic	Home Visiting Model Developers Questions & Prompts
General reactions to MIECHV evidence requirements	Before we dive into some specific questions about the HHS Criteria, we’d like to get your sense of the strengths and challenges of the evidence approach used in MIECHV. What do you think works well in this approach? What are some challenges? [Prompt if necessary]: What has your experience been that lead you to identify those things are strengths/challenges?
Technical issues: Meaningfulness of impacts	Another key issue is how to ensure that the outcomes in evaluations are meaningful. There are three key issues here for the HHS Criteria, as shown on this slide related to the number of impacts in different domains, the magnitude of impacts, and the duration of impacts. As you can see on the slide, the current criteria state that evidence-based models must have statistically significant impacts in two of eight measured domains OR a replicated impact at least one domain, and for RCTs at least one impact must be sustained for at least a year. There are no specifications about the magnitude of the impacts. What are your reflections on these criteria? Are there changes you would recommend and why?
Technical issues: Subgroups	Another issue evidence reviews face is how to consider impacts on different populations or different contexts. Currently, evidence determinations in the HHS Criteria can be based on full samples or on replicated findings for subgroups. What are your reactions to this approach? As a model developer, are there challenges these criteria pose as you are considering developing or testing your models?
Technical issues: Tiered evidence ratings	The MIECHV statute requires the distinction of evidence-based home visiting models. It also includes some specific language about how that distinction should be made, including having findings of positive outcomes in specific domains from “well-designed and rigorous randomized control trials and quasi-experimental studies.” The current HHS Criteria reflect these statutory requirements and create a binary rating; in other words, the review leads to a determination of whether a model is evidence-based versus not. However, the HomVEE review itself does not have to be binary in its rating. For instance, other evidence reviews sometimes have multiple rating levels – for example, the Prevention Services Clearinghouse has four levels of evidence: well-supported, supported, promising, or does not currently meet criteria. What is your perspective on a multi-tiered rating approach? In what ways would this be helpful or unhelpful? And what aspects of a model or its evaluations would you recommend be considered in designing tiers? Prompt: If a tiered system were to be used, what factors should be considered to distinguish those tiers? For instance, how much should they consider study quality, replication of findings, reach to different populations, for instance? And why?

Section 4: Final Reflections and Wrap-Up (8 minutes)

We have a few minutes remaining together. As we wrap up, I’d like to ask for your final thoughts.

Topic	TTA Provider Questions & Prompts
General Reflections	We have discussed several specific technical issues relevant to the HHS Criteria. As we wrap up for today, I’d like to ask each one of you if you have any other thoughts about updating the HHS evidence criteria and its implications for model developers?

Those are all questions we have for today. We want to thank you all for taking the time to participate. We appreciate your insights and comments.

1