Feedback on the HomVEE Criteria for Evidence-Based Early Childhood Home Visiting Models

Focus Group Protocol:

Tribal MIECHV Experts

Introduction and Consent Script (5 minutes)

Hello and thank you for joining this focus group today. My name is [NAME OF FACILITATOR], and I’m joined by my colleagues [NAME OF NOTETAKER AND ANY PROJECT STAFF]. We are working with the Administration for Children and Families (ACF), Office of Planning, Research, and Evaluation, and the Health Resources and Services Administration (HRSA). Both these agencies sit within the U.S. Department of Health and Human Services. We are part of a team exploring the possibility of updating the HHS Criteria for Evidence-Based Early Childhood Home Visiting Models, the criteria HHS uses to determine if early childhood home visiting models are evidence based.

As part of this effort, we are engaging experts in the field to gather input on how well the current criteria are working and priorities for improving them. Along with Tribal MIECHV experts, we will talk with state administrators, experts in evidence-based policy, and home visiting model developers, researchers, and advocates.

Before we begin the focus group, we’d like to do two things: first, we want to be sure that you are aware of your rights to participate – or not participate – in this focus group and the efforts that we will make to protect your privacy to the greatest extent possible. Second, we want to give a brief overview of the evidence criteria that we are considering updating to ensure we are all on the same page about the task at hand for the government.

First, a word about your rights. A Federal agency may not conduct or sponsor, and no individual or entity is required to respond to, nor shall an individual or entity be subject to a penalty for failure to comply with a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless that collection of information displays a currently valid OMB Control Number. The OMB # for this effort is 0970-0531 and the expiration date is 9/30/2025.

Second, it is important that you know how we intend to use the information we collect from you here. We do not intend to release a public report describing the findings of this or other focus groups or engagements with experts. The information gathered here will be used to inform ACF and HRSA. For internal reports and discussions, any results we share from this focus group will be summarized across what we hear from you and the others in this conversation. You will not be identified by name in any form of analysis or report, and data will be reported in a de-identified manner. In addition, when we need to use quotes to help illustrate the findings, we will only identify them by role of the participant (such as, “evidence-based policy expert”). In some cases, our team may follow up with you to request additional feedback about your comments. All data collected for the review will be stored in secure environments, and we will protect the privacy of the information you provide. Finally, your participation in this focus group is voluntary, and you may decline to respond at any time. Your funding from HHS will not be affected by your decision to participate or not participate.

We would like to record this conversation to ensure we accurately capture your comments. If you agree, we will retain the recording only until we can validate our notes, at which point we will destroy the audio recording. Since we are in a group setting today, if you are not comfortable being recorded at this time, we ask that you exit the call, and we will follow up with you at a later time. At this time, I will pause to make sure everyone we have with us agrees to allow us to record the focus group. Please do so by typing “yes” in the Zoom chat or stating “yes” verbally.

At this time, I am going to turn on a recording to capture consent for participation in this focus group and recording.

[Turn on recording]

[Facilitators to make note of anyone leaving the call, confirm that all remaining have consented]

Today, our main topics for discussion will be about the HHS Criteria for Evidence-Based Early Childhood Home Visiting Models, which are the criteria that the U.S. Department of Health and Human Services have outlined to determine which home visiting models are considered evidence-based for the Maternal, Infant, and Early Childhood Home Visiting – or MIECHV -- Program. These criteria have implications for how states and jurisdictions can spend their MIECHV Program dollars. The criteria were developed more than a decade ago, and we want to be sure that they evolve as the broader fields of home visiting and evidence-based policy evolve.

Our plan is to give you an overview of these criteria – including how they were developed and how they are operationalized in a systematic evidence review. We will then facilitate a dialogue with you to get your thoughts on the strengths and challenges of these criteria and their implications for MIECHV and other home visiting programs in the U.S.

We want to be clear about our intention for using the information we gather in this focus group and through other similar engagements. MIECHV was designed to support the use of evidence-based home visiting models. That is not something that will change from this process. However, we will be asking questions that get at how well the current approaches to identifying and determining which models are evidence-based, and how we disseminate that information. Not everything you share or suggest will be within the scope of what HHS can do, but we don’t want to miss anything by focusing our questions too narrowly. We do not expect you to have deep familiarity with the HHS Criteria for evidence-based Early Childhood Home Visiting models. Instead, we will ask you about your experience identifying and using the evidence-based models, with the intention to listen for experiences that relate to the criteria or how they are implemented.

We have scheduled 60 minutes for this focus group. We would like everyone to feel comfortable sharing , so we ask that everyone here keep this conversation private and not share or discuss outside of this group anything that is said. We will be using a tool called PollEverywhere to ask interactive, polling-style questions; responses on PollEverywhere are anonymous and cannot be tied to you; they will only be used to inform our conversation. We also welcome you to use the Zoom chat to type comments in addition to speaking. Before we get started, I would like to allow you all to ask any questions you have about our work or the plan for our time together today.

[Address any comments or questions]

Questions and Prompts

Section 1: Introductions (7 minutes)

To start, we’d like to spend a few minutes getting to know you all and learning about your familiarity with Tribal MIECHV and components related to the determination of evidence-based home visiting models in MIECHV.

Topic	Tribal Home Visiting Experts Questions & Prompts
Introductions	Briefly, can you tell us your name, your organization and your connection to Tribal MIECHV.
Assessing familiarity with MIECHV evidence criteria, statutes, and evidence reviews	We’d like to use a poll to quickly get some information about how familiar you are with Tribal MIECHV, the HHS evidence criteria, and the systematic review used in Tribal MIECHV and MIECHV (called HomVEE). We have shared a link to a website called PollEverywhere in the chat. Please open this link. [PollEverywhere multiple choice style question:] How familiar are you with the Tribal MIECHV program? Not at all familiar A little familiar Somewhat familiar Very familiar How familiar are you with the HHS Criteria for evidence-based early childhood home visiting program? Not at all familiar A little familiar Somewhat familiar Very familiar How familiar are you with the Home Visiting Evidence of Effectiveness evidence review (HomVEE?) Not at all familiar A little familiar Somewhat familiar Very familiar

Section 2: Overview of HHS Criteria for Evidence-Based Early Childhood Home Visiting Models (10 minutes)

Before we begin our discussion, we want to be sure that everyone has some basic information about the content of the HHS Criteria.

There are three terms that we feel are important to clarify before we begin. (Show slide with this language.) These are:

• Maternal, Infant, and Early Childhood Home Visiting Program, called MIECHV for short. You all are familiar with the Tribal MIECHV program, as administrators. The MIECHV Program supports home visiting for expectant and new parents with children up to kindergarten entry age who live in communities that are at-risk for poor maternal and child health outcomes. As you are aware, by law, state and territory awardees must spend the majority of their MIECHV Program grants to implement evidence-based home visiting models, with up to 25 percent of funding available to implement promising approaches that will undergo rigorous evaluation. MIECHV’s authorizing language includes some specific requirements for a home visiting model to be considered evidence-based. This language serves as a foundation for HHS evidence criteria we will be discussing today.

• HHS Criteria for Evidence-Based Early Childhood Home Visiting Models, which we will refer to as the HHS Criteria for short – This is the purpose of our conversation today. It is the language that HHS has issued to indicate what evidence is required for home visiting models to be considered evidence-based. We will share the specific language of these criteria in the next slide.

(Show next slide) On this slide, we present the specific language of the HHS Criteria

“To meet HHS Criteria for an “evidence-based early childhood home visiting service delivery model,” models must meet at least one of the following criteria:

• At least one high- or moderate-quality impact study of the model finds favorable, statistically significant impacts in two or more of the eight outcome domains.

• At least two high- or moderate-quality impact studies of the model using non-overlapping analytic study samples find one or more favorable, statistically significant impacts in the same domain.

• In both cases, the impacts must either (1) be found in the full sample or (2) if found for subgroups but not for the full sample, be replicated in the same domain in two or more studies using non-overlapping analytic study samples.

Additionally, following the MIECHV-authorizing statute, if the model meets the above criteria based on findings from randomized controlled trial(s) only, then two additional requirements apply. First, one or more favorable, statistically significant impacts must be sustained for at least one year after program enrollment. Second, one or more favorable, statistically significant impacts must be reported in a peer-reviewed journal.”

• Home Visiting Evidence of Effectiveness – Called HomVEE for short. This is the systematic review that HHS funds to review the evidence for individual home visiting models to see whether they align with the HHS Criteria. HomVEE has detailed standards – documented in a nearly 200-page handbook – that operationalize the HHS Criteria into far more specific details. For instance, the HHS Criteria say “high- to moderate-quality impact studies”, while the HomVEE review operationalizes those in terms of features of the study design and execution required to meet high or moderate quality.

The reason we wanted to share this with you is so that you see the criteria themselves, but also to help you understand the factors influencing the HHS Criteria – including, most importantly, the MIECHV authorizing language – and also understand the distinction between the HHS Criteria and how they are operationalized through HomVEE. Since you do not all have deep expertise in the HHS Criteria or HomVEE, our conversation today will focus more generally on the way that HomVEE considers evidence in its determinations of evidence-based home visiting models.

Any questions about these terms before we begin?

[Answer questions]

Section 3: Perceptions of HHS Evidence Criteria (30 minutes)

As we noted, the HHS Criteria have significant implications for both how the evidence review – HomVEE – is structured and how states and jurisdictions can spend their MIECHV grant dollars. The purpose of this update is to understand how well the current standards align with the newest information about home visiting and the best practices in evidence-based policymaking and – if necessary – update the criteria to better align to those practices. Below, we’d like to ask you some questions about your general perspectives on the evidence requirements in MIECHV as well as a few specific areas of the HHS Criteria that we will be exploring, with particular attention to the implications for Tribal MIECHV.

Topic	Tribal Home Visiting Experts Questions & Prompts
General reactions to MIECHV evidence requirements	Before we dive into some specific questions about the HHS Criteria, we’d like to get your sense of the strengths and challenges of the evidence approach used in MIECHV. Thinking specifically of tribal communities, what do you think are the strengths of the evidence approach in MIECHV? What are some challenges? [Prompt if necessary]: What has your experience been that lead you to identify those things are strengths/challenges?
Technical issues: Adaptation and enhancement	Several of the studies that ACF has conducted with tribes have shown a strong interest in adapting evidence-based programs for new populations or contexts or enhancing them - such as by adding in a cultural element. Are there ways in which the HHS evidence criteria support or hinder communities in this regard? Prompt: What has your experience been with evaluating adaptations or enhancements to get them added to the list of evidence-based models? Are there improvements you’d suggest to make this process easier?
Technical issues: Subgroups	Another issue evidence reviews face is how to consider impacts on different populations or different contexts. Currently, evidence determinations in the HHS Criteria can be based on full samples or on replicated findings for subgroups. What are your reactions to this approach? Are there ways in which the approach might be enhanced to provide information more directly relevant to AI/AN populations?
Tiered evidence ratings	The MIECHV statute requires the distinction of evidence-based home visiting models. It also includes some specific language about how that distinction should be made, including having findings of positive outcomes in specific domains from “well-designed and rigorous randomized control trials and quasi-experimental studies.” The current HHS Criteria reflect these statutory requirements and create a binary rating; in other words, the review leads to a determination of whether a model is evidence-based versus not. However, the HomVEE review itself does not have to be binary in its rating. For instance, other evidence reviews sometimes have multiple rating levels – for example, the Prevention Services Clearinghouse has four levels of evidence: well-supported, supported, promising, or does not currently meet criteria. What is your perspective on a multi-tiered rating approach? In what ways would this be helpful or unhelpful? And what aspects of a model or its evaluations would you recommend be considered in designing tiers?

Section 4: Final Reflections and Wrap-Up (8 minutes)

We have a few minutes remaining together. As we wrap up, I’d like to ask for your final thoughts.

Topic	Tribal Home Visting Expert Questions & Prompts
General Reflections	We have discussed several specific technical issues relevant to the HHS Criteria. As we wrap up for today, I’d like to ask each one of you if you have any other thoughts about updating the HHS evidence criteria and its implications for Tribal MIECHV researchers, administrators, or model developers?

Those are all questions we have for today. We want to thank you all for taking the time to participate. We appreciate your insights and comments.

1