Instrument 1. Focus Group Protocol for Evidence-Based Policy Experts_clean

Focus Group Protocol:

Evidence-Based Policy Experts

Introduction and Consent Script (5 minutes)

Hello and thank you for joining this focus group today. My name is [NAME OF FACILITATOR], and I’m joined by my colleagues [NAME OF NOTETAKER AND ANY PROJECT STAFF]. We are working with the Administration for Children and Families (ACF), Office of Planning, Research, and Evaluation, and the Health Resources and Services Administration (HRSA). Both these agencies sit within the U.S. Department of Health and Human Services. We are part of a team exploring the possibility of updating the HHS Criteria for Evidence-Based Early Childhood Home Visiting Models, the criteria HHS uses to determine if early childhood home visiting models are evidence based.

As part of this effort, we are engaging experts in the field to gather input on how well the current criteria are working and priorities for improving them. Along with evidence-based policy experts, we intend to speak with MIECHV administrators, and home visiting model developers, researchers, and advocates.

Before we begin the focus group, we’d like to do two things: first, we want to be sure that you are aware of your rights to participate – or not participate – in this focus group and the efforts that we will make to protect your privacy to the greatest extent possible. Second, we want to give a brief overview of the evidence criteria that we are considering updating to ensure we are all on the same page about the task at hand for the government.

First, a word about your rights. A Federal agency may not conduct or sponsor, and no individual or entity is required to respond to, nor shall an individual or entity be subject to a penalty for failure to comply with a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless that collection of information displays a currently valid OMB Control Number. The OMB # for this effort is 0970-0531 and the expiration date is 9/30/2025.

Second, it is important that you know how we intend to use the information we collect from you here. We do not intend to release a public report describing the findings of this or other focus groups or engagements with experts. The information gathered here will be used to inform ACF and HRSA. For internal reports and discussions, any results we share from this focus group will be summarized across what we hear from you and the others in this conversation. You will not be identified by name in any form of analysis or report, and data will be reported in a de-identified manner. In addition, when we need to use quotes to help illustrate the findings, we will only identify them by role of the participant (such as, “evidence-based policy expert”). In some cases, our team may follow up with you to request additional feedback about your comments. All data collected for the review will be stored in secure environments, and we will protect the privacy of the information you provide. Finally, your participation in this focus group is voluntary, and you may decline to respond at any time.

We would like to record this conversation to ensure we accurately capture your comments. If you agree, we will retain the recording only until we can validate our notes, at which point we will destroy the audio recording. Since we are in a group setting today, if you are not comfortable being recorded at this time, we ask that you exit the call, and we will follow up with you at a later time. At this time, I will pause to make sure everyone we have with us agrees to allow us to record the focus group. Please do so by typing “yes” in the Zoom chat or stating “yes” verbally.

At this time, I am going to turn on a recording to capture consent for participation in this focus group and recording.

[Turn on recording]

[Facilitators to make note of anyone leaving the call, confirm that all remaining have consented]

Today, our main topics for discussion will be about the HHS Criteria for Evidence-Based Early Childhood Home Visiting Models, which are the criteria that the U.S. Department of Health and Human Services have outlined to determine which home visiting models are considered evidence-based for the Maternal, Infant, and Early Childhood Home Visiting – or MIECHV -- Program. These criteria have implications for how states and jurisdictions can spend their MIECHV Program dollars. The criteria were developed more than a decade ago, and we want to be sure that they evolve as the broader fields of home visiting and evidence-based policy evolve.

Our plan is to give you an overview of these criteria – including how they were developed and how they are operationalized in a systematic evidence review. We will then facilitate a dialogue with you to get your thoughts on the strengths and challenges of these criteria and their implications for MIECHV and other home visiting programs in the U.S.

We want to be clear about our intention for using the information we gather in this focus group and through other similar engagements. MIECHV was designed to support the use of evidence-based home visiting models. That is not something that will change from this process. However, we will be asking questions that get at how well the current approaches to identifying and determining which models are evidence-based, and how we disseminate that information. Not everything you share or suggest will be within the scope of what HHS can do, but we don’t want to miss anything by focusing our questions too narrowly. We do not expect you to have deep familiarity with the HHS Criteria for evidence-based Early Childhood Home Visiting models. Instead, we will ask you about your experience identifying and using the evidence-based models, with the intention to listen for experiences that relate to the criteria or how they are implemented.

We have scheduled 60 minutes for this focus group. We would like everyone to feel comfortable sharing , so we ask that everyone here keep this conversation private and not share or discuss outside of this group anything that is said. We will be using a tool called “PollEverywhere” to ask interactive, polling-style questions; responses on PollEverywhere are anonymous and cannot be tied to you; they will only be used to inform our conversation. We also welcome you to use the Zoom chat to type comments in addition to speaking. Before we get started, I would like to allow you all to ask any questions you have about our work or the plan for our time together today.

[Address any comments or questions]

Questions and Prompts

Section 1: Introductions (7 minutes)

To start, we’d like to spend a few minutes getting to know you all and learning about your familiarity with the PIR data and how you use it in your work.

Topic	Evidence-Based Policy Expert Questions & Prompts
Introductions	Briefly, can you tell us your name and a little bit about your engagement in evidence-based policymaking and any connection to MIECHV? [Facilitator to call name of individuals from the list]
Assessing familiarity with MIECHV evidence criteria, statutes, and evidence reviews	We’d like to use a poll to quickly get some information about how familiar you are with MIECHV, the HHS evidence criteria, and the systematic review used in MIECHV (called HomVEE). We have shared a link to a website called PollEverywhere in the chat. Please open this link. [PollEverywhere multiple choice style question:] How familiar are you with the MIECHV federal home visiting program? Not at all familiar A little familiar Somewhat familiar Very familiar How familiar are you with the HHS Criteria for evidence-based early childhood home visiting program? Not at all familiar A little familiar Somewhat familiar Very familiar How familiar are you with the Home Visiting Evidence of Effectiveness evidence review (HomVEE?) Not at all familiar A little familiar Somewhat familiar Very familiar

Section 2: Overview of HHS Criteria for Evidence-Based Early Childhood Home Visiting Models (10 minutes)

Before we begin our discussion, we want to be sure that everyone has some basic information about the content of the HHS Criteria.

There are four terms that we feel are important to clarify before we being. (Show slide with this language.) These are:

• Maternal Infant and Early Childhood Home Visiting Program, called MIECHV for short. The MIECHV Program supports home visiting for expectant and new parents with children up to kindergarten entry age who live in communities that are at-risk for poor maternal and child health outcomes. By law, state, tribal, and territory awardees must spend the majority of their MIECHV Program grants to implement evidence-based home visiting models, with up to 25 percent of funding available to implement promising approaches that will undergo rigorous evaluation. MIECHV’s authorizing language includes some specific requirements for evidence-based home visiting models. This language serves as a foundation for HHS evidence criteria we will be discussing today.

• HHS Criteria for Evidence-Based Early Childhood Home Visiting Models, which we will refer to as the HHS Criteria for short – This is the core of our conversation today. It is the language that HHS has issued to indicate what evidence is required for home visiting models to be considered evidence-based. We will share the specific language of these criteria in the next slide.

(Show next slide) On this slide, we present the specific language of the HHS Criteria

“To meet HHS criteria for an “evidence-based early childhood home visiting service delivery model,” models must meet at least one of the following criteria:

• At least one high- or moderate-quality impact study of the model finds favorable, statistically significant impacts in two or more of the eight outcome domains.

• At least two high- or moderate-quality impact studies of the model using non-overlapping analytic study samples find one or more favorable, statistically significant impacts in the same domain.

• In both cases, the impacts must either (1) be found in the full sample or (2) if found for subgroups but not for the full sample, be replicated in the same domain in two or more studies using non-overlapping analytic study samples.

Additionally, following the MIECHV-authorizing statute, if the model meets the above criteria based on findings from randomized controlled trial(s) only, then two additional requirements apply. First, one or more favorable, statistically significant impacts must be sustained for at least one year after program enrollment. Second, one or more favorable, statistically significant impacts must be reported in a peer-reviewed journal.”

• Home Visiting Evidence of Effectiveness – Called HomVEE for short. This is the systematic review that HHS funds to review the evidence for individual home visiting models to see whether they align with the HHS Criteria. HomVEE has detailed standards – documented in a nearly 200-page handbook – that operationalize the HHS Criteria into far more specific details. For instance, the HHS Criteria say “”high- to moderate-quality impact studies”, while the HomVEE review operationalize those in terms of features of the study design and execution required to meet high or moderate quality.

The reason we wanted to share this with you is so that you see the criteria themselves, but also understand the factors influencing the HHS Criteria – including most important, the MIECHV authorizing language – and also understand the distinction between the HHS Criteria and how they are operationalized through HomVEE. Since you do not all have deep expertise in MIECHV, the HHS Criteria or HomVEE, our conversation today will focus more generally on best practices in evidence-based policymaking.

Any questions about these terms or MIECHV before we begin?

[Answer questions]

Section 3: Perceptions of Best Practice in Evidence-Based Policy (30 minutes)

As we noted, the HHS Criteria have significant implications for both how the evidence review – HomVEE – is structured and how states and jurisdictions can spend their MIECHV grant dollars. The purpose of this update is to understand how well the current standards align with best practice in evidence-based policymaking and – if necessary – update the criteria to better align to those practices. Below, we’d like to ask you some questions about your general perspectives on effective approaches to evidence-based policymaking as well as a few specific areas of the HHS Criteria that we will be exploring.

Topic	Evidence-Based Policy Expert Questions & Prompts
Best practices in evidence-based policymaking	As you think about the most effective models of evidence-based policymaking you’ve come across in the federal or state governments, what are key features of those you feel have been more successful? Prompt if needed: How is this different from models that have been less successful? Prompt [if they do not specify what they think success looks like] How do you know that the approaches you mentioned were successful or not successful?
Technical issues: Meaningfulness of impacts	Another key issue is how to ensure that the positive findings in studies reflect meaningful changes. There are three key issues here for the HHS Criteria, as shown on this slide related to the number of impacts in different domains, the magnitude of impacts, and the duration of impacts. As you can see on the slide, the current criteria state that evidence-based models must have statistically significant impacts in two of eight measured domains OR a replicated impact at least one domain, and for RCTs at least one impact must be sustained for at least a year. There are no specifications about the magnitude of the impacts. What are your reflections on these criteria? Are there changes you would recommend and why?
Technical issues: Adaptations and enhancements	Another issue evidence reviews face is how to consider adaptations or enhancements to evidence-based models. So, for instance, there might be a home visiting model that has been deemed to be evidence-based, but it has been adapted or enhanced over time to meet the needs of certain communities or to address issues identified in implementation. Given your experience, what are the best approaches to addressing these circumstances in evidence reviews or in the broader policy for tiered evidence or evidence-based programs? Are there strategies that you would recommend avoiding and why?
Technical issues: Tiered evidence ratings	The MIECHV statute requires the distinction of evidence-based home visiting models. It also includes some specific language about how that distinction should be made, including having findings of positive outcomes in specific domains from “well-designed and rigorous randomized control trials and quasi-experimental studies.” The current HHS Criteria reflect these statutory requirements and create a binary rating; in other words, the review leads to a determination of whether a model is evidence-based versus not. However, the HomVEE review itself does not have to be binary in its rating. For instance, other evidence reviews sometimes have multiple rating levels – for example, the Prevention Services Clearinghouse has four levels of evidence: well-supported, supported, promising, or does not currently meet criteria. What is your perspective on a multi-tiered rating approach? In what ways would this be helpful or unhelpful? And what aspects of a model or its evaluations would you recommend be considered in designing tiers?

Section 4: Final Reflections and Wrap-Up (8 minutes)

We have a few minutes remaining together. As we wrap up, I’d like to ask for your final thoughts.

Topic	Evidence-based Policy Expert Questions & Prompts
General Reflections	We have discussed several specific technical issues relevant to the HHS Criteria. As we wrap up for today, I’d like to ask each one of you if you have any other recommendations or thoughts about setting evidence criteria in a national program like MIECHV.

Those are all questions we have for today. We want to thank you all for taking the time to participate. We appreciate your insights and comments.

1