Instrument 6. Listening Session Protocol for Researchers_clean

Listening Session Protocol:

Researchers

Introduction and Consent Script (10 minutes)

Hello and thank you for joining this listening session today. My name is [NAME OF FACILITATOR], and I’m joined by my colleagues [NAME OF NOTETAKER AND ANY PROJECT STAFF]. We are working with the Administration for Children and Families (ACF), Office of Planning, Research, and Evaluation, and the Health Resources and Services Administration (HRSA). Both these agencies sit within the U.S. Department of Health and Human Services. We are part of a team exploring the possibility of updating the HHS Criteria for Evidence-Based Early Childhood Home Visiting Models, the criteria HHS uses to determine if early childhood home visiting models are evidence-based.

As part of this effort, we are engaging experts in the field to gather input on how well the current criteria are working and priorities for improving them. Along with MIECHV administrators, we intend to speak with experts in evidence-based policy, and home visiting model developers, researchers, and advocates.

Before we begin the listening session, we’d like to do two things: first, we want to be sure that you are aware of your rights to participate – or not participate – in this listening session and the efforts that we will make to protect your privacy to the greatest extent possible. Second, we want to give a brief overview of the evidence criteria that we are considering updating to ensure we are all on the same page about the task at hand for the government.

First, a word about your rights. A Federal agency may not conduct or sponsor, and no individual or entity is required to respond to, nor shall an individual or entity be subject to a penalty for failure to comply with a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless that collection of information displays a currently valid OMB Control Number. The OMB # for this effort is 0970-0531 and the expiration date is 9/30/2025.

Second, it is important that you know how we intend to use the information we collect from you here. We do not intend to release a public report describing the findings of this or other engagements with experts. The information gathered here will be used to inform ACF and HRSA. For internal reports and discussions, any results we share from this group will be summarized across what we hear from you and the others in this conversation. You will not be identified by name in any form of analysis or report, and data will be reported in a de-identified manner. In addition, when we need to use quotes to help illustrate the findings, we will only identify them by role of the participant (such as, “evidence-based policy expert”). In some cases, our team may follow up with you to request additional feedback about your comments. All data collected for the review will be stored in secure environments, and we will protect the privacy of the information you provide. Finally, your participation in this listening session is voluntary, and you may decline to respond at any time.

We would like to record this conversation to ensure we accurately capture your comments. If you agree, we will retain the recording only until we can validate our notes, at which point we will destroy the audio recording. Since we are in a group setting today, if you are not comfortable being recorded at this time, we ask that you exit the call, and we will follow up with you at a later time. At this time, I will pause to make sure everyone we have with us agrees to allow us to record the listening session. Please do so by typing “yes” in the Zoom chat or stating “yes” verbally.

[Turn on recording]

[Facilitators to make note of anyone leaving the call, confirm that all remaining have consented]

At this time, I am going to turn on a recording to capture consent for participation in this listening session and recording.

Today, our main topics for discussion will be about the HHS Criteria for Evidence-Based Early Childhood Home Visiting Models, which are the criteria that the U.S. Department of Health and Human Services have outlined to determine which home visiting models are considered evidence-based for the Maternal, Infant, and Early Childhood Home Visiting – or MIECHV -- Program. These criteria have implications for how states and jurisdictions can spend their MIECHV Program dollars. The criteria were developed more than a decade ago, and we want to be sure that they evolve as the broader fields of home visiting and evidence-based policy evolve.

Our plan is to give you an overview of these criteria – including how they were developed and how they are operationalized in a systematic evidence review. We will then facilitate a dialogue with you to get your thoughts on the strengths and challenges of these criteria and their implications for MIECHV and other home visiting programs in the U.S.

We want to be clear about our intention for using the information we gather in this listening session and through other similar engagements. MIECHV was designed to support the use of evidence-based home visiting models. That is not something that will change from this process. However, we will be asking questions that get at how well the current approaches to identifying and determining which models are evidence-based, and how we disseminate that information. Not everything you share or suggest will be within the scope of what HHS can do, but we don’t want to miss anything by focusing our questions too narrowly. We do not expect you to have deep familiarity with the HHS Criteria for evidence-based Early Childhood Home Visiting models. Instead, we will ask you about your experience identifying and using the evidence-based models, with the intention to listen for experiences that relate to the criteria or how they are implemented.

We have scheduled 60 minutes for this listening session. We would like everyone to feel comfortable sharing, so we ask that everyone here keep this conversation private and not share or discuss outside of this group anything that is said. We will be using a tool called “PollEverywhere” to ask interactive, polling-style questions; responses on PollEverywhere are anonymous and cannot be tied to you; they will only be used to inform our conversation. We also welcome you to use the Zoom chat to type comments in addition to speaking. Before we get started, I would like to allow you all to ask any questions you have about our work or the plan for our time together today.

[Address any comments or questions]

Section 2: Introductions (5 minutes)

To start, we’d like to spend a few minutes getting to know you all. Please respond to the following Poll Everywhere questions, following the instructions on the screen. You will see the results on screen as you respond.

1. What is your organizational affiliation?

• Response options:

  • Academe

  • Research firm or think tank

  • Home Visiting or Service Providing Organization

  • Self-employed

  • Other (please describe)

2. How familiar are you with the MIECHV federal home visiting program?

• Response options:

• Not at all familiar

• A little familiar

• Somewhat familiar

• Very familiar

3. How familiar are you with the HHS Criteria for evidence-based early childhood home visiting program?

• Response options:

• Not at all familiar

• A little familiar

• Somewhat familiar

• Very familiar

4. How familiar are you with the Home Visiting Evidence of Effectiveness evidence review (HomVEE?)

• Response options:

• Not at all familiar

• A little familiar

• Somewhat familiar

• Very familiar

.

Section 3: General Reflections on MIECHV Evidence Criteria (10 minutes)

For these next two questions, we are going to ask your perspective on the successes and challenges of the HHS Criteria and HomVEE.

For each of the following questions, please indicate the degree to which you agree or disagree with the following statements, following the Poll Everywhere instructions you see on the screen. I will give everyone three minutes to answer the questions and then will show the results online. Remember, no one will see anyone’s individual responses, just the average and distribution of your ratings.

5. The HHS Criteria and HomVEE effectively identifies home visiting models that are most likely to promote outcomes for MIECHV children and families.

• Response options:

• Strongly disagree

• Disagree

• Neither agree nor disagree

• Agree

• Strongly agree

5. The HHS Criteria and HomVEE effectively identifies and excludes the home visiting models that are unlikely to promote outcomes for MIECHV children and families.

Response options:

• Strongly disagree

• Disagree

• Neither agree nor disagree

• Agree

• Strongly agree

Discussion: I’d like to hear a little bit about what people were thinking when answering. If you are willing to share, please raise your hand using the hand to share why you responded to the questions the way you did.

Section 4: Reflections on specific aspects of the HHS evidence criteria and HomVEE (15 minutes)

Now we’re going to use PollEverywhere to get your input on different issues relevant to evaluating the evidence for home visiting models. I’ll give everyone about ten minutes to answer these questions and then will show the mean scores – ranked highest to lowest - for each of the items. Then we will reflect and discuss. Again, no one’s individual ratings will be shown on screen; rather, we will only present averages and distributions across respondents.

5. How important do you think the following issues are for determining the evidence base for home visiting models?

	Not at all important	A little important	Somewhat important	Very important
Evaluation design and implementation allows for strong causal inference
Evaluations show statistically significant benefits when averaged across all studies.
The model shows benefits in at least one or two strong studies, even if other studies show no benefits.
The benefits from evaluations are statistically significant.
Findings have been replicated in different outcome domains
Findings have been replicated in different populations or communities
Findings have been replicated with groups of marginalized or minoritized populations
Outcomes are measured with multiple methods.
Outcome measures have demonstrated reliability and validity.
The evaluations have been conducted within the past twenty years
The evaluations have been conducted within the past ten years.
The size of the effects found in evaluations are large enough to be indicate meaningful changes in outcomes
Benefits to participants last at least a year beyond enrollment in the model.
Benefits last longer than a year beyond program enrollment.
The model has no unfavorable impacts when averaged across all studies.
The model has no unfavorable impacts in any of the evaluations.

Discussion: Let’s reflect on these rankings. It looks like (share highest rating item) was thought to be most important, while (share some lower rating items) were rated lower. Why do we think that factor is more important? What about (lower rating items), what makes those less important? If you are willing to share your thinking, please raise your hand.

Section 4: Impact of HHS Criteria and HomVEE on Home Visiting Research Field (13 minutes)

For my last question, I’d like to get your input on the impact that the HHS Criteria and HomVEE have on the broader home visiting research field. I’ll ask for your general reflections and then we can spend some time talking about what you have to say. Please follow the instructions on the screen to address the next question in PollEverywhere. I will share the average score and distribution when we are finished.

5. To what degree do you agree with the following statement Overall, the HHS standards, requirements, and processes for identifying evidence-based home visiting models has had a positive impact on the home visiting research field.

   1. Strongly disagree

   2. Disagree

   3. Neither agree nor disagree

   4. Agree

   5. Strongly agree

Discussion: Is anyone willing to share their thinking on this? Please raise your hand to share why you responded in one way or another?

Section 6: Final Reflections and Wrap-Up (2 minutes)

Those are all questions we have for today. We want to thank you all for taking the time to participate. We appreciate your insights and comments.

1