Touchpoints for Addressing Substance Use Issues in Home Visiting: Performance Measurement Pilot

^{A nchor}

Touchpoints for Addressing Substance Use Issues in Home Visiting

Pilot Study Interview Protocol: LIA Managers and Data Managers, Home Visiting Supervisors, and Home Visitors

February 2021

CHARLOTTE CABILI, STACIE FELDMAN, AND MELISSA AZUR

This page has been left blank for double-sided copying.

Interview protocol: lia managers and data managers, home visiting supervisors, and home visitors¹

I am pleased to be talking with you again!

As you may remember, my firm, Mathematica, has been contracted to conduct a pilot study of two performance measures, called Screening for Substance Misuse (SUD-1) and Follow up for Caregivers At Risk of Substance Misuse (SUD-2). The purpose of the pilot study is to assess the usefulness of the measures to Maternal, Infant, and Early Childhood Home Visiting (MIECHV) awardees and local implementing agencies for improving home visiting services and identify facilitators and challenges with using the measures. Mathematica is conducting this evaluation on behalf of the Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) in collaboration with the Health Resources and Services Administration (HRSA). There are no direct benefits or risks for your participation, but your input will help to determine the feasibility and usefulness of these two performance measures that could be adopted into the MIECHV performance system.

To date, Mathematica has been providing several local implementing agencies with technical assistance on the use of these two performance measures. Today, Mathematica is going to ask you to reflect on your agency’s practices along three broad topics: substance use screening and follow up care practices before the pilot; substance use screening measurement before and during the pilot; and the usefulness of SUD-1 and SUD-2. Your participation is voluntary, and you may skip any question. We may interview other staff members in your agency for their input as well, including a data manager, supervisor, and up to five home visitors. We expect our discussion to last about [90/60] minutes. Mathematica will compile findings from this interview and other local implementing agencies as well as state MIECHV awardees and home visiting model representatives. We will provide our aggregated pilot study findings to OPRE and HRSA. Your information will be kept private to the extent permitted by law. We are not assessing your performance, and we will not name individual staff or agencies in our pilot study report. Our main goal is to learn from you!

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB # is 0970-0355 and the expiration date is 05/31/2021. If you have any comments on this collection of information, please contact Melissa Azur.

Would it be okay if we recorded the interview? We take notes to capture everything you say but sometimes we miss something important. We listen to recordings in case we want to go back and check something in our notes. We do not share recordings outside of the study team and will destroy recordings when our project ends.

[If the respondent does not want to be recorded] That’s no problem. Do you have any questions before we begin?

I. Existing substance use screening and follow up care

1. Let’s begin by talking about your role here, at [AGENCY NAME]. Can you tell me your title and key responsibilities? [S, HV]

2. How long have you worked at this agency, and in your current role? [S, HV]

Note: If agencies did not screen caregivers for substance use before the pilot, as previously identified during recruitment, interviewers should go to question 27 and skip questions 3 to 26.

Let’s discuss the substance use screening process your agency used before the pilot study began. I will refer to screening practices before the pilot, but I recognize that the prior process or some version of it may still be in place. I also recognize that activities that have traditionally been in person may have moved to virtual settings in recent months. As we talk about your agency’s screening process, please point out any adaptations you have made to accommodate your virtual work setting.

3. Can you walk me through the process of screening primary caregivers for substance use that you had in place before the pilot study began? [S, HV, NO MGR]

Probe for elements of the process:

1. How were primary caregivers selected for screening?

2. How and when was the screening conducted?

3. Which staff members conducted the screening (for example, home visitors or agency partner staff)

4. What substance use screening tools were used? [S, HV]

5. Why did your agency choose these screening tools? [S, HV]

6. [Ask if not included on the study team’s list of validated tools] What kinds of substances does the tool screen for? [S, HV]

7. [Ask if not included on the study team’s list of validated tools] Do you know the extent to which the tool has been tested or used by others? [S, HV]

8. What are the strengths and weaknesses of the tools? [S, HV, NO MGR]

9. Did your agency provide you with written policies or procedures for substance use screening? [S, HV]

10. (If yes to #9) What kinds of documentation and resources were available to you, to assist you in adhering to these policies and implementing procedures? [S, HV]

11. In your experience, how long after enrollment did the first primary caregiver substance use screening typically take place? [S, HV, NO MGR]

12. [Ask if more than 30 days after enrollment] What factors might have impeded timely screenings? [S, HV, NO MGR]

13. [Ask if less than 30 days after enrollment] What factors facilitated timely screenings? [S, HV, NO MGR]

14. How often did you screen a primary caregiver for substance use? [S, HV, NO MGR]

15. When were pregnant women screened for substance use, relative to enrollment? [S, HV, NO MGR]

16. When did post-birth substance use screenings typically take place, if any? [S, HV, NO MGR]

17. What was the ideal number and timing of substance use screenings for pregnant women, based on your experience? [S, HV, NO MGR]

We can turn now to activities that take place after substance use screenings are completed, again focusing on activities that your agency had in place before the pilot study began.

18. Can you walk me through the process of what your agency did with the completed screening results, and who at your agency is responsible for the activities? A good place to start is scoring of the results. Then walk me through to the point a caregiver received an assessment from a behavior health provider to assess whether treatment or follow up care is needed. [S, HV, NO MGR]

Probe for activities that follow substance use screenings:

1. Scoring

2. Interpreting results

3. Discussing results with primary caregivers, and the typical amount of time that lapses between screening and this discussion

4. Identifying a referral partner that provides behavioral health assessment

5. Conveying positive screen results to the partner

6. Making referrals for follow up care (internal or with other providers)

7. Making appointments for follow up care (caregiver versus agency staff)

19. In your experience, how long after a positive screening did it take for the caregiver to have met with a behavioral health provider? Was the length of time different if the caregiver was a pregnant mother? [S, HV, NO MGR]

20. [Ask if more than 14 days after screening] What factors might have impeded timely meetings? [S, HV, NO MGR]

21. [Ask if less than 14 days after screening] What factors facilitated timely meetings, and why? [S, HV, NO MGR]

22. Did your agency provide you with written policies or procedures for making referrals and appointments? [S, HV]

23. [Ask if yes] What kinds of documentation and resources were available to you, to assist you in adhering to policies and implementing procedures? [S, HV]

Let’s talk for a few minutes about the resources and supports that home visitors need to interpret and act on substance use screening results in a meaningful way.

24. In your agency, what professional development, training, or resources are provided to home visiting staff to help them interpret and act on the screening results? Where are additional supports needed? [S, HV]

Probe on if/how home visitors acquired the knowledge and skills to:

1. Interpret the screening results

2. Identify appropriate follow up care providers

3. Approach and engage in discussions about screening results with primary caregivers when substance misuse is identified

4. Explain availability and benefits of follow up care

5. Convey positive screening results to referral partner

6. Gather information from behavioral health providers on receipt of follow up care

25. Would you make changes to the way your agency shares substance use screening results with home visiting staff, to enhance their understanding of results or make the results more actionable? [S, HV]

Probe for changes:

1. Customization of results

2. Mode and method of sharing (individual, group; email, phone, in-person)

3. Other

26. How has your agency established relationships with referral partners that provide behavioral health services? [S, HV]

Probe:

1. Methods used with external and in-house referrals

2. What has worked well in establishing partnerships with behavioral health service providers? What has worked well in maintaining partnerships? What has not worked well with maintaining partnerships?

27. Think about education on the effects of substance use on primary caregivers and child development. Does your agency provide education to primary caregivers about the effects of substance use? If so, how would you rate the value of this education?? [S, HV]

28. In your experience, what have been the most beneficial supports for primary caregivers to maintain positive behaviors or make positive changes with substance use? [S, HV]

29. What additional supports would you like for home visitors to provide families to promote positive behavior change? [S, HV]

II. Substance use screening measurement

Note: If agencies did not screen caregivers for substance use before the pilot, interviewers should go to question 4 and skip questions 1 to 3.

I am going to ask about information you collect for the substance use screening measures.

1. I would like to understand your agency’s existing measures to track the receipt of substance use screenings and follow up care to understand how they compare to SUD-1 and SUD-2. I recognize that activities that have traditionally been in person may have moved to virtual settings in recent months. As we talk about your agency’s measures, please point out any adaptations you have made to accommodate your virtual work setting.

First, please tell me each substance use screening measure your agency is using.

2. Now let’s talk about the requirements for each screening measure.

Probe for each screening measure:

1. What is the reference period (that is, the time frame of interest)?

2. What types of substances are included?

3. How is the numerator defined (that is, who in your program is supposed to be screened for substance use)?

4. How is the denominator defined (that is, the pool of eligible primary caregivers)?

5. Who is excluded from the measure?

6. How are these requirements guided by [HOME VISITING MODEL NAME]? How do they differ?

3. What collection and tracking procedures does your agency use for substance use screenings and follow up care? [S, HV]

[If the response to question 3 demonstrates minimal use of collection and tracking procedures, then ask:] If you were to begin formal tracking of substance use screenings, what procedures would you use for data collection and tracking? [S, HV]

Probe on process elements:

1. Database or tracking tool to monitor screening and (as needed) referral to behavioral health assessment

2. Data on paper forms that must be entered into electronic systems

3. Methods for scoring the screening measure

4. Staff who track and score screening and referrals

5. How feedback about incomplete screening goes to home visitors

6. How feedback goes to home visitors about complete screenings or follow up with behavioral health specialists

Now I would like to hear about your more recent experiences during the pilot, implementing the SUD-1 and SUD-2 measures.

4. Please describe the recordkeeping system that your agency uses to compile the results of screenings, including substance use screenings, for reporting to the state. [S, HV]

5. Does your recordkeeping system have the specifications and structure for calculations of SUD-1 and SUD-2?

Probe for individual data elements that are required to calculate SUD-1 and SUD-2:

1. Record identifier

2. Primary caregiver enrollment date in home visiting program

3. If primary caregiver has been enrolled for at least 30 days

4. Screening completed for unhealthy alcohol use or unhealthy drug use with a validated tool

5. Name of screening tool used

6. If screening completed for unhealthy alcohol use or unhealthy drug use occurred within 30 days of enrollment

7. If screen was positive for unhealthy alcohol use or unhealthy drug use

8. Date of positive screen for unhealthy alcohol use or unhealthy drug use

9. Status of primary caregiver participation in substance use treatment program

10. Follow up care with behavioral health specialist completed within 14 days of positive screen

11. Date of follow up care with behavioral health specialist

6. Did your agency modify your recordkeeping system to allow you to collect data elements for SUD-1 and SUD-2? What kinds of changes or accommodations were needed?

Provide examples if needed:

1. Addition of data elements

1. Different sources of data (electronic health records or administrative records, for example)

2. Different data entry requirements, such as replacing open-ended with closed data fields

7. How would you describe your agency’s capacity and ability to collect accurate and complete SUD-1 and SUD-2 data?

8. What have been the key barriers to collecting the required data in the measures reporting tool?

Probe for agency experiences:

1. Understanding instructions for collection

2. Compiling or entering the data

3. Gaining cooperation from agency staff

4. Using service data from behavioral health providers (if needed)

5. Staff ability to obtain primary caregiver self-reports about their receipt of follow up care from a behavioral health specialist

6. [Ask if LIA did not screen for substance use before the pilot] Issues with newly screening for substance use

9. Are these barriers similar or different than those you encounter when collecting other MIECHV performance measures?

10. To what extent have the SUD-1 and SUD-2 collection and reporting added difficulty and preparation time to your typical MIECHV performance measure reporting process?

11. What kinds of resources and supports would have helped you to overcome these barriers or would have generally improved your experience with measures collection and reporting overall?

Probe for supports and resources from the agency and the state, separately.

1. Systems for monitoring and tracking

2. Policies and procedures

3. Organizational leadership

4. Other

12. [Ask if LIA did not screen for substance use before the pilot] What resources and trainings did you need to add to support your agency’s use of screening with validated tools and making referrals to behavioral health providers?

13. Were there any key lessons your agency learned from the process of implementing SUD-1 and SUD-2? [If yes] Please describe these. [S, HV]

Probe on process elements:

1. Choice of validated screening tool

2. Screening frequency and timing

3. Service receipt per self-report or behavioral health agency data

14. Looking to the future, how might your agency use SUD-1 and SUD-2 performance measure scores

Probe:

1. Feedback to home visitors

2. Inform agency decision making

3. Share data other agency staff or stakeholders

15. How would this anticipated use of SUD-1 and SUD-2 performance measure scores compare to the way your agency uses other MIECHV performance measures?

16. Think about your overall experience with collecting information SUD-1 and SUD-2 and using the measures reporting tool. What key lessons, challenges, or unanticipated consequences might you share with another agency who is about to embark on the same process? [S]

We would like to turn now to the costs your agency incurred for the setup and collection of data associated with SUD-1 and SUD-2 measurement. If your agency incurred costs related to adopting a substance use screening tool into your practice, you may describe this as well. Think about agency funds, staff time, and resources that were involved in the setup and use of SUD-1 and SUD-2.

17. What purchased expenses were involved in setting up and using SUD-1 and SUD-2?

Probe for purchases of screening tools, materials, and technology.

Prompt for the estimated amount of each expense.

18. How much staff time was dedicated to the setup and use of SUD-1 and SUD-2? I would also like to understand the roles of staff spending this time on setup, to see how the level of effort was shared among staff at different levels.

Prompt if needed: Staff may spend time on capacity building and developing procedures; the screening itself; monitoring, tracking, managing data; and reporting SUD-1 and SUD-2 to Mathematica.

Thank you! Those are all the questions I have for you today.

[If the interview was not recorded, ask] Would it be okay to contact you after today with a follow up question or two, in case we missed something in our notes?

1 As displayed in the protocol, LIA managers and/or data managers will respond to all questions in the interview protocol, except those marked [NO MGR]. Home visiting supervisors and home visitors will respond to selected interview questions as indicated by [S] and [HV].