Supporting Statement B - Touchpoints Pretesting GenIC

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Touchpoints for Addressing Substance Use Issues in Home Visiting: Performance Measurement Pilot

Pre-testing of Data Collection Activities

0970 - 0355

Supporting Statement

Part B

February 2021

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Nicole Denmark

Part B

B1. Objectives

Study Objectives

The Administration for Children and Families’ Office of Planning, Research, and Evaluation (OPRE) within the U.S. Department of Health and Human Services seeks approval under the umbrella generic, Pre-testing of Data Collection Activities (0970-0355), to conduct a six-month pilot study. The pilot study will include collecting data using a Measures Reporting Tool and semistructured interviews with Maternal, Infant, and Early Childhood Home Visiting (MIECHV) state administrators, local implementing agency (LIA) staff, and home-visiting-model representatives. MIECHV awardees provide home visiting services for pregnant women and parents of young children. The services help prevent child abuse and neglect, support positive parenting, improve maternal and child health, and promote child development.

The findings from the pilot study will be used to refine the way two potential performance measures, Screening for Substance Misuse (SUD-1) and Follow-up for Caregivers at Risk of Substance Misuse (SUD-2), are defined. The findings will also help produce information about the feasibility of collecting substance misuse screening and follow-up performance measures using SUD-1 and SUD-2, the usefulness of SUD-1 and SUD-2 in improving services for families with substance misuse problems, and the supports LIA staff require to implement SUD-1 and SUD-2. Pilot study findings will potentially help MIECHV state awardees and staff at LIAs engage and support families with substance use problems more effectively. The information could also be used in a Home Visiting Collaborative Improvement and Innovation Network that the Health Resources and Services Administration may convene to accelerate improvements in processes and outcomes related to behavioral health, including substance use. The information collected is not intended to serve as the principal basis for a decision by a federal decision maker, nor is it expected to meet the threshold of influential or highly influential scientific information.

Generalizability of Results

This study is intended to present an internally valid description of two substance use measures within home visiting programs in select sites, not to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

The pilot study is a formative study intended to provide information to states with home visiting models and the agencies implementing those models. To understand the usefulness and feasibility of collecting the two measures (SUD-1 and SUD-2) and the supports required to implement the measures, qualitative and quantitative data sources will be used to capture the experience of participating stakeholders (for example, frontline staff, agency leaders, and state representatives). The information collected will be used to address the research questions (noted in A2) and provide insight into the overarching study objectives.

The study design and methodology are appropriate given the objectives of the data collection, target population, and quality of data needed for the intended uses. None of the study instruments will ask for information that can be reliably obtained from alternative data sources, including administrative data. For more information on specific data collection efforts, please refer to item A2 in Supporting Statement A.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information. Given the small sample size, the data are not representative of all LIA programs, staff, and caregivers and the data should not be used to draw conclusions regarding LIA performance. Written products associated with the study will include key limitations of the findings.  

B2. Methods and Design

Target Population

The universe of programs eligible for the pilot study include MIECHV state awardees in the United States and their LIAs. The target respondents include MIECHV state administrators; LIA staff, including managers, supervisors, and home visitors; and representatives of home visiting models. Two state administrators, 42 members of the LIA staff, and six home-visiting-model representatives will be asked to participate in the pilot, for a total of 56 respondents across two sites.

Site Selection

The contractor will target MIECHV state awardees that (1) implement two or more home visiting models using MIECHV funds, (2) do not currently require that all LIAs screen for substance use with a validated tool, (3) have initiatives planned or underway to address substance use (suggesting they might be interested in participating in the pilot), and (4) anticipate having LIAs that are interested in participating in the pilot and have the capacity to do so. We will focus on LIAs that have a mix of experience with screening caregivers for substance use. Specifically, we aim to recruit two LIAs that currently universally screen caregivers for substance use with a validated tool, two LIAs that currently screen caregivers for substance use but do not screen universally or with a validated tool, and two LIAs with limited or no experience with screening caregivers for substance use. We will prioritize LIAs that serve at least 100 families and implement more than one home visiting model. We will also aim to include LIAs that operate in both urban and rural locations.

OPRE and the Health Resources and Services Administration will identify two MIECHV state awardees that meet the pilot study’s criteria. The contractor will work with the MIECHV state administrators to identify and recruit LIAs to participate in the data collection. Each MIECHV state awardee will identify five eligible LIAs to invite to participate with the expectation that three of five LIAs will agree to participate. The contractor will invite up to six home-visiting-model representatives to participate in the study.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

The measures reporting tool (Instrument 1) and the data collection instruments for the site phone interviews (Instruments 2 to 4) were developed based on the essential data required to answer the priority research questions. Table B.1 presents a crosswalk between the data collection instruments and study objectives.

Table B.1. Crosswalk Between Data Collection Instruments and Study Objectives

	Objective
Instrument	Understand feasibility of using and reporting on the measures	Understand perceived usefulness of the measures for improving home visiting services	Identify supports needed to collect and report on measures
Instrument 1: SUD-1 and SUD-2 Measures Reporting Tool	X		X
Instrument 2: MIECHV State Administrator Interview Protocol	X	X	X
Instrument 3: LIA Staff Interview Protocol	X	X	X
Instrument 4: Home-Visiting Model Representative Interview Protocol	X	X	X

B4. Collection of Data and Quality Control

Instrument 1: SUD-1 and SUD-2 Measures Reporting Tool

The contractor will work with MIECHV state awardees to identify and recruit LIAs to participate in data collection for the pilot study. Using the SUD-1 and SUD-2 Measures Reporting Tool, LIAs will collect and track data to inform SUD-1 and SUD-2 throughout the pilot study. LIAs will provide these data to the contractor via a secure file transfer platform such as Box, which encrypts data in transit and at rest. They will transmit the data once toward the end of the pilot study.

To enhance the quality of data collected in the SUD-1 and SUD-2 Measures Reporting Tool, the contractor will provide LIAs with (1) a 90-minute training webinar; (2) a list of validated substance use screening instruments, should they adopt new screening, referral, and follow-up practices to align with the SUD-1 and SUD-2 measures; (3) the Measures Reporting Tool that standardizes the format of the data collected; and (4) technical support to respond to LIAs’ questions and needs regarding data abstraction. The contractor will work with LIAs to determine the appropriate staff person to collect the data, such as a data manager or quality improvement director.

Instruments 2 to 4: Interviews with LIAs and MIECHV state administrators and recruiting home visiting model representatives

The contractor will conduct semistructured telephone interviews with LIA staff (including managers, supervisors, and home visitors), MIECHV state administrators (such as program directors or data managers), and representatives of home visiting models (such as implementation specialists).

The contractor will reach out to representatives with whom it has an established relationship. The contractor will schedule interviews with MIECHV state administrators, key LIA staff, and home-visiting-model representatives toward the end of the pilot study. The interviews with LIA staff and MIECHV state administrators will ask about practices related to existing substance use screening and follow-up care, substance use screening measurement, and using information from SUD-1 and SUD-2. Interviews with home-visiting-model representatives will gather information on how the measures align with their current recommendations and existing requirements for substance use screening and follow-up services. See Appendix C for the interview questions that will be used to address each research question. Interviewers will describe the pilot study, review the purpose of the interview, describe how the data will be used, and assure respondents that information discussed will be kept private and reported in a manner that will not identify individual respondents.

To ensure data quality, interviewers will be trained on the objectives of the pilot study and research questions related to the interview protocol questions. This will encourage thorough and appropriate probing. Interviewers will also ask respondents for permission to record the interview. Recordings will allow interviewers to clarify portions of the interview if notes were unclear. Interviewers will clean their notes and organize their documentation using the structure of the interview protocol for standardization and consistency.

B5. Response Rates and Potential Nonresponse Bias

The purposive selection and recruitment of MIECHV state administrators and LIA staff is designed to achieve a 100-percent response rate. Participants will be selected based in part on their willingness to participate. We expect model representatives to continue their engagement throughout the pilot as key stakeholders in the study. As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. The contractor will schedule interviews to accommodate the needs of respondents.

B6. Production of Estimates and Projections

No descriptive or statistical estimates will be produced for this study.

B7. Data Handling and Analysis

Data Handling and Analysis

Qualitative data collected from semistructured interviews will be synthesized to address predetermined research questions related to substance use screening and referrals to follow-up care, including information about the feasibility and use of the measures and their implementation supports. Information collected from interviews will be summarized with narrative and supporting tables. Quantitative data collected from the Measures Reporting Tool will be reviewed for data quality, including completeness and accuracy of data entry. LIAs will provide Measures Reporting Tool data to the contractor via a secure file transfer platform. Information collected from the Measures Reporting Tool’s data quality review will inform recommendations for refinements to SUD-1 and SUD-2 measures specification and the supports required for their implementation. No quantitative data analysis will occur.

Data Use

The contractor will summarize the results from this data collection effort in a report for OPRE. Results will help refine SUD-1 and SUD-2 as potential performance measures that could serve to inform development of a Home Visiting Collaborative Improvement and Innovation Network that the Health Resources and Services Administration may convene to accelerate improvements in processes and outcomes related to behavioral health, including substance use. ACF may share the report or other summaries of pilot study findings they develop with external stakeholders. No interview data or Measures Reporting Tool data will be released to the public and individual participants and their agencies will not be named in the report or any other documents produced.

B8. Contact Persons

The contractor, Mathematica, developed the plans for this data collection.

The following contractor staff can answer questions about the purpose and design of this data collection:

• Melissa Azur, mazur@mathematica-mpr.com

• Patricia Del Grosso, pdelgrosso@mathematica-mpr.com

• Charlotte Cabili, ccabili@mathematica-mpr.com

Staff from the contractor will collect, process, and analyze the information for OPRE.

Attachments

Instrument 1: SUD-1 and SUD-2 Measures Reporting Tool

Instrument 2: Interview Protocol: State Awardee

Instrument 3: Interview Protocol: LIA Managers and Data Managers, Home Visiting Supervisors, and Home Visitors

Instrument 4: Interview Protocol: Home-Visiting-Model Representatives

Appendix A: Pilot Study FAQs

Appendix B: Invitations to Participate in the Study

Appendix C: Research Questions Addressed by Interview Protocols

6