Supporting Statement B

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Family Level Assessment and State of Home Visiting (FLASH-V) Outreach and Recruitment Study

OMB Information Collection Request

0970 - New Collection

Supporting Statement

Part B

September 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Laura Nerenberg

Kelly Jedd McKenzie

Part B

B1. Objectives

Study Objectives

The purpose of these new data collection activities is to gather information about how Maternal, Infant, and Early Childhood Home Visiting (MIECHV) state and territory local implementing agencies (LIAs) and tribal grantees (hereafter referred to collectively as MIECHV funded LIAs) recruit families for program participation and work with their community referral partners to recruit families. The project is designed to examine challenges programs experience reaching caseload capacity and how challenges might be overcome. This descriptive work will capture how LIAs identify and refer families to home visiting services. The goal is to help the Administration for Children and Families (ACF) and the Health Resources and Services Administration (HRSA) in identifying potential bottlenecks in the outreach and recruitment process to inform the development and future testing of strategies to improve the delivery of MIECHV funded services.

Generalizability of Results

This study is intended to present internally-valid descriptions of how MIECHV funded LIAs engage in program outreach and recruitment, not to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

The design includes a survey sent to all MIECHV funded LIAs followed by semi-structured interviews with a smaller purposive sample. A similar design was used for the survey portion of the recently completed Home Visiting Career Trajectory Study (OMB # 0970-0512). The use of a national survey of MIECHV funded LIAs is appropriate to provide a snapshot of the landscape of program capacity, challenges, and accomplishments across MIECHV funded LIAs. We are proposing to send the survey to all MIECHV funded LIAs, as opposed to a sample of LIAs, to maximize variability across LIAs with respect to program characteristics (such as program model, locale, size, type of implementing agency). While we will not be obtaining a representative sample, we expect that sending the survey to all MIECHV funded LIAs does allow us to better approximate the landscape of program capacity across diverse contexts and LIAs in order to help ensure that the results are relevant. It also allows exploration of the impact of and changes related to the COVID-19 pandemic on program capacity, outreach, and recruitment. The use of semi-structured interviews is appropriate because the nature of semi-structured interviews allows collection of data across a consistent set of questions and concepts, while at the same time also allows for deeper probing of particular issues that may emerge over the course of the conversation. The document review portion of the study will help describe the content and nature of program outreach and recruitment materials. Information gathered will be used to identify potential bottlenecks in the outreach and recruitment process to inform the development and future testing of strategies to improve the delivery of MIECHV funded services. A limitation of the study design is that the semi-structured interviews and document review will not yield a representative sample of MIECHV home visiting programs; however, it will allow us to learn in detail from a subset of programs about challenges and promising strategies related to program outreach and recruitment.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target Population

We propose to gather information from MIECHV funded LIAs. The universe of MIECHV funded LIAs is approximately 779. One staff member at each of these LIAs will be invited to complete the survey. We will follow up with a subset of survey respondents and request that their LIA be represented in a semi-structured interview. We plan to conduct semi-structured interviews at 40 LIAs.

Because LIAs participating in semi-structured interviews will be purposively selected, they will not be representative of the population of MIECHV funded LIAs. Instead, we aim to obtain variation in LIA’s experiences recruiting families for program participation and working with their community referral partners to recruit families. For each of the 40 LIAs selected for semi-structured interviews, we will collect information from up to three staff designated by the LIA to participate in the interview. Target respondents at each LIA include staff who have responsibility for recruitment and enrollment of families. These staff may include dedicated outreach workers or other staff (such as home visitors or program managers).

Respondent Recruitment and LIA Selection

In an effort to reach the universe of MIECHV funded LIAs, the Study Team will reach out to MIECHV leads requesting contact information for each of their MIECHV funded LIAs. This request will occur for MIECHV leads for which that information is not already available (see Supporting Statement A).

The goal is to achieve a survey response rate of at least 55% (estimated to equate to at least 428 MIECHV funded LIAs nationally). We estimate burden for the full approximate universe of MIECHV funded LIAs, up to 779 potential respondents, to account for the possibility that more LIAs than estimated respond to our request (see Supporting Statement A). We expect one respondent per LIA to complete the survey.

For the interviews, we will select a total of 40 LIAs from the approximately 428 LIAs that we expect to complete the survey. A purposive sampling frame will be used for interview respondent recruitment to ensure variation in the final sample of LIAs selected, as described below.

LIAs will be eligible for study participation if they receive MIECHV funding. Outside of this eligibility requirement, LIAs will be prioritized for study participation based on a variety of characteristics such as program model diversity and variation in LIA urbanicity, LIA size, and LIA staff type conducting outreach and recruitment efforts.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

Data collection includes: (1) a national survey of MIECHV funded LIAs and (2) semi-structured interviews with a subset of MIECHV funded LIAs and a review of program outreach and recruitment materials.

The survey and interview protocol were guided by the project’s study aims and were also informed by the Study Team’s knowledge of existing research and literature on family outreach and recruitment as well as common challenges programs may experience in reaching out to and recruiting families. We also reviewed findings from select MIECHV grantee evaluations focused on the topic of family outreach and recruitment. The Study Team ensured the survey and interview protocol included aspects of implementation systems (such as staff training and support delivery systems) to help the Study Team document an LIA’s overall process of identifying, reaching out to, and referring families for program services. They survey and interview protocol were designed to allow the Study Team to gather a detailed picture of each LIA’s intended and actual outreach and recruitment processes and analyze findings across the LIAs. Instruments were carefully designed to allow the Study Team to identify promising strategies, contextual issues, potential bottlenecks, as well as structural constraints related to the outlined research questions. Careful attention was given to avoid duplication across instruments and to inform which items are appropriate for survey collection versus a semi-structured interview format.

The survey and interview protocols were each pilot tested with fewer than 9 individuals and revised accordingly. All pilot participants were selected for their ability to provide a LIA perspective when completing the data collection. Pilot testing for the survey and the interview was completed to assess clarity of the protocol and to monitor length. Pilot testing for the survey was completed with one non-study team JBA staff member as well as two external contacts; pilot testing for the interview was completed with one non-study team JBA staff member. As a result of this testing, clarifying items were added or reorganized to ensure the instruments accurately reflect the information intended to be collected and to improve flow. Unnecessary or duplicative items were deleted.

B4. Collection of Data and Quality Control

The Study Team will collect survey and interview data from LIA staff consenting to participate (see Instrument 2 for survey consent and Appendix K for interview consent). ACF has contracted with JBA and partner MDRC to conduct the study. JBA will lead the qualitative interview portion of the study, with MDRC assisting in conducting interviews as well. MDRC will lead the survey portion of the study.

Recruitment of LIAs into the study will involve outreach to MIECHV leads to provide LIA contact information (Instrument 1). The Study Team will then send an email directly to each LIA point of contact that includes a list of interview topics to help them decide who is best suited to participate in the interview (Appendix J), invites survey participation (Appendix F) and includes a study project description (Appendix G). LIAs will then decide who is best suited to complete the survey and the selected staff member will complete the survey using the survey link provided in the email (Instrument 2).

Following survey data collection, for each LIA selected for interviews (Instrument 3), the staff member who completed the survey will receive an email from the Study Team inviting and assessing their willingness to participate in an interview and to share their LIA’s outreach and recruitment materials before the scheduled interview (Appendix H). Participants will receive emails confirming interviews and thanking them for their participation (Appendix I) and will be provided a list of interview topics in advance of the interview (Appendix J). Additionally, we will ask participants to send their outreach and recruitment materials before the scheduled interview.

The survey will be programmed into Qualtrics web-based software. The Study Team will generate unique survey links for all potential survey respondents in Qualtrics and include the unique survey link in an email sent to the LIA requesting survey participation.

For the qualitative interviews, data will be collected via telephone interviews, which will be recorded if participants agree to audio recording. If the respondent does not consent to being recorded, a member of the project team will take notes during the interview.

To monitor surveys for quality and consistency, the Study Team will review reports on survey completion rates and item completion rates periodically during the data collection period.

Prior to data collection, the Study Team will conduct initial trainings with interviewers to cover general interview guidelines, provide tips on conducting successful interviews, and review the specific interview protocol. During data collection phase for LIA interviews, weekly check-in calls with interviewers will be held to monitor qualitative interviews for quality and consistency. While the interviews will be one-time only, we will request participant’s permission to contact them if we have any individualized follow-up questions after the interview. If we follow up with a respondent after the interview, we will give the respondent the option to respond to follow-up questions by phone or in writing.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

Based on past experience with similar studies, such as the Home Visiting Career Trajectory Study (OMB # 0970-0512), we anticipate being able to successfully recruit the number of respondents described in section B2: Sampling and Site Selection. We assume a minimum of 55% of the individuals who are invited to participate will complete the web-based survey. This projected sample of 428 respondents will help ensure we collect data from a sufficient number of LIAs to ensure results are relevant and are inclusive of diverse perspectives and contexts. It will also help ensure a sufficient range of programs that will allow the purposive sampling in the next stage of interview. All potential respondents are staff at LIAs that receive MIECHV funding. We anticipate this will motivate participation since LIAs and grantees will likely want to be included in a national survey on outreach and recruitment. The sample will be treated as nonrepresentative since we will be unable to compare the sample characteristics with those of the universe of LIAs.

In order to conduct interviews with 40 LIAs that have sufficient variation in their characteristics, the Study Team assumes that we will need to invite more than 40 LIAs to complete the interviews. The Study Team will use the selection factors described above (See B2: Respondent Recruitment and LIA Selection) to ensure diversity in type and number of models implemented, organization type, urbanicity, LIA size, staff type in the selection of 40 LIAs for study participation. The study team also assumes that not all LIAs selected for participation in the study will ultimately complete the interviews for a variety of possible reasons (e.g., scheduling challenges, staff turnover, competing priorities) and that half of the LIAs contacted will agree to participate. To account for these participation challenges, the study team will identify 20 alternate LIAs in addition to the 40 LIAs selected for participation in order to reach 40 LIA interviews and will reach out to alternates as necessary. The interviews are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.

NonResponse

As participants will not be randomly sampled and findings are not intended to be representative, nonresponse bias for the survey and semi-structured interviews will not be calculated. LIA demographics will be documented and reported in written materials associated with the data collection, however, we will not have demographic information on the universe of MIECHV funded LIAs to compare with. Refusal rate demographics will be collected for LIAs that decline participation in semi-structured interviews.

LIAs will be selected for semi-structured interviews to ensure variation in the interview sample selected. If we are unable to conduct 40 interviews with the LIAs originally selected for participation, the Study Team will begin reaching out alternate LIAs in a way that seeks to preserve variation in LIA characteristics.

B6. Production of Estimates and Projections

Data will not be used to generate population estimates, either for internal use or dissemination.

B7. Data Handling and Analysis

Data Handling

To minimize errors in survey data processing and analysis, the technical staff involved on the project have considerable experience working with survey data. Additionally, the Study Team employs a process of initial data processing and secondary data checking to mitigate data processing errors whereby a data manager is involved in reviewing the programming and resulting output.

All interviews (upon participant agreement) will be audio recorded and transcribed before analysis is completed to mitigate errors. For participants who do not consent to being audio recorded, a note taker will be used to ensure responses are recorded. During data analysis, double coding of data will be employed to reduce analysis errors and coder drift.

Data Analysis

A mixture of descriptive and qualitative methods will be used to analyze the data. The Study Team will explore relationships between the qualitative and quantitative data collected to triangulate data between surveys, interviews, and reviews of outreach and recruitment materials. This will allow the Study Team to compare information across the data sources and strengthen the quality and accuracy of the findings. A variety of analytic methods will be used across the data collected. The general analytic techniques to be used are described below.

Descriptive Analysis

The Study Team data analytics team members typically use SAS, RStudio, or Jupyter to perform data processing and analysis tasks. One or more of these software platforms will be employed to run frequency distributions for all variables of interest and descriptive analyses to generate summaries as well as to examine the variability in the data. The Study Team will also look for trends or patterns in the data that inform interpretations or conclusions related to LIA’s outreach and recruitment processes.

Qualitative Analysis

LIA interviews will yield rich information about outreach and recruitment processes that is best analyzed in a qualitative way. Qualitative data analysis software, Dedoose, will be used to facilitate data abstraction and coding. Open coding techniques will be employed to analyze the interview data. A codebook will be developed to guide the first round of thematic coding of interview transcripts. This first round of coding will be determined deductively (i.e. predetermined based on research questions and interview protocols). Once coding begins, inductive codes will also be identified through patterns and themes emerging in the data. After coding data into larger thematic codes, sub coding will be completed to identify patterns within each thematic code.

Data Use

One or more dissemination products will be developed. The products developed will depend on study findings and expected interest in those findings. Limited or no findings, for example, may result in a single report written for a small circle of federal staff and not widely disseminated. Findings with widespread implications for the home visiting field may be presented via multiple products and presentations customized by audience. Possible dissemination products might include a full-length report, a summary document, a brief, or a webinar. All dissemination products will include information about limitations of the data and how to interpret the information.

B8. Contact Person(s)

The information for this study is being collected by James Bell Associates and MDRC on behalf of ACF. Principal Investigator Jill Filene (filene@jbassoc.com), and Co-Project Directors Susan Zaid (szaid@jbassoc.com) and Kristen Faucetta (Kristen.Faucetta@mdrc.org) led development of the study design plan and data collection protocols and will oversee collection and analysis of data gathered through interviews.

The agency responsible for receiving and approving contract deliverables is:

The Office of Planning, Research, and Evaluation (OPRE),

Administration for Children and Families (ACF)

U.S. Department of Health and Human Services

The Federal project officers for this project are Laura Nerenberg and Kelly Jedd McKenzie.

Attachments

Instrument 1: Request for LIA Contact Information from MIECHV Leads

Instrument 2: LIA Survey

Instrument 3: LIA Interview Protocol

Appendix A: 60 Day FRN

Appendix B: 60 Day FRN Public Comment 1

Appendix C: 60 Day FRN Public Comment 2

Appendix D: 60 Day FRN Public Comment 3

Appendix E: Family Level Assessment and State of Home Visiting (FLASH-V) Outreach and Recruitment Study Design Changes in Response to Public Comments

Appendix F: Email Templates to LIAs Requesting Survey Participation

Appendix G: Project Description

Appendix H: Email Templates to LIAs Requesting Interview

Appendix I: Email Templates Providing Interview Information, Confirmation, and Thank You

Appendix J: Interview Topics and Sub-Topics

Appendix K: Interview Consent Form

7