Supporting Statement B_R3

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Expanding Evidence on Replicable Recovery and Reunification Interventions for Families (R3)

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part B

NOVEMBER 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Calonie Gray

Kelly Jedd McKenzie

Part B

B1. Objectives

Study Objectives

The objectives of the overarching Expanding Evidence on Replicable Recovery and Reunification Interventions for Families (R3) project are to:

• Compile and disseminate to the field current evidence on recovery and reunification interventions that use recovery coaches in child welfare settings.

• Determine the feasibility of conducting a rigorous evaluation of one or more potentially replicable and scalable interventions.

The purpose of this information collection is to conduct a feasibility study of one or more promising family recovery and reunification interventions that use recovery coaches. Recovery coaching is a promising approach to support parents who are working toward treatment completion, recovery, and ultimately reunification with their children when possible.

Through phone calls and site visits with programs that use recovery coaching in a child welfare setting, the study team will 1) select sites for participation in a feasibility study, and 2) conduct a feasibility study by assessing sites’ readiness for a future rigorous evaluation.

The current request is the first submission for the R3 project.

Generalizability of Results

This study is intended to assess the feasibility of a future rigorous impact evaluation in selected sites, not to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

Using a feasibility study design for the R3 project allows ACF to understand sites’ evaluation readiness before initiating a rigorous impact evaluation. This design is appropriate based on the early state of practice and research on interventions using recovery coaching in a child welfare setting. With the exception of a few long-standing interventions, recovery coaching for child welfare-involved families has only recently gained interest, and there is relatively little research base or other published material from which to draw information. Collecting information directly from sites will provide current, detailed information to assess their readiness for rigorous evaluation.

B2. Methods and Design

Target Population

The feasibility study’s target population is programs and sites implementing recovery coaching interventions for families involved in the child welfare system.

Sampling and Site Selection

The R3 project is not connected to any child welfare or substance use treatment services funding stream, so it will draw from any available and suitable sites across the US. The R3 project team will select up to four sites to participate in a feasibility study. We will select the sites based on information gathered through the informational phone calls and site visits. The initial criterion for selection is that the sites are currently implementing or interested in implementing the intervention(s) of interest for the R3 project. Other examples of characteristics we will consider for sites selection include: the programs’ interest in participating; design of program components; service environment; and experience with and scale of program implementation. More detail on the information that will be collected to inform site selection is provided in Instruments 1 and 2.

Relevant participants for the discussions will be identified by each site’s project director in consultation with the R3 project team.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

The R3 project team developed the instruments for R3 based on the project team’s experience with prior feasibility studies and the information it gathered through a systematic scan of recovery coaching interventions for families involved in the child welfare system. The instruments will be used to guide semi-structured discussions with programs and sites. The project team will not ask all questions of every program or site, but instead use the guides to fill gaps in the information from other sources.

B4. Collection of Data and Quality Control

The R3 project team (staffed by Abt Associates and its subcontractors) will collect all data through phone and in-person discussions.

During the phone calls or site visits, one R3 team member will ask questions and lead the discussion. A second team member will take notes and ask for clarification where necessary, aiming to capture the information as close to verbatim as possible. The team member who leads the discussion will be responsible for reviewing the notes to check for accuracy and identify any missing information. A final, clean version of the notes will be saved after review.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The site visits are not designed to produce statistically generalizable findings and participation is wholly at the site directors’ discretion. Response rates will not be calculated or reported.

Non Response

As sites will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Site demographics will be documented and reported in written materials associated with the data collection.

B6. Production of Estimates and Projections

Our data are not representative and are not generalizable. We will not make any estimates or projections, either for internal use or dissemination.

B7. Data Handling and Analysis

Data Handling

All data will be collected through phone or in-person discussions, and final versions of notes from the discussions will be saved on Abt Associates’ secure server. The R3 project team will use the original notes documents for reference, and will extract key information into Excel spreadsheets as a cumulative central repository for information collected from sites.

Data Analysis

The data for this information collection will be in the form of notes from phone calls and site visits with programs. The discussions that will be conducted as part of this data collection will be used for internal feasibility study purposes, i.e., to assist in the identification and selection of feasibility study sites and subsequently assess each site’s feasibility for a future impact evaluation. The information will be interpreted in conjunction with existing documentation the project team collected through a systematic scan of recovery coaching interventions for families involved in the child welfare system. The project team will assess and synthesize the information to identify the evaluability of the specific sites under consideration, and summarize the results in a final technical report.

Data Use

No documentation of how to use these data will be released to the public.. Findings from the current information collection request will not be used for the primary purpose of publication, but may be incorporated into documents or presentations that are made public, such as research design documents or reports; research or technical assistance plans; background materials for technical workgroups; or contextualization of research findings from a follow-up data collection that has full PRA approval.

B8. Contact Person(s)

Name	Organization	Role on Contract	Phone/email
Calonie Gray	OPRE, ACF, HHS	Co-Contracting Officer’s Representative	Calonie.Gray@acf.hhs.gov (202) 565-0205
Kelly Jedd McKenzie	OPRE, ACF, HHS	Co-Contracting Officer’s Representative	Kelly.McKenzie@acf.hhs.gov (202) 245-0976
Kimberly Francis	Abt Associates	Project Director	Kimberly_Francis@abtassoc.com (617) 520-2502

Attachments

• Instruments

  • Instrument 1: R3 Discussion Guide for Informal Informational Calls with Potential Program Sites

  • Instrument 2: R3 Discussion Guide for Virtual Site Selection Visits

  • Instrument 3: R3 Discussion Guide for Feasibility Data Collection Visits

• Appendices:

  • Appendix A: Text from SUPPORT Act Section C. 8082

8