Instrument 3
R3 Discussion Guide for Feasibility Study Visits
PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN: Through this information collection, ACF is gathering information to conduct a feasibility study of one or more promising family recovery and reunification interventions that use recovery coaches. Public reporting burden for this collection of information is estimated to average six hours per grantee, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. This is a voluntary collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB # is 0970- 0356 and the expiration date is 06/30/2021. If you have any comments on this collection of information, please contact Kimberly Francis, R3 Project Director, at Kimberly_Francis@abtassoc.com or (617) 520-2502.
Instrument 3: R3 Discussion Guide for Feasibility Study Visits
Interviewer Instructions: This guide is intended to build on Instruments 1 and 2. Please review notes from prior calls and customize the topics below, eliminating any redundancies.
Introduction
Thank you for taking the time to meet with the R3 study team. As you know, [name of organization or program] is participating in the R3 study. R3 is a study of recovery and reunification interventions that use recovery coaches in child welfare settings. Abt Associates and its partners are conducting the study on behalf of the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS). The study will help us determine the feasibility of conducting a future rigorous evaluation of one or more potentially replicable and scalable interventions. This feasibility study is the first step toward the goal of conducting a rigorous evaluation of recovery coaching interventions.
We are visiting to collect more detailed information about your program for the feasibility study. The information we discuss will help the study team and ACF assess whether the program has the potential to be part of a future rigorous evaluation. We expect these visits to last for a total of four to eight hours. Lastly, please know that your participation in all discussions with our team is completely voluntary. Should you choose to participate, your responses will be discussed internally among the study team and the funding agency (the Administration for Children and Families) but that, to the extent allowable by law, individual identifying information will not be disseminated publicly.
Intervention:
Program Name:
Program Location:
Organization(s):
Individual(s) Interviewed: (names and titles)
Date of Communication:
Individual’s background
Please describe your current role in this program.
How long have you worked with the program?
What is your prior experience in this area?
[If front-line staff/manager] Please tell me about training you have received in this program.
Please tell me any other background information that we should know about this program, and your involvement in it.
Program locations and settings
Review and confirm where the program operates (e.g., statewide; one county; multiple counties; multiple locations within one county)
Review the number of locations for enrolling parents. To what extent does the program differ by location? (e.g., enrollment process; program services; partners involved)
Please describe the key partners at each location and their role(s) in the program. Do you have a formal or informal arrangement with this/these partner(s)?
How is the program coordinated across multiple locations? (probe for day-to-day oversight, overall management, shared processes and systems across locations)
Community context
Please tell me about the characteristics of the communities in which the program operates.
Review and confirm population size, SES, race/ethnicity, major industries.
Are there major cultural practices or influences?
Are there major issues or challenges facing the community?
How would you describe the attitudes and knowledge in the community about substance use disorder? What perceptions do community members have about parents with SUD and their treatment/recovery process? (probe for attitudes/understanding among local child welfare service providers)
Review the number and percentage of cases referred to child welfare with parental substance use as a risk factor in 2020 and 2019.
Review the rate of out-of-home care placement for child welfare referrals with parental substance use as a risk factor?
Please describe the range of SUD treatment services available in the communities which the program operates.
Service environment/Counterfactual
Review and confirm other related programs or services available to the target population within organization or the broader community. If multiple communities, review for each.
What are the “usual services” if participants were not offered this program?
Has implementation of this program changed the nature of “usual services”? E.g., improved systems coordination; changing perspectives within the court system
Target population
Review and confirm the characteristics of the target population.
At what point in the child welfare system does the program identify eligible parents/families? (e.g., at risk of entering CW system; report of CA/N; children already removed from home)
What are the program’s eligibility requirements? Are any of the requirements subjective or flexible? Please describe.
Eligibility determination and intake
Let’s talk about your process for determining eligibility for the program.
When are parents/families screened for eligibility?
Who conducts the eligibility screening?
What screening tools or instruments are used?
Where does the eligibility determination take place?
Let’s talk about your program enrollment process.
Once eligibility is determined, when do you conduct enrollment with parents/families?
Who conducts the enrollment?
What are the goals/objectives of the enrollment process?
Where does the enrollment process take place?
What happens to parents/families after eligibility determination and enrollment are complete?
Program capacity [ask for each location where the program operates]
Does the program have enrollment targets? If so, what are they?
How many parents/families did the program serve in 2020 and 2019? How did that break down by month? How did that break down by site/enrollment location? (probe for changes in service volume due to the COVID-19 pandemic)
What percentage of available program spots were full in 2020 and 2019? Was the program at capacity, or could more parents/families have enrolled?
If the program was full, approximately how many parents/families were eligible but were not able to be enrolled due to capacity?
How many parents/families do you expect to serve this year (2021)? What about future years?
Are there any known or potential issues on the near horizon that might change those projections? If yes, please describe them. (probe for any expected changes related to COVID-19 pandemic or its abatement)
Enrollment and retention [ask for each location where the program operates]
Of the parents/families that are found eligible for the program, what percentage enroll? What are common reasons for not enrolling?
Of the parents/families that enroll in the program, what percentage complete it? [Confirm how program defines “completion”]. Does the program have completion targets? What are the main barriers to completion?
Are there key milestones and activities as parents/families move through the program? What percentage of parents/families complete these? Are there steps where parents/families tend to drop off or stall in the program?
What is the average duration of a parent/family’s participation in the program? In other words, how long do parents/families typically remain enrolled?
How does the program determine when enrollment ends? What is considered a successful completion? (probe for child welfare case closure, SUD treatment completion, prescribed program duration) Under what circumstances might enrollment end for other reasons?
Review demographic characteristics of the parents/families enrolled in the program.
Administrative data sources
Please describe the administrative data systems that your organization and your partners use for this program.
What data elements are captured for the program by these administrative data systems?
Review available data elements against the outcome measures required for the impact study and identify any that are not currently captured.
Inquire as to whether those uncaptured data elements may exist elsewhere (other sources) or if it might be possible for the program to capture those data.
How frequently are data entered? What activities trigger data entry?
Does the program have specific data collection requirements, mandated by certain funding streams or agencies?
Which individuals at each organization/enrollment location are responsible for entering the data? For monitoring the data?
Is there a staff member designated to oversee data collection for the program as a whole? What are that person’s responsibilities?
How does the program use the data it collects? (e.g., reporting; internal monitoring; program evaluation)
What would be required for the study team to have access to the data?
Review current data use agreements (DUA) and identify any new or amended DUAs that would be necessary for the impact study.
Discuss state/local IRB requirements for participating in an impact study.
Staffing and training
Please describe the program’s staffing structure. In addition to the recovery coaches, what are the other roles on the team? (probe for roles within the organization and among partners)
Review and confirm what education, experience, or certification recovery coaches are required to have.
How have you recruited program staff? What organization hires the recovery coaches? Please describe the recruitment and hiring process.
Has recruitment been easy or difficult? Have there been any challenges? How long does it typically take to fill a position?
Who supervises recovery coaches?
What training and support do recovery coaches receive once they’re hired? Please describe the training topics and content; length and timing of training; who conducts the training.
What ongoing training and support do recovery coaches receive?
What is your experience with program staff retention?
How frequent is staff turnover? How long do staff stay with the program? What leads to turnover?
Integration of study design with enrollment procedures
As you know, R3 is exploring the feasibility of conducting a future impact study—in particular, a study with a random assignment design.
[As needed, describe how random assignment operates.]
Let’s discuss some options for how random assignment could be integrated with your current processes.
Discuss at what point in the eligibility and enrollment process random assignment of parents/families could take place.
Discuss what services a control group would receive.
Discuss if “services as usual” could be provided to some parents/families and the program provided to others, within the same location. (i.e., control group contamination).
Discuss random assignment at the level of intake location. Discuss what services a control group would receive.
Challenges and future plans
Review plans to operate the program in the future.
Review status of funding to continue or expand the program.
Discuss interest in moving forward from the feasibility study to a potential future rigorous evaluation. Confirm understanding of any barriers, concerns.
What are the primary challenges the program might face?
What resources or technical assistance would be needed to help overcome these challenges?
Feedback on research questions and potential study designs
Review the R3 research questions together. Do you have any feedback on the research questions? Is anything missing that you think is important? Which research question do you think would be most challenging to answer with this study? Why?
Review the potential random assignment approaches discussed during this visit. Do you have any feedback on the potential approaches? Which ones seem most feasible for your program?
Do you anticipate any challenges associated with random assignment? How do you think [frontline staff/colleagues/referral and service delivery partners/community members] will feel about random assignment?
What could help mitigate those challenges?
Do you have any other comments or advice for project team? We appreciate all your input and will continue to discuss these topics with you going forward.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Jill Hamadyk |
| File Modified | 0000-00-00 |
| File Created | 2021-01-12 |