Topic Guide – Program Managers

R3-Impact Program Managers Topic Guide

Introduction. We are from the research firm [Abt Associates/Child Trends] and we are conducting the [STUDY NAME] evaluation. The study is overseen by the Office of Planning, Research, and Evaluation (OPRE), in collaboration with the Children’s Bureau, in the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS). As part of the evaluation, we are conducting an implementation study to learn how counties implementing [PMP or START] are approaching their work.

As part of this study, we are talking with program managers to learn more about how [PMP/START] is being delivered in your community. During our conversation, we will ask questions to understand more about your community, your agency / organization, and how [PMP/START] is being implemented in your community, including partners involved in implementing the program, factors influencing implementation of [PMP/START], and your thoughts about how well the program is working.

Before beginning our discussion, we want to thank you for agreeing to talk with us today. We know you are busy and we appreciate the valuable time you are spending with us today. The interview will take about 90 minutes. Your participation is voluntary, and there are no penalties for choosing not to take part in the interview. You can refuse to answer any questions or stop the interview at any time. Our aim is to learn from your insights and experience. There are no right or wrong answers. Your answers will be kept private to the extent permitted by law. The information you provide will not be shared with other staff at your agency or organization. Your name will not be listed in any published reports, and comments will not be attributed to you. Instead, your answers will be combined with answers provided by other people in your same role that we interview across the country for this study. However, because of the relatively small number of agencies/organizations participating in the study, there is a possibility that a response could be correctly attributed to you. Your data will only be used for research purposes. The legal authority for this project is 42 U.S.C. § 1310a and Public Law 115-271. For System of Records Notice (SORN) information, please see the SORN number 09-80-0361, “OPRE Research and Evaluation Project Records.

We would like to record this discussion with a digital recorder so we can listen to it later when we write up our notes to make sure we captured everything accurately. No one besides our research team and the transcription vendor will listen to the recording. If you want to say anything that you don’t want recorded, please let us know and we will be glad to pause the recorder. Do you have any objections to being part of this interview or to us recording our discussion?

This collection of information is voluntary and will be used to understand programs that provide peer mentoring for parents involved in the child welfare system. Public reporting burden for this collection of information is estimated to average 90 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: XXXX, Exp: XXXX. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Kimberly Francis (Abt Associates); kimberly_francis@abtassoc.com.

Do you have any questions before we begin?

1. Respondent Information

• Name, title, organization/affiliation, length of involvement with the organization and with the program

• Overall role/job responsibilities

2. Local Context (CFIR Domain: Outer Setting)

• Local political, economic, and geographic conditions

• Barriers families face to stability in their lives

  • Transportation to/from neighborhoods where families live

    • To areas where jobs are available

    • To schools

  • Housing affordability and availability

  • Availability of jobs suitable for people in recovery

  • Availability of employers providing support to people in recovery/“Recovery Oriented Workplaces”

  • Education

  • Health care (including mental health care)

  • Policy context, such as occupational licensing restrictions

• Families most / least affected by these barriers

• Programs / resources in the community to remove barriers

• Recent local policy efforts to remove these barriers, such as fair chance hiring

• Facilitators of family stability

  • Social supports

  • Community sense of collective efficacy

  • Differential access to facilitators

• Local substance use disorder conditions

• Trends in substance use disorder, including types of substances, in areas served by program (e.g., arrests, hospitalizations, overdose, etc.) [pre-populate from secondary sources, SAMSHSA, CDC prior to visit]

• Treatment and recovery service landscape, including availability and accessibility of ASAM levels of care and culturally appropriate services specifically

• Policy shifts in recent years on substance use issues, such as decriminalization of certain substances, increased use of telemedicine, increased MOUD access (e.g. through buprenorphine waivers)

  • Effects of decriminalization on organizational approach around abstinence and harm reduction

  • Implications for child welfare or criminal justice involvement due to now-legal drug possession/use

• Prior grants and initiatives to address high rates of SUD, overall and for CW-involved parents specifically

• History of inter-agency efforts to address high rates of SUD, overall and for CW-involved parents specifically

• Local child welfare system conditions

• Perceptions of CW system perspective and biases

  • Views of parents

  • Views of SUD

  • Views of recovery process

• Trends in family preservation, removals and reunifications, including shifts during/after COVID-19 lockdowns

• Policy context and shifts, e.g. status of Family First implementation, state-specific ICWA laws)

• How intervention aligns with broader CW system strategy

• Fluctuations in funding and how they have affected service delivery

3. Agency Background (CFIR Domain: Inner Setting)

[Site visitors: Pre-populate with information collected through regular calls with sites, materials sent ahead of time (such as an org chart), organizations’ websites, etc. Confirm and/or obtain any missing information.]

• Other programs and services offered by organization/agency, including prior experience offering peer support services.

• Organization’s major sources of funding

• Organization’s total number of paid staff, including number of staff and percent FTE

• Types of clients served or targeted, including prior experience serving CW-involved parents with substance use concerns.

• Annual number of clients served overall

• Organization’s approach to supporting a recovery-oriented workplace

• Organization’s approach to promoting racial equity, including through staff training and hiring

4. Program Background (CFIR Domains: Process of Implementation, Individuals)

• Prior knowledge of recovery coaching interventions generally

• Prior knowledge of PMP/START

• Concerns about implementing START/PMP and how these concerns have been addressed

  • Concerns about CW system buy-in and attitudes

  • Concerns about implementing a new intervention (for MI/MN/VA)

  • Concerns about agency capacity for implementation

• Reasons for interest in implementing START/PMP

• How START/PMP aligns with broader organizational goals, including around racial equity

• Leadership support for implementing START/PMP

• Staff involved in deciding to implement START/PMP

• How and why, if at all, organization has tailored START/PMP to adapt to local context

  • How, if at all, organization has tailored START/PMP to help address disproportionality in CW system involvement

• Challenges meeting fidelity standards

• Technical assistance and training received on START/PMP (including feedback on technical assistance and training)

5. Program Structure (CFIR Domain: Innovation, Process of Implementation)

• Management structure

• How program is situated/managed within the larger organization or institution

• Role of parents and others with lived experience in program operations, decision making, continuous quality improvement

• Who has authority over the program

• Number of staff and positions

• Required experience and qualifications (training and lived experience) for mentor supervisors

• Primary responsibilities for mentor supervisors

• Supervisor demographics and how they compare to mentors and the parents served in the program

• Number of FTEs by position

• Location (at provider organization or CW branch)

• Staff turnover and shortages (generally and due to COVID-19)

• How staff demographics compare to CW population generally and parents served in the program specifically

• Funding

  • Other resources required to operate the program such as facilities, technology, and treatment/recovery services

  • Stability of funding

  • Sustainability and prospects for future funding

• Partnerships

• Key partners involved in implementation

  • Child welfare system

  • Treatment providers

  • Other community partners

• Prior existing relationships with these partners and how relationship has evolved during implementation of the intervention

• Partner roles and responsibilities

• Staff at organization responsible for coordination with partners

• Schedule and format for partner coordination meetings

• Process and frequency of any data exchange about parents between partners to ensure coordination of services

• Partnership challenges and how they have been addressed

• Gaps in partnerships

• Benefits of partnerships

• Parent characteristics and participation in services

• Barriers to parents’ success and engagement in the program

• Families most/least affected by these barriers

• Strategies to mitigate barriers to parent success and engagement

• Parent strengths and facilitators of success in the program

• Engagement and reengagement strategies

• Disparities in completion of services and achievement of key outcomes

• Documentation and evaluation

• Data entry processes: who enters, at what point, on what information

• Use of data for service planning

• Use of data for program improvement

• Use of data to promote equitable access and outcomes

• Successes and challenges tracking data

6. Operational Challenges and Successes

• Ways the program has exceeded or fallen short of its goals

• Challenges the program has encountered and how they were overcome

• Areas for improvement; plans for changes and modifications

• Advice for organizations implementing similar interventions

7