Site visit and key informant data collection: Program manager/supervisor individual interviews (Appendix B)

APPENDIX B

TOPIC GUIDE FOR SITE VISIT and phone INTERVIEWS

This page has been left blank for double-sided copying.

Regional Partnership grants (RPG) cross-site evaluation

TOPIC GUIDE FOR site visit and phone interviews

The RPG cross-site evaluation will include site visits to 11 projects and phone interviews with the remaining 6 in 2020. Researchers will interview RPG project directors, partners, managers, supervisors, and frontline staff who work directly with families during the site visits. Interviews will be conducted either individually or in small groups, depending on staffing structure, roles, and the number of individuals in a role. Researchers will interview RPG project directors and partners by phone for those projects that do not receive a site visit. This topic guide includes the full breadth of topics that will be covered in the multiple interviews, although each individual or small group interview may not include all topics.

Topic	Subtopic
Informant characteristics
Informant characteristics	Job title Education background/licensing qualifications Years in current position and with agency Role on RPG and prior experience with RPG project
Partnerships
Goal setting	Organizations/stakeholders that participated in planning (during proposal stage and planning phase), if not the same as current partners Child welfare and substance use treatment agencies’ involvement in RPG planning How partners were involved in developing a shared vision and goal setting How partners and other community organizations/stakeholders were involved in the planning and decision-making processes, and how concerns were addressed Key design decisions made during the planning phases and rationale for those decisions Challenges encountered during the planning process and how/if they were resolved Grantees’ and partners’ prior experience with similar programs and how prior experience informed the RPG project design
Partnership composition and roles	How and why particular partners were selected How partnerships came to be/developed, such as partnerships with organizations prior to RPG project; type and length of prior relationship Grantee and partner organizations’ roles in RPG Child welfare and substance use treatment agencies’ roles and responsibilities in RPG Development and maintenance of formal or informal agreements Changes in partnerships and the rationale for those changes (such as turnover of partner organizations and key staff within partners) Changes in grantee, partner, or RPG project leadership staff that occurred during the grant period and may have affected the direction of the RPG project
Interagency collaboration and service coordination	Whether and how partners collaborate on joint activities (such as training) Competing priorities for partner organizations Process for decision making and resolving conflicts within the partnership Policy or process changes within partner agencies (such as mental health service providers, courts) resulting from collaboration on RPG Process to share data and information about families across partners Process for coordination of screening, assessment, referrals, treatment, or other services Partnership successes, challenges, and lessons learned about interagency collaboration/partnerships
Child welfare and SUD treatment agencies’ collaboration	Child welfare and substance use treatment agencies’ history of working together; successes, challenges, and lessons learned Child welfare and substance use treatment agencies’ views on the goals of RPG Process for defining and delineating the roles and responsibilities of each agency to meet the goals of the RPG project Clarity of roles for each agency while families were served during RPG Guidelines and delineation of roles for each agency especially for follow-up of services referrals Child welfare agency’s capacity to offer SUD assessment and treatment improved or changed resulting from collaboration with SUD treatment agency The extent of collaboration between the two agencies on four collaboration activities with drug and alcohol service providers (as defined in He, 2015)1: (a) a memorandum of understanding (MOU) or other formal interagency agreement, (b) cross-training of staff, (c) colocation of staff, and (d) joint budgeting or resource allocation Intensity of collaboration, that is the number of collaboration activities Types of collaboration, such as policy (such as having an MOU) versus practice collaboration (such as colocation of staff). Alignment of RPG goals with child welfare and substance use treatment agencies’ goals and priorities Process for reconciling competing priorities (if applicable) Process for reconciling differing treatment and permanency timelines; how child welfare and SUD treatment RPG staff at every level interact with each other (such as frontline staff, managers/supervisors, and administrators/directors across the two agencies) How lessons learned from prior collaborations between child welfare and SUD treatment agencies have been integrated into the RPG project Process for child welfare and SUD treatment agencies to identify and address challenges related to RPG collaboration Policy or practice changes within the child welfare and substance use treatment agencies resulting from collaboration on RPG
Perceptions of RPG project partners	Perceptions of partnership quality; frequency of partner interaction Partners’ views about the grantee organization as a convener (an organization with enough credibility to bring together stakeholders across sectors)
Implementation teams2	Organizational structure of the implementation team for the RPG project Development of implementation team for the project partnership; timing of development, relative to project implementation How grantee determined members of project partnership implementation team; qualifications established for team membership; member characteristics Roles and responsibilities of project partnership implementation team and its members Duration of operation of implementation team, frequency of meetings, forms of communication by team members Barriers and facilitators to effectiveness of implementation team in RPG project Accomplishments of implementation team Staff perception of usefulness of implementation team
Services
Implementation plans	Development of plans and procedures used to monitor project activities for compliance with plans and consistency; what details were included in plan (such as types of tasks, timeline for activities, partners responsible for tasks) Modifications to the RPG project implementation plan that have occurred since implementation began; reasons for modifications; whether they were planned or unplanned; how decisions about modifications were made How RPG projects developed strategies to address barriers to the delivery of high-quality services Staff perceptions of whether implementation plan was communicated sufficiently, executed successfully, and useful in proactively identifying roadblocks to implementation Barriers and facilitators to success of implementation plan
Service selection process	Process by which RPG project selected planned RPG services, including Identified community need to be addressed Conducted needs and resource assessment (including need for and availability of space, technology, financial, and other resources, including in-kind contributions by grantee and/or partners) Assessed alignment of services with planned target population Assessed organization capacity/readiness Considered other programs or services Considered need for adaptation of planned service Assessed alignment of services with grantee and partners’ goals and mission
Referral processes to RPG services	How and when RPG project determined referral pathways Sources of referrals, length of relationship with these referral sources, how relationships were established, relative size of enrollment from each referral source Referral sources that consistently refer individuals that meet eligibility criteria and engage in the RPG project Process used by partners to refer potential participants to RPG Any changes to outreach and referral strategies and why How staff accept referrals for RPG services Barriers and facilitators to establishing pathways and translating referrals into participation
Referral processes from RPG services	Types of community services to which RPG project staff refer participants Extent to which needed services are available and accessible in the community Plan for conducting initial and ongoing assessments of participants’ needs and linking them to appropriate services Extent to which participants follow up on referrals and take up the services Process for tracking referrals, how often progress is monitored, and who is responsible for monitoring
Staffing roles and perceptions	Staffing structure for the RPG project, including frontline staff and those who support implementation (project directors, managers, and supervisors) Responsibilities and expectations for each staff role Staff perception on extent to which RPG project goals and purpose of partnership were explained Involvement of frontline staff in the planning and decision-making processes, and how concerns were addressed Who and how RPG staff interact with other RPG staff in partner organizations Staff perceptions of the utility of an integrated approach to the provision RPG services RPG project’s ability to provide sufficient training to all necessary staff
Internal evaluation and continuous quality improvement	RPG project expectations about the quality of services delivered through RPG; how RPG project defines high quality delivery for RPG services, and why project defines service quality in this manner Efforts to monitor RPG service delivery, quality, and project performance; adherence to curricula or other programming, engagement, participation, and participant outcomes; who completes monitoring; what is monitored and how often; how information is used by staff Strategies for identifying successes and challenges to implementation for purposes of continuous project improvement Use of improvement cycles or other continuous quality improvement strategies Use of data systems to monitor progress toward goals and RPG project performance How staff use the data to make project decisions Staff perception of the relevance and usefulness of project data and management information system Barriers and facilitators to using systems and conclusions derived from data Facilitators and barriers to ongoing evaluation and project improvement
Sustainability
Plans and activities to sustain services and partnership after grant period ends	Decision-making process for sustaining services/partnership Strategies to engage external systems (such as health, education, housing) that may not have been involved in partnership in provision of financial, organizational, or other resources after the grant period ends How partners identified funding sources/financing to sustain services Mix of financial resources (such as, state, local, federal, private) Sustainability of implementation team for scale-up Sustainability of implementation plan as grant period ends Sustainability of referral pathways RPG project’s ability to provide sufficient training to all necessary staff for sustainability Sustainability of data systems and processes needed to monitor and improve project implementation
Federal, state, local, tribal, and community context
Federal, state, local, tribal, and community context	Federal, state, or local policies and policy climate, and how they impeded or supported project development Role of the courts and willingness of family court judges to support and participate in RPG For projects who serve many American Indian/Alaska Native participants, How partnerships formed, operate, and serve clients Characteristics of communities in which RPG is offered Unexpected events that altered RPG project activities; how they affected the project and how they were addressed

¹ He, A. S. (2015). Examining intensity and types of interagency collaboration between child welfare and drug and alcohol service providers. Child Abuse & Neglect, 46, 190–197. http://dx.doi.org/10.1016/j.chiabu.2015.07.004

² An implementation team is a team of individuals focused on supporting the implementation of services. The team may help increase the buy-in and readiness of staff, coordinate the supports staff may need to implement the services (particularly evidence-based programs or practices [EBPs]) with fidelity, assess the fidelity of the implementation of the services, and problem-solve implementation challenges. Collectively, the team possesses an in-depth knowledge of the services, knowledge of implementation best practices, and experience using data to improve program quality (Metz & Bartley, 2012).

This page has been left blank for double-sided copying.