Form 1 Interview Topic Guide

OMB No.: 0970-0444

Expiration date: 03/31/2017

Regional Partnership grants (RPG) cross-site evaluation

TOPIC GUIDE FOR implementation study site visit interviews

The core implementation study for the Regional Partnership Grants (RPG) cross-site evaluation will include two multi-day site visits to each grantee. Researchers will interview grantee program directors, managers, supervisors, and frontline staff who work directly with families during the site visits. Interviews will be conducted either one-on-one or in small groups, depending on staffing structure, roles, and the number of individuals in a role.

Topic	Sub-Topic
Informant Characteristics
Informant Characteristics	Job title Education background/licensing qualifications Years in current position and with agency Role on RPG Relevant prior experience with the RPG grant program, target population, and evidence-based programs (EBPs) being implemented by the grantee
Pre-Implementation
Selection of EBPs	Grantees’ prior experience with similar programs and how prior experience informed the RPG design Knowledge of evidence-based practices (EBPs) Involvement of partners and other community organizations/stakeholders in the planning and decision-making processes, and how concerns were addressed Involvement of frontline staff in the planning and decision-making processes, and how concerns were addressed Key design decisions made during the planning phases and rationale for those decisions Process by which grantees selected the planned interventions, including: Community need to be addressed by the EBP Needs and resource assessment (including need for and availability of: space, technology, financial and other resources, including in-kind contributions by grantee and/or partners) Alignment with planned target population Assessment of organization capacity/readiness Whether other programs were considered Champions for certain EBPs Need for adaptation Alignment with grantee and partners’ goals and mission Challenges encountered during the planning process and steps taken to address them
Referral Processes to RPG services	How and when grantee determined referral pathways Sources of referrals, length of relationship with these referral sources, and how relationships were established, relative size of enrollment from each referral source Referral sources that consistently refer individuals that meet eligibility criteria and engage in the RPG program Process used by other agencies to refer potential participants to RPG Any changes to outreach and referral strategies and why Barriers and facilitators to establishing pathways and translating referrals into participation Sustainability of referral pathways
Staff Selection and Hiring and Retention	Staffing structure for the RPG program, including frontline staff and those who support their implementation (program directors, managers, and supervisors) Responsibilities and expectations for each staff role Timeline and process for hiring new staff or reassigning staff to fill RPG roles Re-assignment of existing staff to implement RPG services and/or support implementation Whether job postings specified specific qualifications required for implementation of EBPs selected; recruitment methods used to identify likely qualified candidates; protocols and criteria used to identify qualified candidates Number of full-time equivalent (FTE) staff in RPG program devoted to administration and direct service provision Extent to which staff in the RPG program had other responsibilities in addition to RPG Current staff vacancies; length of vacancies; efforts to fill vacancies Extent of staff turnover since initiating program operations; reasons for staff turnover (or staff retention); effects on remaining staff when turnover occurs; length of process to replace departing staff Effect of staff turnover on enrollment and service delivery; programmatic adjustments and accommodations as a result of turnover Efforts to prevent future turnover and retain current direct service staff and supervisors Expectation of continued rate of turnover for sustainability Likelihood of identifying individuals with necessary qualifications for sustainability and/or scale-up
Pre- and In-service Training	Plan for and approach to providing supervision and training to direct service staff, including the intended frequency, duration, and focus Initial and in-service training plan for new and ongoing RPG program staff, including the frequency, content, length, and format of training, and individual or organization providing the training (includes whether the EBP’s developer or purveyor was involved in training, whether training covers key components of EBPs and whether trainees were given time to practice implementation with feedback) Grantee’s ability to provide sufficient training to all necessary staff, at start-up and for sustainability and/or scale-up Staff perception on extent to which training(s) provided necessary information on theory of intervention(s), goals of RPG, as well as competencies needed to implement Whether staff received the planned level of initial and ongoing training and guidance
Implementation Teams	Organizational structure for the RPG project Development of implementation team; timing of development, relative to program implementation How grantee determined members of implementation team; qualifications established for team membership; member characteristics Roles and responsibilities of team and its members Strongest advocate for RPG program and how demonstrated; role of advocate and how individual emerged as advocate Existence and role of advisory committee and/or steering committee Duration of operation of implementation team; frequency of meetings; forms of communication by team members; Turnover of team membership; reasons for turnover; impact on implementation Barriers and facilitators to fully installing implementation team in RPG project Accomplishments of implementation team Staff perception of usefulness of team Sustainability of team for scale-up
Implementation Plans	Development of plans and procedures used to ensure that all staff carry out program activities as planned and in a consistent manner; what details were included in plan (e.g. types of tasks, timeline for activities, staff responsible for tasks) Modifications to the grantee’s RPG implementation plan that have occurred since implementation began; reasons for modifications; whether they were planned or unplanned Development of strategies to address barriers to the program’s ability to deliver high-quality services Staff perceptions of whether implementation plan was communicated sufficiently, executed successfully, and useful in proactively identifying roadblocks to implementation Barriers and facilitators to success of implementation plan Sustainability of implementation plan as RPG programs adapt
Early and On-Going Implementation
Facilitative administrative support	Grantee oversight of RPG activities and partner services Changes in the demonstration’s organizational structure Changes in grantee, partner, or RPG program leadership staff that occurred during the demonstration and may have impacted the direction of the RPG program Strategies to reduce administrative barriers, develop communication and feedback protocols, implement program improvement based on data or staff suggestions Staff perception of availability of these strategies Staff perception of administration’s commitment to supporting the implementation of EBPs Sustainability of leadership approach Facilitators and barriers to providing administrative support
Supervision and Feedback	Whether protocols were established for providing feedback Use of staff performance assessments for frontline staff Sources of data for performance assessments Facilitators and barriers to supervisory and feedback mechanisms
Technical Assistance and Coaching	Grantee and partners’ capability to provide ongoing TA for duration of RPG and beyond Use of external TA providers, including curriculum developers, Children’s Bureau, and other entities; whether staff have accessed these resources and, if so, helpfulness of the technical assistance Extent to which grantee believed TA providers were aware of and receptive to goals for RPG Whether TA and coaching led to further adaptation of program model Whether TA and coaching led to need for further training sessions Topics on which staff needed more training and technical assistance Plans for when, how, and why TA or coaching would be provided Sustainability of TA and coaching networks, including financial and other resources
Internal evaluation and continuous program improvement	Grantee expectations about the quality of services delivered through RPG; how grantee defines high quality delivery for core services, and why program defines service quality in this manner Efforts to monitor service quality, adherence to curricula or other programming, client engagement, participation, and participant outcomes; who completes monitoring; what is monitored and how often; how information is used by staff Strategies for identifying successes and challenges to implementation for purposes of continuous program improvement Use of improvement cycles or other continuous quality improvement strategies Dissemination of RPG implementation to policymakers: frequency of exchanges, extent to which such exchanges are purposeful and part of usual practice Dissemination of RPG implementation to partners: frequency of exchanges, extent to which such exchanges are purposeful and part of usual practice Dissemination of RPG implementation to researchers, other practitioners: plans to publish findings Facilitators and barriers to ongoing evaluation and program improvement Sustainability of systems needed to monitor and improve program implementation
Decision Support Data Systems	Use of data systems to monitor progress toward goals and partner performance Plan for monitoring program performance and for tracking service delivery and quality, adherence to curricula or other programming, client engagement and participation, and participant outcomes Data sources and frequency of data collection and analysis How staff use the data to make program decisions Dissemination plans for: partners/stakeholders, administrators, support staff, frontline staff; perception of purpose of feedback dissemination Staff perception of the relevance and usefulness of program data, management information system Sustainability of data systems Barriers and facilitators to using systems and conclusions derived from data
Referral Processes from RPG services	Types of community services to which RPG program staff refer participants Extent to which needed services are available and accessible in the community Plan for conducting initial and ongoing assessments of participants’ needs and linking them to appropriate services Extent to which participants follow up on referrals and take up the services Process for tracking referrals, how often progress is monitored, and who is responsible for monitoring
Interventions with external systems	Strategies to engage external systems in provision of financial, organizational, or other resources Types of external systems/organizations engaged Staff perception of alignment of organizations with grantee’s goals for RPG services
Adherence/Fidelity
Fidelity	How grantee defines high quality delivery of core components of the EBP, and why grantee defines quality in this manner The extent to which staff adhere to the EBP guidelines (during service delivery) Consistency with which services are provided, per EBP guidelines Grantee expectations about the quality of services delivered through the EBP Attitudes expressed by staff towards the use of the EBP Extent to which program staff think that the EBP will improve outcomes Staff understanding of EBP’s theory of change (how program services are linked to desired outcomes)
Staff Attitudes Toward Implementation	Staff perceptions as to whether these multiple roles had an effect on their ability to implement the program as designed Extent to which RPG program staff “bought in” to the idea that providing substance abuse treatment, family strengthening, parenting education, and/or in an integrated package would improve participant outcomes Staff perceptions of the EBP’s fit to the target population, strengths, and weaknesses Staff perceptions about how well the model has worked in practice; benefits and challenges of this approach to demonstration leadership Staff perceptions of the strengths and weaknesses of the demonstration’s approach to leadership Staff perceptions of the utility of an integrated approach to the provision of core services
Community, State, and National Context
State and Local Context	State or local policies and policy climate, and how they impeded or supported program development Other state or community organizations providing parenting or employment services; how the services provided by these organizations differ from the RPG program; whether and how these services may have affected the RPG program; and use of these other services by participants Role of the courts and willingness of family court judges to support and participate in RPG Physical, social, and economic characteristics of communities in which RPG is offered Unexpected events that altered RPG program activities; how they affected the program and how they were addressed

THE PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) Public reporting burden for this collection of information is estimated to average of 90 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.

2