Implementation Survey Interview Topic Guide (State-level Respondents)

PREP MULTI-COMPONENT EVALUATION omb supporting statement: INSTRUMENT 1

iNSTRUMENT 1

Implementation Survey Interview Topic Guide

A. Introduction

Thank you for speaking with me today. As you know, my organization, Mathematica Policy Research is leading the Personal Responsibility Education Program (PREP) Multi-Component Evaluation on behalf of the Administration for Children and Families. The study’s purpose is to provide information to policymakers and administrators about how PREP-funded programs are operationalized in the field, to collect and analyze program performance measure data, and to assess the effectiveness of selected PREP-funded programs at achieving program goals.

As part of the PREP Evaluation, the study team is gathering information from four states to produce a detailed description of the structures and supports that are in place to assist in PREP program implementation. To achieve this goal, we are speaking with staff from various organizations involved with PREP program implementation and support to learn about the different structures and practices designed to support and monitor the PREP program.

The purpose of this interview is to collect information on supports for program implementation; it is not an assessment or monitoring effort. Everything that you say will be kept strictly private within the study team and participation is voluntary. The study report will include a list of the states that participated in these interviews, but all interview data will be reported in the aggregate. In our reports, we will not identify a specific person, program, or state unless we are highlighting a promising practice. If you do not feel comfortable answering a particular question, please let me know and we will move on to the next one.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0398 and it expires XX/XX/2017.

This discussion should take about one hour. Do you have any questions before we begin?

[The research team will not ask all questions of all respondents. For example, some questions pertain only to the work of certain staff (such as the state agencies responsible for overseeing the PREP program, and so on). Therefore, interviews will also be customized based on each respondent’s job duties and knowledge.]

B. Topic Guide for Telephone Interviews in Four Grantee States

Construct	Interview Topics
PREP Structure and Planning
Implementation structure	Characteristics of organizations involved in PREP implementation, such as mission and length of operation Changes in staffing structures and responsibilities for the PREP program, by organizational affiliation, since initial design and planning interviews (summer of 2012) Modifications or adaptations to PREP implementation plans that has occurred and why Potential future modifications or adaptations to implementation plans based on experience, and why
Model fit for service providers	Lessons learned in the selection of PREP models at state grantee and provider levels Lessons learned in the selection of providers to deliver PREP services Level of experience (and success) in delivering the selected model by providers prior to PREP; extent of similar in any challenges experienced prior to and during PREP implementation Level of experience delivering teen pregnancy prevention programs Level of experience in serving the targeted population for PREP Successes and challenges in transitioning to PREP implementation Organizational characteristics and role in community and how characteristics and role provide advantages or challenges in delivering PREP services Perceived degree of overall “fit” of PREP model(s) within local provider organization and community (the degree to which the model aligns with the organization’s objectives and service capacity)
Lessons from PREP planning	Time from planning to early implementation and factors that determined the timeframe Lessons learned during planning and early implementation of PREP that informed program refinement Retrospective view of approaches to PREP planning (and potential differences in decisions or process) based on experience
PREP Program Support: Training and Technical Assistance
Implementation support structure	Changes in the role of state grantee agency in supporting implementation; number and responsibilities of staff supporting PREP implementation since initial design and planning interviews (summer of 2012) Use and number of intermediary organizations to support PREP training and technical assistance (T/TA); number and responsibilities of staff supporting PREP implementation T/TA provider’s experience with and exposure to the PREP model they are supporting; experience working with grantee organization, and service providers implementing PREP Role of PREP model developer(s) in supporting implementation Strengths and weaknesses to the T/TA approach pursued to support PREP implementation Adequacy of resources to support T/TA and changes in funding for T/TA over time
Training to support PREP	Training or guidance from model developer that informed implementation support for PREP Provision, duration, format, and frequency of training for T/TA providers to prepare them to support PREP Provision, duration, format, and frequency of training for facilitators and supervisors to prepare them to deliver PREP Specific requirements or benchmarks that facilitators must meet to deliver PREP sessions Sufficiency of training in preparing staff to deliver or support PREP, or what additional training may be needed Helpfulness of training to prepare staff to deliver or support PREP Successes, challenges, lessons learned about training
Ongoing support and technical assistance	Use of data or other methods to inform TA and implementation support needs (to include federal performance measures, fidelity data, supervision and performance assessment, state grantee activities) Initiation, frequency, content, format and level of satisfaction of TA received from the model developer or national model purveyor Initiation, frequency, content, format and level of satisfaction of TA received from the state grantee agency Initiation, frequency, content, format and level of satisfaction of TA received from an organization designated to support PREP in the state (or from any other entities) Timing and means of first introduction of TA provider to program providers Level of satisfaction with ongoing level of communication between TA provider and program providers Feedback loops following TA (information to state grantee agency; feedback from program provider to TA provider about usefulness of TA; follow-up from TA provider with program provider to track or assess use of TA in practice) Areas of unmet needs for technical assistance or ongoing implementation support Key successes in technical assistance for the PREP program
Implementation Drivers: Competency Drivers
Staff selection	Changes needed in organizational staffing to implement PREP; whether new roles or new staff were needed Skills, abilities, and other qualifications sought for PREP staff; whether these differ for existing staff or new hires Whether guidance was received to specify the qualifications for PREP staff Methods for communicating expectations for PREP during recruitment or hiring process Ease or difficulty in hiring staff with the qualifications for PREP; successful methods for getting staff in place Qualifications of current PREP facilitators and consistency in qualifications across all facilitators Key lessons from the initial staff selection and recruiting process
Staff turnover and retention	Degree of turnover among PREP staff at all levels How the turnover for PREP compares to other programs within organization Main reasons for degree of turnover Methods and timing of filling vacated positions Operational challenges due to staff turnover Strategies for retaining staff and preventing future turnover Successes and challenges in staffing the PREP program
Staff expectations and receptiveness	Methods of orienting staff to the requirements and expectations of the PREP program Receptiveness of staff to use of formal curriculum and specifications of PREP Aspects of PREP that staff are resistant to Perceived level of consistency in which PREP staff (particularly facilitators) deliver PREP
Staff supervision and performance assessment	Expectations for supervision or oversight of PREP program facilitators; frequency and methods for supervisory activities Degree to which expectations for supervision are met Common concerns or issues raised by facilitators during supervisory activities and ability of supervisors to address them Methods of performance assessment of PREP facilitators; use of observations, intended frequency, and focus Feedback loop following performance assessment; types of discussions, usefulness for staff in improving skills and fidelity to the PREP model Use of performance assessment information by administrators to support PREP implementation to improve practice and fidelity
Implementation Drivers: Organization Drivers
Decision-support data systems	Data collected (beyond performance measures) to inform or support PREP implementation; what is collected, why, and how How data are used to support PREP implementation and benefits from use of data Specific ways in which data has informed changes or improvements to PREP implementation or services Procedures for promoting timeliness and consistency in data entry related to PREP implementation; frequency or timing of data entry after a PREP activity Documentation, guidance, or training for data entry and use Type of data system used to track PREP implementation and support Perceived quality of data collected for the PREP program
Facilitative administration	Shared understanding among administrators and staff within organization about the goals of the PREP program and what it means to effectively implement an evidence-based program Presence of champion within organization who promotes shared understanding of PREP goals and expectations Methods of developing a shared understanding (or buy-in) of PREP among PREP staff as well as among other staff in the organization Resistance to PREP within the organization Perception of role of state grantee agency as a partner in supporting PREP implementation Perceptions of commitment of each implementation partner in supporting the level of staffing, training, and ongoing assistance necessary to implement PREP Willingness to adjust or develop new policies and procedures to support PREP model Methods to identify and reduce administrative or operational barriers to PREP implementation
Communication, decision-making, and feedback loops	Description of lines of communication between service delivery staff and management within program provider organizations to facilitate or support PREP implementation Participation of service delivery staff in developing or modifying procedures or processes or identifying implementation challenges; formality of their involvement and regularity of consultation or feedback Examples, if applicable, of policies, procedures, or processes that have changed within program provider organizations as a result of service delivery staff input Description of frequency and mode of communication between state grantees and program providers to support PREP implementation, and changes in this communication over time Examples, if applicable, of policies, procedures, or processes that have changed at the state grantee level as a result of program provider input Means of identifying issues or problems in PREP implementation across all entities involved in service delivery and support, and methods of developing strategies across entities to address issues
Systems interventions	Methods of connecting with other systems or partners to support PREP implementation at the state and local levels Change in partners over time Capacities that partners bring or help build to support PREP implementation Identification of barriers or problems across partners and methods of addressing Ability to gain buy-in and cooperation with organizational partners and within the community at large to support the PREP model Resistance to the approach or content of the PREP program within the state/community or from particular organizations such as schools or other community organizations; ways this has affected implementation Changes in existing service delivery structures across entities or within communities to support PREP implementation Perception of PREP program contribution to infrastructure to support youth education programs to prevent teen pregnancy in state/community
Fidelity Assessment and Monitoring
Program modifications or adaptations	Description of PREP model modifications or adaptations that were implemented, including type and reasons needed, degree of adaptations and use of existing adaptation kit to guide adaptations Process for obtaining approval of modifications or adaptations and entity that had final approval Process for rolling out modifications or adaptations (if made after implementation began); and process for monitoring implementation of modification or adaptation Collaboration with model developer, state grantee, or T/TA provider to inform modification or adaptation Plans for additional modifications or adaptations to the PREP model Type and degree of unplanned model adaptations that occurred (and why) Guidance and support provided by state grantee or T/TA provider with regard to implementation of modifications or adaptations
Adherence to service model	Methods that state grantee agency uses to convey expectations regarding quality and consistency of PREP services Efforts to ensure that all staff carry out program activities in a consistent manner Obstacles that impede the delivery of consistent services across program staff; strategies to remove obstacles Access to materials (such as curriculum, supporting materials, procedures manuals) to support ongoing program delivery and address questions or issues related to practice among PREP staff Concerns by program model developer about the ability of program providers to provide the PREP model as designed Easiest and most challenging aspects of the PREP program to implement
Monitoring service delivery	How high quality delivery of PREP services is defined, and by whom Use of existing fidelity indicators (from model developer) or extent of development of indicators such as fidelity monitoring logs Ways that fidelity data are used to support or improve implementation and by whom Aspects of implementation that are monitored on a regular basis, with what frequency, how monitored, by whom, for what purpose, to whom reported Methods of collection of fidelity data (self-report, observation, data system) Documentation and tracking of planned and actual content and activities of each PREP session, and adaptations made Methods of assessing quality of PREP sessions, frequency, by whom, and how is information used Changes in PREP service delivery due to monitoring efforts Lessons learned, successes or challenges identified through monitoring
Evaluation, Sustainability, and Lessons Learned
Evaluation capacity	Evaluation activities focused on PREP program Role of external entities in monitoring and evaluating PREP implementation; number and type of organizations and staff involved Methods of communicating evaluation information to PREP stakeholders at the state and local levels Methods of using evaluation information to improve PREP implementation Key findings from evaluation that are currently available Level of interaction and communication between evaluators and state grantee agency, program providers and T/TA providers
Sustainability	Sustained readiness of organization (financial, organizational, logistical capacities) to implement PREP Efforts to sustain the implementation infrastructure and funding to deliver PREP services after the federal grant period ends Level of coordination in efforts to sustain the PREP program across partners at the state and local level Opportunities and challenges in ability to sustain the implementation or supports for the PREP program
Perceptions and lessons learned about supporting PREP implementation	Opinion of current number and type of PREP program models as right for organization’s ability to implement (or support) PREP with quality and fidelity Opinion of current number of program providers as right for organization’s ability to implement (or support) PREP with quality and fidelity Factors that facilitate ability to implement PREP Challenges that affect ability to implement PREP and how they have been addressed Based on experience, how might the organization have done things differently to implement or support PREP Advice for other organizations implementing evidence-based teen pregnancy prevention programs Advice for other state and federal agencies about how to support evidence-based programs generally or PREP specifically

I1.4