Key Program Staff Interview

OMB No. 0990-XXXX

Exp. Date XX/XX/20XX

INSTRUMENT 2:

KEY PROGRAM STAFF INTERVIEW TOPIC GUIDE

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-XXXX. The time required to complete this information collection is estimated to average 1 hour per response, including the time to review instructions, search existing data resources, gather the data needed, to review and complete the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Health & Human Services, OS/OCIO/PRA, 200 Independence Ave., S.W., Suite 336-E, Washington D.C. 20201, Attention: PRA Reports Clearance Officer

INSTRUMENT 2. TOPIC GUIDE FOR PROGRAM SUPERVISORS AND FRONTLINE STAFF

INTRODUCTION TO THE INTERVIEW

Thank you for agreeing to meet with us. I am from Mathematica Policy Research. I’m part of an independent research team that is studying the implementation of the programs funded by Office of Adolescent Health (OAH) as part of the 2018 Teen Pregnancy Prevention (TPP) grant program.

The purpose of our discussion today is to learn more about your experiences planning and implementing [insert program name] in Phase I of your 2018 OAH grant. Your point of view is valuable. The interview should last about 60 minutes, and we will take notes about our conversation so we can accurately represent your experience and views. We would also like to record this discussion to ensure our notes are accurate, if that is okay with you.

Your responses will be kept private, and the notes and recording from this discussion will not be shared with anyone beyond the research team. The recording will be erased once we have finalized our notes. We will combine most information from this conversation with information from other discussions we conduct.

We will report most information based on these discussions in the aggregate. We may use quotes to illustrate findings, but in these instances, we will not report any information that will allow a quote to be identified with you.

Please keep in mind:

There are no right or wrong answers to these questions. We just want to learn about your experience and perspective.

Your participation in this conversation is completely voluntary. You don’t have to answer any questions you don’t want to answer during our discussion today.

Do you have any questions for us before we get started?

PRE-DISCUSSION QUESTIONS

I want to emphasize again that there are no right or wrong answers to our questions.

By voluntarily agreeing to participate in this study, you are agreeing to answer these questions with responses that are true for you.

Do you understand the purpose of our conversation today?

Do you have any additional questions before we begin?

PROJECT CONTACT INFORMATION

If you have questions about this project, or if you have a research-related problem, you may contact the project director, Dr. Jean Knab, at (609) 945-3367 or JKnab@mathematica-mpr.com.

TOPICS FOR DISCUSSION

(Note: Topics and questions will be tailored to each grantee’s individual context and stage of readiness by the end of Year 1, based on TA calls, discussions with grantee leadership, and the quarterly reports. When topics are more relevant to a particular respondent that is noted below).

A. Program readiness: Program design and readiness for implementation

1. Theory of change

• Target outcomes of program

• Process for selection or development of the program(s)

• Target population(s) for whom the program is intended

2. Core components

• Implementation setting(s)

• Essential elements or core components of the program

• Protective factors that the program targets and strategies used to address them

• Extent that elements of optimal health, risk avoidance, and/or risk reduction are incorporated into the intervention

• Program structure (frequency, duration, number of sessions, etc.)

3. Standardized program operations

• Description of program materials (e.g., facilitator manual, student workbooks, handouts)

• Defined fidelity and quality benchmarks

• Staffing requirements and training processes

B. Organizational readiness: Preparation and planning for implementation

1. Enabling organizational context

• Organization's history of working in target communities

• Process for assessment of community needs and demand for program and/or services

• How input from youth or community stakeholders was incorporated into program planning

• Key challenges or barriers to getting community support or buy-in

• Other similar programs and services available in the community

• Assessment of leadership support and buy-in to the need for the program (e.g., staff involvement in decision making or "champions")

• Extent that a shared vision was used to guide intervention planning among organization or agency staff

• Extent that staff at various levels were involved in intervention development, selection and/or planning

2. Infrastructure and implementation supports

• Description of partnerships utilized during Year 1 (e.g., key partners, process for involvement)

• Staff recruitment and hiring procedures

• Staff role and background

• Staff qualifications and credentials

• Description of the current staffing structure and capacity

• Timeline for recruitment and hiring

• Descriptions and benchmarks of what staff are expected to do for optimal implementation (i.e., staff performance)

• Extent that implementation procedures and training are operationalized for repetition

• Youth recruitment plan (e.g., inclusion/exclusion criteria, setting, methods of outreach)

• Plans for initial (and refresher) staff training for optimal implementation (e.g., timeline, content, delivery of training)

• Plans for individual and group supervision

• Technical assistance and other supports available to staff during implementation

C. Organizational readiness: Preparing for program evaluation

1. Continuous quality improvement (CQI)

• Extent that CQI approaches were utilized in program development and refinement

• Process for incorporating community stakeholder, youth, and family feedback into program improvement during Phase I

• Description of data that were used for CQI

• How data collected are used for program improvement

2. Formative evaluation (Supervisors only)

• Description of the measures and data used for the formative evaluation

• Description of how data were used to influence planning in Phase I

• Advantages of having the Phase I funding to prepare for a summative evaluation

• Disadvantages of having the Phase I funding

• Accomplishments due to the Phase I funding that would not have been feasible otherwise

• Plans for scaling up, if any

3. Process evaluation (Supervisors only)

• Plans for process data collection: types of data included and who will collect

• Plans to collect data based on coaching or technical assistance to assess staff needs and effectiveness

• Fidelity measurement overview (e.g., types of measures, analysis of data, collecting and using data for program improvement)

D. Promising evidence: Early implementation experience

1. Implementation supports

• Staff recruitment and hiring procedures

• Staff training overview and experiences

• Lessons learned from staff training

• Extent that training was provided to staff as intended

• Extent that coaching or technical assistance was provided as needed

• Overview of staff supervision and support (e.g., frequency, extent of supervision and TA)

• Successes and challenges of staff performance

• Successes and challenges of supervision structure

• Lessons learned regarding staff supervision and support

• Overview of youth recruitment and retention process (e.g., timing, reaching targets)

• Recruitment and retention successes and challenges

• Lessons learned from youth recruitment, engagement, and retention

2. Community Need and Demand

• Main unmet needs in the target communities

• Community perceptions about and desire for teen pregnancy prevention programming

• Efforts made to engage and solicit input from youth, families, and other local stakeholders related to demand for program

• Response from youth and families during Phase I (i.e., descriptive or quantitative feedback from end users on program fit, challenges, successes)

3. CQI

• Staff experiences with the CQI process

• Data collected for CQI and how it was used

• Adjustments or changes to the original CQI plan

• Improvements made as a result of CQI process

4. Fidelity

• Process for measuring and monitoring program fidelity and quality

• Extent that the program is being implemented with fidelity

• Overview of adaptations or adjustments that were made during implementation

• Lessons learned related to measuring and monitoring fidelity and quality to prepare for the summative evaluation

5. Participant outcomes

• Changes in participant engagement and participation in program

• Challenges related to participant engagement and participation in program

E. Lessons learned

• Overall lessons related to program and organizational readiness for implementation and evaluation

• Successes, challenges and lessons based on phased grant structure

This page has been left blank for double-sided copying.