Grantee Leadership Interview Follow-up

OMB# 0990-XXXX

Expiration Date: XX/XX/XXXX

INSTRUMENT 1B

FOLLOW-UP GRANTEE PROGRAM LEADERSHIP STAFF INTERVIEW TOPIC GUIDE

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-XXXX. The time required to complete this information collection is estimated to average 1 hour per respondent, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Health & Human Services, OS/OCIO/PRA, 200 Independence Ave., S.W., Suite 336-E, Washington D.C. 20201, Attention: PRA Reports Clearance Officer.

INSTRUMENT 1B. TOPIC GUIDE FOR FOLLOW UP INTERVIEWS WITH GRANTEE LEADERSHIP

INTRODUCTION

Thank you for agreeing to meet with us. I am from Mathematica Policy Research. I’m part of an independent research team that is studying the implementation of the programs funded by the Office of Adolescent Health (OAH) Teen Pregnancy Prevention (TPP) grant in 2018.

The purpose of our discussion today is to learn more about your experiences planning and implementing [insert program name] in Phase I of your 2018 OAH grant. Your point of view is valuable. The interview should last about [60 minutes], and we will take notes during our conversation so we can accurately represent your experience and views in our reporting. We would also like to record this discussion to make sure our notes are accurate, if that is okay with you.

Your responses will be kept private, and the notes and recording from this discussion will not be shared with anyone beyond the research team. The recording will be erased once we have finalized our notes. We will combine most information from this conversation with information from other discussions we conduct.

We will report most information based on these discussions in the aggregate. We may use quotes to illustrate findings, but if we do, we will not report any information that will allow a quote to be identified with you.

Please keep in mind:

There are no right or wrong answers to these questions. We just want to learn about your experience and perspective.

Your participation in this conversation is completely voluntary. You don’t have to answer any questions you don’t want to answer during our discussion today.

Do you have any questions for us before we get started?

PRE-DISCUSSION QUESTIONS

I want to emphasize again that there are no right or wrong answers to our questions. By voluntarily agreeing to participate in this study, you are agreeing to answer these questions with responses that are true for you.

Do you understand the purpose of our conversation today?

Do you have any questions before we begin?

PROJECT CONTACT INFORMATION

If you have further questions about this project or if you have a research-related problem, you may contact the project director, Dr. Jean Knab, at (609) 945-3367 or JKnab@mathematica-mpr.com.

TOPICS FOR DISCUSSION

[Note to interviewer: Topics and questions will be tailored to each grantee’s individual context and stage of readiness, based on program materials, TA calls and the quarterly reports.”].

A. Program Readiness: Updates to Program Design and Readiness for Implementation

1. Changes (if any) to the program model in Year 2

• Changes to the program’s theory of change or logic model

• Changes to the core components

• Changes to the program’s standardized operations

B. Organizational readiness: Preparing for program evaluation

1. Continuous quality improvement (CQI)

• How extensively CQI approaches were utilized in developing and refining programs

• Process for incorporating feedback from community stakeholders, youth, and families into program improvement during Phase I

• Description of data that were used for CQI

• How data that were collected are used for program improvement

2. Formative evaluation

• Description of the measures and data used for the formative evaluation

• Description of how data were used to influence planning in Phase I

• Advantages of having the Phase I funding to prepare for a summative evaluation

• Disadvantages of having the Phase I funding

• Accomplishments due to the Phase I funding that would not have been feasible otherwise

• Plans for scale-up, if any

3. Summative evaluation

• Plan for the summative evaluation (e.g., timeline, target outcomes, plans for data collection and analysis)

• Factors that were helpful or challenging in designing a rigorous evaluation plan

• Extent of community commitment to rigorous evaluation

• Inclusion of an economic evaluation

4. Process evaluation

• Changes to plans for collecting process data: types of data included, and who collected them

• Updates related to data collection based on coaching or technical assistance to assess staff needs and effectiveness

• Updates to fidelity measurement process (e.g., measure, analysis of data, using data for program improvement)

B. Promising evidence: Early implementation experience

1. Implementation supports

• Updates on staff recruitment and hiring in Year 2

• Staff training overview and experiences in Year 2

• Changes in staff recruitment, hiring, or training in Year 2

• Lessons learned from staff training in Year 2

• Changes in supervision structure in Year 2

• Successes and challenges of supervision structure

• Changes in methods used for youth recruitment and engagement in Year 2

• Lessons learned from youth recruitment and engagement in Year 2

• Updates to program delivery structure (frequency, methods, setting, length, etc.)

• Lessons learned from program delivery in Year 2

2. Community need and demand

• Changes or updates to main unmet needs in the target communities

• Changes in community perceptions about and desire for programs focused on preventing teen pregnancy

• Lessons learned from efforts to engage and request input from local stakeholders and beneficiaries on the demand for the program

• Updates on response from youth and families during Phase I (i.e., descriptive or quantitative feedback from end users on program fit, challenges, successes)

3. CQI

• Update on staff experiences with the CQI process

• Update on data collected for CQI and how they were used

• Adjustments or changes to the original CQI plan

• How program was improved as a result of CQI

4. Fidelity

• Early data on fidelity measures and how these were used

• Data on program quality and how they were used

• Adjustments made to program or implementation infrastructure based on fidelity data

5. Participant outcomes

• Indicators of youth engagement and participation, and how these changed over time

• Reasons for changes in response from youth and families over time

E. Lessons learned

• Overall lessons related to program and organizational readiness for implementation and evaluation

• Successes, challenges, and lessons based on phased grant structure

This page has been left blank for double-sided copying