0990-New Sustainability SSB Passback_Revised_(3)

PART B: SUSTAINABILITY STUDY OF PROGRAMS Funded by OAH in 2010

Part B: Justification for the Collection of Data—Sustainability Study of Programs Funded by OAH in 2010

June 2016

Submitted to:

U.S. Department of Health and Human Services

Office of Adolescent Health
Department of Health and Human Services

1101 Wootton Parkway, Suite 700
Rockville, MD 20852

Project Officer: Amy Farb
Contract Number: GS-10F-0050L/HHSP233201300416G

CONTENTS

PART B: INTRODUCTION 1

B1. Respondent universe and sampling methods 2

1. Universe 2

2. Site selection 3

3. Selection of respondents for each grantee 3

B2. Procedures for collection of information 4

B3. Methods to maximize response rates and deal with nonresponse 5

B4. Test of procedures or methods to be undertaken 5

B5. Individuals consulted on statistical aspects and individuals collecting and/or analyzing data 5

ATTACHMENTS

ATTACHMENT A: PERSONS CONSULTED ON INSTRUMENT DEVELOPMENT

ATTACHMENT B: EMAIL REQUESTING PARTICIPATION IN THE STUDY

ATTACHMENT C: EMAIL Reminder for study interviews

INSTRUMENTS

INSTRUMENT #1: IN-DEPTH INTERVIEW MASTER TOPIC GUIDE

B1. Respondent universe and sampling methods

1. Universe

Our overall study sample will draw on the set of grantees who received grant funding or cooperative agreements through one of three OAH initiatives in 2010, but were subsequently not refunded. This sample consists of 7 former PAF grantees, 64 former TPP grantees, and 1 former CDC Communitywide programs and services grantee.

Each of the three federal funding programs are designated for particular types of programs and approaches, and have specific implementation, evaluation, and reporting requirements. For example, within the TPP program, grantees were funded under four separate “tiers”: Tiers 1A and 1B are reserved for capacity-building and replication of evidence-based TPP program models. Tier 2 provides funding for supporting and rigorously evaluating innovative new programs. Similarly, PAF grantees were awarded grants under four separate categories of program types. Categories 1 and 2 focus on programs serving expectant and parenting teens in different settings (universities or colleges, community-based organizations, high schools). Category 3 is for programs that provide services to pregnant women who are victims of domestic violence, sexual violence, sexual assault, and stalking, and Category 4 is for programs aiming to increase public awareness and education services for expectant and parenting teens, women, fathers, and their families.

The team will administer two rounds (one in 2016 and one in 2017) of in-depth telephone interviews with up to 50 former grantees so that we may learn from the varied trajectories leading to both sustained and non-sustained programs over time. This sample size will also reduce respondent burden and permit a focus on those former grantees most relevant to understanding what makes a program sustainable, and the varied ways in which respondents approached sustainability for their programs within the context of their organizations over time.

2. Site selection

The sustainability study is not intended to select a representative sample of all former grantees. Rather, the site selection process will use “judgmental” or “purposive” sampling techniques designed to answer particular research questions or meet certain analytic needs. In addition to examining which programs were sustained, our selection process for the in-depth interviews will pay special attention to the varied typologies among grantees (different funding streams, organizational characteristics, number or types of programs implemented) and how they affected program sustainability. For example, we may use the interviews to explore the differences in paths former TPP grantees followed to sustain evidence-based programs that have been rigorously evaluated (Tier 1), compared with new programs, many of which are still in the process of being scaled up and tested (Tier 2). The two types of programs would likely require different strategies and resources to sustain. Similarly, since some of the Cohort 1 grantees received funding for multiple programs, we may explore how the former grantees implementing more than one program fared as compared to grantees who were implementing just one program. Some of the particular dimensions along which grantees could be selected are as follows:

• Type of funding (for example: TPP Tiers 1 and 2, PAF Categories 1 to 4, CDC)

• Program setting (for example: school, community, clinic, home, university or college, and so on)

• Geographic location or region

• Types of populations served (for example: expectant or parenting youth, foster care youth)

• Current program status (that is, whether still operating as an identifiable entity or no longer operational)

3. Selection of respondents for each grantee

In collaboration with OAH, the study team will contact each organization to ascertain the appropriate contact for the interview, based on their knowledge of program sustainability efforts and outcomes and invite them to participate in the study (Attachment B).

B2. Procedures for collection of information

To determine which programs were sustained and which were not, it is important to articulate what is meant by sustainability. For the purpose of this study, the team will focus on whether the program or set of services funded by the 2010 grant have continued operating as an identifiable entity inside the same or a new organization.

To prepare for data collection, the study team will review technical assistance resources developed by OAH, administrative documents, as well as grantee-specific reports to define grantee organization characteristics such as size, partnerships, and funding sources, as well as relevant information about the program(s) being implemented. Documentation of this information using existing sources will help to minimize the burden imposed on respondents.

The primary data collection for the study will involve the In-Depth Interview Master Topic Guide. The study will begin with a pilot of the instrument with a small set of grantees in summer 2016 (for example: the seven PAF grantees whose federal funding ended in 2013). The pilot will include fewer than 10 respondents. Based on the pilot, the team may make minor adjustments to the instrument as needed to improve data collection processes, participation rate, and the quality of data gathered.

The former project officer, in conjunction with the study team, will send an introductory email to each former grantee, describing the study, requesting participation, and confirming the appropriate contact for the interviews (Attachment B).

In-Depth Interview Master Topic Guide: The study team will identify a subsample of up to 50 former grantees for two rounds of qualitative, semi-structured telephone interviews (Instrument #1). Selected respondents will reflect a diverse group of grantee types, including those from both different funding tiers and categories, those that reapplied and were not refunded, those that did not reapply for federal funding, and a mix of program settings and geographical locations.

It is expected that the first in-depth interviews will occur beginning in November 2016, although the exact timing will depend on the schedule of program activities and grantees’ availability. We plan to conduct a second round of in-depth interviews with respondents from the same subsample, beginning in October 2017. Interviews will focus on topics listed in the Master Topic Guide (Instrument #1), to assess changes and progress on sustainability, depending on the data gathered in the first round of interviews. Two-person teams, consisting of a lead interviewer and a note taker, will conduct both rounds of the 90-minute semi-structured telephone interviews. Before the start of the interview, the study team will ask respondents’ permission to record the interviews to help the note takers ensure that their notes are accurate. Recordings will not be heard by anyone outside of the study team and will be destroyed once the interview notes are finalized. The research questions will guide the topics discussed, aiming to elicit key lessons specifically related to sustaining teen pregnancy prevention programs or those that support expectant and parenting teens. We will aim to understand what makes a program sustainable, describe the varied ways in which respondents approached sustainability for their programs within the context of their organizations, and present the outcomes of their efforts over time.

B3. Methods to maximize response rates and deal with nonresponse

To ensure high response rates to data collection efforts associated with the sustainability study, OAH will work closely with former project officers to conduct outreach via email to former grantees and inform them about the purpose and importance of participating in the study. The study will rely on telephone interviews and provide sufficient time (up to four weeks) for respondents to schedule and complete the interview. To maximize the response rate for data collection activities, the study team will send follow-up emails to remind respondents to schedule the calls. Study team members and the project officer will do additional targeted follow-up with non-respondents after the four-week period. The team will plan all telephone interviews well in advance so that identified respondents can participate within the anticipated three-month interview periods. The study team will also send reminder emails before the scheduled interviews to encourage high participation (Attachment C).

However, even in the rare event of nonresponse, the study findings will not be significantly affected because the sites are not intended to be a representative probability sample of all OAH grantees; rather, the study is focused on lessons learned from an interesting array of previous grantees.

B4. Test of procedures or methods to be undertaken

The study team will begin with a pilot of the data collection instrument with a small set of up to nine grantees in spring 2016. One possibility is to use a naturally existing subsample such as the seven not refunded PAF grantees whose federal funding ended in 2013. These respondents comprise a small subset of the respondent universe and are expected to have common experiences with the rest of the universe of grantees regarding program sustainability efforts. The results of this test will inform the subsequent rounds of data collection for PAF, CDC, and TPP grantees. OAH plans to inform OMB of any changes in the data collection instruments for former grantees as a result of this test.

B5. Individuals consulted on statistical aspects and individuals collecting and/or analyzing data

Mathematica, OAH’s contracting organization, will conduct interviews for the proposed study. (Attachment A) lists the individuals whom OAH consulted on the development of the instrument for the sustainability study.

iii