PREP PAS+Baseline - Attachment A - Overview of the PREP Evaluation - 6-7-12

Attachment A

Overview of the PREP Evaluation

Overview of the PREP Multi-Component Evaluation and Data Collection Efforts Previously Approved by OMB

The goal of the PREP Multi-Component Evaluation is to document how programs funded through the PREP grants program are designed and implemented in the field and to assess selected PREP-funded programs’ effectiveness. The evaluation will include three primary, independent study components: (1) the Design and Implementation Study (DIS), (2) The Performance Analysis Study (PAS), and (3) the Impact and In-depth Implementation Study (IIS). Each study component is described in more detail below.

Design and Implementation Study (DIS). States must decide how to distribute grant funds to local sub-awardees, what program models to authorize, and how to support implementation. Under­standing the range of decisions made to meet the funding requirements and how implementation unfolds can contribute to stronger future program design for pregnancy prevention and other programs.

Documentation of states’ PREP design decisions and implementation experiences will be based on two rounds of data collection. In spring 2012, the contractor began a Design Survey. The contractor reviewed documents that outline state PREP program plans and is currently conducting telephone interviews with state PREP officials to learn more about what programming decisions have been made, and why.

• The Design Survey conducted as part of the DIS received approval on March 7, 2012 (OMB Control #0970-0398). The data collection instrument is a discussion guide to be used with states administrators in states that received PREP state grant funding (ie, “PREP State-Level Coordinators”). The survey focuses on issues such as: how program and policy decisions were made at the state level, the context in which the state is operating, how PREP programs are being integrated with other efforts to reduce teen pregnancy in the state, the relationship between the state and PREP sub-awardees, and how states intend to meet the legislative requirements of the funding.

In summer 2013, the team will begin an Implementation Survey, which will focus on program implementation, including training and technical assistance provided, monitoring activities to ensure that programs are being replicated with fidelity, adaptations made to fit local contexts, and youth enrollment and retention. For the Implementation Survey, the contractor will conduct interviews with state PREP officials, and selected sub-awardees. The information collected about sub-awardees through the Design Survey - including program type, region, urbanicity, experience, population size served, target population, and setting - will inform the selection of sub-awardees to be included in the Implementation Survey.

Performance Analysis Study (PAS). All PREP grantees will be required to submit standard information on program structure and program delivery. The performance measures will be aligned, to the extent possible, with other federally funded teenage pregnancy prevention programs, and will also reflect the unique feature of the PREP funding. These measures will help ACF understand whether the PREP objectives are being met and whether technical assistance may be needed to support program improvement.

• This current ICR pertains to the Performance Analysis Study, and this study component is described in more detail in Part A of this package.

Impact and Implementation Study (IIS). In four or five sites (a site could be an entire state or local sub-awardees within a state), this component of the evaluation will provide rigorous estimates of program effectiveness on key outcomes, such as rates of sexual initiation and abstinence, contraceptive use, and teen pregnancy, and include a detailed look at program delivery.

The impact study will be based on a random assignment design. For sites that can support random assignment and are selected for the impact and implementation study, the evaluation team will work collaboratively with grantees to develop a plan for either randomly assigning individuals to a group that will receive the program or to a control group, or ran­domly assigning program sites (such as schools, clinics, or group homes) to deliver the program or to be control sites.

• The Field Data Collection as part of the IIS was approved on November 6, 2011 (OMB Control #0970-0398). The field collection activity involves the collection of information from observations of program activities and interviews with a range of experts and persons involved with programs about various aspects of existing prevention programs, topics the experts view as important to address through evaluation, and assessments of feasibility for conducting a random assignment study in a particular site. These data will be used to help enhance decisions about the types of programs to be evaluated in the IIS and which sites can support the rigorous study.

Once sites have been selected and the sample enrolled, the evaluation team will administer surveys to study youth shortly before the programs begin, and about 6 and 18 months after the programs end.

• This current ICR pertains to the baseline survey, which will be administered before the programs begin.

The implementation study will focus a scientific lens on program operations to identify factors that affect large-scale replication of program models. The evaluation team will make multiple visits to the selected sites, conduct interviews with stakeholders and program staff, hold focus groups with program participants, and review program documents. Team members will also observe program sessions to document the quality of delivery and fidelity to the program models.