PREP Eval - PAS+Baseline - Supp Stat B - Clean - 11-28-12

U.S. Department of Health and Human Services Office of Planning, Research and Evaluation, Administration for Children and Families Family and Youth Services Bureau 7th floor West Aerospace Building 370 L'Enfant Promenade, SW Washington, DC 20047 Project Officer: Clare DiSalvo, Dirk Butler

PART B: Statistical Methods for the Collection of Performance Measures and Baseline Data - Personal Responsibility Education Program (PREP) Multi-Component Evaluation 0970-0398 November 2012

CONTENTS

PART b Introduction 1

B1. Respondent Universe and Sampling Methods 1

B2. Procedures for Collection of Information 3

B3. Methods to Maximize Response Rates and Deal with Non-response 4

B4. Test of Procedures or Methods to be Undertaken 5

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 5

TABLES

B1.1 Respondent Universe and Expected Response Rates for the Performance Analysis Study 2

B1.2 Minimum Detectible Impacts with Sample of 1,500 Youth 3

INSTRUMENTS

INSTRUMENT #1 – PARTICIPANT ENTRY SURVEY (PAS)

INSTRUMENT #2 – PARTICIPANT EXIT SURVEY (PAS)

INSTRUMENT #3 – BASELINE SURVEY (IIS)

INSTRUMENT #4 – PERFORMANCE REPORTING SYSTEM DATA ENTRY FORM

INSTRUMENT #5 – SUBAWARDEE DATA COLLECTION AND REPORTING

INSTRUMENT #6 – IMPLEMENTATION SITE DATA COLLECTION

ATTACHMENTS

ATTACHMENT A: OVERVIEW OF THE PREP EVALUATION

ATTACHMENT B: ANALYSIS PLAN FOR IMPACT STUDY

ATTACHMENT C: QUESTION BY QUESTION SOURCE TABLE FOR THE BASELINE
SURVEY

ATTACHMENT D: SOURCES REFERENCED FOR THE BASELINE QUESTIONNAIRE

ATTACHMENT E: 60-DAY FEDERAL REGISTER NOTICE

ATTACHMENT F: PERSONS CONSULTED ON COLLECTION AND/OR ANALYSIS OF THE PAS AND IIS BASELINE SURVEY

ATTACHMENT G: CONSENT LETTERS AND FORMS AND YOUTH ASSENT FORM

ATTACHMENT H: CONFIDENTIALITY PLEDGE

PART b

B1. Respondent Universe and Sampling Methods

Performance Analysis Study. As part of the PAS of the PREP evaluation, the participant entry and exit surveys will provide data on the demographic and behavioral characteristics of program participants and participants’ perceptions of program effects and their responses to the program. Administrative data reported by the grantees for performance measurement will include annual data provided by the grantees on program features and structure, allocation of funds, participant numbers, levels of participant engagement, fidelity to evidence-based program models, and staff perceptions of quality challenges and needs for technical assistance.

Grantees or their sub-awardees will administer participant entry surveys to all program participants, except those enrolled at sites participating in the IIS, at the time they are enrolled in the program. Data collected in the entry survey are also covered in the IIS baseline survey. To avoid duplication of data collection among youth enrolled in programs selected for inclusion in the impact study, these youth will complete only the baseline survey at program entrance.

An estimated 69,000 new participants will enroll in the study each year. Of those, we expect 65,550 (95 percent) will complete the PAS entry survey each year. Exit surveys will be administered to all participants who are still in the program at completion. We assume approximately 80 percent of youth who enroll in these programs will complete them.¹ Of those, we estimate that 52,440 youth (95 percent) will complete the exit survey annually. Because the PAS participant surveys will be administered to all participants who are active at the time of entry and exit, no sampling is required for the PAS component of the evaluation.

As a funding requirement of their PREP grant, all grantees will be required to report the administrative performance measurement data described above through the PREP reporting system. Grantees will collect substantial parts of this data from their sub-awardees, estimated to number 350 across all grantees, and sub-awardees will collect some data from their implementation sites, estimated to be 1,400. Table B1.1 presents the respondent universe with expected response rates (see section B1.2 for details) for each respondent population.

Table B1.1. Respondent Universe and Expected Response Rates for the Performance Analysis Study

Data Collection	Type of respondent	Number of Respondents	Expected response rate	Total expected responses
Instrument 1: Participant entry survey	Youth participant	69,000	95%	65,550
Instrument 2: Participant exit survey	Youth participant	55,200	95%	52,440
Instrument 4: Performance Reporting System Data Entry Form	Grantee Administrator	65	100%	65
Instrument 5: Sub-awardee data collection and reporting	Sub-Awardee Administrator	350	100%	350
Instrument 6: Implementation site data collection	Site Facilitator	1,400	100%	1,400
Estimated Totals		126,015		119,805

Impact and In-depth Implementation Study. From the universe of PREP grantees, ACF will select four or five program sites to participate in the Impact and In-depth Implementation Study. The sites are not meant to be representative of PREP-funded programs as a whole. Rather, site selection is focusing on grantees that (1) are large enough to support an impact and in-depth implementation study, (2) are implementing programs in a way that is amenable to random assignment for the program impact study (discussed below), and (3) address priority gaps in the existing research literature on evidence-based approaches to teen pregnancy prevention. These gaps include evidence on effective programs for high-risk populations such as youth living in rural areas or youth in the foster care or juvenile justice systems.

In each site, ACF expects to recruit and enroll a sample of 1,200 to 1,500 youth (for a total of 6,000 youth across four or five sites).² Each site will be analyzed separately, so the relatively large samples of 1,200 to 1,500 youth per site are needed to detect policy-relevant impacts on key behavioral outcomes. Table B1.2 reports minimum detectible impacts on two illustrative outcomes—one with 50 percent prevalence (such as the proportion of high-risk teens that have had sex in the past three months) and one with 20 percent prevalence (such as the proportion of high-risk teens that have been pregnant or gotten someone pregnant or had an STI). Separate estimates are presented assuming either (1) random assignment of individuals to treatment and control groups or (2) random assignment of clusters of individuals (such as schools, clinics, or group homes). Separate estimates are also presented for analyses of full sample versus subgroup impacts. The table reports minimum detectible impacts for an assumed sample of 1,500 youth per site. However, smaller samples of 1,200 youth per site might, in some instances, be sufficient—for example, if the main research questions are limited to full sample impacts, not subgroup analyses.

Sample enrollment is expected to begin in one or more sites in September 2012. All eligible youth will be considered for enrollment until we reach the target sample of 1,200 to 1,500 youth per site. ACF does not expect needing to conduct any sampling of youth prior to enrollment.

Table B1.2. Minimum Detectible Impacts with Sample of 1,500 Youth

	Percentage Point Impacts for Illustrative Binary Outcomes
	Recent Sexual Activity (Mean 50%)		Pregnancy or STI (Mean 20%)
	Full Sample	50% Subgroup	Full Sample	50% Subgroup
Individual Random Assignment	7.0	9.0	5.6	7.9
Cluster Random Assignment	9.1	11.4	7.3	9.2

Notes: Sample size of 1,500 youth refers to program and control groups combined. Figures assume that the sample is evenly divided between the program and control groups, a response rate of 75 percent, and that covariates explain 30 percent of the variance at the individual level. The figures also assume a two-tailed t-test with 80 percent power and a 95 percent confidence interval. For sites with cluster random assignment, the figures further assume a total of 16 clusters (evenly divided between the program and control groups), an intra-class correlation (ICC) of 0.01, and that covariates explain 30 percent of the variance at the cluster level.

B2. Procedures for Collection of Information

Performance Analysis Study. Each grantee and their sub-awardees and implementation sites will make decisions regarding procedures for collecting the participant entry and exit surveys. Some grantees have elected to work with local evaluators that will administer the surveys for performance measure purposes; the local evaluators could decide to use paper-and-pencil or web-based surveys. For those grantees not working with local evaluators, it is likely that the program staff at the implementation sites will administer the entry and exit surveys using paper and pencil in group or individual settings. Grantees will inform their individual program participants that participation is voluntary and that they may refuse to answer any or all of the questions in the entry and exit questionnaires. The response rate for both surveys is expected to be 95 percent. As indicated in Table B1.1, the estimated number of respondents is less for the exit survey because we expect about 20 percent of the participants to drop out of the program prior to completion.

Grantees will report separately on levels of participant attendance, reach, dosage, and retention. Data on these measures will be collected by implementation site facilitators (Instrument 6). Administrative data on program features and structure, allocation of funds, fidelity to evidence-based program models, and staff perceptions of quality challenges will be collected by grantees and sub-awardees through their administrators (Instruments 4 and 5). Grantees will prepare and submit their final data sets in aggregate form to ACF through the PREP reporting system. The Performance Reporting System Data Entry Form (Instrument 4) contains the list of all data elements grantees will report, collected from among their sub-awardees and implementation sites. Because collecting and reporting data for performance measures is a funding requirement of the PREP grants, the grantee and sub-awardee response rate is expected to be 100 percent.

The timing of PAS participant survey data collections will be customized for each site depending upon the start and end dates of each cohort of participants. Administrative performance measurement data will be submitted annually by grantees following the end of each grant year.

Impact and In-depth Implementation Study. In each of the four or five sites selected for the IIS, all eligible youth will be considered for enrollment in the study (discussed in Section B.1). Each site will be responsible for providing the evaluation team with a list of eligible youth. The evaluation team will then work collaboratively with each site to identify youth for the study and obtain active written consent from the responsible parent or guardian for youth under age 18 and from the youth themselves for those age 18 or older. Draft consent forms are included in Attachment G. The evaluation team will then prepare a final roster of youth at each site for whom it has consent.

The baseline survey will be administered to all consented youth shortly after study enrollment. The evaluation team will work individually with each site to determine the best mode and procedures for survey administration. As discussed in Part A of this information collection request, wherever possible, the evaluation team will conduct group administration of a self-administered pencil and paper survey instrument (PAPI). When necessary to increase response rates or accommodate specific populations, this method will be augmented with individual administration of PAPI surveys or web or hard-copy telephone interviews.³

For group administration, the evaluation team will begin by handing out pre-identified survey packets to the youth whose names are on the packets, and obtaining youth assent. Each packet will consist of the PREP baseline survey and a sealable return envelope. The survey and envelope will have a label with a unique ID number (no personally identifying information will appear on the survey or return envelope). Youth will self-administer the survey. Part A of the survey asks for background information and concludes with a single screening question about sexual experience. Youth with sexual experience will complete Part B1 and those without will complete Part B2. Two members of the evaluation team will monitor activities in each survey room. At the end of the survey administration, youth will place the entire survey in the return envelope, seal it, and return it to a member of the evaluation team. Completed surveys will be immediately shipped via FedEx to Mathematica’s Survey Operations Center for receipting, and then checked for completeness. Any forms with identifying information (consent and assent forms) will be shipped separately from the surveys. All surveys that pass the check will be sent to a vendor for scanning. All scanned data will be electronically transmitted back to the evaluation team.

For make-up sessions or when group administration is not feasible, the evaluation team will work collaboratively with each site to determine the best alternative mode of survey administration. Three options will be considered: (1) individual administration of the PAPI survey, either by members of the evaluation team or trained local staff members, (2) individual administration of a web-based version of the survey, which youth would access with unique PIN/password combinations, or (3) individual administration of the PAPI survey over the telephone. Since ACF plans to analyze data from each site separately, it is feasible to use different combinations of survey modes in each site.

B3. Methods to Maximize Response Rates and Deal with Non-Response

Performance Analysis Study. Response rates for PAS participant surveys will be maximized through the administration of entry surveys to all participants at enrollment and administration of the exit surveys during final program sessions. Where feasible, exit surveys will be administered on an individualized basis to program exiters who are absent during final sessions when the surveys are completed.

To reduce grantee burden and maximize grantee response rates, ACF is streamlining the PAS administrative data reporting process by providing common data element definitions across PREP program models and collecting these data in a uniform manner through the PREP reporting system (see Instrument 4). Because the submission of the performance measures data is a grant requirement, except in the cases when waivers are extended for the sensitive questions on the participant entry and exit surveys, ACF does not expect problems with non-response.

Impact and In-depth Implementation Study. ACF expects to achieve a response rate of 95 percent for the baseline survey. A high response rate is expected because survey administration will occur shortly after active parental consent is received. This timing will ensure our contact data are current (no location problems) and that surveys can be administered to most youth in the location where the program would take place. Methods to achieve high response rates at follow-up will be discussed in future information collection requests.

B4. Test of Procedures or Methods to be Undertaken

Performance Analysis Study. Cognitive pretesting with nine youth ages 13 to 18 has been conducted for both the PAS entry and exit surveys. The cognitive pretest sample included males and females and included youth from a mix of racial and ethnic backgrounds. Survey questions have been revised based on the results of these tests.

To ensure that the PREP reporting system functions as intended and in a user-friendly manner for grantees’ entry of administrative performance measurement data, the system will be tested by an internal team of PAS reviewers who designed the system specifications, independent of the system developer. This team will review all data import/entry, reporting, calculation, and extract functions of the system to ensure that grantee end-users will find the system to be efficient and user-friendly. Prior to deployment, the system will also undergo beta testing with nine or fewer grantee staff.

Impact and In-depth Implementation Study. As discussed in Part A of this information collection request, many of the items included on the baseline survey are taken directly from the similar survey OMB has already approved for use on the ongoing Evaluation of Adolescent Pregnancy Prevention Approaches (PPA). The PPA baseline survey was pre-tested prior to receiving OMB clearance and has since been administered to approximately 3,500 adolescents. Any new items added specifically for PREP were generally drawn from established sources (see Attachments C and D). The evaluation team will carefully monitor the sample enrollment, consent, and data collection procedures in each site, and our formal Memorandum of Agreement (MOA) with each site will reserve the possibility of suspending sample enrollment and baseline data collection in the event of unforeseen problems.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Data for the PAS will be collected by grantees and their sub-awardees. In some cases, grantees will have engaged local evaluators who will assist them in performance measure data collection. Grantees will report these data in aggregate form into the PREP-PM reporting system that will be maintained by ACF’s contractor, RTI International. RTI International will provide data extracts from this reporting system to ACF’s evaluation contractor, Mathematica Policy Research. Mathematica and its subcontractor, Child Trends, will use these extract files to analyze PREP performance data and to generate performance measurement reports for ACF.

Baseline survey data for the IIS will be collected and analyzed by ACF’s prime contracting organization, Mathematica Policy Research.

Attachment F lists the individuals whom ACF consulted on the collection and/or analysis of the PAS and IIS baseline survey data.

1 Based on our review of state PREP plans and other documents, we estimate that 60 percent of youth served in PREP programs will be in school-based programs and that 40 percent will be served in out-of-school programs. We assume that 90 percent of youth in school-based PREP programs will complete the program and that 65 percent of youth in out-of-school PREP programs will complete the program. These assumptions yield an overall completion rate for the program of 80 percent.

2 Some youth or their parents will not consent to be part of the PREP evaluation. These sample size estimates are for youth who complete the consent process to be part of the evaluation.

3 We assume a 95 percent response rate for the IIS Baseline Survey.