PMAPS- PM Supp Stat B_012717_clean

U.S. Department of Health and Human Services Office of Planning, Research and Evaluation, and Family and Youth Services Bureau Administration for Children and Families Switzer Building, 4th Floor 330 C Street, SW Washington, DC 20201 Project Officer: Caryn Blitz

PART B: Statistical Methods for the Collection of Performance Measures Data - Personal Responsibility Education Program XXXX-XXXX September 2016

CONTENTS

PART b Introduction 1

B1. Respondent Universe and Sampling Methods 1

1. Youth Participants 1

2. Grantees and Sub-Awardees 1

B2. Procedures for Collection of Information 3

B3. Methods to Maximize Response Rates and Deal with Non-response 4

B4. Test of Procedures or Methods to be Undertaken 5

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 5

TABLES

B1 1. Annual Respondent Universe and Expected Response Rates for the Study of Performance Measures 2

2. Annual Respondent Universe, Expected Response Rates, and Power Analyses for the Core Measures for PREIS Grantees’ Local Evaluations Study of Performance Measures 3

INSTRUMENTS

INSTRUMENT #1 – PARTICIPANT ENTRY SURVEY

INSTRUMENT #2 – PARTICIPANT EXIT SURVEY

INSTRUMENT #3 – PERFORMANCE REPORTING SYSTEM DATA ENTRY FORM

INSTRUMENT #4 – SUBAWARDEE DATA COLLECTION AND REPORTING FORM

INSTRUMENT #5 – CORE MEASURES FOR PREIS GRANTEES’ LOCAL EVALUATIONS

ATTACHMENTS

ATTACHMENT A: 60-DAY FEDERAL REGISTER NOTICE

PART b

B1. Respondent Universe and Sampling Methods

Performance measures. The participant entry and exit surveys will provide data on the demographic and behavioral characteristics of program participants and participants’ perceptions of program effects and their responses to the program. Administrative data reported by the grantees for performance measurement will include semi-annual data on program features and structure, allocation of funds, participant numbers, levels of participant engagement, fidelity to evidence-based program models, and staff perceptions of quality challenges and needs for technical assistance.

1. Youth Participants

Instrument 1: Participant Entry Survey. PREP grantees are expected to serve approximately 436,575 participants over the three year OMB clearance period, for an average of about 145,525 new participants per year.¹ Once we apply a 95 percent response rate to the participants, we anticipate 138,249 respondents to the entry survey each year (145,525 x 0.95 =138,249).

Instrument #2: Participant Exit Survey. It is estimated that about 20 percent of the participants will drop out of the program prior to completion, leaving approximately 116,420 (145,525 x .80 = 116,420) participants at the end of the program annually.² Of those, we expect 95 percent, or approximately 110,599 participants, will complete the participant exit survey each year.³

2. Grantees and Sub-Awardees

Instruments 3-4: Performance Reporting System Data Entry Form and Subawardee Data Collection and Reporting Form. The 93 grantees⁴ will report performance measures data into a national data warehouse developed for the PREP initiative. They will gather this information with the assistance of their sub-awardees (estimated to be 416 across all grantees)⁵. The grantee and sub-awardee data collection efforts are record-keeping tasks.

Table B1.1. Annual Respondent Universe and Expected Response Rates for the Study of Performance Measures

Data Collection	Type of respondent	Number of Respondents	Expected response rate	Total expected responses
Instrument 1: Participant Entry Survey	Youth participant	145,525	95%	138,249
Instrument 2: Participant Exit Survey	Youth participant	116,420	95%	110,599
Instrument 3: Performance Reporting System Data Entry Form	Grantee Administrator	93	100%	93
Instrument 4: Sub-awardee Data Collection and Reporting Form	Sub-Awardee Administrator	416	100%	416
Estimated Totals		262,454		249,357

Instrument 5: Core Measures for PREIS Grantees’ Local Evaluations.

PREIS grantees will be asked to have all participants complete brief surveys at program exit, short-term follow-up and long-term follow (Instrument 5). The respondent universe for these surveys will be all youth participating in programming supported by PREIS grants. ACF estimates that PREIS grantees will be supporting programming in approximately 16 sites, with an estimated 6,400 enrolled annually at baseline across all sites. This estimate is assuming a great number of youth will be enrolled in PREIS programs, given this is the second round of grant funding. We assume that 95 percent of youth (N = 6,080) will complete the exit survey. We assume approximately 80 percent of youth who enroll in these programs (N = 5,120) will complete the short-term follow-up. Of those, we assume that approximately 75 percent of youth who enroll in these programs (N = 4,800) will complete the long-term follow-up survey. These response rates are conservative follow-up estimates and based off comparable studies examining adolescent pregnancy prevention programs. See below for the anticipated minimum detectable impacts for our binary measures.

Table B1.2 Annual Respondent Universe, Expected Response Rates and Power Analyses for the Core Measures for PREIS Grantees’ Local Evaluations

Data Collection	Type of respondent	Average number of total respondents	Expected response rate	Total annual expected responses per grantee	Minimum detectable impact for binary outcome with mean of 50%
Instrument 5: Core Measures for PREIS Grantees’ Local Evaluations	Youth participant	16,000	Post-Program: 95% Short-term follow—up: 80% Long-term follow-up: 75%	Post-program: 6,080 Short-term follow-up: 5,120 Long-term follow-up: 4,800	Post-program: 7.7 pp Short-term follow-up: 8.3pp Long-term follow-up: 8.666

B2. Procedures for Collection of Information

Performance Measures

Instruments 1-2: Participant Entry and Exit Surveys. Each grantee and their sub-awardees will make decisions regarding procedures for collecting the participant entry and exit surveys. Some grantees have elected to work with local evaluators that will administer the surveys for performance measures purposes; the local evaluators could decide to use paper-and-pencil or web-based surveys. For those grantees not working with local evaluators, it is likely that the subawardees’ program facilitators will administer the entry and exit surveys using paper and pencil in group or individual settings. Grantees will inform their individual program participants that participation is voluntary and that they may refuse to answer any or all of the questions in the entry and exit questionnaires. The response rate for both surveys is expected to be 95 percent. As indicated in Table B1.1, the estimated number of respondents is less for the exit survey because we expect about 20 percent of the participants to drop out of the program prior to completion.

Instruments 3-4: Performance Reporting System Data Entry Form and Subawardee Data Collection and Reporting Form. Grantees will report separately on levels of participant attendance, reach and dosage (see Figure 1). Data on these measures will be collected by subawardees (i.e., providers and programs; see Figure 1) (Instrument 4). Administrative data on program features and structure, allocation of funds, fidelity to evidence-based program models, and staff perceptions of quality challenges will be collected by grantees and sub-awardees through their administrators (Instruments 3 and 4). Grantees will prepare and submit their final data sets in aggregate form to ACF through the PREP Data warehouse. The Performance Reporting System Data Entry Form (Instrument 3) contains the list of all data elements grantees will report, collected from among their sub-awardees. Because collecting and reporting data for performance measures is a funding requirement of the PREP grants, the grantee and sub-awardee response rates are expected to be 100 percent.

Figure 1. Levels of PREP Performance Measures Data Collection

The timing of participant survey data collections will be customized for each site depending upon the start and end dates of each cohort of participants. Administrative performance measurement data and participant information will be reported twice a year.

Instrument 5: Core Measures for PREIS Grantees’ Local Evaluations. Each PREIS grantee will make decisions regarding procedures for collecting the core measures. All grantees will be working with a local evaluator and may decide to use paper and pencil, web based surveys, or another method to reliably collect data. Grantees will inform their individual program participants that participation is voluntary and that they may refuse to answer any or all of the questions in the entry and exit questionnaires.

The specific timing of the short-term and long-term follow-ups will be customized for each PREIS grantee. It is anticipated that, on average, the short-term follow-up will occur around 6 months post-programming; the long-term follow-up will, on average, occur, 9 months post-programming. The response rate for program exit is 95 percent; short-term follow-up is expected to be 80% percent; long-term follow-up is expected to be 75 percent. As indicated in Table B1.2 the estimated number of respondents is fewer for the program exit and the follow-up surveys because we expect a small level of attrition. These response rates are conservative estimates and based off of comparable studies examining adolescent pregnancy prevention programs. For example, in the PREP Multi-Component study, response rates in the Kentucky impact site, which is working with middle school males in schools, are as follows: 90 percent for 12-months post-baseline; and 85 percent for 24 months post-baseline. In the Iowa site, which is working with high school youth in schools, response rates are as follows: 95 percent at 12-months post-baseline and 91 percent at 24 months post-baseline. In programs that are not conducted in schools, response rates are slightly lower. For example, in the site in San Angelo, TX working with pregnant and parenting mothers in a home visiting program, short-term response rates, one year post-baseline are 79% and at two-years post-baseline are 72%. Because we don’t know whether the PREIS sites will be conducted in schools or outside of schools, are expected response rates are being more conservative, assuming a lower response rates. This also allows us to be conservative with our power analyses.

B3. Methods to Maximize Response Rates and Deal with Non-Response

Performance Measures

Instruments 1-2: Participant Entry and Exit Surveys. Response rates for participant surveys will be maximized through the administration of entry surveys to all participants at enrollment and administration of the exit surveys during final program sessions. Where feasible, exit surveys will be administered on an individualized basis to program exiters who are absent during final sessions when the surveys are completed.

Instruments 3-4: Performance Reporting System Data Entry Form and Subawardee Data Collection Reporting Form. To reduce grantee burden and maximize grantee response rates, ACF is providing common data element definitions across PREP program models and collecting these data in a uniform manner through the PREP data warehouse (see Instruments 3-4). Because the submission of the performance measures data is a grant requirement, except in the cases when waivers are extended for the sensitive questions on the participant entry and exit surveys, ACF does not expect problems with non-response.

Instrument 5: Core Measures for PREIS Grantees’ Local Evaluations. ACF expects to achieve a response rate on the core measures of 95% at program exit, 80 percent for the first follow-up survey and 75 percent for the second follow-up. We can expect to achieve these completion rates for several reasons. The first follow-up survey administration will occur, on average, no later, than 6 months post-program. The second follow-up will occur, on average, no later than 9 months post-program. This timing will ensure contact data are quite current, which should minimize location problems. In school-based sites, in many cases, youth will be enrolled in the same schools at follow-up that they were enrolled in at baseline, simplifying locating efforts and improving response rates.

In addition, we expect that obtaining the grantees’ willing assistance will be very important to maximizing the response rate. ACF and its contractors will invest significant effort in gaining their active cooperation and collaboration from the beginning of the study, minimizing burden on sites and assuring privacy to the youth participants, and will provide thorough training and technical assistance to all grantees. Grantees will be given detailed information about the surveys. By applying identical methods for maximizing the response rates of the treatment and control groups, the evaluation team does not anticipate differences in response rates across research groups.

B4. Test of Procedures or Methods to be Undertaken

Performance measures; and core measures for PREIS grantees’ local evaluations. Cognitive pretesting with nine youth ages 13 to 18 will be conducted for the Participant Entry and Exit Surveys (Instruments 1 and 2), as well as the Core Measures for PREIS grantees (Instrument 5). The cognitive pretest sample will include males and females, as well as youth from a mix of racial and ethnic backgrounds. Survey questions will be revised based on the results of these tests. Any resulting updates will be submitted to OMB as a nonsubstantive change request. If substantive changes result from cognitive pretesting of Instruments 1, 2, and 5, we will publish a 30 day Federal Register Notice allowing for public comment and submit the revised instruments to OMB for review and approval.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Performance measures. Data for the performance measures will be collected by grantees and their sub-awardees. In some cases, grantees will have engaged local evaluators who will assist them in performance measure data collection. Grantees will report these data in aggregate form into the PREP Performance Measures Reporting System that will be maintained by ACF’s contractor, Mathematica Policy Research. Mathematica will use these data to analyze PREP performance data and to generate performance measurement reports for ACF.

In reconsidering the performance measures, and in identifying core measures for PREIS grantees’ local evaluations, ACF consulted with FYSB program staff (grantees’ project officers) and 9 tribal grantees.

Core measures for PREIS grantees’ local evaluations. Data for the PREIS local evaluations will be collected by each individual PREIS local evaluator. Grantees will not be required to submit data on the core measures to ACF on a regular basis. PREIS grantees’ local evaluation analysis plans will be pre-approved by ACF to ensure that they are rigorous and accurate.

1 The three year period for which we are requesting clearance covers the first three years of the PMAPS project.

2 Based on our review of data from the PREP Multicomponent Evaluation Performance Analysis Study (PAS), we estimate that 60 percent of youth served in PREP programs will be in school-based programs and that 40 percent will be served in out-of-school programs. We assume that 90 percent of youth in school-based PREP programs will complete the program and that 65 percent of youth in out-of-school PREP programs will complete the program. These assumptions yield an overall program completion rate of 80 percent.

3 We are currently requesting clearance for three years; over the three years for which we are requesting clearance, we expect that 349,260 youth will complete the programs and 331,797 will complete a participant exit survey.

4 The 93 grantees include 49 states and territories, 10 grants to tribes and tribal communities, 21 grants under Competitive PREP, and 13 PREIS grantees.

5 Our estimates are based upon the number of sub-awardees observed through the PREP Multi-Component evaluation and the growth in sub-awardees annually.