Supporting Statement B - Revision_043020_clean_final

Personal Responsibility Education Program (PREP) Performance Measures and Adulthood Preparation Study (PMAPS)

OMB Information Collection Request

0970 - 0497

Supporting Statement

Part B

APRIL 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Caryn Blitz, Tia Brown

Part B

B1. Objectives

Study Objectives

The objective of the Personal Responsibility Education Program (PREP) performance measures (PM) effort is to document how PREP-funded programs are operationalized in the field and assess program outcomes. This performance measures effort includes collection and analysis of performance measures data from State PREP (SPREP), Tribal PREP (TPREP), Competitive PREP (CPREP), and Personal Responsibility Education Innovative Strategies (PREIS) grantees. ACF will use the PM data to continue to (1) track how grantees are allocating their PREP funds; (2) assess whether PREP objectives are being met (e.g., in terms of the populations served); and (3) help drive PREP programs toward continuous improvement of service delivery. In addition, ACF will use this information to fulfill reporting requirements to Congress and OMB concerning the PREP initiative. ACF will also continue to share grantee and provider level findings with each grantee to inform their own program improvement efforts.

Generalizability of Results

This study is intended to present internally-valid description of PREP, not to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

The PREP PM data collection is designed to describe the implementation and outcomes of the PREP program. The PM data collected through this descriptive study provide necessary information to ACF and grantees to effectively manage their programs. Entry and exit surveys of youth participating in PREP are necessary to collect information on the demographic and behavioral characteristics of program participants, their experiences in the program, and their perceptions of program effects. Administrative data from grantees and their sub-awardee program providers are needed to understand the structure and features of PREP programs, participant numbers, implementation supports, and staff perceptions of quality challenges and needs for technical assistance. Because these are performance measures, data are required on the universe of grantees, programs, and participants.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target Population

The target population for the PREP PMAPS PM data collection includes all PREP grantees, their sub-awardee program providers, and youth participants. The numbers may vary somewhat by year, but for the first year under this extension they include 51 State PREP (SPREP), 9 Tribal PREP grantees (TPREP), 25 Competitive PREP grantees (CPREP), and 13 PREIS grantees. The number of sub-awardee providers is estimated to be 416 across all grantees. Grantees are expected to serve approximately 336,498 participants over the three-year OMB clearance period, for an average of about 112,166 new participants per year. Program participants are youth in middle school through age 20. The data collection instruments to be used by each target population are as follows:

• Instrument 1a/b: Participant Entry Survey -- youth participants

• Instrument 2a/b: Participant Exit Survey -- youth participants

• Instrument 3: Performance Reporting System Data Entry Form – grantees

• Instrument 4: Sub-awardee Data Collection and Reporting Form – subawardee providers

Sampling and Site Selection

The PREP PMAPS PM data collection includes all PREP grantees, sub-awardee program providers, and participants. ACF will use the performance measures data to monitor and report on progress in implementing PREP programs and support continuous quality improvement of the programs. In addition, the information will be used by grantees themselves to inform improvements to their program(s). All PREP grantees must be included in the study so that (1) the measures reflect the complete scope of the program and (2) the data can be used to promote program improvement among all grantees.

B3. Design of Data Collection Instruments

Development of Data Collection Instrument(s)

In April 2017, OMB approved the collection of PREP performance measures data through participant entry and exit surveys, and the grantee and subawardee reporting forms (OMB # 0970-0497). Internal discussions with new leadership at ACF led to some proposed revisions to the approved measures. In Fall 2018, cognitive pretesting of revised entry and exit instruments was conducted with nine youth, including a mix of males and females, ages 11 to 18. The survey questions were revised based on the results of these pretests and additional discussions with leadership. In August 2019, OMB approved a non-substantive change request that included revised versions of the participant entry and exit surveys, which include the following survey changes:

• Removed sensitive items (by dropping items about oral sex or anal sex, modifying items asking about “vaginal sex” to ask about “sexual intercourse,” and deleting definitions of sexual behaviors).

• Modified and expanded existing items related to adulthood preparation subjects.

• added items to capture information from youth relevant to the success sequence for poverty prevention.^1,2

Based on feedback from grantees on the 2019 versions of the surveys, as well as additional discussions with ACF leadership, ACF proposes the following revisions to the planned data collection:

1. Two Versions of the Entry and Exit Surveys (Instruments #1 and #2): One for middle school-age groups and one for high school- and older age groups.

• Middle school versions (Instruments #1a and #1b): We developed versions of the surveys that exclude items on sensitive subjects for use with State, Tribal and Competitive PREP participants in middle school. The following changes were:

  • Removed items on sexual activity and incidence of pregnancy (questions 14-20 in the previous Entry Survey); participants’ perceptions of PREP’s influence on their plans to engage in sexual activity and the importance of various reasons in decisions to not have sexual intercourse (questions 15-17 in the previous version of the Exit Survey).

• Removed and added response categories for older ages and grades (questions 1 and 2 in the previous versions of the Entry and Exit Surveys) to reflect the other changes to the surveys.

• High school and older versions (Instruments #1b and #2b):

  • Removed response categories for younger ages and grades (questions 1 and 2 in the previous versions of the Entry and Exit Surveys) as appropriate.

• For all versions of the middle-school and high school surveys, the sexual orientation item (question 6 in the previous versions of the Entry and Exit Surveys) was removed.

1. Edits to the Performance reporting system data form (Instrument #3): updates were made to reflect changes to the Entry and Exit Surveys.

Each of the four data collection instruments will address each of the study’s objectives described in Section B1 above. Instruments 1 and 2 will capture information on the characteristics of the youth participating in the program and their perceptions of program effects, and Instruments 3 and 4 will capture information on grantees’ and sub-awardees’ implementation of the PREP programs. Both types of data will be used to monitor program implementation and outcomes, inform program improvement, and provide status and progress updates.

B4. Collection of Data and Quality Control

Instruments 1-2: Participant Entry and Exit Surveys. Each grantee and their sub-awardees will make decisions regarding procedures for collecting the participant entry and exit surveys. Some grantees have elected to work with local evaluators that will administer the surveys for performance measures purposes; the local evaluators could decide to use paper-and-pencil or web-based surveys. For those grantees not working with local evaluators, it is likely that the subawardees’ program facilitators will administer the entry and exit surveys using paper and pencil in group or individual settings. Grantees will inform their individual program participants that participation is voluntary and that they may refuse to answer any or all of the questions in the entry and exit questionnaires.

Instruments 3-4: Performance Reporting System Data Entry Form and Subawardee Data Collection and Reporting Form. Grantees will report separately on levels of participant attendance, reach and dosage (see Figure 1 in SSA). Data on these measures will be collected by providers, which could be grantees or subawardees (Instruments 3 and 4). Administrative data on program features and structure, allocation of funds, and staff perceptions of quality challenges will be collected by grantees and sub-awardees through their administrators (Instruments 3 and 4). Grantees will prepare and submit their final data sets in aggregate form to ACF through the Performance Measures Management System (PMMS). The Performance Reporting System Data Entry Form (Instrument 3) contains the list of all data elements grantees will report, collected from among their sub-awardees.

The timing of participant survey data collections will be customized for each site depending upon the start and end dates of each cohort of participants. Administrative performance measurement data and participant information will be reported twice a year³.

ACF’s contractor will continue to provide training and technical assistance to ensure that grantees and program providers understand the measures, instruments, and data collection processes.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

Instruments 1-2: Participant Entry and Exit Surveys. Response rates for participant surveys will be maximized through the administration of entry surveys to all participants at enrollment and administration of the exit surveys during final program sessions. Where feasible, exit surveys will be administered on an individualized basis to program exiters who are absent during final sessions when the surveys are completed. The response rate for both surveys is expected to be 95 percent.

Instruments 3-4: Performance Reporting System Data Entry Form and Subawardee Data Collection Reporting Form. To reduce grantee burden and maximize grantee response rates, ACF is providing common data element definitions across PREP program models and collecting these data in a uniform manner through the PMMS (see Instruments 3-4). Because collecting and reporting data for performance measures is a funding requirement of the PREP grants, the grantee and sub-awardee response rates are expected to be 100 percent.

Table B1.1. Annual Respondent Universe and Expected Response Rates for the Study of Performance Measures

Data Collection	Type of respondent	Annual number of respondents	Expected response rate	Annual expected respondents
Instrument 1: Participant Entry Survey	Youth participant	112,166	95%	106,558
Instrument 2: Participant Exit Survey	Youth participant	102,324	95%	97,208
Instrument 3: Performance Reporting System Data Entry Form	Grantee Administrator	98	100%	98
Instrument 4: Sub-awardee Data Collection and Reporting Form	Sub-Awardee Administrator	416	100%	416
Estimated Totals		215,004		204,280

NonResponse

Analyses will be conducted based on the respondents providing data for a given measure, with no imputation or weight adjustments to address missing data.

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented and reported in written materials associated with the data collection.

B6. Production of Estimates and Projections

The performance measures data will primarily be used internally by ACF and PREP grantees but will also be used to inform other stakeholders. For example, performance measures data will inform ACF’s annual reporting to Congress on the progress of the PREP program, and end-of-cohort reports will be made available to the public.

The performance measures data are to be collected from all PREP grantees, program providers, and youth participants. The analyses will include computation of statistics such as percentages and means based on the respondents; we will not produce estimates intended to apply to any broader population.

B7. Data Handling and Analysis

Data Handling

Data for the performance measures will be collected by grantees and their sub-awardees. In some cases, grantees will have engaged local evaluators who will assist them in PM data collection. Grantees will report these data in aggregate form into the PREP PMMS that will be maintained by ACF’s contractor, Mathematica. Mathematica will use these data to analyze PREP performance data and to generate performance measurement reports for ACF.

The entry screens of the PMMS include a series of automated validity checks to identify some types of errors as the data are entered. Error messages alert grantees to inconsistencies between data elements, values beyond the expected range, and similar issues, and provide an opportunity for the grantee to correct such errors before the data are submitted. The system also conducts automated checks to ensure that the full set of performance measures are entered.

Additional quality checks are conducted to identify remaining issues. Cases with unresolved data issues may be omitted from analyses that rely on the problematic data elements. If suspect data are included in any tabulations, caveats will be included alongside the reported data.

Data Analysis

The PM effort involves collecting performance measures data that will be used to monitor and analyze grantee performance.

Using the performance data for accountability requires constructing indicators for many of the same measures, but separately for each grantee, and in some cases each sub-awardee provider or program. Indicators at the grantee level help fulfill federal responsibilities to hold grantees accountable for performance. Indicators at the sub-awardee and program levels will help grantees in their efforts to hold accountable those to whom they are providing resources for PREP implementation. The structure of the data will also allow for examining several of these questions by program model to better understand successes and challenges implementing the various programmatic approaches.

The results of the performance measures analysis will help ACF and grantees pinpoint areas for possible improvement of program implementation. ACF will learn which implementation challenges are most evident to grantees and their sub-awardees, and which are seen as topics for technical assistance. Over time, data can demonstrate which grantees and sub-awardees are improving with respect to elements of program delivery and which areas of technical assistance require on-going attention.

Data Use

In addition to the availability of the data to ACF and grantees in the dashboard, the performance measures data will be used to develop end-of-cohort reports, which will synthesize performance measures data across years. These reports will include summaries of the data collection and analysis methods as well as appropriate caveats and data limitations. ACF will also use the data provided for annual reporting to Congress and OMB.

B8. Contact Person(s)

• Caryn Blitz OPRE, ACF, Caryn.Blitz@acf.hhs.gov

• Tia Zeno, OPRE, ACF, Tia.Zeno@acf.hhs.gov

• Lara Hulsey, Mathematica (contractor), lhulsey@mathematica-mpr.com

• Lauren Murphy, Mathematica (contractor),lmurphy@mathematica-mpr.com

• Melissa Thomas, Mathematica (contractor), mthomas@mathematica-mpr.com

Attachments

Instrument 1: Participant Entry Survey

1a: Participant Entry Survey (middle school-aged youth)

1b: Participant Entry Survey (high school-aged and older youth)

Instrument 2: Participant Exit Survey

2a: Participant Exit Survey (middle school-aged youth)

2b: Participant Exit Survey (high school-aged and older youth)

Instrument 3: Performance Reporting Data Entry Form – Grantees

Instrument 4: Sub-awardee Data Collection and Reporting Form – subawardee program providers

Appendix A: 60-day FRN

Appendix B: Consent forms

1 The success sequence was first discussed by Haskins and Sawhill in Creating an Opportunity Society (Washington, DC: Brookings Institution; 2009. https://www.brookings.edu/book/creating-an-opportunity-society/), which identified a correlation between having income above the poverty level and three “norms:” (1) completing high school, (2) working full time, and (3) waiting until age 21 and marrying before having children.

2 The proposed items related to adulthood preparation subjects and the success sequence are also included in the performance measures for the Sexual Risk Avoidance Education Performance Analysis Study (30 day FRN posted on 05/23/19).

3 Measures of structure, cost, and support for program implementation will be submitted once per year, and all other measures will be submitted twice per year.

8