Personal Responsibility Education Program (PREP) Multi-Component Evaluation – Extension
OMB Information Collection Request
0970 - 0398
Supporting Statement
Part A
November 2017
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officer:
Caryn Blitz
A1. Necessity for the Data Collection
The Family and Youth Services Bureau (FYSB) and the Office of Planning, Research, Evaluation (OPRE) in the Administration for Children and Families (ACF) are requesting a one-year extension without change of a currently approved information collection (OMB No. 0970–0398). The purpose of the extension is to complete the ongoing follow-up data collection for the Personal Responsibility Education Program (PREP) Multi-Component Evaluation, which was designed to document how PREP programs are designed and implemented in the field, collect performance measure data for PREP programs, and assess the effectiveness of selected PREP-funded programs.
The PREP Multi-Component Evaluation contains three components: A Design and Implementation Study, a Performance Analysis Study, and an Impact and In-Depth Implementation Study. Data collection related to both the Design and Implementation Study and Performance Analysis Study is complete. This request for a one-year extension without change is limited to data collection activities for the Impact and In-Depth Implementation Study, which is being conducted in four sites. The proposed extension is necessary to complete ongoing follow-up data collection. The resulting data will be used in a rigorous program impact analysis to assess the effectiveness of each program in reducing teen sexual activity and associated risk behaviors.
In March 2010, Congress authorized the Personal Responsibility Education Program (PREP) as part of the Patient Protection and Affordable Care Act (ACA). PREP provides grants to states, tribes, tribal communities, and local organizations to support evidence-based programs to reduce teen pregnancy and sexually transmitted infections (STIs). The programs are required to provide education on both abstinence and contraceptive use. The programs also offer information on adulthood preparation subjects such as healthy relationships, adolescent development, financial literacy, parent–child communication, education and employment skills, and healthy life skills.
In line with PREP’s emphasis on evidence-based programming, Congress also mandated a federal evaluation of the PREP program. To meet this need, FYSB and OPRE contracted with Mathematica Policy Research and its subcontractors to conduct the PREP Multi-Component Evaluation, a seven year evaluation to document how PREP-funded programs are operationalized in the field, collect performance measure data from PREP grantees, and assess the effectiveness of selected PREP-funded programs on reducing teenage pregnancies, sexual risk behaviors, and STIs.
The evaluation includes three complementary components, each with distinct data collection activities:
Design and Implementation Study, a broad descriptive analysis of how states are using PREP grant funding to support evidence-based teen pregnancy and STI prevention programs;
Performance Analysis Study, focused on the collection and analysis of performance management data from state grantees, tribal grantees, and Competitive PREP grantees;
Impact and In-Depth Implementation Study, designed to assess the impacts and implementation of funded programs in four selected PREP sites.
OMB approved the initial information collection request for activities related to the PREP Evaluation in November 2011 (OMB No. 0970–0398). ACF subsequently requested and received approval for several revisions to the information collection. These prior approvals covered activities related to all three components of the PREP Evaluation: the Design and Implementation Study, the Performance Analysis Study, and the Impact and In-Depth Implementation Study.
Data collection related to both the Design and Implementation Study and Performance Analysis Study is now complete. In this submission, ACF is requesting a one-year extension without change specific to the follow-up data collection for the Impact and In-Depth Implementation Study. This follow-up data collection involves longitudinal surveys administered to the individual youth enrolled in three of the four impact study sites. Follow-up data collection is ongoing and expected to continue beyond the current OMB expiration date of November 30, 2017. ACF is requesting a one-year extension to the current expiration date to complete the follow-up data collection as planned.
On March 23, 2010 the President signed into law the Patient Protection and Affordable Care Act, H.R. 3590 (Public Law 111-148, Section 2953). In addition to its other requirements, the act amended Title V of the Social Security Act (42 U.S.C. 701 et seq.) to include formula grants to states and territories, and competitive grants to tribes and local organizations to “replicate evidence-based effective program models or substantially incorporate elements of effective programs that have been proven on the basis of scientific research to change behavior, which means delaying sexual activity, increasing condom or contraceptive use for sexually active youth, or reducing pregnancy among youth.” The legislation mandates that the Secretary evaluate the programs and activities carried out with funds made available through PREP. To meet this requirement, FYSB and OPRE within ACF have contracted with Mathematica Policy Research and its subcontractors to conduct the PREP Multi-Component Evaluation.
A2. Purpose of Survey and Data Collection Procedures
Overview of Purpose and Approach
ACF’s request for a one-year extension without change is specific to the PR EP Impact and In-Depth Implementation Study, which is assessing the impacts and implementation of funded programs in four selected PREP sites. Data collection is complete in one site, but is ongoing in the other three sites. In each site, the study team is using a random assignment evaluation design and longitudinal survey data to determine the effectiveness of PREP-funded programs in improving key outcomes related to teen pregnancy, STIs, and associated sexual risk behaviors. Data on key outcomes are collected through the administration of follow-up surveys to participating youth.
The follow-up data collection focuses on two types of outcomes, both of which can be measured only through surveys of youth. The first are sexual risk outcomes, including the extent and nature of sexual activity, use of contraception (if sexually active), pregnancy, and testing for and diagnoses of STDs. The second are a series of intermediate outcomes that may be associated with the sexual risk outcomes and therefore important to measure as potential pathways of any program effects on sexual risk behavior. Examples of these outcomes include participation in and exposure to pregnancy prevention programs and services, intentions and expectations of sexual activity, relationships with family and friends, knowledge of contraception and sexual risks, dating behavior and alcohol and drug use. In addition, for sample youth who report not being sexually active, the survey includes questions to support a descriptive analysis of these youth and a future investigation of their potential transition into sexual activity.
Research Questions
The follow-up survey data will be used to address the following research questions on program impact:
Are the approaches effective at meeting their immediate objectives (for example, improving knowledge of pregnancy risks)?
Are the approaches effective at reducing adolescent pregnancy?
What are their effects on related outcomes, such as postponing sexual activity and reducing or preventing sexual risk behaviors and STDs?
Do these approaches work better for some groups of adolescents than for others?
Study Design
The PREP Impact and In-Depth Implementation Study uses a random assignment evaluation design to measure the effects of selected PREP-funded programs on youth outcomes. This component of the evaluation is being conducted in four selected sites around the country: Iowa, Kentucky, New York, and Texas. In each of the four participating sites, consented sample members are randomly assigned to either a treatment group offered a PREP-funded teen pregnancy prevention program or to a control group that is not. Both groups of youth complete a baseline survey administered before random assignment and follow-up surveys approximately 12 and 24 months later. In all four sites, sample enrollment and baseline data collection has been completed. Follow-up data collection is still ongoing in three sites; one site (New York) has completed follow-up data collection. ACF’s request for a one-year extension without change is specific to the second follow-up survey data collection in the three sites, which is expected to continue beyond the current OMB expiration date of November 30, 2017.
The mode of follow-up data collection varies by site. Wherever possible, the evaluation team administers the follow-up surveys in groups using a paper-and-pencil instrument (PAPI). When necessary to increase response rates or accommodate specific populations, this method has been augmented with or replaced by a computer assisted telephone interview (CATI) follow-up or a telephone follow-up with hard copy. For the telephone follow-up with hard copy, trained interviews read the PAPI survey aloud to respondents over the phone, and the interviewers record the respondent’s answers on a hard copy survey. The previously approved follow-up data collection instruments that will continue to be used are attached as Instruments 1 and 2.
Universe of Data Collection Efforts
The extension to follow-up survey data collection is specific to youth participating in the Impact and In-Depth Implementation Study component of the PREP evaluation in three sites: Iowa, Kentucky, and Texas. In each site, all consented sample members are eligible to participate in the follow-up surveys. The previously approved follow-up data collection instruments include:
Master Follow Up Survey: This follow up survey is administered at all but one of the four sites.
HFSA Follow Up Survey: This follow up survey was modified to be administered at Healthy Families San Angelo (HFSA). This is a modified version of the Master Follow-up Survey, modified to reflect the fact that the sample is all female and that all youth have already had sexual intercourse, and therefore do not need to be asked about sexual initiation. In addition, there are questions about subsequent pregnancies, parenting behaviors, and couple relationships to assess program effectiveness on these outcomes, which are goals of that program.
A3. Improved Information Technology to Reduce Burden
ACF is not proposing any changes to the previously approved data collection procedures or instruments as part of this one-year extension request. The previously approved procedures for the follow-up surveys were designed to balance the related issues of efficiency, accuracy, and respondent burden. Wherever possible, the follow-up surveys are administered in group using a self-administered pencil and paper survey instrument (PAPI). The advantages of PAPI over other data collection approaches, such as laptops or personal digital assistants (PDAs), are that it enables respondents to set their own pace (allowing for more accurate responses to sensitive questions); reduces costs; and simplifies administration logistics. Studies have shown no difference between PAPI and computer-assisted self-interviewing (CASI) in reports of most measures of male-female sexual activity, including reports such as ever having had sexual intercourse, recent sexual activity, number of partners, condom use, and pregnancy.1,2,3,4,5 This method is also consistent with other national youth surveys (for example, the National Youth Risk Behavior Survey) and the Evaluation of Adolescent Pregnancy Prevention Approaches (PPA), sponsored by the Office of Adolescent Health (OAH) within HHS.
In those instances in which the survey cannot be administered in a group-based setting, respondents are surveyed via telephone. For example, one of the PREP Impact and In-Depth Implementation study sites is assessing the effectiveness of a home visiting program for teen mothers. The structure of the home visiting program does not provide a natural group setting for survey administration. Therefore, the follow-up surveys are conducted via computer assisted telephone interviewing (CATI). Telephone interviewing is more cost efficient than CASI and has been used successfully on other teen pregnancy prevention evaluations, including the federal PPA study for OAH.
A4. Efforts to Identify Duplication
In planning for PREP Impact and In-Depth Implementation study, ACF carefully reviewed the information collection requirements to avoid duplication with either existing studies or other ongoing federal teen pregnancy prevention evaluations. ACF believes that the follow-up surveys for the PREP Impact and In-depth Implementation study complement, rather than duplicate, the existing literature and the other ongoing federal teen pregnancy prevention evaluations.
A5. Involvement of Small Organizations
Programming in some of the PREP Impact and In-Depth Implementation study sites is delivered by local community-based organizations. The data collection plan for the follow-up surveys was designed to minimize burden on such sites by providing staff from Mathematica Policy Research to manage the group administered data collection. For respondents who do not complete the follow-up survey in the group setting, Mathematica will conduct a telephone data collection, thus minimizing requirements for extensive “sample pursuit” by site staff.
A6. Consequences of Less Frequent Data Collection
Outcome data collected through the follow-up surveys are essential to conducting a rigorous evaluation of PREP programs supported under Public Law 111-148. Without completing the follow-up survey data collection, ACF cannot answer the primary research questions and estimate the effectiveness of the four selected program.
A7. Special Circumstances
There are no special circumstances for the proposed data collection effort.
A8. Federal Register Notice and Consultation
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on August 10, 2017, Volume 82, Number 153, page 37453, and provided a sixty-day period for public comment. A copy of this notice is attached as Attachment A. During the notice and comment period, no comments were received.
The names and contact information of the persons consulted in the drafting and refinement of the previously approved follow-up surveys are found in Attachment B.
A9. Incentives for Respondents
ACF is not requesting any change to the previously approved incentives provided to respondents. Gift cards are provided to study participants in appreciation of their participation in the study. These gift cards are important because many of the respondents are members of hard-to-reach populations, such as pregnant and parenting teens. In addition, the surveys include highly sensitive questions, and thus impose additional burden on respondents. Research has shown that incentives are effective at increasing response rates for populations similar to participants in PREP programs.6,7,8 Research also suggests that providing an incentive for earlier surveys may contribute to higher response rates for subsequent surveys.9
For the group survey administrations, respondents receive $15 gift cards for completing a 12-month follow-up survey and $20 gift cards for completing a 24-month follow-up survey. For surveys completed by telephone, respondents receive $20 gift cards for completing a 12-month follow-up survey and $25 gift cards for completing a 24-month follow-up survey. Slightly larger gifts are offered to respondents who complete surveys outside of group administration because of the additional burden associated with phone administration, requiring greater initiative and cooperation on behalf of the respondent, as well as additional time outside of school or their ordinary day. For both group survey administrations and telephone surveys, slightly larger gifts are offered to respondents for the 24-month follow-up surveys to promote high response rates. Attrition from surveys tends to increase over time due to mobility of participants and study fatigue. Higher incentives are needed to continue to ensure participant responses. Research has shown that gifts of this size are effective at increasing response rates for populations similar to those participating in this study.10,11 Throughout the study, the evaluation team has offered these incentives and maintained response rates of more than 80 percent for all sites and for each of the two follow-up waves.
A10. Privacy of Respondents
Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law.
As specified in the contract, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ personally identifiable information. The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. Field staff administering the follow-up surveys to youth are required to sign a Confidentiality Pledge (see Attachment C).
As specified in the evaluator’s contract, the Contractor shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access.
Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.
A11. Sensitive Questions
ACF is not requesting any changes to the previously approved follow-up survey instruments. Because a primary focus of PREP-funded programs is to prevent teen pregnancy through a decrease in sexual activity and/or an increase in contraceptive use, some questions on the follow-up surveys for the PREP Impact and In-Depth Implementation study relate to these issues. ACF drew on questions from previously-successful youth surveys and evaluations in developing the follow-up survey instruments that were previously approved. Table A11.1 provides a list of the sensitive questions found on the PREP follow-up surveys, along with a justification for their inclusion.
Table A11.1. Summary of Sensitive Questions on the Follow-Up Surveys and Their Justification
Topic |
Justification |
Sexual activity, incidence of pregnancy and STDs, and contraceptive use Master follow-up survey questions 4.12, 5.1 in B1 and B2; 5.2-5.21 in B1; 6.1-6.7 in B1; 6.1-6.4 in B2; 7.6.f in B1 and B2 |
Sexual activity, incidence of pregnancy and STDs, and contraceptive use are all key outcomes for the evaluation. The majority of these questions are asked only of youth who report being sexually active. |
Intentions regarding sexual activity Master follow-up survey question 5.13 in B2 |
Intentions regarding engaging in sex and other risk-taking behaviors are extremely strong predictors of subsequent behavior (Buhi and Goodson, 2007). Intentions are strongly related to behavior and will be an important mediator predicting behavior change. |
Drug and alcohol use Master follow-up survey - questions 7.1–7.5 in B1 and B2 |
There is a substantial body of literature linking various high-risk behaviors of youth, particularly drug and alcohol use, sexual intercourse, and risky sexual behavior. The effectiveness of various program strategies is expected to differ for youth who are and are not experimenting with or using drugs and alcohol (Tapert et al., 2001; Li et al., 2001; Boyer et al., 1999; Fergusson and Lynskey, 1996; Sen, 2002; Dermen et al., 1998; Santelli et al., 2001.) |
A12. Estimation of Information Collection Burden
The previously approved burden for follow-up survey data collection for the Impact and In-Depth Implementation Study component of the PREP Evaluation was estimated to be 5,288 hours.
We estimate that through November 2017, a total of 4,872 burden hours will have been used for follow-up data collection, with 416 hours remaining. It is expected that 325 youth will complete follow-up surveys between December 1, 2017 and November 30, 2018. Based on previous experience with the follow-up questionnaire, it is estimated that it will take youth 45 minutes (0.75 hour) to complete the follow-up survey, on average. Therefore, the total burden for the follow-up survey during the one-year extension period we are requesting is estimated to be 244 hours (325 respondents x 0.75 hours burden; see Table A12.1).
Table A12.1. Total Burden Requested Under this Information Collection
Instrument |
Total/Annual Number of Respondents |
Number of Responses Per Respondent |
Average Burden Hours Per Response |
Annual Burden Hours |
Average Hourly Wage |
Total Annual Cost |
Follow-up survey |
325 |
1 |
.75 |
244 |
$7.25 |
$884.50 |
Estimated Annual Burden Total |
244 |
|
$884.50 |
|||
Based on the ages of sample members at sample intake, we assume that 50 percent of the remaining respondents will be 18 or older at the time of the follow-up survey. Therefore, the cost of this burden during this one-year extension period we are requesting is estimated to be $884.50 (= 244 hours * .5 for the proportion of youth 18 or over * $7.25).
A13. Cost Burden to Respondents or Record Keepers
There are no additional costs to respondents.
A14. Estimate of Cost to the Federal Government
The total cost for follow-up survey data collection under the PREP Multi-Component Evaluation is estimated at $2,004,021 (across four years for follow-up data collection). Annual costs to the Federal government for the ongoing follow up data collection under this extension will be $501,005..
A15. Change in Burden
OMB previously approved the burden associated with the follow-up survey for the PREP Impact and In-Depth Implementation Study component of the PREP Evaluation. ACF’s request for a one-year extension without change to complete the follow-up survey data collection does not increase the overall burden previously approved, it just extends the time for data collection.
A16. Plan and Time Schedule for Information Collection, Tabulation and Publication
Program impacts will be analyzed separately for each site using data from the baseline and follow-up surveys. Site-specific impact analyses began in 2015 and will continue through 2018 as the remaining follow-up data collection is completed. For each site, regression-adjusted impact estimates will be estimated for each primary outcome in each site, drawing on baseline and follow-up data. The set of primary analyses for each site will be limited to a small set of key outcomes, including measures of sexual risk behavior and its health consequences. To support these analyses, the follow-up surveys include measures of pregnancy, STIs, and associated sexual risk behaviors. Subgroup analyses will be performed according to characteristics captured in the baseline survey data, including prior sexual experience and other risk factors.
The results of the analyses will be reported in a series of site-specific impact reports. ACF released the first site-specific report, for the Kentucky site, in May 2017. The remaining impact reports will be released on a rolling basis through early 2019. All of the site-specific implementation reports associated with the PREP Impact and In-Depth Implementation Study have been completed and released.
A17. Reasons Not to Display OMB Expiration Date
All instruments will display the expiration date for OMB approval.
A18. Exceptions to Certification for Paperwork Reduction Act Submissions
No exceptions are necessary for this information collection.
1 Turner, C.F., L. Ku, S.M. Rogers, L.D. Lindberg, J.H. Pleck, and F.L. Sonenstein. “Adolescent Sexual Behavior, Drug Use, and Violence: Increased Reporting with Computer Survey Technology.” Science, vol. 280, 1998, pp. 867–873.
2 Beebe, Timothy J., Patricia A. Harrison, James A. McCrae Jr., Ronald E. Anderson, and Jayne A. Fulkerson. “An Evaluation of Computer-Assisted Self-Interviews in a School Setting.” Public Opinion Quarterly, vol. 62, 1998, pp. 623–632.
3 Beebe, Timothy J., Patricia A. Harrison, Eunkyung Park, James A. McRae, Jr., and James Evans. “The Effects of Data Collection Mode and Disclosure on Adolescent Reporting and Health Behavior.” Social Science Review, vol. 24, no. 4, 2006, pp. 476–488.
4 Brener, Nancy D., Danice K. Eaton, Laura Kann, JoAnne Grunbaum, Lori A. Gross, Tonja M. Kyle, and James G. Ross. “The Association of Survey Setting and Mode with Self-Reported Health Risk Behaviors Among High School Students.” Public Opinion Quarterly, vol. 70, 2006, pp. 354–374.
5 Webb, P.M., G.D. Zimet, J.D. Fortenberry, and M.J. Blythe. “Comparability of a Computer-Assisted Versus Written Method for Collecting Health Behavior Information from Adolescent Patients.” Journal of Adolescent Health, vol. 24, no. 6, 1999, pp. 383–388.
6 Berlin, Martha, Leyla Mohadjer, Joseph Waksberg, Andrew Kolstad, Irwin Kirsch, D. Rock, and Kentaro Yamamoto. 1992. An experiment in monetary incentives. In JSM proceedings, 393–98. Alexandria, VA: American Statistical Association.
7 James, Jeannine M., and Richard Bolstein. 1990. The effect of monetary incentives and follow-up mailings on the response rate and response quality in mail surveys. Public Opinion Quarterly 54 (3): 346–61.
8 Singer, Eleanor, and Richard A. Kulka. 2002. Paying respondents for survey participation. In Studies of welfare populations: Data collection and research issues, eds. Michele Ver Ploeg, Robert A. Moffitt, and Constance F. Citro, 105–28. Washington, DC: National Academy Press.
9 Singer, Eleanor, John Van Hoewyk, and Mary P. Maher. 1998. Does the payment of incentives create expectation effects? Public Opinion Quarterly 62:152–64.
10 Singer, E., and R.A. Kulka. “Paying Respondents”.
11 U.S. Department of Education. National Center for Education Statistics. Technical Report and Data File User’s Manual For the 1992 National Adult Literacy Survey, NCES 2001–457, by Irwin Kirsch, Kentaro Yamamoto, Norma Norris, Donald Rock, Ann Jungeblut, Patricia O’Reilly, Martha Berlin, Leyla Mohadjer, Joseph Waksberg, Huseyin Goksel, John Burke, Susan Rieger, James Green, Merle Klein, Anne Campbell, Lynn Jenkins, Andrew Kolstad, Peter Mosenthal, and Stéphane Baldi. Project Officer: Andrew Kolstad. Washington DC: 2001.
| File Type | application/msword |
| File Title | OPRE OMB Clearance Manual |
| Author | DHHS |
| Last Modified By | SYSTEM |
| File Modified | 2017-11-13 |
| File Created | 2017-11-13 |