Supporting Statement A_6.11.19(final)

This is a request for Office of Management and Budget (OMB) approval for new data collection to evaluate sexual risk avoidance (SRA) curricula that are aligned with the Systematic Method for Assessing Risk Avoidance Tool (SMARTool).^¹ The SMARTool is a technical assistance guide for use by schools, youth-serving organizations, and other agencies interested in delivering SRA education to youth.^² It was developed by the Center for Relationship Education (CRE) as part of a five-year cooperative agreement with the Centers for Disease Control and Prevention and was released in 2008. The Office of the Assistant Secretary for Health (OASH) is sponsoring a project to update the SMARTool to include emerging themes important to achieving optimal health outcomes within the adolescent SRA approach.^³ OASH also plans to conduct an evaluation to assess the impact of two SRA curricula that are aligned with the updated SMARTool. The aims of the evaluation are to (1) produce preliminary evidence regarding the effects of two SMARTool-aligned SRA curricula on proximal youth outcomes^⁴, and (2) derive lessons learned to improve SRA program implementation. The evaluation will use quantitative and qualitative methods. OASH is authorized to conduct this study by the Public Health Service Act (42 U.S.C.241) and the FY2017 Consolidated Appropriations for General Departmental Management to evaluate and refine programs that reduce teen pregnancy, including programs that implement SRA education.

To implement this project, OASH contracted with the MITRE Corporation, the operator of the Health Federally Funded Research and Development Center (FFRDC). The Health FFRDC team also includes two MITRE subcontractors, RTI International (RTI) and Atlas Research, Inc. Under the auspices of this project, MITRE has awarded subcontracts to two implementing organizations (IOs) that have established partnerships with approximately 14 secondary schools in 11 school districts to implement SRA education programs. For the purposes of this project, the FFRDC team defines IOs to be organizations focused on health, education, and social services that have experience working with schools or school districts to implement SRA programming. One IO has established partnerships with approximately 4 schools in 1 school district for this project, and the other has established partnerships with 10 schools in 10 different school districts. Each IO will be teaching a different SMARTool-aligned SRA curriculum to youth. Accordingly, results for each IO will be analyzed and reported separately, rather than pooled. Exhibit 1 displays these details.

Exhibit 1. Details about Participating Two Implementing Organizations (IOs)

IO and SRA curriculum

Length of curriculum

Expected # of districts

Expected # of schools

Estimated # of classrooms

Estimated # of students

IO A

(REAL Essentials)

6 sessions (7.5 hours)

17-20 classrooms (30-35 students per classroom)

1,200

IO B (Represent®)

5 lessons (3.75 hours)

56-60 classrooms
(20-25 students per classroom)

1,200

*The REAL Essentials curriculum used in IO A may require 1-3 weeks to complete, depending on school scheduling constraints (e.g., longer class block schedules vs. shorter class period schedules). The represent® curriculum used in IO B is approximately 1 week long.

Purpose and Use of Information Collection

The proposed formative evaluation will seek evidence of the effect of SMARTool-aligned SRA curricula on proximal youth outcomes (i.e., SRA-related knowledge, attitudes, intentions, skills, and peer and parental influences) that shape short- and longer-term sexual health behaviors. These proximal outcomes are aligned with (a) the 10 SMARTool “targets” or themes that correspond to protective factors associated with the sexual behaviors of youth, and (b) an SRA logic model developed as part of this project. A list of the 10 SMARTool targets is presented in Exhibit 2.

Exhibit 2. Overview of the 10 SMARTool Targets

Target Number	Description
1	Enhance knowledge of physical development and sexual risks
2	Healthy relationship development
3	Support personal attitudes and beliefs that value sexual risk avoidance
4	Acknowledge and address common rationalizations for sexual activity
5	Improve perception of and independence from negative peer and social norms
6	Build personal competencies and self-efficacy to avoid sexual activity
7	Strengthen personal intention and commitment to avoid sexual activity
8	Identify and reduce the opportunities for sexual activity
9	Strengthen future goals and opportunities
10	Partner with parents

The SRA logic model is presented in Exhibit 3, with annotation to indicate which factors will be measured in the evaluation.

Exhibit 3. SRA Logic Model

Factors from the SRA logic model that will be examined in this study are shown in Exhibit 4.

Exhibit 4. SRA Logic Model Factors that Will Be Examined in This Study

Background and Control Factors

Proximal Outcomes (Risk and Protective Factors)

Biological sex
Mother’s education
Father’s education
Receipt of school lunch/other welfare
Household composition (e.g., mother, father, older and younger sibs)
Religiosity
Sexual orientation
Receipt of education on relationships, not having sex, birth control or STDs in past

Knowledge of physical development, sexual risks, contraception, and healthy relationships
Attitudes, beliefs, values regarding not having sex (not starting or refraining from having sex if already non-virgins) and its importance for future goals including relationships and marriage
Intentions and commitment to not have sex
Skills to resist peer pressure, refuse sex, and avoid risky situations
Peer influences: perception of peer attitudes, norms, and behavior regarding sex and other risk behaviors
Parental influences: parental connectedness and attitudes, communications, and monitoring regarding sex and other risk behaviors

The baseline and follow-up data collection instruments for youth in this evaluation are designed to measure these proximal outcomes. The data collection instruments are also designed to measure background (control or covariate) variables that have been theorized (in the SRA logic model) or empirically shown to account for some variance in youth proximal outcomes. Appendices A-J provide copies of the data collection instruments; including the Baseline and Follow-up outcome surveys (i.e., Youth Outcome Questionnaire), which include measures of the proximal outcome constructs described above.

This formative evaluation will focus on examining changes in proximal outcomes, or individual risk and protective factors (e.g., short term SRA-related knowledge, attitudes, and skills that may result from the SMARTool-aligned SRA curricula). Exhibit 5 shows the proximal outcome variables we will examine in this study, drawn from the 10 SMARTool targets and the SRA logic model.

Exhibit 5. Proximal Outcomes That Will Be Examined in This Study

From 10 SMARTool Targets

From SRA Logic Model

Knowledge of physical development and sexual risks
Knowledge of healthy relationship development
Personal attitudes and beliefs that value sexual risk avoidance
Attitudes/beliefs about common rationalizations for sexual activity
Perception of and independence from negative peer and social norms
Personal competencies and self-efficacy to avoid sexual activity
Personal intention and commitment to avoid sexual activity
Attitudes/beliefs about reducing opportunities for sexual activity
Future goals and opportunities
Parent influences and involvement

Knowledge of physical development, sexual risks, contraception, and healthy relationships
Attitudes, beliefs, values regarding not having sex (not starting or refraining from having sex if already non-virgins) and its importance for future goals including relationships and marriage
Intentions and commitment to not have sex
Skills to resist peer pressure, refuse sex, and avoid risky situations
Peer influences: perception of peer attitudes, norms, and behavior regarding sex and other risk behaviors
Parental influences: parental connectedness and attitudes, communications, and monitoring regarding sex and other risk behaviors

It is important to note that given the short time frame between the baseline and follow-up youth surveys, we do not anticipate observing changes in longer-term outcomes such as youth sexual behavior, so this formative evaluation does not include research questions related to the impact of the SRA curricula on youth sexual behavior. The major driver of this decision is the available funding and period of performance of MITRE’s contract for this project. Given the timeline for data collection and analysis for this study, it will not be feasible to conduct a second, longer-term follow-up data collection to assess longer-term outcomes such as behavior change. Limitations associated with conducting an impact analysis that focuses on youth attitudes and beliefs, but not behaviors will be discussed in the final report.

Overview of Study Design

The study design to assess whether such changes have occurred is depicted in Exhibit 6.

Exhibit 6. Quasi-Experimental Study Design: Within-School Comparison Group

This study design is known as a quasi-experimental design with a within-school comparison group. Classes of students at each school will be assigned to either the intervention or comparison (i.e., non-intervention) group. The intervention group classes will receive the curriculum in Fall 2019, and the comparison group classes will not. The project team will collect data on proximal outcomes and background characteristics from youth in both groups (intervention and comparison group) at two time points: (a) Baseline: immediately before the intervention group receives the curriculum, and (b) Follow-up: immediately after the intervention group receives the curriculum.

The curriculum will be provided to the comparison group after follow-up data collection is complete, but this curriculum delivery is not part of the study design.

The primary research question being tested in this design is as follows:

What are the effects of SRA curricula on participants’ SRA-related proximal outcomes (e.g., knowledge, attitudes, intentions, skills, and perceptions or experiences of peer and parental influences)?

As noted above, our study design examines the effects of SRA curricula on SRA participants’ proximal outcomes relative to a comparison group that did not receive the SRA curricula during the field period of data collection.

The analytic approach to answer this question will involve a multilevel analysis to account for clustering (nesting) of students within classrooms as the level of assignment to intervention vs. comparison groups. The analysis will compare the change in youth outcomes in the intervention group over time to the change in youth outcomes in the comparison group over time. If the change in the intervention group is larger than the change in the comparison group, controlling for key covariates (e.g., baseline sexual experience, demographics), then the study can be said to support the hypothesis that SRA curricula have some effect on the proximal youth outcomes measured in this study.

In addition to this research question, the process evaluation will seek evidence of potential explanations for youth proximal outcome findings and generate descriptive data and qualitative insights on promising practices for IOs in delivering SRA programs. Process evaluation data will be collected on adherence (what is being taught), quality (how the content is taught), and implementation (logistics responsible for a conducive learning environment, such as context, dosage, and participant responsiveness).^⁵ Descriptive and qualitative data will also be collected on IOs and their implementation to inform collection and dissemination of good practices for SRA program providers and the broader field of adolescent pregnancy prevention. Process evaluation data will be collected from youth participants in the program, facilitators who deliver the curriculum, and observers who monitor the fidelity/quality of program delivery.

The IOs participating in the evaluation may use the results of the evaluation to improve their SRA curriculum and program implementation. The results of the evaluation will also inform HHS on the extent to which two curricula affect youth proximal outcomes that are aligned with the SMARTool Targets.

The data collection instruments may be shared with program providers or researchers who want to evaluate the impact of SRA curricula on the SMARTool targets. If resources permit, the project team will also conduct psychometric analysis (e.g., Cronbach’s alpha, factor analysis) and report basic psychometrics on the youth data collection instruments to inform HHS.

Limitations of the study design.

The limitations of a within-school comparison group design measuring short-term effects on proximal outcomes are described here and will be described in the final report for the evaluation.

One limitation of a within-school comparison group design is the potential for unmeasured spillover or contamination effects, or the possibility that students in the intervention group classes will communicate with students in the comparison group classes in a manner that influences comparison students’ attitudes, beliefs, or intentions before follow-up data are collected.. Spillover and contamination effects result in potentially underestimating the impact of the SRA curricula.^⁶

The study team explored the possibility of mitigating the potential for spillover or contamination effects by employing a study design that used a between-school comparison group (rather than a within-school comparison group). However, the power calculations for a design with a between-school comparison group revealed a prohibitively large number of schools (at least 22 intervention and 22 comparison schools) would be required to adequately power this design, given the project constraints (e.g., funding, time).^⁷ Further, designs with a between-school comparison group typically have their own limitations— “business as usual” in which schools in the comparison group are implementing their own pregnancy prevention curricula.^⁸ Nonetheless, the limitations of the within-school comparison group design will be discussed in the final report for the evaluation.

Use of Improved Information Technology and Burden Reduction

Where feasible, available technology will be used to reduce burden and improve efficiency and accuracy. For example, student-level school records (such as gender, race/ethnicity, free and reduced price meal eligibility, English language learner status) will be provided to the study team in electronic format via a secure file transfer protocol (SFTP) environment that is approved by the MITRE Privacy Office for PII storage and transfer and that is only accessible to the project team. Data submitted in hard copy or via fax or PDF file will be manually keyed for storage in this secure environment.

Because of variable access to computers and the Internet in school settings, all surveys will be administered with paper-and-pencil questionnaires.

Efforts to Identify Duplication and Use of Similar Information

The evaluation does not duplicate other studies. Several studies have evaluated SRA curricula, including evaluation of curricula such as “Big Decisions” (Realini et al., 2010), “Choosing the Best” (Lieberman & Su, 2012), “Yes You Can!” (Donnelly et al., 2016), and others (Borawski et al., 2005; Jemmott et al., 2010; Kirby, 2008; Markham et al., 2012; Piotrowski & Hedeker, 2016; Weed et al., 2008). However, no studies have specifically evaluated the impact of the SRA curricula on youth outcomes that are aligned with the SMARTool targets.

Impact on Small Businesses or Other Small Entities

Schools included in this study may be part of small school districts. Burden will be minimized wherever possible. Contractor staff will conduct all data collection activities and will assist with parental consent and other study tasks as much as possible within each school. Schools will be able to provide student data (e.g., attendance data) in the format that is most efficient for them.

For the purposes of this information collection request, we make the conservative assumption that 100 percent of the schools providing student rosters and administrative data are part of districts that will meet OMB’s definition of small entities: a small government jurisdiction which is a government of a city, county, town, township, school district, or special district with a population of less than 50,000.

Consequences of Not Collecting the Information or of Collecting Less Frequently

If the youth surveys were not conducted, it would not be possible to assess changes in program participants’ proximal outcomes.

Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

The proposed data collection is consistent with guidelines set forth in 5 CFR 1320.5.

Comments in Response to the Federal Register Notice, and Outside Consultation

Federal Register Notice Comments. A 60-Day Federal Register Notice was published in the Federal Register on July 2, 2018, in Volume 83, Number 127, pages 30942–30943 (see Appendix N). There were three public comments. A summary of public comments received in response to the 60-Day Federal Register Notice and a response to these comments are in Appendix R. No changes to this information collection request were made as a result of the public comments.

A 30-Day Federal Register Notice was published in the Federal Register on February 8, 2019, in Volume 84, Number 27, pages 2884-2885 (see Appendix L). There was only one public comment requesting copies of the data collection instruments, which were provided to the requestor. No changes to this information collection request were made as a result of the public comments.

Changes were made to this information collection after the publication of the 30-day Federal Register Notice due to consultation with IOs on community standards for sensitive questions and follow-on expert review to minimize youth burden in completing the updated versions of the instruments. IOs provided guidance regarding acceptable language in their communities for youth surveys on sensitive topics such as sexual and pre-coital behaviors. This guidance was solicited to help to increase parental consent and youth response rates in communities where the study will be conducted. After IOs provided guidance, the youth surveys were updated to reflect community standards, and the updated survey versions were provided an expert review to minimize youth burden (e.g., clarifying instructions and question wording).

Outside Consultation. To inform the design of the data collection, OASH consulted with a panel of technical experts on SRA education; the organization responsible for conducting the data collection, the MITRE Corporation; and its subcontractors, RTI, and Atlas Research. Exhibit 7 presents the name, affiliation, and contact information of members of the technical expert panel.

Exhibit 7. Outside Consultation with Technical Experts

Nanci Coppola, D.P.M., M.S.

Principal

Coppola and Company

ncoppola@coppolaandcompany.com

William Jeynes, Ph.D.

Professor, California State University, Long Beach

Senior Fellow, Witherspoon Institute

William.jeynes@csulb.edu

Alma Golden, M.D.

Deputy Assistant Administrator

Bureau for Global Health, United States Agency for International Development

Agolden@usaid.gov

Peggy Pecchio

Executive Director

Operation Keepsake, Inc.

PPecchio@operationkeepsake.com

Wade F. Horn, Ph.D.

Managing Director

Deloitte Consulting

whorn@deloitte.com

Lisa Rue, Ph.D.

Vice President, Marketing and Strategic Partnerships

Preventative Technology Solutions

lisa@mypreventtech.com

Explanation of Any Payment/Gifts to Respondents

A particular challenge to conducting research about youth is obtaining parental consent to participate in the data collection activities. To increase response rates, we will offer small incentives to individual students, classes or groups of students, and/or public school teachers for the return of signed consent forms (regardless of whether the parents provide consent to participate). Which type of incentive may be most appropriate for each school will be determined in collaboration with the schools and the IOs. Options include:

A $25 gift card to each public school teacher for each classroom in which at least 90% of the parental consent forms are returned. Gift cards may be used to purchase classroom supplies or to pay for a class party. (These gift cards will go to school employees, not employees of the IO administering the curriculum.)
Eligibility to participate in two $25 gift card drawings for all youth who return a signed parental consent form.
A small incentive, equivalent to $1 or less in value (e.g., pen, notepad), for each youth who returns a signed form.
$50 for a party for the class from each grade at the school with the highest return rate.

Exhibits 8 and 9 list studies that have employed similar individual and school-level incentives.

Exhibit 8. RTI Studies Involving Incentives for Class-Level Return of Consent Forms

RTI study	Incentive provided
Safe Schools/Healthy Students Initiative (1999–2004)	$25 per classroom achieving a minimum of 70% active parental consent forms signed and returned
Middle School Coordinator Initiative (1999–2004)	$25 per classroom achieving a minimum of 70% active parental consent forms signed and returned
Impact Evaluation of a School-Based Violence Prevention Program (2004–2009)	$25 per classroom achieving a minimum of 90% active parental consent forms signed and returned

Exhibit 9. RTI Studies Involving Incentives to Youth for Return of Consent Forms

RTI study	Incentive provided
Evaluating the Prevention Effects of Men of Strength (MOST) Clubs on Sexual Violence and Teen Dating Violence Perpetration	$1 token incentive for signed permission form
The Comprehension of Emergency Operations Protocols Study (CEOP)	$1 token incentive for signed permission form
Redesigned High Schools for Transformed STEM Learning Study (TSL)	$1 token incentive for signed permission form

Assurance of Confidentiality Provided to Respondents

Parental consent and youth assent forms are provided in Appendices A and B, respectively. These forms discuss participants’ rights and privacy protections.

Personally identifiable information (PII) about participating youth—including name, date of birth, and student identification number—will be collected from their school or school district office along with school records data (e.g., gender, race/ethnicity, grade, homeroom teacher, eligibility for free and/or reduced-price lunches, and English as a Second Language (ESL) designation). PII will be used to construct youth rosters for the purpose of tracking parental consent forms for the youth surveys, distributing survey booklets to youth, and tracking attrition. This latter issue of tracking attrition is particularly important in program evaluation, so that researchers and other stakeholders can understand any differences between youth who participated in the study versus those who did not. Without receiving rosters of all eligible students in the school, researchers would only have demographic data from the youth who did participate in the study and would be unable to determine whether these youth differed from those who did not participate in the study.^⁹ Some PII (e.g., demographics) from the rosters will also be used as covariates or control variables for the analysis.

This PII data will be shared with the project team by schools via a secure file transfer protocol (SFTP) environment that is approved by the MITRE Privacy Office for PII storage and transfer and that is only accessible to the project team. Privacy training and controlled administrative access approvals will be required for project team members to access this environment.

To better protect youth identities, each youth will be assigned a unique study identification number that corresponds with information they provide in the surveys and data from school records. Before the roster data is merged with the survey data, analysts will assign this unique study identification number to each student. They will then delete student names and other identifiers (e.g., student ID assigned by school) from the roster and survey data files. Date of birth will also be transformed into a simple “age” variable (e.g., 14, 15) and then date of birth will be deleted from the roster file. After the student names, student ID from the school, and date of birth are deleted from the roster file, the roster data will be merged with the survey data. Personal identifiers such as name, date of birth, and student ID number will be stripped from the datasets and destroyed.

All other data will be collected in paper form.

To protect youth’s privacy and data, for each of the youth surveys (baseline and follow-up, and process evaluation participant survey) a barcode label will be affixed to the cover of the survey booklet, on which students will either write their name or have their name pre-assigned to that specific barcode. After the survey booklets are distributed to the youth, the youth will be instructed to remove the entire cover sheet containing their name or the part of the label containing their name, leaving only the ID and barcode on the cover sheet and/or on all subsequent pages of the survey booklet. This enables the study team to link youth’s baseline and follow-up data for analysis, but without using PII to do so.

No PII will be stored with the data collected from the youth surveys or associated with the school records data. Only the study identification number will be stored with the data. Only members of the project team conducting the data collection will have a crosswalk that links study IDs to youth names. Once data collection has been completed and the final analytic datasets produced, the crosswalk will be destroyed. No study participant names or other identifying information will be included in any reports or data sets.

The RTI Research Operations Center (ROC) will process data from all paper forms. The SFTP environment will be used to store and analyze all data. The ROC complies with the confidentiality and data security guidelines set forth by the National Institute of Standards and Technology (NIST) in Recommended Security Controls for Federal Information Systems and Organizations. The ROC can accommodate both low- and moderate-risk projects, as defined by NIST’s Federal Information Processing Standards Publication 199, Standards for Security Categorization of Federal Information and Information Systems. After ROC processing is complete, the ROC will produce final datasets that are stripped of identifiers (e.g., name, DOB) and include only the study identification number. The ROC will then transfer these datasets to MITRE using the SFTP environment. All data will be stored and analyzed within this secure environment at MITRE. For paper forms, access to the keys to locked cabinets where paper data is stored will be formally controlled to ensure that only authorized individuals are allowed access to the paper data.

To further protect student and school identities, school names and locations will not be shared with the Agency sponsoring the study (OASH) or with anyone other than the Contractors performing the research. The data will be aggregated and reported for each of the two organizations delivering the two different curricula. The aggregated results may be shared with the organizations delivering the curricula for program improvement purposes.

Procedures for maintaining privacy will include: notarized nondisclosure affidavits obtained from all study personnel who will have access to individual identifiers; personnel training regarding the meaning of privacy; controlled and protected access to computer files; built-in safeguards concerning status monitoring and receipt control systems; and a secure, staffed, in-house computing facility.

Per the Protection of Pupil Rights Amendment (PPRA) (https://studentprivacy.ed.gov/topic/protection-pupil-rights-amendment-ppra), copies of the surveys will be made available in the school (or site) office for parents to review before they sign the consent form.

Data will be kept private to the extent allowed by law. Student-level records for participating youth will be requested from school districts and/or schools under the Family Educational and Privacy Act (FERPA) of 1974 (34 CFR Part 99) exception to the general consent requirement that permits disclosures to “organizations conducting studies for, or on behalf of, educational agencies or institutions” (34 CFR §§ 99.31(a) and 99.35). This information will be securely destroyed when no longer needed for the purposes specified in 34 CFR §99.35.

Data from key informant interviews and facilitator surveys will be provided to IOs and schools only at the aggregate level to protect respondents’ privacy. Because the facilitator Session Logs are part of the IOs’ ongoing program monitoring efforts, the IOs will have access to the data from the Session Logs at the individual level, but the schools will not.

Per HHS 45 CFR 46, all data will be retained for 3 years after MITRE submits its final expenditure report. Then it will be destroyed. A de-identified dataset may be provided to OASH upon completion of the study. Before delivery of any de-identified datasets to the OASH, the project team will conduct a privacy analysis and will aggregate reporting categories for demographic variables to ensure that no fewer than 5 individuals are in any combination of key demographic categories (e.g., Black female 15-year-olds), to mitigate the risk of re-identification.

RTI IRB has determined that the evaluation is not research involving human subjects as defined by DHHS regulations. We have also submitted the study to the MITRE Privacy Office and MITRE IRB for review, and we plan to submit an application for a Certificate of Confidentiality after receiving the MITRE IRB’s determination. As required by school districts, we will also submit the study to individual school district IRBs for review.

Justification for Sensitive Questions

The primary purpose of SRA curricula is to reduce adolescent sexual risk behaviors and related outcomes such as pregnancy and sexually transmitted diseases. Although this study is not assessing behavioral outcomes due to timeline constraints, it is necessary to ask sensitive questions about these behavioral constructs to contextualize the findings and include as covariates, or control variables, for the study. Questions in the youth baseline and follow-up surveys will include whether youth have ever engaged in voluntary sexual intercourse. The questionnaire will not ask about any non-voluntary sexual experiences they may have had. Youth who respond that they have never engaged in sexual intercourse will be skipped out of all further questions about sexual activity; youth who respond that they have engaged in sexual activity will be asked additional questions about age of first sexual intercourse and number and sex of partners. Questions about sexual activity are necessary to assess risk factors for sexually transmitted diseases and pregnancy.

Estimates of Annualized Hour and Cost Burden

Calculation of the estimated annualized hour burden of responding to this information collection is presented in Exhibit 10 and the estimated annualized cost to participants is presented in Exhibit 11. Evaluation data collection will be conducted in one year (May 2019–January 2020), so the total burden and costs are the same as the annualized burden and costs. The total estimated annualized burden hours are 2,243, and the total estimated cost is $30,983.30.

The 2,243 burden hours are lower than the 3,135 estimated in the 30-day FRN. This is because we finalized the number of IOs (2 rather than the 4 previously assumed) and provided an updated estimate of the number of participating schools that would provide class rosters and administrative data (14). We also updated the number of teachers, facilitators, and sessions based on these final numbers of IOs and schools.

For the youth proximal outcome evaluation, parents will be asked to sign a parental consent form, youth will be asked to review and sign a youth assent form, and youth will complete a baseline and follow-up survey. We will also collect class rosters and administrative data from sites. The numbers in the estimated annualized burden hours table (Exhibit 10) represent the burden hours for intervention and comparison groups for both IOs. Exhibit 11 provides the estimated annualized cost to respondents.

Parental consent. An anticipated 1,680 parents will complete a 5-minute parental consent form, for a total burden of 140 hours. Using the U.S. Department of Labor’s Bureau of Labor Statistics’ 2017 estimate of an average hourly wage of $24.34, the total cost will be $3,407.60.
Youth assent. An anticipated 1,680 youth will complete a 5-minute assent, for a total burden of 140 hours and a cost of $1,015.00. The figure of $7.25 per hour (an approximation of the hourly wage that students could earn) is used to value the time cost of participation. However, because the assent and surveys will be administered during the school day, the real cost to students is the opportunity cost of filling out the survey instruments in lieu of formal classroom instruction.
Baseline youth survey. The baseline survey will use Teleform scannable questionnaires and will take 30 minutes on average to complete. An anticipated 1,596 youth will complete the survey, for a burden of 798 hours. Based on the figure of $7.25 per hour, the estimated annual cost to students for the baseline survey is $5,785.50.
Follow-up youth survey. The intervention group follow-up survey will use Teleform scannable questionnaires and will take 10 minutes on average to complete. After consulting with IOs, we estimate that 5% of youth who complete the baseline survey will not complete the follow-up survey (attrition). An anticipated 1,596 youth will complete the survey, for a burden of 798 hours. The estimated annual cost to intervention students for the follow-up outcome survey is $5,785.50.
Classroom roster report. An anticipated 14 schools will provide administrative data and rosters of youth in the school who are eligible for the SRA program. We estimate that producing administrative data and rosters will take an average of two hours per school for a total burden of 28 hours. Based on an adjusted hourly wage of $45.38 for school administrative staff, the total cost for the rosters would be $1,270.64.

For the process evaluation, data will be collected through youth and facilitator surveys, key informant interviews, a Session Log completed by program facilitators, and an attendance form.

Facilitator Session Logs. To obtain information about the fidelity with which the curriculum is administered, and to document the level of youth engagement, facilitators (employed by the IOs) will complete a Session Log for the sessions they facilitate. We anticipate that each of the 8 facilitators will complete approximately 75 Session Logs (assuming 120 classes of 20 students each at 5 sessions per class), and that each Session Log will take approximately 15 minutes to complete. The total burden will be 150 hours. Using the adjusted hourly wage for high school teachers of $60.12, the total cost will be $9,018.00.
Facilitator surveys. An anticipated 8 facilitators will complete a 25-minute survey about their SRA knowledge and attitudes and their experiences implementing the SRA program. The total burden is expected to be 3 hours. Using the adjusted hourly wage for high school teachers of $60.12, the total cost will be $180.36.
Process evaluation participant surveys. An anticipated 756 youth (half of the total follow-up survey youth – i.e., those in the intervention group) will complete a 10-minute survey about their experience with the SRA course and facilitator. The total burden is expected to be 126 hours. Based on the figure of $7.25 per hour, the estimated annual cost to students for the participant surveys is $913.50.
Key informant interviews. To obtain a more in-depth understanding of the implementation process, the contractor will conduct in-person key informant interviews with facilitators as well as representatives of the study sites. For each of the two IOs, two representatives of the sites and three program facilitators/staff will be interviewed, for a total of 10 key informant interviews across both IOs. Each interview is anticipated to last one hour, so the total burden is 10 hours. Using the adjusted hourly wage for high school teachers of $60.12, the total cost will be $601.20.
Attendance forms. We anticipate that 120 teachers will provide attendance data on 5 class sessions each, and that each attendance form will take 5 minutes to complete. Using the adjusted hourly wage for high school teachers of $60.12, the total burden is 50 hours, and the cost is $3,006.00.

Exhibit 10. Estimated Annualized Burden Hours

Respondents	Form name	No. of respondents	No. of responses per respondent	Average burden per response (in hours)	Total burden (hours)
Parents	Parental consent	1,680	1	5/60	140
High school students	Youth assent	1,680	1	5/60	140
	Baseline survey	1,596	1	30/60	798
	Follow-up survey	1,596	1	30/60	798
School or school district administrative staff	Classroom roster report	14	1	120/60	28
Program facilitators	Process evaluation facilitator Session Log	8	75	15/60	150
Program facilitators	Process evaluation Facilitator survey	8	1	25/60	3
High school students	Process evaluation participant survey	756	1	10/60	126
Program facilitators, site representatives	Key informant interviews	10	1	60/60	10
Teachers	Attendance form	120	5	5/60	50
Total burden					2243

Exhibit 11. Estimated Annualized Cost to Respondents

Respondents	Form name	Total burden (hours)	Adjusted hourly wage	Total respondent costs
Parents	Parental consent	140	$24.34	$3,407.60
High school students	Youth assent	140	$7.25	$1,015.00
	Baseline survey	798	$7.25	$5,785.50
	Follow-up survey	798	$7.25	$5,785.50
School or school district administrative staff	Classroom roster report	28	$45.38	$1,270.64
Program facilitators	Facilitator Session Log	150	$60.12	$9,018.00
Program facilitators	Facilitator survey	3	$60.12	$180.36
High school students	Process evaluation participant survey	126	$7.25	$913.50
Program facilitators, site representatives	Key informant interviews	10	$60.12	$601.20
Teachers	Attendance form	50	$60.12	$3,006.00
Total cost				$30,983.30

Respondents will include youth participants in each of the IOs’ SRA programs (9th- or 10th-grade students), their parent(s), and samples of the program facilitators and of representatives from participating sites (e.g., teachers, school principals, administrative staff). There will be no cost to participants other than their time.

Wage Estimates. To derive wage estimates, we used data from the U.S. Bureau of Labor Statistics’ (BLS) May 2017 National Occupational Employment and Wage Estimates. We have adjusted the school employee hourly wage estimates by a factor of 100 percent to reflect current HHS department-wide guidance on estimating the cost of fringe benefits and overhead (see Exhibit 12). For school employees, we use the adjusted hourly wages from one BLS occupation title, Education, Training, and Library Workers, All Other.

In addition, to calculate parent time costs, we have used wage estimates for all occupations, using the same BLS data discussed above. To calculate youth time costs, we have used the federal minimum wage. We have not adjusted these costs for fringe benefits and overhead because direct wage costs represent the “opportunity cost” to parents and youth for time spent on form completion.

Exhibit 12. Adjusted Hourly Wages Used in Burden Estimates

Occupation Title	Occupational Code	Mean Hourly Wage ($/hr.)	Fringe Benefits and Overhead ($/hr.)	Adjusted Hourly Wage ($/hr.)
Education, Training, and Library Workers, All Other	25-9099	$22.69	$22.69	$45.38
Secondary School Teachers	25-2030	$30.06*	$30.06	$60.12
All Occupations	00-0000	$24.34	N/A	$24.34
Not applicable (youth wage)	N/A	$7.25 (federal minimum wage)	N/A	$7.25

Source: “Occupational Employment and Wage Estimates May 2017,” U.S. Department of Labor, Bureau of Labor Statistics. http://www.bls.gov/oes/current/oes_nat.htm

* Wages for some occupations that do not generally work year-round, full time, are reported either as hourly wages or annual salaries depending on how they are typically paid. Annual mean wage for Secondary School Teachers is $62,730. Utilizing the Office of Personnel Management’s 2,087 annual standard work hours, mean hourly wage for Secondary School Teachers was calculated to be $30.06.

Estimate of Other Total Annual Cost Burden to Respondents or Recordkeepers and Estimate of Capital Costs

This information collection entails no respondent costs other than the cost associated with response time burden, and no non-labor costs for capital, startup or operation, maintenance, or purchased services.

Annualized Cost to Federal Government

The cost estimate for the design and conduct of the evaluation will be $2.850 million annually. This total cost covers all evaluation activities, including the estimated cost of coordination among OASH, the contractors, and the IOs; compensation to schools for school records data; project planning and schedule development; Institutional Review Board applications; study design; technical assistance to IOs; data collection; data analysis and reporting; and progress reporting.

Program Changes or Adjustments

This will be a new data collection.

Plans for Tabulation and Publication and Project Time Schedule

Data will be collected between May 2019 and January 2020. Data will be tabulated and presented to OASH in May 2020. See Exhibit 13 for the anticipated schedule.

Exhibit 13. Estimated Schedule for the Evaluation

Activity	Start date	End date
Research approvals from participating schools, as needed	December 2018	May 2019
Collection of parental consent	May 2019	September 2019
Youth data collection	September 2019	December 2019
Data analysis	October 2019	April 2020
Summary report	‒	May 2020

Display of Expiration Date for OMB Approval

The expiration date for OMB approval will be displayed on all data collection instruments.

Exceptions to Certification Statement.

There are no exceptions to the certification.

References

Borawski, E. A., Trapl, E. S., Lovegreen, L. D., Colabianchi, N., & Block, T. (2005). Effectiveness of abstinence-only intervention in middle school teens. American Journal of Health Behavior, 29, 423–434. http://dx.doi.org/10.5993/AJHB.29.5.5

Coyle, K. K., & Glassman, J. R. (2016). Exploring alternative outcome measures to improve pregnancy prevention programming in younger adolescents. American Journal of Public Health, 106(Suppl 1), S20–S22. https://doi.org/10.2105/AJPH.2016.303383

Donnelly, J., Horn, R. R., Young, M., & Ivanescu, A. E. (2016). The effects of the Yes You Can! curriculum on the sexual knowledge and intent of middle school students. Journal of School Health, 86(10), 759–765. https://doi.org/10.1111/josh.12429

Family and Youth Services Bureau. Making Adaptations Tip Sheet. https://www.acf.hhs.gov/sites/default/files/fysb/prep-making-adaptations-ts.pdf.

Francis, K., Woodford, M., Kelsey, M. (2015). Evaluation of the Teen Outreach Program in Hennepin County, MN: Findings from the Replication of an Evidence-Based Teen Pregnancy Prevention Program. https://www.hhs.gov/ash/oah/sites/default/files/ash/oah/oah-initiatives/evaluation/grantee-led-evaluation/reports/hennepin-final-report.pdf.

Jemmott III, J. B., Jemmott, L. S., & Fong, G. T. (2010). Efficacy of a theory-based abstinence-only intervention over 24 months: A randomized controlled trial with young adolescents. Archives of Pediatrics and Adolescent Medicine, 164(2), 152–159. https://doi.org/10.1001/archpediatrics.2009.267

Kirby, D. B. (2008). The impact of abstinence and comprehensive sex and STD/HIV education programs on adolescent sexual behavior. Sexuality Research and Social Policy, 5(3), 18–27. https://doi.org/10.1525/srsp.2008.5.3.18

Lieberman, L., & Su, H. (2012). Impact of the choosing the best program in communities committed to abstinence education. SAGE Open, 2(1), 215824401244293. https://doi.org/10.1177/2158244012442938

Markham, C. M., Tortolero, S. R., Peskin, M. F., Shegog, R., Thiel, M., Baumler, E. R., & Robin, L. (2012). Sexual risk avoidance and sexual risk reduction interventions for middle school youth: A randomized controlled trial. Journal of Adolescent Health, 50, 279–288. https://doi.org.10.1016/j.jadohealth.2011.07.010

Office of Adolescent Health, U.S. Department of Health and Human Services. (2017). Should Teen Pregnancy Prevention Studies Randomize Students or Schools? The Power Tradeoffs Between Contamination Bias and Clustering. Evaluation Technical Assistance Brief for OAH Teenage Pregnancy Prevention Grantees. https://www.hhs.gov/ash/oah/sites/default/files/contamination-bias-and-clustering-brief.pdf.

Office of the Assistant Secretary for Planning and Evaluation, Office of Human Services Policy, U.S. Department of Health and Human Services. (2015). Improving the Rigor of Quasi-Experimental Impact Evaluations. ASPE Research Brief. https://tppevidencereview.aspe.hhs.gov/pdfs/rb_TPP_QED.pdf.

Piotrowski, Z. H., & Hedeker, D. (2016). Evaluation of the Be the Exception sixth-grade program in rural communities to delay the onset of sexual behavior. American Journal of Public Health, 106(S1), S132–S139. https://doi.org/10.2105/ajph.2016.303438

Realini, J. P., Buzi, R. S., Smith, P. B., & Martinez, M. (2010). Evaluation of “Big Decisions”: An abstinence-plus sexuality curriculum. Journal of Sex and Marital Therapy, 36(4), 313–326. https://doi.org/10.1080/0092623x.2010.488113

Schochet, P. Z. (2008). Statistical Power for Random Assignment Evaluations of Education Programs. Journal of Educational and Behavioral Statistics, 33(1), 62-87. https://journals.sagepub.com/doi/abs/10.3102/1076998607302714.

Schochet, P. Z. (2005). Statistical Power for Random Assignment Evaluations of Education Programs. https://cire.mathematica-mpr.com/~/media/publications/pdfs/statisticalpower.pdf.

Weed S. E., Ericksen, I. E., Lewis, A., Grant, G. E., & Wibberly, K. H. (2008). An abstinence program’s impact on cognitive mediators and sexual initiation. American Journal of Health Behavior, 32(1), 60–73. https://doi.org/10.5993/ajhb.32.1.6

What Works Clearinghouse. (2018). What Works Clearinghouse Standards Handbook, Version 4.0. Washington DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education.

1 The original SMARTool was developed in 2010 by the Center for Relationship Education through a cooperative agreement with the Centers for Disease Control and Prevention, Division of Adolescent School Health: 1-U58-DP000452.

2 The SMARTool defines SRA education as follows: “Sexual risk avoidance education is defined in legislation and while there is some variance in the language, there are several core elements: (1) SRA education must focus on the benefits of voluntarily refraining from nonmarital sexual activity. Within that context SRA education must, (2) present medically-accurate and age-appropriate information; (3) implement an evidence-based approach; (4) provide skills on healthy relationship development; (5) help students develop personal responsibility, self-regulation, and healthy decision making; (6) include teaching on other health risk behaviors such as substance use; and (7) promote parent involvement and communication.” (p. 6).

3 The updated SMARTool is available on CRE’s website: https://www.myrelationshipcenter.org/getmedia/dbed93af-9424-4009-8f1f-8495b4aba8b4/SMARTool-Curricula.pdf.aspx

4 Note that this study is not comparing the SMARTool-aligned SRA curricula vs. curricula that are not aligned with the SMARTool. The comparison of different SRA curricula could potentially be part of a broader research agenda for OASH.

5 https://www.acf.hhs.gov/sites/default/files/fysb/prep-making-adaptations-ts.pdf

6https://www.hhs.gov/ash/oah/sites/default/files/contamination-bias-and-clustering-brief.pdf

7 https://cire.mathematica-mpr.com/~/media/publications/pdfs/statisticalpower.pdf https://journals.sagepub.com/doi/abs/10.3102/1076998607302714.

8 See for example: https://www.hhs.gov/ash/oah/sites/default/files/ash/oah/oah-initiatives/evaluation/grantee-led-evaluation/reports/hennepin-final-report.pdf

9 What Works Clearinghouse. (2018). What Works Clearinghouse Standards Handbook, Version 4.0. Washington DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Bernstein, Shampa
File Modified	0000-00-00
File Created	2021-01-15