Supporting Justification for the OMB New Request for Clearance of the “Cohort 3 Teen Pregnancy Prevention Performance Measure Data Collection”
Part A: Justification for the Collection of the Data
Submitted December 2018
Revised July 2019
Submitted by:
Office of Adolescent Health
U.S. Department of Health and Human Services
1101 Wootton Parkway, Suite 700
Rockville, MD 20852
Project Officer: Tara Rice
A1. Need and Legal Basis
OAH supports two types of grants through the TPP program: (1) Tier 1 projects that replicate programs, and (2) Tier 2 projects that develop and test innovative models and strategies to prevent teen pregnancy and promote healthy adolescence. Funding for the TPP Program is authorized under the Continuing Resolution for FY 2018 (Public Law 115-141). The Act provides $100 million in FY 2018 for making competitive awards to public and private entities to fund programs that reduce teen pregnancy. In FY 2018, OAH released new funding opportunity announcements (FOAs) for TPP Tier 1 and Tier 2; phase 1 awards for Tier 2 grants were made in September 2018 to 14 organizations for a two-year period of performance
The collection of performance measures from its grantees is critical to OAH because it provides the agency with data to both effectively manage the TPP program, and to comply with accountability and federal performance requirements for the 1993 Government Performance and Results Act (P.L. 103-62). Moreover, collecting and reporting on data for performance measures are funding requirements for the grants, as stated in the funding opportunity announcement. Finally, data helps grantees to make continuous quality improvements in programming,
This new ICR request only pertains to the cohort 3 TPP grants, and is a separate request from the ICR for cohort 2 TPP grantees (OMB # 0990-0438, expiration date 10/31/2021).
The performance measures to be reported to OAH by the 3rd TPP cohort are summarized in Exhibit 1. The specific questions that grantees will answer to address these measures are found in the information collection (IC) forms.
Exhibit 1: Performance Measures for the TPP Cohort 3 FY 2018-2020
Performance Measure Constructs |
Data source |
Grant Recipient-Level Measures |
|
General Program Information
|
Grantee/Sub-awardee Administrative Records |
Partnerships: Implementation Sites
|
Grantee/Sub-awardee Administrative Records |
Staffing & Training
|
Grantee/Sub-awardee Administrative Records |
Program-Model Level Measures |
|
Participant Reach (Output)
|
Grantee/Sub-awardee Administrative Records |
Quality and Fidelity
|
Derived from OAH Observation Forms, Program-model Fidelity Logs, and OAH Fidelity Process Form |
Dosage
|
Grantee/Sub-awardee Attendance Records |
|
|
Dissemination**
# presentations (national, state, local)** |
Grantee/Sub-awardee Attendance Records |
‡New measure for the FY2018 cohort
* SMARTool or Tool to Assess Characteristics of Effective HIV/STI programs
** Proposed for Phase 2 only (that is, FY2020-2021)
A.2 Purpose and Use of Information
The performance measures data collection provides information to OAH leadership and project officers to help them to more effectively manage the cohort 3 TPP grants. We anticipate that the data will be made available to Congress by the Office of Management and Budget (reach, dosage, training and partners), and the public at large (all measures) to assess program performance. Use of these data is vital for ensuring on-going improvement of the TPP program and through dissemination efforts, broader understanding and support of programs designed to prevent teen pregnancy.
Performance measures have been a requirement of the TPP grant program since its inception in 2010. Several of the measures that are included in this information collection request previously received OMB clearance (0990-0392 and 0990-0438) and have been used over the past 8 years with the initial 2 cohorts of TPP grantees. Additional measures specific to the goals of the FY2018 TPP FOAs have been added to this collection request.
A.3. Use of Information Technology and Burden Reduction
Grantees will enter performance measure data into a multi-use, Web-based reporting system (similar to the system used by the previous TPP grant cohort), either by using point and click entry or by uploading spreadsheets using a template. The Web-based system can reduce burden for respondents by programming in skip patterns, so that grantees only have to look at questions or uploading data that are relevant for them. Programming automatically performs necessary calculations for respondents, and will validate responses. For point and click entry, a branching mode of presentation allows respondents to go directly to the sections they need, without having to go through the system in a linear progression. The system also automatically produces a data set of measures across all grantees using relevant filters (such as for a specific grant Tier), which saves time on preparation of the data for analysis. Data are also available for grantees to export into Excel files to customize reports. Screenshots of the web-based reporting system used in cohort 2 are included in Appendix E; it is expected that cohort 3 grantees would use a similarly designed system.
A.4. Efforts to Identify Duplication and Use of Similar Information
The OAH cohort 3 FY2018 TPP performance measures data collection is the only data collection that will provide information on the performance of the cohort 3 FY 2018 TPP grant program. The data collection will make use of existing data to the extent possible. Most of the performance measures would be collected by grantees as part of their routine administrative records (e.g., numbers of participants, and attendance at program sessions). Please see Exhibit 2 for data sources for each performance measure and see the Appendix for the complete list of performance measures).
The current request is a new information collection request specific to the cohort 3 TPP grant recipients that receive a new grant award beginning in September 2018.
A.5. Impact on Small Businesses
No small businesses will be involved in the collection of data in this study, other than any that are funded as grant recipients. The impact on small businesses should be minimal. The information being requested or required has been held to the absolute minimum required for the intended use of the data.
A.6. Consequences of Not Collecting the Information/Collecting Less Frequently
GPRA requires that government agencies report on their performance measures annually. Therefore, it is essential that grantees report on these performance measures at least annually to OAH. In addition, collection and reporting of performance measure data is a requirement of all TPP grantees as stated in the Funding Opportunity Announcement. Collecting the data biannually helps grantees to use data for continuous quality improvement and also informs OAH’s program planning and technical assistance for the grant program.
A.7. Special Circumstances
There are no special circumstances that occur when collecting this information.
A.8. Federal Register Comments and Persons Consulted Outside the Office of Adolescent Health
A 60-day notice was published in the Federal Register Vol. 83, No. 191, pages 49560-1, on Tuesday, October 2, 2018, and provided a 60-day period for public comments. A single public comment was received.
Dear Ms. Funn,
I appreciate the opportunity to comment on the proposed information collection Cohort 3 Teen Pregnancy Prevention (TPP) Program Performance Measures (0990-New-60D). I am concerned that the proposed information collection excludes important items, and that these omissions compromise the utility of the information collection. I urge the Office of Adolescent Health (OAH) to modify its proposed information collection to improve the quality and utility of the data it will generate.
Specifically, I recommend that OAH:
1) Modify the Pre-Post Survey to include questions assessing knowledge of the full range of contraceptive methods, and ensure that the TPP program as a whole retains a focus on measuring the outcomes and behaviors described in the 2015-2016 HHS Teen Pregnancy Prevention Evidence Review: pregnancy, vaginal intercourse without a condom, or sexual activity without contraception.
2) Restore the question asking grantees to report on the number of referrals made by program staff to youth-friendly off-site providers or school-based health centers for reproductive health care, mental health services, primary health care, vocational education/workforce development, intimate partner violence prevention, or healthy relationship training. This question appears in the performance measures for FY 2015-2019 TPP grantees.2
3) Restore the gender categories in the demographics section to “Male; Female; Transgender; Do not identify as male, female or transgender; Unknown/unreported.”
Pre-Post Survey
The “Communicating Risk” section of the Pre-Post Survey includes questions that address the benefits of condoms (“Using condoms helps prevent STDs” and “Using condoms helps prevent pregnancy”), but none addressing the benefits of other contraceptive methods for pregnancy prevention. OAH should add questions that assess participants’ knowledge of the full range of contraceptive methods.
Under the 2010 Consolidated Appropriations Act, TPP programs must be “proven effective through rigorous evaluation to reduce teenage pregnancy, behavioral risk factors underlying teenage pregnancy, or other associated risk factors.” For the TPP Evidence Review, Mathematica Policy Research has applied criteria that “require programs to show evidence of at least one favorable, statistically significant impact on at least one sexual risk behavior or reproductive health outcome of interest (sexual activity, number of sexual partners, contraceptive use, STIs, or pregnancy).” In order to effectively assess grantee performance, TPP performance measures should address these behaviors
or outcomes. The proposed Pre-Post Survey does not ask program participants about any of these behaviors or outcomes; rather, it contains questions on knowledge, beliefs, and predicted future actions. OAH should ensure that the TPP program as a whole continues to involve ongoing, rigorous evaluation that focuses on behaviors and outcomes.
Referrals to Youth-Friendly Services
Given that research suggests youth-friendly family planning services may have positive effects on reproductive health outcomes, asking grantees to report on referrals to youth-friendly providers remains a worthwhile metric. A recent literature review from authors at CDC and the Office of Population Affairs called for additional research into outcomes associated with youth-friendly services and concluded: “[Y]oung people continue to value confidentiality, supportive provider interaction, specialized provider training, and the removal of logistic barriers to access family planning services. Further, [the review]
sheds new light on the importance young people put on receiving complete and unbiased
contraceptive counseling from providers, with supportive tools and a client-centered approach, as a critical part of the clinic visit. Youth-serving professionals and researchers should consider these factors when developing, implementing, and evaluating reproductive health services for young people.”
OAH should restore the question about referrals that was asked of previous grantees.
Participants’ Gender Identities
Given high rates of HIV infection among transgender individuals,7 it is important for TPP grantees to assess how well they are serving the needs of transgender adolescents. Grantees may wish to base the questions administered to participating students on recommendations from the Gender Identity in U.S. Surveillance (GenIUSS) group, which notes that questions administered to adolescents should define
terminology related to sex and gender. In its information collection, OAH should ask grantees to report the number of transgender participants. The proposed information collection gives only four categories: Male, Female, Does not identify, Did not report. This is in contrast to the previous performance measures, which reported on participants in five categories: Male, Female, Transgender, Do not identify as male, female or transgender, Unknown/unreported. OAH should restore the Transgender response option for this question.
I thank OAH for the opportunity to comment on this proposed information collection. If you have any questions, please contact me at borkowsk@gwu.edu or 202-994-0034.
Sincerely,
Liz Borkowski
Liz Borkowski, MPH
Managing Director
Jacobs Institute of Women’s Health
Milken Institute School of Public Health
George Washington University
950 New Hampshire Avenue NW, 6th Floor
Washington, DC 20052
OAH reviewed the letter and supporting documentation; no changes to the main grantee-level measures ICR form was made. The response to this letter was to note that the cohort 3 performance measures reflect HHS departmental priorities, as described in the FY 2018 funding opportunity announcements (FOAs). The cohort 3 performance measures collection indicates the data that the FY2018 grantees would be required to report to the federal government. FY2018 TPP grantees may collect additional data beyond the performance measures as part of their individual program evaluations. Additionally, after consultation with OMB, the pre-post test assessment was dropped due to mutual concerns about privacy and low sample size.
In 2010 and 2011, OAH consulted with staff of RTI International, which was the contractor responsible for assisting OAH in developing the performance measures and performance measure reporting system, and a panel consisting of experts in the fields of performance measurement, teen pregnancy prevention, and evidence-based practice. In addition, OAH presented information on the performance measures to TPP grantees and their evaluators at two conferences, and solicited their input. OAH also consulted and received feedback from other Federal staff working in the area of teen pregnancy prevention from ASPE, ACF, and CDC. The bulk of the performance measures that are part of this clearance package have been successfully used during the past 4 years in the data collection for the “Office of Adolescent Health and Administration for Children, Youth and Families Teen Pregnancy Prevention Performance Measure Collection.”
A list of individuals in the expert panel who provided input regarding into the initial measures is found in Exhibit 2.
Exhibit 2. Persons Consulted Outside the Agency for Performance Measure Collection for FY 2010 Grantees |
||||||||||||||||||||
Expert Work Group |
||||||||||||||||||||
|
|
|||||||||||||||||||
|
|
|||||||||||||||||||
|
|
|||||||||||||||||||
|
|
|||||||||||||||||||
A.9. Payments to Respondents
There are no payments to staff of grantee organizations completing the performance measure reporting form.
A.10. Assurance of Confidentiality
All data on TPP program participants will be kept private to the extent allowed by law. All participant data are reported de-identified and aggregated to the section (group or class) level. No personal identifiers will be used in the reporting of any program or participant data.
The Web-based reporting system is designed to ensure the security of the data obtained. Electronic data are stored in a location within the contractor’s network that provides the appropriate level of security based on the sensitivity or identifiability of the data.
Individual users designated by the grantees will be assigned user names and passwords that will grant them access to the project website. There, users will have the opportunity to provide data that will be stored in a secure Microsoft SQL Server database utilizing a relational table structure, facilitating expedient data retrieval and analysis. The database server, located at the contractor’s site, will be accessible only to the statisticians and analysts assigned to this project. Electronic communications will occur via a secure Internet connection. All transmissions will be encrypted with 128-bit encryption through secure socket layers (SSL) and verified by a VeriSign®, the leading SSL Certificate authority.
A.11. Sensitive Questions
TPP program participants will be asked about their race and ethnicity; these items are collected routinely on HHS data collection instruments and provide important information about the reach of the TPP program.
For all items, grantees will inform their individual respondents that their participation is voluntary and that they may refuse to answer any or all of the questions. All of the performance measures data will be reported to OAH in the aggregate, de-identified, however, and there will be no means to identify responses by individuals.
A.12 Burden Estimate (Total Hours & Wages)
A.12A Estimated Annualized Burden Hours
Calculation of the total estimated annualized burden hours is shown in Exhibit 3. Data are reported twice a year; the average burden per response in the table is for each reporting period.
Exhibit 3. Estimated Annualized Burden Hours
Type of Respondent |
No. of Respondents |
No. Responses per Respondent |
Average Burden per Response (in hours) |
Total Burden Hours |
Grantee program staff, performance measures form |
19 |
2 |
7 |
266 |
Total |
19 |
2 |
7 |
266 |
Average burden hours for grantees or their designates
All of the data will be reported by the grantees or their designated reporters twice a year. Grantees will collect most of these data for their own administrative purposes and are expected have their own systems in place to track the data. Thus, the only additional burden to grantees for reporting the grantee-organization performance measures is the time it takes them to assemble the necessary data and enter the data into the reporting systems. Respondents can either upload spreadsheets into the web-based system or enter the data directly into the system using a point and click method. For Tier 1 and Tier 2 grant recipients, grantees (or their designated reporter, such as a sub-awardee or an independent evaluator) will be responsible for reporting all data for each grant organization. Exhibit 4 provides an estimated number of respondents for each grant type.
Exhibit 4. Estimated Total Respondents for cohort 3 FY 2018 Phase 1 TPP Grants1
TPP Program Funding Stream |
Estimated # Grant Awards |
Estimated # of Respondents |
Tier 1 |
14 |
14 |
Tier 2 |
14 |
14 |
TPP Grantee Recipient Performance Measures Form
The measures on the TPP grantee form are expected to take a total of 7 hours per response to report to OAH.
General Program Information: Grantees will report on measures of general program information, such as the type of program, the name of the program being replicated, the name of all curricula being used, the setting of implementation. Some items, such as name of the program model and type of program, are descriptive metrics that would not change and thus would only need to be entered once. We estimate that this information will take approximately 0.25 hours to summarize and report each reporting period, for a total of 0.5 hours per year.
Partnerships. Grantees will report on measures of number and retention of partners. We estimate that it will take each respondent approximately 0.25 hour to summarize and report these data each reporting period, for a total of 0.5 hours per year.
Facilitators Training & Retention. Grantees will report on the number of facilitators who receive initial and follow up training. We estimate that it will take each respondent approximately 0.25 hour to summarize and report these data each reporting period, for a total of 0.5 hours per year.
Dissemination. Grantees will report on measures of dissemination such as manuscripts published and papers presented. We estimate that this will take each respondent approximately 0.25 hour to summarize and report these data each reporting period, for a total of 0.5 hours per year. The dissemination metric would be collected and reported beginning in phase 2.
Participants’ Reach. These data indicate the number of participants, by different background factors that the program is reaching (defined as participating in at least one program activity). We anticipate that it will take each respondent 2.8 hours per reporting period, or a total of 5.6 hours year to report these data.
Dosage. Grantees will collect attendance data on participants as part of their program management. The attendance data that are reported will be used to determine the percentage of core program components participants receive. We estimate that it will take respondents 1.7 hours to summarize and report these data per reporting period, for a total of 3.4 hours per year.
Fidelity and Quality. All cohort 3 grant recipients will collect several types of data related to fidelity as part of their ongoing administration of their programs. Grantees will report measures of adherence to the program model and overall quality of implementation, based on observations of up to 10% of the sessions implemented using the OAH Observation Form (See Appendix). Measures of adherence to the program model will be based on self-assessment fidelity log forms completed by session facilitators. Grantees will report a measure of adherence based on the number of sessions implemented and planned. Grantees will report the total for the fidelity process form (Supplemental Documents) that tracks the extent to which grantees have implemented the necessary processes to ensure fidelity to selected programs as well as adherence to the public health priorities listed within the FOA. The collection of these fidelity data was a requirement stated to grantees in the funding opportunity announcement to which they responded, thus, only the actual reporting of these data to OAH is included in the burden calculation. We estimate that it will take respondents approximately 1.5 hours to summarize and report these data per reporting period, for a total of 3 hours each year.
The estimated 1-year annualized cost to respondent is shown in Exhibit 5. Salaries of the grantee staff collecting data, entering data, and summarizing and reporting data will vary widely. We estimate an average hourly rate of $30 for grantee staff respondents.
Exhibit 5. Estimated 1-Year Annualized Cost to Respondents
Forms
|
Type of Respondent |
Estimated Number of Respondents |
Total Burden Hours |
Hourly Wage Rate |
Total Respondent Costs |
Grantee-level Responses : |
Grantee staff |
19 |
266
|
$30.00 |
$ 7980 |
|
|
|
|
|
|
|
|
|
|
|
|
Total |
|
19 |
266 |
|
$ 7980 |
A.13 Capital Costs (Maintenance of Capital Costs)
There are no capital costs associated with this study.
A.14 Cost to Federal Government
The cost to the federal government is estimated at $600,000 over 3 years. This cost was estimated by using the previous contract and includes the estimated cost of coordination with OAH, the contractor’s IRB and OMB applications, revision of the data reporting system for the new cohort of grantees, on-going maintenance of the data reporting system, training and technical assistance to the grantees and OAH staff in the use of the data reporting system, and data analysis and reporting. Annual cost to the government is $200,000 per year.
A.15 Program or Burden Changes
There is no change in burden requested, as this is a new information collection.
16. Tabulation of Data and Schedule
The TPP grantees funded in FY 2018 will first report data in the fall 2019. The cohort will continue to collect and report data twice each year until the fall 2020 at the semi-annual required reporting periods (April 30 and October 30). In 2020, if funds are available, 2 year, phase 2 grants would be awarded to successful phase 1 recipients; performance measures would continue to be collected twice per year during phase 2. OAH or its designated contractor will then analyze the data and prepare a written report, summarizing findings and data trends, on an annual basis. Grantees funded in FY19 would also report data twice annually. Data will be summarized overall and/or broken down by tier (Tier 1 and Tier 2).
A.17. Display of Expiration Date for OMB Approval
The expiration date for OMB will be displayed on all data collection instruments.
A.18. Exceptions to Certification Statement.
There are no exceptions to the certification statement.
1 The annualized burden estimate includes 14 Tier 2 grants awarded in FY2018, as well as an estimated 14 Tier 1 phase 1 grants that potentially may be awarded in FY2019, subject to availability of funding. Thus, the annualized (across the 3 years of the ICR) number of respondents is 19 in the burden tables. (14 grants funded in FY18, estimated 28 grants in FY19, and estimated 14 in FY20). OAH will update these tables accordingly.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement for |
| Author | wallace |
| File Modified | 0000-00-00 |
| File Created | 2021-01-16 |