SSA - Pretesting Generic IDEAS

Alternative Supporting Statement Instructions for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Youth Empowerment Information, Data Collection, and Exploration on Avoidance of Sex (IDEAS)—Pre-Test

OMB Information Collection Request

0970 - 0355

Supporting Statement

Part A

June 2019

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Federal Project Officer:

Caryn Blitz

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a Generic Information Collection under the OPRE’s Generic Clearance for Pre-testing of Evaluation Data Collection Activities (0970-0355).

• Description of Request: ACF has designed the Youth Empowerment Information, Data Collection, and Exploration on Avoidance of Sex (IDEAS) project as a national study to identify effective messages and approaches to empower youth to make healthy and informed decisions related to sexual risk avoidance, teen pregnancy prevention, and their well-being.

This generic ICR supports pre-testing of new survey items and item adaptations in Parent, Youth and Young Adult survey instruments. The focus of the pretest is on item validity and reliability across demographically diverse populations and several geographic locations. Pre-test data are not intended to produce generalizable statistical information.

A1. Necessity for Collection

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services seeks approval for a pre-test of the Parent and Youth survey instruments for the Youth Empowerment Information, Data Collection, and Exploration on Avoidance of Sex (IDEAS) project. This proposed information collection meets the primary goals of ACF’s generic clearance for pre-testing (0970-0355): to develop and test information collection instruments and procedures.

The Youth Empowerment IDEAS project is a planned descriptive study designed to identify strategies and approaches that resonate with youth and empower them to make healthy and informed decisions related to sexual risk avoidance, teen pregnancy prevention, and their well-being. Data for the project will be collected through three survey instruments, a Youth Survey (ages 14 to 18), a Young Adult Survey (ages 19 to 24), and a Parent Survey of parents of youth ages 14 to 18. ACF will use the pre-test data from this generic information collection to refine the surveys prior to submitting final instruments to the Office of Management and Budget (OMB) as a new full information collection request for the Youth Empowerment IDEAS project.

Study Background

The consequences of adolescent sexual activity remain a critical social and economic issue in the United States, shaping the lives of thousands of teens and their families every year. Despite its decline over the past 25 years, the teen birthrate in the United States remains higher than in other industrialized countries, and varies widely across geographic regions and racial and ethnic groups.¹ Further, adolescents and young adults account for half of all new cases of sexually transmitted infections each year.² Sexual activity in youth is also associated with other risk behaviors, such as alcohol and substance use. The IDEA project’s Parent, Youth, and Young Adult survey instruments are designed to identify appropriate strategies, skills, messages, and themes likely to resonate with youth for use in designing educational programs and policies to address these risk behaviors.

Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

The Youth Empowerment IDEAS project surveys will collect national-level data about youth and parent attitudes, opinions, knowledge, and experiences with communication between youth and parents; attitudes and beliefs about life-event sequencing; sex; and sex education. Prior to the full national study, we propose pre-testing these survey instruments with a diverse group of youth, young adults, and parents of youth to ensure that survey items are comprehensible and carry the same meaning across a range of participants.

When designing the IDEAS project surveys, the study team reviewed survey items from 12 existing national surveys and 7 previously used research and evaluation instruments (Attachment A). Where possible, the team drew survey items from these existing instruments, but the review identified that several relevant constructs for the study were not previously measured. Following item review by subject matter experts and ACF leaders, the study team also modified several previously used items to better suit the goals of the IDEAS project. Modified items can no longer be considered validated and must be treated as new survey items.

The primary goal of the pre-test is to ensure final questionnaire items are appropriate for their intended population (youth, young adults, and parents); are clearly written; provide sufficient response options; and are not culturally biased. Interviews with test respondents will offer a clearer understanding of participants’ comprehension, recall, judgment, and response strategies for selected items. This information will be used to improve item wording and format. ACF will incorporate findings from the pre-test into instruments submitted for a full OMB information collection request for the Youth Empowerment IDEAS project.

Research Questions or Tests

ACF proposes to examine the following research questions for this pre-test effort:

1. Are the questions on the survey instruments appropriate for their intended populations?

2. Are the questions and response options clearly written, easy to understand, and free of bias?

3. If questions are misunderstood, are there ways to improve the question wording so respondents better understand what is being asked?

4. Are the questions on the survey instruments reliably measuring the intended constructs?

Study Design

Each phase of pre-testing will consist of survey administration and a small group debrief session to collect information about respondents’ reaction to the survey instrument. We propose an iterative testing process, of up to three phases:

(1) We will administer the entire survey, and the debrief will focus on the new or newly modified items.

(2) We will administer questions that were revised based on the first phase.

(3) Due to the large number of survey items that were new or newly modified, it may be necessary to conduct a third phase of cognitive pre-testing to ensure that all survey items are reliable and appropriate for the intended populations. If necessary, the third round would focus on revisions, from the second pre-testing phase.

See Supporting Statement B for additional information about the target population and the procedures for collection of data.

Study Instruments and Protocol

The pre-test consists of four instruments, based upon respondent type and age:

• Parent Pre-Test Protocol: Survey and Debrief Topic Guide (Instrument 1)

This instrument will be tested with parents of youth aged 14-18.

The IDEAS project parent survey collects information about demographics of the parent and focal child; the relationship and communication with the focal child; the child’s exposure to sexual education and parental beliefs about sexual education; parental attitudes and beliefs about youth sexual behavior; and parent’s knowledge of youth sexual risk behaviors, including the focal child’s behaviors.

The Parent Pre-Test session will take up to two hours. The first 45 minutes will consist of a group introduction to the pre-testing/discussion task; conducting the consent procedures; and time for parents to complete the survey instrument. The parent instrument is designed to take 20 minutes to complete, but we allow more time for parents to respond during the pre-test, including writing notes on items that are difficult to understand or raise concerns for discussion. From previous experience pre-testing surveys, we have found that allowing additional time for completion improves the quality of resulting information and enables more productive group discussion. The remaining 75 minutes of the session will be a structured small group discussion, following the pre-test protocol and focusing on new items and previously used items modified for this study.

• Youth Pre-test Protocol: Survey and Debrief Topic Guides (Instruments 2 and 3)

These instruments will be pre-tested with youth aged 14-18. Given the instrument length, sensitive topics, and cognitive demands of the debrief, youth respondents will receive either Part 1 or Part 2 of the survey for pre-testing.

Questions in the Youth Survey Part 1 focus on youth demographics; relationship and communication with parents; aspirations about education and romantic relationships; and attitudes and beliefs about youth sexual behavior.

Questions in the Youth Survey Part 2 focus on youth’s knowledge of prevalence of risk behaviors and risk prevention; exposure to and beliefs about sexual education; and sexual risk behaviors.

Each Youth Survey pre-test session will take 90 minutes to complete. The first step will be obtaining parental consent for youth participation. After consent is obtained, youth will be taken to a separate room from parents and the next 30 minutes will consist of introduction, assent, and the time for the youth to complete the survey instrument. The remaining 60 minutes will be used for the structured small group discussion, following the pre-test protocol and focusing on new and previously-used items that were modified for the IDEAS project.

• Young Adult Pretest Protocol: Survey and Debrief Topic Guide (Instrument 4)

This instrument will be pre-tested with young adults aged 19-24.

Questions in the Young Adult Survey are similar to the Youth Survey Part 1 and 2, with wording changes and some additional questions more appropriate for this older age group.

The Young Adult Pre-Test will take up to two hours to complete. The first 45 minutes will consist of the introduction, consent, and the time for the respondent to complete the survey instrument. The remaining 75 minutes will be used for the structured small group discussion, following the pre-test protocol and focusing on previously used items that were modified as well as new items.

A3. Use of Information Technology to Reduce Burden

The surveys for the final Youth Empowerment IDEAS national data collection will ultimately be programmed as web surveys. At this stage of the design, questions will likely change based on pre-test feedback, so it is inefficient to program the pre-test survey instrument into a web survey system. Respondents will complete the survey on hardcopy so they can easily mark items that were difficult to comprehend. Qualitative information from the small group discussions will be collected via written notes onto laptops and will be audio recorded.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

ACF has carefully reviewed the information collection requirements to avoid duplication with existing studies or other ongoing federal teen pregnancy prevention evaluations. As mentioned above, when designing the survey instruments, the study team reviewed 12 national surveys and 7 previously used research and evaluation instruments. Although a number of survey questions from these sources were added to the current instruments, in order to address all the research questions of the Youth Empowerment IDEAS project, the study team modified many previously used items and designed new items. As noted, the pre-test is focused on modified and new items.

A5. Impact on Small Businesses

This data collection has no impact on Small Businesses.

A6. Consequences of Less Frequent Collection

The purpose of this data collection is to pre-test survey instruments for use in a planned national survey. Pre-testing in several geographic regions is necessary to ensure questions are appropriate for respondents across the country. A smaller pre-test of fewer than 10 participants per instrument is not sufficient given the length of the instrument; the sensitive questions; the number of new or modified questions; and the diversity of the expected set of national study participants.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for Pre-testing of Evaluation Data Collection Activities (0970-0355). This notice was published on October 20, 2017, Volume 82, Number 202, page 48820, and provided a sixty-day period for public comment. During the notice and comment period, no substantive comments were received.

Consultation with Experts Outside of the Study

To develop the Parent and Youth Survey, ACF consulted with national subject matter experts in adolescent health, sexual education, and instrument development. Table A8 lists the experts consulted and their affiliation.

Table A8. Experts Consulted in Development of IDEAS Parent and Youth Surveys

Name	Affiliation
Brooke Hempell	Senior Vice President of Research, Barna Research
John Vessey	Associate Professor of Psychology, Wheaton College
Lisa Rue	Senior Advisor and Strategic Partnerships at Cliexa
Peter Scales	Senior Fellow, Search Institute
Scott Gardner	Professor, Brigham Young University – Idaho
Tim Rarick	Professor, Brigham Young University – Idaho
Wade Horn	Director, Deloitte Consulting LLP’s Public Sector Practice; Former Assistant Secretary, ACF

A9. Tokens of Appreciation

To recruit and conduct the pre-testing activities efficiently in three geographic regions of the United States, our contractor, Mathematica, will collaborate with three small organizations that maintain large and diverse pools of research participants. This approach simultaneously improves the diversity of participants and reduces costs to the federal government by using on-the-ground facilities with connections to the community. It is standard practice for these research organizations to provide tokens of appreciation to their panel participants for attending study sessions. The amounts vary by organization; age of participant; and the length of time the participant is expected to attend a pre-test session.

We expect that these organizations will pay tokens of appreciation directly to the participants, with amounts of $25 for youth and $50 for young adults and parents. Tokens of appreciation are intended to offset the costs of participation in the data collection, including travel to and from the research facility for scheduled group meetings and arranging child care, if necessary. This amount is consistent with similar pre-testing activities approved and conducted under ACF’s generic pre-testing clearance process. For example, the Pretest of the Annual Survey of Refugees offered participants a $40 gift card for a 1 hour survey and 1 hour debrief. Similarly, youth participating in a 30 minute pre-test for the Pre-testing of Evaluation Surveys: An Examination of the Intersection of Domestic Human Trafficking with Child Welfare and Runaway and Homeless Youth Programs received a $25 gift card.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

Respondent name will be recorded on consent and/or assent forms. Demographic information, such as age, race, and ethnicity will be collected from the recruiting organizations. Respondents will be instructed to not record their name on survey instruments. This data collection is not subject to the Privacy Act, as information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.

Assurances of Privacy

Information provided during this data collection will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data; that their participation is voluntary; and that their information will be kept private to the extent permitted by law. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.

Data Security and Monitoring

As specified in the contract, the contractor will protect respondents’ privacy to the extent permitted by law and comply with all federal and departmental regulations for private information. The contractor has developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ personally identifiable information. The contractor will ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor who perform work under this contract or subcontract are trained on data privacy issues and comply with the above requirements. All study team staff involved in the project will receive training on (1) limitations of disclosure; (2) safeguarding the physical work environment; and (3) storing, transmitting, and destroying data securely. All Mathematica staff sign the Mathematica Confidentiality Agreement, complete online security awareness training when they are hired, and receive annual refresher training thereafter.

As specified in the evaluator’s contract, the contractor will use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The contractor will securely generate and manage encryption keys to prevent unauthorized decryption of information in accordance with the Federal Processing Standard. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology requirements and other applicable federal and departmental regulations. In addition, the contractor has a plan for minimizing the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access.  

A11. Sensitive Information ³

The goal of the IDEAS project is to identify age-appropriate strategies, skills, messages, and themes likely to resonate with youth on topics of sexual risk avoidance and teen pregnancy prevention education. Questions pertaining to sexual attitudes and the behaviors of youth and sexual attitudes of their parents can be very sensitive in nature; but the questions are necessary to meet the main study objectives. Table A11 indicates the survey topics and questions deemed sensitive on the Parent and Youth surveys. IRB review of the pre-test protocol, consent, and assent forms is currently in process.

At the start of the survey, all pre-test participants will be informed of the sensitive nature of the questions and that they can refuse to answer any questions they feel uncomfortable answering. For youth younger than 18 years old, notifications will be provided during the parental assent process. Additionally, participants will be informed that the goal of the pre-test is about their understanding of the questions, not to capture their specific answers. As mentioned in Section A10, respondents will be instructed to not record their name or any other identifying information on the hard copy surveys instruments, which will prevent sensitive information in the survey from being linked to individual respondents.

Table A11. Summary of Sensitive Questions to be Included on the Parent and Youth Surveys and Their Justification

Topic and Survey Questions	Justification
Gender identity and sexual attraction Youth & Young Adult Survey Part 1 A5, A6, A7	ACF has a strong interest in improving programming that serves lesbian, gay, bisexual, transgender, and questioning youth and is sensitive and inclusive of all youth. To better understand these subgroups, questions on gender identity and sexual attraction are included in the survey.
Religion Youth & Young Adult Survey Part 1 A12a, A12b; Parent Survey H20a, H20b	Research has shown that religion or religiosity has a protective effect on youth sexual behaviors.4 Understanding how religious belief might moderate respondents’ behaviors and attitudes will be important when interpreting survey findings and translating them to actionable steps in improving programming.
Attitudes and beliefs related to sexual activity Youth & Young Adult Survey Part 1 Section D Parent Survey Section F	To understand what sexual risk avoidance and teen pregnancy prevention messages connect with youth and their parents, it is necessary to understand their attitudes and beliefs related to sexual activity. Responses to questions in these sections can guide the direction of future programming, ensuring it addresses common attitudes and beliefs among youth and parents.
Sexual activity (including intentions), incidence of pregnancy, and contraceptive use Youth & Young Adult Survey Part 2 Section G Parent Survey G3	The surveys are designed to measure knowledge, attitudes, behaviors, communication, education, and expectations related to teen sexual behaviors. These questions address the behavioral component and are necessary to provide context to other topical areas of the survey, such as attitudes and expectations. The majority of these questions are asked only of youth and young adults who report being sexually active. For respondents who do not report sexually activity, we ask their intentions regarding sexual activity. Intentions for engaging in sex and other risk-taking behaviors are extremely strong predictors of subsequent behavior.5 Intentions are strongly related to behavior and will be an important mediator predicting behavior change.

A12. Burden

Explanation of Burden Estimates

Table A12 summarizes the estimated reporting burden and costs for each instrument being pre-tested. We will conduct pre-testing in up to three phases. Phase 1 will include up to three pre-testing sessions in three different geographic areas. If substantive changes are made to the survey following the first round of pre-testing, we propose conducting phase two by returning to the same three geographic locations to pre-test the revised instruments using the same process and design described above. Based on the outcome of phase two, we propose a final phase three of testing, returning to up to all three sites, if extensive revisions are made to the instruments based on the phase two testing, to pre-test the revised instruments, again using the same process and design as previous phases. Different respondents would participate in each phase. In each geographic area, we will pre-test with up to 12 participants in the following groups: parents of a youth ages 14 to 18, youth ages 14 to 18 (part 1 of survey), youth ages 14 to 18 (part 2 of survey), and young adults ages 19 to 24. Pre-tests with parents and young adults will take up to two hours to complete, and each pre-test with youth will take an hour and a half. The youth pre-tests are expected to take less time because youth sessions will be split between Part 1 and Part 2 of the youth survey to reduce the length of time for this younger age group.

Estimated Annualized Cost to Respondents

We estimate the average hourly wage for parents and young adults is the average hourly wage of “all occupations” taken from the May 2018 National Occupational Employment and Wage Estimates, U.S. Bureau of Labor Statistics ($24.98).

Table A12. Total Burden Requested Under this Information Collection

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total Burden (in hours)	Annual Burden (in hours)6	Average Hourly Wage Rate	Total Annual Respondent Cost
Pre-test Phase 1
Parent Pretest Protocol: Survey and Debrief Topic Guide	36	1	2	72	24	$24.987	$599.52
Part 1 Youth Pretest Protocol: Survey and Debrief Topic Guide	36	1	1.5	54	18	$0	$0
Part 2 Youth Pretest Protocol: Survey and Debrief Topic Guide	36	1	1.5	54	18	$0	$0
Young Adult Survey Pretest Protocol: and Debrief Topic Guide	36	1	2	72	24	$24.98	$599.52
Estimated Annual Burden Phase 1 Pre-test Total					84		$1,199.04
Pre-test Phase 2
Parent Pretest Protocol: Survey and Debrief Topic Guide	36	1	2	72	24	$24.98	$599.52
Part 1 Youth Pretest Protocol: Survey and Debrief Topic Guide	36	1	1.5	54	18	$0	$0
Part 2 Youth Pretest Protocol: Survey and Debrief Topic Guide	36	1	1.5	54	18	$0	$0
Young Adult Pretest Protocol: Survey and Debrief Topic Guide	36	1	2	72	24	$24.98	$599.52
Estimated Annual Burden Phase 2 Pre-test Total					84		$1,199.04
Pre-test Phase 3
Parent Pretest Protocol: Survey and Debrief Topic Guide	36	1	2	72	24	$24.98	$599.52
Part 1 Youth Pretest Protocol: Survey and Debrief Topic Guide	36	1	1.5	54	18	$0	$0
Part 2 Youth Pretest Protocol: Survey and Debrief Topic Guide	36	1	1.5	54	18	$0	$0
Young Adult Pretest Protocol: Survey and Debrief Topic Guide	36	1	2	72	24	$24.98	$599.52
Estimated Annual Burden Phase 3 Pre-test Total					84		$1,199.04
Estimated Annual Burden Total					252		$3,597.12

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

Table A14 provides the estimates of the annualized costs to the Federal government for the work associated with this Information Collection Request.

Cost Category	Estimated Costs
Instrument Development and OMB Clearance	$98,980
Field Work	$250,951
Publications/Dissemination	$49,548
Total costs over the request period	$399,479
Annual costs	$133,160

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella generic clearance for pre-testing (0970-0355).

A16. Timeline

The pre-testing expected to begin in summer 2019 after obtaining OMB approval and continue for about 3 months. Pre-test results and proposed revisions to the survey will be reported to OPRE in an internal memo about 1 month after data collection is complete. The pre-testing procedures will be documented as part of the main study’s methodological report following the national data collection.

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Attachment A: Sources Used in the Development of the Parent and Youth Surveys

Attachment B: Consent and Assent Forms

Instrument 1: Parent Pre-Test Protocol- Survey and Debrief Topic Guide

Instrument 2: Part 1 Youth Pre-Test Protocol- Survey and Debrief Topic Guide

Instrument 3: Part 2 Youth Pre-Test Protocol- Survey and Debrief Topic Guide

Instrument 4: Young Adult Pre-Test Protocol- Survey and Debrief Topic Guide

1 Martin, J. A., Hamilton, B. E., Osterman, M. J. K., Driscoll, A. K., & Matthews, T. J. (2017). Births: Final data for 2015. National Vital Statistics Report, 66(1).

2 Centers for Disease Control and Prevention. (2017). Sexually transmitted disease surveillance 2016. Atlanta, GA: U.S. Department of Health and Human Services.

3 Examples of sensitive topics include (but are not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

4 Vesely, S. K., Wyatt, V. H., Oman, R. F., Aspy, C. B., Kegler, M. C., Rodine, S., Marshall, L., & McLeroy, K.R. (2004). The potential protective effects of youth assets from adolescent sexual risk behaviors. Journal of Adolescent Health, 34(5), 356–365.

Sinha, J. W., Cnaan, R. A., & Gelles, R. J. (2007). Adolescent risk behaviors and religion: Findings from a national study. Journal of Adolescence, 30(2), 231–249.

5 Buhi, E. R., & Goodson, P. (2007). Predictors of adolescent sexual behavior and intention: A theory-guided systematic review. Journal of Adolescent Health, 40(1), 4–21.

6 Annualized over the three-year period of the umbrella generic.

7 The average hourly wage for parents and young adults is the “All Occupations” wage from the May 2018 Bureau of Labor Statistics https://www.bls.gov/oes/current/oes_nat.htm#00-0000.

21