SSB - Pretesting Generic IDEAS

Alternative Supporting Statement Instructions for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Youth Empowerment Information, Data Collection, and Exploration on Avoidance of Sex (IDEAS)—Pre-Test

OMB Information Collection Request

0970 - 0355

Supporting Statement

Part B

June 2019

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Caryn Blitz

Part B

B1. Objectives

Study Objectives

The objective of this pre-test is to ensure that the Youth Empowerment Information, Data Collection, and Exploration on Avoidance of Sex (IDEAS) Project Youth, Young Adult and Parent survey items will provide reliable and valid data on the constructs of interest. The pre-test will help us identify potential issues with the survey, and respondent feedback will help us generate potential solutions to address areas of concern. This includes, but is not necessarily limited to, revising the questionnaire so that respondents can more accurately comprehend and respond to questions, reduce burden, increase response rates, and improve the overall quality and efficiency of the survey data collected. The activities in this information collection will focus on modified and new survey items. ACF will use the pre-test data to refine the Youth Empowerment IDEAS surveys as part of finalization of the instruments to be submitted to the Office of Management and Budget (OMB) as a new full information collection request for the Youth Empowerment IDEAS project.

Generalizability of Results

This study is intended to produce descriptive findings about the reliability and validity of survey items for the target populations in chosen sites to inform questionnaire development, not to promote statistical generalization to other sites or populations.

Appropriateness of Study Design and Methods for Planned Uses

The pre-test, as described in B2, is designed to efficiently collect information on new and newly modified survey items from a diverse sample of respondents across multiple geographic locations. The study is also designed to help detect any sensitivity issues for special groups, including whether respondents with different linguistic backgrounds have difficulty understanding specific terms or phrases in survey items. Working with research organizations who have access to a wide range of respondents allows for efficiently recruiting a diverse testing pool. As discussed in B2, the design is iterative, providing the study team a chance to test modifications made to the instrument based on initial feedback. This will result in an instrument that is fully tested prior to fielding in the national study.

Because it is a convenience sample, the results from the pre-test will not be considered representative of parents of youth ages 14 to 18 or youth and young adults 14 to 24 nationally. Results from the pre-test will be used to revise survey instruments prior to submission of the Youth Empowerment IDEAS information collection request.

B2. Methods and Design

Target Population

Our goal is to pre-test the instruments across a range of respondent demographic characteristics (varying in age, race, ethnicity, and sex (male or female)) for the purpose of establishing question reliability and validity. Because the final IDEAS surveys will be administered to a national sample, we propose conducting pre-tests in up to three distinct geographic sites. Testing within different geographic areas chosen for demographic diversity will be sufficient for the purpose of this pre-test.

For each phase of pre-testing, described in Supplemental Statement A2 (Study Design), we propose recruiting 12 participants per instrument, per site. Each round will thus involve 36 parents of youth (Instrument 1), 72 youth aged 14 to 18 (36 each for Instruments 2 and 3), and 36 young adults aged 18-22 (Instrument 4).

Sampling and Site Selection

The IDEAS pre-test will employ a convenience sample. The three pre-testing groups align with the target population of the surveys; which will ultimately be a national sample of parents with youth ages 14 to 18, youth ages 14 to 18, and young adults ages 19 to 24. Parents of youth ages 14 to 18 and youth ages 14 to 18 will be recruited together to facilitate the parental consent process needed for the youth to participate. This process is similar to the way parents and youth will be recruited for the national survey. We will recruit young adults separately in each geographic location. As explained in A2, youth will be administered either Part 1 or Part 2 of the survey. In order to hit recruitment targets and not over pretest with the parent population, half of parents recruited will not be asked to complete a pretest.

We are working with professional research organizations that specialize in recruiting diverse participants for focus groups. We have identified research firms that recruit pre-test participants and have facilities from which the study’s research team can work in geographically diverse areas, such as Washington, DC; Dallas, Texas; and Kansas City, Missouri. Each site will identify participants from its respondent pools who meet our demographic criteria and who would be interested in participating in the pre-test activities. For each phase of the pre-test, each research organization will recruit approximately 12 respondents per target population group, varying in age, race, ethnicity, and sex.

After making changes to the instruments based on feedback from the three pre-testing sites in phase one, if needed, we will commence with phase two pre-testing to assess the changes made on the instruments. If more revisions are made to the instrument based on round 2 of the pre-test, we will return to the same three locations and conduct a final phase of pre-testing.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

When designing the Parent ,Youth and Young Adult surveys, the study team reviewed 12 national or regional, public health and public opinion surveys, surveys and 7 previously used research and evaluation instruments that cover topics relevant to adolescent decision making and risk behavior (Attachment A).

Although the study team drew many of the items in the IDEAS instruments from these existing surveys, our process for developing the item pool identified questionnaire constructs that existing items lacked. In addition, after subject matter experts and ACF leaders reviewed the instruments, the study team modified some previously used items to better suit the goals of the IDEAS project. These modified items must be treated as if they are new survey items, and appropriately validated prior to national fielding for the IDEAS project.

Given the sensitivity of the questions and cognitive burden associated with the debrief, the pre-test breaks the Youth survey into two instruments for youth aged 14-18. These youth will be given only one of the two protocols during the pre-test.

B4. Collection of Data and Quality Control

ACF is contracting with Mathematica for this pre-test data collection. The researchers at Mathematica have extensive experience pre-testing surveys with youth, young adult and adult populations for prior studies sponsored by the U.S. Department of Health and Human Services and other federal agencies¹. One researcher will conduct each pre-test session with one note-taker from the study team also in the room. With the permission of respondents, pre-tests will be audio recorded for note-taking and quality assurance purposes.

As mentioned in B2, recruitment of respondents will be done through professional research organizations that specialize in recruiting diverse participants for focus groups. Pre-testing will occur in small group sessions of three or four respondents. For the youth (14-18 year old) population, the groups will be organized by sex because of the sensitive nature of the questions. At the start of each session, the study team will introduce the study and explain the pre-test process. For the youth pre-test, parents will be asked to provide consent for their child to participate in the pre-test. The consent form (Attachment B) will provide parents with information about the survey topics and the pre-test process and ask whether permission is granted for the youth to participate in the pre-test. Youth with parental consent will be read the assent form and parents and young adults will be read the consent form prior to starting their sessions, and they can decide whether they want to participate.

After completing the consent and assent process, respondents in each session will be asked to individually complete the draft survey, marking questions or words that are unclear or that they have questions about. After everyone finishes, the pre-test lead will facilitate a semi structured small group discussion following the pre-test protocol. Following the small group discussions, the pre-test lead and the note taker will debrief together on key points, discussing the feedback received and clarifying the notes before the next pre-test session begins.

If substantive changes are made to the survey following the first round of pre-testing, we propose returning to the same geographic locations to pre-test the revised instruments using the same process and design described above. Based on findings from this second phase of pre-testing, we may revise items again, and pre-test revised times in all three sites. This third phase will also use the same process and design as previous phases.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

Participants in the pre-test will be identified through a convenience sample recruited by local organizations that maintain a standing pool of interested study participants. As respondents are not randomly sampled, we will not calculate a response rate for the pre-test.

Non-Response

As participants will not be randomly sampled and findings are not intended to be representative, survey non-response bias will not be calculated. We will ask the local organizations to track refusals by reason for refusal and demographic characteristics as this may provide information about potential challenges and assess whether nonresponse bias may be an issue for the planned Youth Empowerment IDEAS data collection. This information and the participant demographics will be documented in the memo of pre-test findings.

The study team will examine item non-response as an indicator of the sensitivity and age appropriateness of survey questions. This information will be used in revisions of questions across the phases of the pre-test, with a goal of minimizing item non-response in the final instrument.

B6. Production of Estimates and Projections

The purpose of this clearance request is for pre-testing data collection instruments and procedures to evaluate and improve their quality for use in future Youth Empowerment IDEAS data collection.

Statistical estimation is not intended with the pre-test survey data.

B7. Data Handling and Analysis

Data Handling

There are no procedures for editing data, data entry, coding or data processing, as these are outside of the scope of this proposed data collection and analysis. Notes will be checked against recordings to ensure accuracies.

Data Analysis

Qualitative data from the pre-test debrief interview notes will be reviewed for problematic questions, and feedback might (or might not) lead to suggestions for modifications to survey items.

Data Use

We will use the pre-test results to revise the Youth and Parents surveys. These revisions, and an accompanying memo, will be included with the full information collection request for the Youth IDEAS project.

B8. Contact Person(s)

To develop the Parent and Youth Surveys, ACF contracted with Mathematica. Mathematica will lead the pre-testing data collection activities described in this Information Collection Request. Table B8 lists the individuals responsible for the pre-test procedures and data collection and includes their affiliation and email address.

Table B8. Individuals Responsible for Survey Development and Pre-test Data Collection Procedures

Name	Affiliation	Email address
Caryn Blitz	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	Caryn.Blitz@acf.hhs.gov
Kathleen McCoy	Business Strategy Consultants Staff, Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services,	Kathleen.McCoy@acf.hhs.gov
Susan Zief	Mathematica	SZief@mathematica-mpr.com
Tiffany Waits	Mathematica	TWaits@mathematica-mpr.com
Sarah Forrestal	Mathematica	SForrestal@mathematica-mpr.com

Attachments

Attachment A: Sources Used in the Development of the Parent and Youth Surveys

Attachment B: Consent and Assent Forms

Instrument 1: Parent Pre-Test Protocol- Survey and Debrief Topic Guide

Instrument 2: Part 1 Youth Pre-Test Protocol- Survey and Debrief Topic Guide

Instrument 3: Part 2 Youth Pre-Test Protocol- Survey and Debrief Topic Guide

Instrument 4: Young Adult Pre-Test Protocol- Survey and Debrief Topic Guide

1 Pregnancy Assistance Fund Study (OMB Control Number 0990-0424); Federal Evaluation of Making Proud Choices (OMB Control Number 0990-0452), The Strengthening Relationship Education and Marriage Services Evaluation (OMB Control Number 0970-0481), Head Start Family and Child Experiences Survey 2019 (OMB Control Number 0970-0151), Middle Grades Longitudinal Study of 2017-18 (OMB Control Number 1850-0911), Evaluation of Demonstration Projects to End Childhood Hunger (OMB Control Number 0584-0603), School Nutrition and Meal Cost Study (OMB Control Number 0584-0596), Regional Partnership Grants National Cross-Site Evaluation and Evaluation Technical Assistance (OMB Control Number 0970-0444).

7