Ss_b_tthy_073019_final

Underage Drinking Prevention’s “Talk. They Hear You.”^® Campaign Parent Questionnaire

Supporting Statement, Part B

B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The Substance Abuse and Mental Health Services Administration (SAMHSA) seeks to administer a questionnaire among parents and caregivers of children ages 9-20. The convenience sample of respondents (N = 5,000) will be drawn from a nonprobability online panel. The panel employs a double opt-in process for individuals to participate in a questionnaire—they must opt in to become panelists, and they must also opt in to each questionnaire. The panel will gather information for prescreening and ask participants to review a privacy policy statement and a terms and conditions statement that outlines the parameters for their participation. The prescreening is designed to screen out persons who are not parents and/or caregivers in the United States of children ages 9–20, per the target specifications of the campaign effort being evaluated.

Respondents will be recruited to participate in this online questionnaire from a Qualtrics^©-sourced online panel. Demographic information for these panelists was collected when they were recruited to participate in the panels and will be collected again for validation purposes at the end of the Parent Questionnaire ¹. Respondents will be drawn from the pool to ensure representation across notable sociodemographic and demographic variables of interest to ensure a broad set of opinions and experiences regarding the “Talk. They Hear You.” campaign.

Using web-based data collection provides a number of methodological advantages including increased specificity in capturing key variables of interest at the individual level (e.g., perceived effectiveness of campaign materials) and reduced burden on participants, because it is automated and electronic and uses other technological collection techniques (e.g., skip patterns). Limitations of this approach include the potential correlation between any in the population of interest who lack the ability to participate in a web-based questionnaire (due either to lack of a computer and/or to lack of broadband service). However, this approach yields significant cost efficiencies compared to other modes of data collection such as telephone questionnaires. These advantages include but are not limited to:

• Convenience and reduced burden to the respondent. Compared to telephone interviewing, web-based questionnaires may reduce vulnerability to socially desirable questionnaire responses, particularly on sensitive subjects such as alcohol and other substances. Questionnaires are self-administered, so the participants may choose a private setting, convenient location, and time, reducing disruption to their daily activities. Computerized skip patterns also minimize respondent burden by avoiding the presentation of unnecessary questions and information collection based on response logic for questionnaire items.

• Flexible and timely data collection. Because web questionnaires do not involve human interviewers and all ensuing requirements for interviewer training and quality control, it is relatively easy and inexpensive to launch web questionnaires very quickly.

• Inclusion of diverse campaign media material, including video streaming of campaign ads, streaming of radio ads, and presentation of print materials, within the questionnaire. This inclusion significantly enhances the ability to more fully evaluate the range of materials in the “Talk. They Hear You.” campaign suite. By comparison, telephone questionnaires do not allow for direct exposure to campaign messages and stimuli.

2. Information Collection Procedures

SAMHSA will conduct the procedures described in the following text to collect data for the questionnaire during the base year (2019). An additional approval will be requested to conduct this questionnaire annually.

Parent/Caregiver Questionnaire

Potential respondents who fit the appropriate requirements for the convenience sample will be recruited from Qualtrics^©’ active academic and market research panels. These potential respondents will be contacted for participation via email invitations. Assuming that the response rate for electronic questionnaires will be 8 percent² and that some percentage of respondents will be screened out because they no longer have children in the desired age range, SAMHSA will likely send between 150,000 and 160,000 questionnaire invitations.

Based on a time test conducted by SAMHSA in August 2017, questionnaires will take an estimated 10 minutes to complete. Each respondent will be asked generally the same number of questions, with an equal amount of time to respond. Nonresponse follow-up will be administered at least three times during the field period; those who have not responded to the initial request will be sent at least three reminders encouraging them to complete the questionnaire.

The questionnaire will be conducted entirely online through the Qualtrics^© Survey Suite, an online questionnaire platform that will collect and store questionnaire responses during questionnaire administration. Once the desired 5,000 interviews have been completed, all questionnaire data submitted online will be downloaded into Excel spreadsheets and made available for data analysis.

3. Methods To Maximize Response Rates

SAMHSA proposes conducting the Parent Questionnaire online in order to provide the most efficient data collection process. Because online questionnaires do not require live interviewers for administration, and questionnaires through the Qualtrics^© platform are optimized to be completed on desktop, tablet, or mobile devices, the questionnaire process will be as convenient as possible for respondents and should produce 5,000 completed interviews in a timely manner. As an added measure to maximize response rates within the sampling frame, 1 week after the initial questionnaire invitation, reminder invitation emails will be sent to parents/caregivers who have not completed the questionnaire. Each parent/caregiver who has not yet completed the questionnaire may receive up to three reminder emails during the course of the questionnaire administration. The efficacy of this repeated contact approach has made it standard procedure throughout the questionnaire/survey and evaluation industries.

4. Tests of Procedures

The questionnaire instrument used for this evaluation was time tested to accurately estimate evaluation costs and burden to respondents (see the “Time Test Findings” report for more details, available upon request). Before questionnaire launch, the final questionnaire instrument will be data optimized by both the Summit Research Associates, NYC and the Qualtrics International teams contracted to design and conduct the analysis for this evaluation. In 2013, following the pilot launch of the campaign, SAMHSA conducted a pilot campaign evaluation survey (OMB [Office of Management and Budget] No. 0930-0196; expired: September 30, 2013), which asked parents and caregivers located in five pilot sites about their current attitudes, behaviors, and concerns about underage drinking.

As described more fully in Part A of this OMB package, data will be reviewed to assess campaign development and expansion efforts. Quantitative data will be analyzed using univariate statistics to determine overall feedback regarding the campaign (e.g., general campaign appeal, perceived usefulness of materials) as well as patterns in how different subgroups of parents/caregivers respond to the campaign materials.

5. Federal Contact Information:

Robert Vincent, M.S.Ed.

Public Health Analyst

Center for Substance Abuse Prevention, SAMHSA

Phone: (240) 276–1582

Email: robert.vincent@samhsa.hhs.gov

Ralph Hingson, Sc.D., M.P.H.

Director

Division of Epidemiology and Prevention Research,

National Institute on Alcohol Abuse and Alcoholism

Phone: (301) 443–1274

Email: rhingson@mail.nih.gov

Aria Davis-Crump, Sc.D.

Deputy Branch Chief

Prevention Research Branch,

National Institute on Drug Abuse

Phone: (301) 435–0881

Email: acrump@nida.nih.gov

The following individuals outside of the Agency, and independent of SAMHSA and “Talk. They Hear You.” campaign development, have been consulted on the evaluation design, questionnaire development, and sampling plan. Regular meetings with Summit Research Associates, NYC and Qualtrics International are held to provide updates on the campaign and evaluation efforts.

Ryan Parks

Project Director, Synergy Enterprises, Inc.

8757 Georgia Avenue, Suite 1440

Silver Spring, MD 20910

Phone: (240) 485–3278

Email: RParks@seiservices.com

Jodie Abbatangelo-Gray, Sc.D., M.S., M.A.

Independent Evaluator, Summit Research Associates, NYC

267 West 89th Street, Suite 6D

New York, New York

Phone: (917) 710–5219

Email: jodie@sra-nyc.com

Steven Snell, Ph.D.

Qualtrics International

333 River Park Drive

Provo, Utah 84604

Phone: (609) 759–0897

Email: stevens@qualtrics.com

7. References

Southwell, B., Barmada, C., Hornik, R., & Maklan, D.M. (2002). Can we measure encoded exposure? Validation evidence from a national campaign. Journal of Health Communication, 7(5), 445–453.

List of Attachments

Attachment 1: Screener and Questionnaire Tool

1 Pre-screening panelists ensures that respondents are unable to opt into the questionnaire when they do not qualify. Furthermore, questions related to qualifying criteria embedded within the questionnaire, such as confirmation of eligibility, are presented neutrally to keep questionnaire participation requirements hidden. This approach minimizes the risk of self-selection bias that could result in false conclusions.

2 According to Qualtrics^©’ most recent reporting statistics, the average response rate was 8.34 percent across 100 national studies conducted by various researchers using Qualtrics^© and its standard sample sources.

5