Supporting Statement part B-Testing Message Evaluation

U.S. Food and Drug Administration

Center for Tobacco Products

Testing Message Evaluation Measures Using Youth-focused Vaping Prevention Messages

0910-0810

Supporting Statement: Part B

B. Statistical Methods

1. Respondent Universe and Sampling Methods

The respondent universe for this study is youth aged 13-17 who are susceptible to, or currently use vaping products. Data will be collected via a one-time cross-sectional survey using a participant panel of adult parents with children in this age range. This study is designed to determine valid message evaluation (ME) measures to use in evaluating anti-tobacco messaging. The results will be used to inform FDA, prevention practitioners, and researchers of optimal ME measures for evaluating the relative potential effectiveness of campaign messages.

As this study is considered part of formative research for campaign development and planning, these methods are not intended to generate nationally representative samples or precise estimates of population parameters. The sample drawn here is designed primarily to provide information that may be used in formative evaluation of future tobacco prevention campaigns.

Specifically, we will conduct a message testing study to examine whether key ME measures (e.g., perceived message effectiveness, reactance in response to messages, perceived argument strength) often used in formative research demonstrate criterion validity by investigating the strength of their relationship with outcomes of message exposure, such as beliefs and behavioral intentions. We will also aim to compare ME measures on the strength of their association with AME and to examine whether the relationship between ME measures and message outcomes varies depending on characteristics of the vaping prevention messages and user groups.

Sampling Methods

Study participants will be recruited from a national online panel of adults who have children ages 13-17 managed by Lightspeed. Lightspeed panel members will receive an invitation to participate in the study and will determine if they are interested in their child participating in the survey. Parents who consent to have their child participate will give parental permission (Appendix A) and their child will then be asked for their assent (Appendix B) and directed to the screener (Appendix C). The screening criteria are based on age, vaping status, and intention to vape in the future. Those qualifying for the study will proceed to the survey (Appendix C).

We will monitor the distributions of age, gender, education, and ethnicity/race among the completed study sample. However, FDA does not intend to generate nationally representative results or precise estimates of population parameters from the study; generating a representative sample of the size necessary for this study, using Random Digital Dialing or other similar method, would be cost prohibitive. The study will use convenience samples rather than probability samples. Despite the attempt to match the study’s sample to national demographic categories, this is used solely to produce a sample with a reasonable degree of diversity in key demographic characteristics. Despite best efforts to have the study population reflect the demographic makeup of the U.S. population, the nature of convenience samples still limits the generalizability of the results from this study. Thus, the conclusions drawn from this sample, while representative of the sample, may not necessarily generalize to other populations not included in the study. These limitations in generalizability do not affect the internal validity of the study and will be noted in the context of describing the results of the study.

Sample Size

To obtain a final sample of 2,400 youth aged 13-17 who have either experimented with vaping or are at risk of initiating vaping, we will need to screen approximately 8,000 potential respondents. This is because we anticipate approximately 40% of youth have either experimented with vaping or are at risk of initiating vaping (NYTS, 2020). Based on experience from previous surveys, we anticipate about 75% of screened contacted respondents will provide both parental permission and child assent to participate in the study. Exhibit 1 lists the study activities and sample size assumptions to yield the needed number of completes.

Table 1. Study Activities and Sample Size Assumptions to Yield the Needed Number of Completes

Activity	Sample Size (Expected Yield)
Adult panelists who review consent	8,000 (75%)
Youth who are screened	6,000 (40%)
Survey completes	2,400

2. Procedures for the Collection of Information

This section describes the procedures for the survey data collection. The data will be conducted via a self-administered, web-based survey disseminated by Lightspeed. Lightspeed will contact panel members who have indicated in their panel profile that they have a child in the eligible age range to invite their child to participate in the study. Adolescents (aged 13-17 years) will then be invited to participate. To be eligible, the parent must give their permission, the youth must give their assent, and the youth (age 13-17) must be either a person who has experimented with vaping or a youth (ages 13-17) who is susceptible to vaping in the future. The screener is provided in Appendix C. The survey instrument (also Appendix C) will include the survey questions and contact information for technical staff at RTI who will be available to respond to questions posed by participants. The survey will be hosted on Lightspeed’s cloud-based servers.

RTI International, the research organization contracted to conduct this research, will analyze the data collected from this study, the results of which will inform FDA’s efforts to determine valid ME measures.

Summary of Protocol

The list of study procedures is as follows:

1. Adult panelists who have indicated in their panel profile that they have a child in the eligible age range will receive an initial invitation that will direct them to the parent permission form. The parent permission form provides information about the length of the survey, confidentiality standards, and incentive the parent will receive if the child qualifies and completes the survey.

2. If the parent confirms they have a child in the specified age range and determines that they would like their child to participate in this survey, they will be asked to provide parental consent.

3. The parent is then asked to allow their child to complete the remainder of the survey in privacy.

4. The youth will review the assent form, which provides a description of the purpose and confidentiality standards associated with the study.

5. If the youth gives their assent, then they will be redirected to the online screener questions.

6. If the respondent qualifies for the survey, he or she will begin the survey. Respondents will be asked questions about tobacco use, watch four 15- to 30-second anti-tobacco videos, and answer questions about their beliefs, behavioral intentions, and reactions to the messages.

7. If the respondent does not qualify for the survey, he or she will be thanked for their time and explain that they do not qualify for the survey.

Due to Lightspeed’s protected panel technology, it will not be possible for anyone to enter the survey who has not been recruited through their parent, or for a respondent to complete the survey more than once.

Unusual Problems Requiring Specialized Sampling Procedures

No specialized sampling procedures are involved.

Use of Periodic Data Collection Cycles to Reduce Burden

This is a one-time survey data collection effort.

2b. Degree of Accuracy Required for the Study

We conducted a power analysis to determine the effect size needed to achieve 80% power given a sample size of 2,400 and different distributions assumptions for the variables and error term. We used a simulation approach to estimate effect sizes for a linear regression of the following form:

Where:

• = a scaled measure of intention to vape (range of 1-5), assessed following experimental ad exposure.

• = a scaled measure of intention to vape (range of 1-5), assessed before experimental ad exposure. We assumed a mean of 3 and standard deviation (SD) of 0.6.

• = Aggregate message evaluation measure (range = 1-5), derived from the average scores of 80 ads in a simulation. We assumed a normal distribution with a mean of 3 and SD of 0.20 (the SD estimate is based on a compilation of perceived effectiveness scores from previous ad testing surveys RTI has conducted as part of its media campaign evaluations for the New York State and Utah Departments of Health).

• = An indicator for long ad (e.g., 30-second) vs. short ad (e.g., 15-second).

• = Error term. We evaluated two different assumptions for the error term in the model: low (0.60) and high (0.92).

We simulated the data 300 times, with the effect of selected so that the value of achieved significance (p-value = 0.05) in 80% of the iterations. Results demonstrate the following:

• When , the value of needed to achieve 80% power is 0.27; i.e., adjusting for and , for every unit increase in there needs to be a 0.27 unit change in .

• When the value of needed to achieve 80% power is 0.18; i.e., adjusting for and , for every unit increase in there needs to be a 0.18 unit change in .

Next, we used simulation methodology and the results of the first power analysis to investigate the power to detect the interaction ( ). We applied the same assumptions as the first analysis for the distributions of , , , and . Results demonstrate the following:

• When , we have 80% power to reject the null hypotheses when is 0.57 or greater.

• When , we have 80% power to reject the null hypotheses when is 0.37 or greater.

2c. Estimation Procedures

Statistical analyses will be conducted to address the study’s primary research questions. We will develop multivariable regression models to estimate statistical associations between exposure to messaging of varied message evaluation performance and key beliefs, attitudes, and intentions targeted by the ads. Model results will facilitate a comparison of ME measures’ relative association with actual message effectiveness to identify optimal ME measures for use in evaluating the potential effectiveness of campaign messages.

3. Methods to Maximize Response Rates

To maximize participation, we will incorporate best practices from similar online surveys into our data collection procedures. These include:

• Implementing a soft launch of the online survey to a small number of selected panel members to detect and resolve any technical difficulty.

• Keeping the questionnaire at a reasonable length to minimize break-offs.

• Including a brief introduction to the study that identifies FDA as the sponsor, states the purpose of the study, and provides toll-free telephones numbers for participants to call RTI with any questions about the study or their rights as a study participant.

• Inviting panel members who appear to be eligible based on their member profile. As part of the process of registering with the survey panel, panelists provide information about a range of sociodemographic characteristics, including whether or not they have children, that can be used to target particular groups. Lightspeed actively manages panelist profiles, requesting updated information on an ongoing basis to ensure that profile information is up to date.

• Recruiting verified panelists. Lightspeed uses a double opt-in registration process whereby panelists are invited to participate and then must sign up through an opt-in confirmation e-mail. This process protects against fraudulent account registrations and ensures that panelists are actively motivated to participate in surveys.

• To minimize nonresponse, Lightspeed will conduct ongoing monitoring of response levels and drop-off rates. Lightspeed will work with RTI project staff to address any problems that arise throughout the course of the collection of information.

• Providing incentives to recognize the time burden placed on participants, encourage their cooperation, and convey appreciation for contributing to this important study. Numerous empirical studies have shown that incentives can significantly increase response rates in cross-sectional surveys and reduce attrition in longitudinal surveys (e.g., Abreu & Winters, 1999; Castiglioni, Pforr, & Krieger, 2008; Jäckle & Lynn, 2008; Shettle & Mooney, 1999; Singer, 2002).

4. Tests of Procedures or Methods

The contractor RTI will conduct rigorous internal testing of the online survey instrument prior to its fielding. RTI will review the online test version of the instrument that we will use to verify that instrument skip patterns are functioning properly, the videos are working properly, and that all survey questions are worded correctly and are in accordance with the instrument approved by OMB. Lightspeed will begin data collection with a soft launch during which they will send invitations to a small subset of panel members and review their responses to ensure the online survey is working properly.

5. Individuals Involved in Statistical Consultation and Information Collection

The following individuals inside the agency have been consulted on the design and statistical aspects of this information collection as well as plans for data analysis:

Matthew Walker

Office of Health Communication & Education

Center for Tobacco Products

Food and Drug Administration

10903 New Hampshire Avenue

Silver Spring, MD 20993-0002

Phone: 240-402-3824

E-mail: matthew.walker@fda.hhs.gov

Megan Wall

Office of Health Communication & Education

Center for Tobacco Products

Food and Drug Administration

10903 New Hampshire Avenue

Silver Spring, MD 20993-0002

Phone: 240-750-5844

E-mail: megan.wall@fda.hhs.gov

Emily Peterson

Office of Health Communication & Education

Center for Tobacco Products

Food and Drug Administration

10903 New Hampshire Avenue

Silver Spring, MD 20993-0002

Phone: 202-964-4011

E-mail: emily.peterson@fda.hhs.gov

Xiaoquan Zhao

Office of Health Communication & Education

Center for Tobacco Products

Food and Drug Administration

10903 New Hampshire Avenue

Silver Spring, MD 20993-0002

Phone: 240-402-0296

E-mail: Xiaoquan.Zhao@fda.hhs.gov

The following individuals outside of the agency have been consulted on questionnaire development and/or will be collecting and/or analyzing data:

Susana Peinado

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: 919-316-3190

E-mail: speinado@rti.org

Laura Baum

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: 770-234-5017

E-mail: lbaum@rti.org

Matt Eggers

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: 919-541-6683

E-mail: meggers@rti.org

James Nonnemaker

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: 919-541-7064

E-mail: jnonnemaker@rti.org

Jesse Thompson

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: 919-485-5650

Email: jmthompson@rti.org

References

Abreu, D. A., & Winters, F. (1999). Using monetary incentives to reduce attrition in the survey of income and program participation. In Proceedings of the Survey Research Methods Section (pp. 533-538).

Castiglioni, L., Pforr, K., & Krieger, U. (2008, December). The effect of incentives on response rates and panel attrition: Results of a controlled experiment. In Survey Research Methods (Vol. 2, No. 3, pp. 151-158).

Centers for Disease Control and Prevention (2021). 2020 National Youth Tobacco Survey (NYTS) Dataset. Atlanta, GA.

Jäckle, A., & Lynn, P. (2008). Offre de primes d’encouragement aux répondants dans une enquête par panel multimodes: effets cumulatifs sur la non-réponse et le biais. Techniques d’enquête, 34(1), 115-130.

Shettle, C., & Mooney, G. (1999). Monetary incentives in US government surveys. Journal of Official Statistics, 15(2), 231.

Singer, E. (2002). The use of incentives to reduce nonresponse in household surveys. Survey nonresponse, 51, 163-177.