Experimental Study of Cigarette Warnings SS Part B - CTP Clean 3.8.19

Experimental Study of Cigarette Warnings

0910-NEW

FDA SUPPORTING STATEMENT PART B

B. Statistical Methods

1. Respondent Universe and Sampling Methods

The respondent universe for this study is (1) adolescent current cigarette smokers aged 13-17 years, adolescent non-smokers who are susceptible to initiation of cigarette smoking aged 13-17 years; (2) young adult current cigarette smokers and non-smokers aged 18-24 years; and (3) older adult current cigarette smokers and non-smokers aged 25 years and older.

Study participants will be recruited from a national online panel of adults managed by Lightspeed. (http://www.lightspeedresearch.com). Lightspeed panel members are recruited from various sources (both online and not) and doubly opt-in to the be a part of the overall panel. Lightspeed provides publicly-available information about the general demographic and geographic characteristics of their panel members. The U.S. panel had approximately 1.3 million members as of 2017. As of 2018, the U.S. panel was 70% female and diverse in terms of U.S. geography (22% Midwest, 16% Northeast, 43% South, 19% West), age (26% ages 15-24, 22% ages 25-35, 16% ages 35-44, 14% ages 45-54, and 22% ages 55+), and income (37% <$20,000, 27% $20,000-39,000, 17% $40,000-74,000, 19% >$75,000). Lightspeed does not provide more granular information about the size and demo/geographic composition of targeted subgroups (e.g., young adult smokers), and this information is not available for youth aged 13-17 who are not panelists but who are recruited via parent panelists.

As a condition of their participation on the Lightspeed panel, panelists periodically are asked to provide information about their demographic characteristics and other lifestyle information. Lightspeed uses this profile information to identify and recruit potential study participants. For the current study, Lightspeed will identify pools of panelists who have indicated in their panel profile that they meet the criteria for a particular subgroup (e.g., young adult smoker, adult 25+ nonsmoker). To recruit the youth sample, Lightspeed will identify panelists who indicated that they have a child in the eligible age range. Next, Lightspeed will generate random samples within each group and send invitation e-mails in batches until each sub-group quota is met.

The amount of sample pulled for a particular study is determined by assumed metrics, such as expected incidence rate, response rate, and completion rate. The current study includes recruitment of “low incidence” populations, such as smokers and young adults, who may be challenging to recruit in large numbers because of their relatively low incidence in the population and/or in the Lightspeed panel. For these sub-groups, we are attempting to recruit the estimated maximum completed sample size according to Lightspeed’s projected feasibility (based on assumed metrics described above). Thus, we are limited in the extent to which we can set additional quotas or sample stratifications to align the completed sample with population demographic and geographic benchmarks.

Nevertheless, we plan to set some broad initial quotas to prevent skewing in demographic characteristics for which we anticipate the sample may differ substantially as compared to population distributions. For example, given the over-representation of female participants on the panel, we will set an initial sub-quota to limit the overall proportion of female respondents to 50% of the overall sample. We also know from previous experience on a previous study (Experimental Study on Warning Statements for Cigarette Graphic Health Warnings – OMB # 0910-0848) that adult smokers tend to skew older in age; thus, we will set an initial quota such that no more than 25% of the adult (aged 25+) sample will be aged 65 or older. These quotas may need to be relaxed depending on sample feasibility while in the field (for example, if we are unable to reach our target sample sizes for study sub-groups because large numbers of otherwise eligible participants are being screened out as these secondary sex and age quotas are filled). While in the field, we will monitor the incoming sample to ensure that we are able to reach our target sample sizes while maintaining a reasonable degree of demographic diversity in the final sample.

Lightspeed panel members will receive an email inviting them to participate in the study. Adolescent children of adult panel participants will be invited to participate in the study through an email invitation to their parents asking for their consent to solicit their child’s opinions. Potential participants will be screened (via a screener questionnaire linked from the email invitation), and those qualifying for the study will proceed to Session 1.

We estimate a total of 9,760 respondents will complete Session 1, between 4,880 and 9,760 respondents will complete Session 2, and between 2,440 and 9,760 respondents will complete Session 3 (the exact total will depend on the retention rate).

As with any study conducted using opt-in online panels, this study may be subject to several threats to external validity that limit the generalizability of study results. Panelists are recruited into the online panel using convenience sampling methods and thus do not have a known probability of selection into the panel. As such, FDA will monitor the distributions of age, gender, education, and ethnicity/race among the completed study sample to ensure a reasonable degree of diversity in key demographic characteristics of the targeted sub-populations and readjust the number of recruitment emails being sent out to specific groups to ensure that the final sample is reasonable diverse to account for the underlying demographic profile of the overall panel. However, FDA does not intend to generate nationally representative results of the targeted sub-population in this experimental study. Note that generating a representative sample for such a three-step experiment of the size necessary to identify small effect sizes would be cost prohibitive. The study will use convenience samples rather than probability samples, and despite the diversity in the sample and the use of quota to ensure a reasonable diverse sample, the sample in the study is nevertheless still a convenience sample and not representative of national estimates or necessarily of the underlying study panel used. These limitations in generalizability do not affect the internal validity of the study. Additional limitations are that conclusions can only be drawn based on the stimuli presented. The results of this study will inform the FDA’s broader efforts to develop cigarette GHW with the goal of implementing the mandatory graphic warning label statements as required by section 4(d) of FCLAA. Such limitations will be noted in the context of describing the results of the study.

2. Procedures for the Collection of Information

For the information collection, Lightspeed will send email invitations (with links to the screener) to the target audiences using their market research panel (as described above). Adult (aged 18 years and older) panel members will be sent an email directly inviting their participation in the study and instructions for accessing the secure website for the screener. Adolescents (aged 13-17 years) will be invited to participate through an email invitation to an adult panel member who has indicated in their panel profile that they have a child in the eligible age range. Parents or guardians will be asked to provide permission before allowing their child to participate. Once a panel member or child of a panel member enters the secure web site, the respondent will access the screener to determine eligibility based on the study inclusion and exclusion criteria. Those respondents who are determined to be eligible to participate will be presented with a brief introduction informing them of the private and voluntary nature of the study. Those respondents who are determined to be eligible and provide consent to participate will then be randomly assigned to a condition and complete the study (beginning with Session 1, immediately following the screener, and followed by Sessions 2 and 3 at later dates, as described below). As data collection progresses for Session 1, study staff will check the distribution profile of those completing the study. Lightspeed may adjust the targeting of invitations to those in the research panel to better ensure a final sample that is more in line with demographic characteristics of the subsample. Additionally, demographic and tobacco use information will be collected from the respondents to understand the extent to which the characteristics of the participants reflect those of participants in national studies of adolescents and adult smokers (e.g., National Health Interview Survey for adults and the Population Assessment of Tobacco and Health) for youth.

This experimental study will be conducted using an Internet panel and questionnaires designed to measure responses to cigarette GHW that explain the negative health consequences of cigarette smoking. Specifically, this study is designed to determine if the GHW being tested increase understanding of the negative health consequences of cigarette smoking among study participants. The study is not designed, nor is it the intent of the study, to investigate the effect of these warnings on behavior, behavioral intentions, or emotional reactions.

Those who screen into the study and are chosen (i.e., participants) will be randomized to one of 16 experimental conditions or a control condition with variation in exposure to cigarette warnings (Table 1 shows the study conditions and planned allocations). Participants in each experimental condition will be exposed to one graphic health warning, with each condition corresponding to a unique warning from a set of 16; participants in the control condition will be exposed to a random selection of one of four Surgeon General’s (SG) warnings. All stimuli will be presented sequentially on a mock cigarette package and a mock cigarette advertisement, with the stimuli formats presented in a random order. When viewing the mock cigarette packages, participants will be able to manipulate the package electronically to rotate it 360 degrees to ensure they can view all sides of the package and zoom in on the pack or advertisement. The warnings on the mock packages will be placed either on the upper portion of the front and rear panels of the mock package comprising the top 50 percent of the front and rear panels of the package (for GHW) or on the side of the package (for Surgeon General’s Warnings), and all warning language will appear as black text on a white background. The mock advertisements will be full-page magazine-style advertisements, and the warnings will appear on the top 20 percent of the advertisement (for GHW) or the bottom of the advertisement as per current warning placement practice in accordance with FCLAA and FTC formatting specifications (for Surgeon General’s Warnings). The stimuli for all conditions can be found in the Stimuli document.

In the first part of the study (Session 1), all participants will complete a baseline assessment of their beliefs about the negative health consequences of cigarette smoking, followed by initial cigarette warning stimuli exposure according to study condition assignment and an assessment of (a) if the information presented in the warning was new; (b) self-reported learning from the warning; (c) if the warning was easy to understand; (d) if the warning was perceived to be a fact or an opinion; (e) if the warning was informative; (f) if the warning grabbed their attention; and (g) if the warning made them think about the health risks of smoking.

In the second part of the study (Session 2), one to two days after Session 1, participants will be re-exposed to the stimuli (based on condition assignment; see Table 1) and complete a follow-up questionnaire assessing beliefs about the negative health consequences of cigarette smoking.

In the final part of the study (Session 3), approximately 14 days after Session 2, participants will complete items assessing recall of the warnings and a delayed post-test on beliefs about the negative health consequences of cigarette smoking.

Analyses involve (1) a pre-post comparison to assess the extent to which exposure to either the GHW or SG warnings result in a change in understanding about the negative health consequences of cigarette smoking; and (2) a comparison of exposure to the GHW condition to the SG warnings condition to assess the ability of the GHW to increase understanding about the negative health consequences of cigarette smoking.

For warnings to be considered for future regulatory action, those individual warnings must demonstrate statistically significant improvements, as compared to the control condition, on both the two outcomes of New Information and Self-Reported Learning (knowledge gain). The warnings will then be evaluated not only on the presence of a statistically significant effect on both outcomes, but also on the magnitude of the effect as an indicator of the conceptual significance of the findings. Other outcomes being measured will also be considered when interpreting the results of the study to provide context and additional support. These findings will be considered in the context of the strengths and limitations of the sample and study design in order to ultimately inform FDA's implementation of section 201 of the Tobacco Control Act.

Table 1. Condition Assignment and Session 1 Sample Size, by Study Population

Condition	Exposure	Sample Size
		Adolescents	Young Adults	Older Adults	Total
0 (Control)	Random selection of 1 of the following SG warnings: SURGEON GENERAL’S WARNING: Smoking Causes Lung Cancer, Heart Disease, Emphysema, and May Complicate Pregnancy. SURGEON GENERAL’S WARNING: Quitting Smoking Now Greatly Reduces Serious Risks to Your Health. SURGEON GENERAL’S WARNING: Smoking by Pregnant Women May Result in Fetal Injury, Premature Birth, and Low Birth Weight. SURGEON GENERAL’S WARNING: Cigarette Smoke Contains Carbon Monoxide.	492	564	1,024	2,080
1	WARNING: Cigarettes are addictive. [GHW]	113	131	236	480
2	WARNING: Tobacco smoke can harm your children. [GHW]	113	131	236	480
3	WARNING: Smoking can kill you. [GHW]	113	131	236	480
4	WARNING: Tobacco smoke causes fatal lung disease in nonsmokers. [GHW]	113	131	236	480
5	WARNING: Quitting smoking now greatly reduces serious risks to your health. [GHW]	113	131	236	480
6	WARNING: Smoking causes head and neck cancer. [GHW]	113	131	236	480
7	WARNING: Smoking causes bladder cancer, which can lead to bloody urine. [GHW]	113	131	236	480
8	WARNING: Smoking during pregnancy stunts fetal growth. [GHW]	113	131	236	480
9	WARNING: Smoking can cause heart disease and strokes by clogging arteries. [GHW]	113	131	236	480
10	WARNING: Smoking causes COPD, a lung disease that can be fatal. [GHW – IMAGE 1]	113	131	236	480
11	WARNING: Smoking causes COPD, a lung disease that can be fatal. [GHW – IMAGE 2]	113	131	236	480
12	WARNING: Smoking reduces blood flow, which can cause erectile dysfunction. [GHW]	113	131	236	480
13	WARNING: Smoking reduces blood flow to the limbs, which can require amputation. [GHW]	113	131	236	480
14	WARNING: Smoking causes type 2 diabetes, which raises blood sugar. [GHW]	113	131	236	480
15	WARNING: Smoking causes age-related macular degeneration, which can lead to blindness. [GHW]	113	131	236	480
16	WARNING: Smoking causes cataracts, which can lead to blindness. [GHW]	113	131	236	480
TOTAL		2,300	2,660	4,880	9,760

RTI International, the research organization contracted to manage data collection, will, in collaboration with FDA investigators, analyze the data collected from this study, the results of which will inform FDA’s efforts to implement section 201 of the Tobacco Control Act, which requires FDA to issue regulations to implement textual warning statements accompanied by color graphics depicting the negative health consequences of cigarette smoking. The analysis will be guided by a pre-specified Statistical Analysis Plan that lays out the specific analytic approaches to be performed and the power analysis that informed the design of the study, both of which are also described below.

Power

As part of the planning for this study, FDA worked with the contractor managing the data collection to generate power calculations to confirm that the overall sample size (shown in Table 1) is sufficiently powered and to determine the optimal sample size and allocation of sample across study conditions. To control for Type 1 error taking into account multiple comparisons, power calculations were based on the false discovery rate (FDR).¹ Assuming the tests are independent, the FDR is the expected proportion of significant results that are falsely declared as statistically significant. Controlling the FDR is controlling the expected proportion of falsely declared differences (i.e., false discoveries). Controlling the FDR is a more powerful method for dealing with multiple comparisons than other methods that control the family-wise error rate such as the Bonferroni procedure.²

To inform the overall study sample size, FDA worked with RTI to calculate the power to detect a difference from Session 1 to Session 2 between treatment and control groups (i.e., difference in difference) (Table 2 provides power estimates for Session 2 across various scenarios). The power analysis calculated power to detect a 0.3 difference on a 7-point scale (assuming a standard deviation of 1) under different scenarios with variation in FDR, within-person correlation between Session 1 and 2, and sample allocation. A conservative 50% retention was used as an assumption between Session 1 to Session 2 for the purposes of the power calculations. Power calculations were computed using 100 simulations for each sample allocation in SAS v9.4.

Across various assumptions of correlation and FDR, the power analysis showed generally higher levels of power using an optimized sample allocation with between 1,760 and 2,400 participants assigned to the control condition at Session 1 (880-1,200 participants at Session 2, assuming 50% retention).

Based on this analysis showing that higher power is achieved with an unbalanced allocation, the final design allocates 2,080 to the control group and 480 to each treatment group at Session 1.

Table 2. Study Power by Sample Allocation at Session 2

Sample Allocation at Session 2		Correlation	FDR					Unadjusted Power
Control	Treatment		0.05	0.1	0.15	0.2	0.25
287	287	0	0.60	0.71	0.83	0.89	0.91	0.68
880	250	0	0.80	0.89	0.94	0.95	0.97	0.85
1,200	230	0	0.77	0.87	0.90	0.94	0.95	0.81
1,520	210	0	0.88	0.92	0.94	0.96	0.98	0.88
1,840	190	0	0.72	0.84	0.88	0.92	0.93	0.78
287	287	0.2	0.64	0.77	0.83	0.89	0.90	0.73
880	250	0.2	0.95	0.95	0.97	0.99	0.99	0.96
1,200	230	0.2	0.89	0.94	0.95	0.98	1.00	0.91
1,520	210	0.2	0.87	0.92	0.94	0.96	0.98	0.89
1,840	190	0.2	0.85	0.88	0.94	0.94	0.99	0.87
287	287	0.4	0.85	0.91	0.96	0.98	0.99	0.89
880	250	0.4	0.97	0.98	0.99	0.99	1.00	0.98
1,200	230	0.4	0.97	1.00	1.00	1.00	1.00	0.97
1,520	210	0.4	0.95	0.97	0.97	0.97	0.98	0.95
1,840	190	0.4	0.91	0.94	0.97	0.97	0.97	0.91
287	287	0.6	0.97	0.98	0.98	1.00	1.00	0.98
880	250	0.6	1.00	1.00	1.00	1.00	1.00	1.00
1,200	230	0.6	1.00	1.00	1.00	1.00	1.00	1.00
1,520	210	0.6	1.00	1.00	1.00	1.00	1.00	1.00
1,840	190	0.6	1.00	1.00	1.00	1.00	1.00	1.00
287	287	0.8	1.00	1.00	1.00	1.00	1.00	1.00
880	250	0.8	1.00	1.00	1.00	1.00	1.00	1.00
1,200	230	0.8	1.00	1.00	1.00	1.00	1.00	1.00
1,520	210	0.8	1.00	1.00	1.00	1.00	1.00	1.00
1,840	190	0.8	1.00	1.00	1.00	1.00	1.00	1.00

Note: FDR = False Discovery Rate

3. Methods to Maximize Response Rates and Deal with Non-response

Experience with online experimental studies suggests that about 10 percent of adults and six percent of adolescents who are sent invitations will complete a study. FDA will ensure that the contractors implement several procedures to maximize participation. Specifically, the following procedures will be used to maximize cooperation and to achieve the desired response rates:

• A brief introductory paragraph will identify FDA as the sponsor of the study, state the purpose of the study, and encourage participation.

• Lightspeed will provide toll-free telephone numbers to all sampled individuals and invite them to call with any questions or concerns about any aspect of the study. RTI will provide a toll-free telephone number for an RTI project member and a toll-free telephone number for the RTI IRB hotline should participants have any questions about the study or their rights as a study participant.

• Lightspeed data collection staff will conduct ongoing monitoring of response levels and drop-off rates and will work with project staff to address any problems that arise throughout the course of the collection of information.

• Non-respondents will receive an initial email invitation and up to two email reminders from Lightspeed requesting their participation in the study.

• In recruiting panelists, Lightspeed uses a double opt-in registration process whereby panelists are invited to participate and then must sign up through an opt-in confirmation email. This process protects against fraudulent account registrations and ensures that panelists are actively motivated to participate in studies.

• As part of the registration process, panelists provide information about a range of sociodemographic characteristics, including age and smoking status, which can be used to target particular groups in study recruitment and maximize eligible responses. Lightspeed actively manages panelist profiles, requesting updated information on an ongoing basis to ensure that profile information is up to date.

• Participants will be informed that personally identifiable information (PII) is not linked to data and is not shared with RTI or FDA.

In addition, as discussed in Part A, this study will also employ incentives to increase participation and reduce attrition. The incentive plan calls for equal incentives ($10, in the form of Lightspeed’s “Lifepoints”) at each wave, even though questionnaire lengths vary, for a total potential incentive of $30. Total time for the three sessions is expected to be about 27 minutes.

As with any study conducted using opt-in online panels, this study may be subject to

several threats to external validity that limit the generalizability of study results. Panelists are recruited into the online panel using convenience sampling methods and thus do not have a known probability of selection into the panel. Recruitment of the study sample from the online panel is also subject to bias resulting from potential differences between responders (i.e., panelists who received the invitation and opted to participate in our study) and non-responders (i.e., panelists who were invited but chose not to participate) in characteristics that may be associated with study outcomes. Because of these limitations, the relationship between treatment and outcomes found in this study may not generalize to the broader U.S. population. Nevertheless, the experimental design of the study, including random assignment to condition, enhances the internal validity of the study (i.e., the ability to establish a causal relationship between treatment and outcomes). While random assignment does not rule out every threat to internal validity, it does rule out or minimize most. In this case, the internal validity threat that potentially remains is differential attrition between treatment and control groups (i.e., differences between treatment and control groups in the frequency of respondents who begin the study but drop out before completing it). However, there is no reason to believe this design would lead to those in the treatment groups dropping out of the study at a different frequency than those in the control group. In addition to randomization, the design, measures, power, and analysis plan are appropriate to ensure that we can draw valid statistical conclusions about the relationship between treatment and outcomes.

We do not anticipate significant item non-response and thus have no plans to utilize imputation procedures. All analyses will be conducted using Stata 14.1 with specific estimators determined by the measurement of the outcome variable and model used.

4. Test of Procedures or Methods to be Undertaken

Measures included are drawn from previously used and/or validated instruments to ensure that instruments are not ambiguous, burdensome, or confusing. Stimuli used in the study have been previously tested in a study conducted between May and June 2015, entitled “Qualitative Study on Cigarettes and Smoking: Knowledge, Beliefs, and Misperceptions,” OMB #0910-0674, and revised based on feedback from respondents representing the same populations as will be included in this study. Additionally, there will be two pretests conducted with panelists from Lightspeed to thoroughly test the programmed questionnaires and ensure there are no programming issues that affect the quality of the data. At the conclusion of the pretest, all strategies, algorithms, and programs for sampling, administration, and data compilation will be tested, validated, and readied for launch of the main data collection. The questionnaires and study protocol will be revised, if necessary, based on the pretest findings.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

RTI International will manage the information collection on behalf of FDA. Dr. James Nonnemaker is the project director at RTI. RTI will subcontract to Lightspeed to collect the data. John Garvey is the project manager at Lightspeed.

Analysis and dissemination of the data will be led by Dr. David Portnoy at FDA’s Center for Tobacco Products and Dr. James Nonnemaker at RTI International.

David B. Portnoy, Ph.D., MPH

Food and Drug Administration

Center for Tobacco Products

Document Control Center

Building 71, Room G335

10903 New Hampshire Avenue

Silver Spring, MD 20993-0002

(301) 796-9298

david.portnoy@fda.hhs.gov

James M. Nonnemaker, Ph.D.

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709-2194

919-541-7064

jnonnemaker@rti.org

John Garvey

501 Boylston Street, Suite 6101

Boston, MA 02116

(617) 733-2809  

John.Garvey@lightspeedresearch.com

1 Benjamini Y & Hochberg Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J Roy Stat Soc. Series B, 57(1), pp. 289–300.

2 Id.

2