Supporting Statement Part B

Information Collection Request

New

National Tobacco Education Campaign

Rough-Cut Testing of Television, Radio, Print, and Digital Advertisements

(OMB No. 0920-0910)

Supporting Statement: Part B

Program Official/Contact

Michelle O’Hegarty, PhD

Office on Smoking and Health

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

Atlanta, Georgia 30341

770-488-5582

FAX: 770-488-5939

Email: mohegarty@cdc.gov

May 7, 2020

TABLE OF CONTENTS

B. STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

2. Procedures for the Collection of Information

3. Methods to Maximize Response Rates and Deal with No response

4. Tests of Procedures or Methods to be Undertaken

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or

Analyzing Data

LIST OF ATTACHMENTS

Attachment 1: Online Questionnaire Email Invitation to Potential Respondents

Attachment 2: Online Questionnaire Recruitment Screener

Attachment 3: Online Questionnaire

Attachment 4: Online Questionnaire Main Screenshots

Attachment 5: Toluna Terms and Conditions

Attachment 6: Battelle Institutional Review Board Approval

Attachment 7: Toluna Privacy Policy

Notes on Excluded Attachments. In this information collection request (ICR), CDC outlines a plan to test rough-cut advertisements (ads) with content that may be considered sensitive. The draft materials are not included because the near final, “rough-cut” ads have not been approved for public distribution by HHS/Assistant Secretary for Public Affairs (ASPA). To support adequate review of this Gen IC by OMB, the Centers for Disease Control and Prevention requests permission to provide OMB with a secure link to the draft materials.  

Part B: Statistical Methods

B.1 Respondent Universe and Sampling Methods

This is a request for a quantitative data collection. In this GenIC, the Centers for Disease Control and Prevention (CDC) requests OMB approval to collect information for rough cut testing of 10 rough-cut ads developed for a future Tips From Former Smokers^® (Tips^®)¹ campaign. This testing will provide information that will inform whether changes must be made to the ads before finalizing, to help ensure that the ads are credible, persuasive, clear, and will motivate the largest number of people who smoke to quit smoking conventional cigarettes completely. The proposed information collection will involve testing rough-cut ads among adults who smoke and adults who don’t smoke, 18-54 years of age. The Plowshare Group, and subcontractors Qualtrics and Battelle, will conduct the data collection and analysis for this proposed project.

The sample of respondents for the quantitative data collection will be drawn from Toluna’s online panel (see https://us.toluna.com/#/ for more detail on this panel). Toluna employs a double opt-in process for individuals to participate in a survey – they must opt-in to become panelists, and they must also opt-in to each survey. For this proposed project, Toluna will gather information for pre-screening as well as ask participants to review both a privacy policy and a terms and conditions statement that outlines the parameters for their participation. The pre-screening is designed to screen out persons < 18 and > 54 years of age for this proposed project. Although the sample will be a convenience sample, the panel sample has a reasonable degree of diversity in key demographic characteristics such as age, gender, region of residence, race/ethnicity, education, and income. The findings from this proposed project will have high internal validity, but are not expected to be widely generalizable to the universe of people who smoke and people who don’t smoke in the U.S. As this proposed project is part of formative work for campaign development and planning, these methods are not intended to generate nationally representative samples or precise estimates of population parameters. However, the design allows for high internal validity to provide information on the perceived effectiveness of rough-cut ads under test.

Power analyses were run to determine the sample sizes needed to detect statistically significant differences on key measures (e.g., PE scores for each rough-cut ad) for people who smoke

and people who don’t smoke based on the observed differences in previous rounds of rough- cut testing.

The primary outcome of interest is perceived effectiveness of ads among people who smoke. Due to the number of comparisons being proposed between ads across multiple media formats (TV, radio, print, digital), Bonferroni correction was applied to reduce the risk of false positives to 0.05 in people who smoke and people who don't smoke. To attain 80% power to detect a 0.20 standard deviation difference between any pair of ads, using a two-sided t-test, a total of 800 people who smoke and 800 people who don't smoke are needed per ad.

Another aim of this study is to compare ad perceptions across paired subpopulations (e.g. young adults who smoke and older adults who smoke). Stratified sampling will be used to target 400 participants from each subpopulation of adults who smoke and adults who don't smoke for each rough-cut ad.

Specifically, we are targeting a sample of 400 respondents to view or listen to each ad from the following groups:

• Young adults who smoke (18-26 years of age), half of whom will be low-SES and half of whom will be non-low-SES

• Older adults who smoke (27-54 years of age), half of whom will be low-SES and half of whom will be non-low-SES

• Young adults who don't smoke (18-26 years of age)

• Older adults who don't smoke (27-54 years of age)

Through stratified sampling, half of the young adults who smoke will be low-SES and half will be non-low-SES. Half of the older adults who smoke will be low-SES and half will be non-low-SES. Recruitment of people who don't smoke will also be stratified such that half of those recruited will be young adults who don't smoke and half will be older adults who don't smoke. Recruitment for each stratum will close when that stratum is filled. Using this method, the sample size needed to have adequate power to detect differences by SES and age among people who smoke, and by age among people who don't smoke, is minimized. This will provide 80% power to detect a 0.20 standard deviation difference between pairs of subpopulations of interest within each ad with an alpha of 0.05.

Overall, a sample size of 16,000 people who smoke and people who don’t smoke is needed. Table B.1 provides additional information by subpopulation about the ads to be tested, the format of the ad, the number of ads, and the target number of respondents associated with each ad to be tested. To achieve this sample size, we conservatively anticipate screening 20,116 respondents; this estimate is based on two factors from prior experiences in the field. First, it is anticipated that roughly 18 percent of screener respondents (n=3,621) will be deemed ineligible for the proposed project because of not meeting inclusion criteria.  Second, of those deemed eligible (n=16,495), an estimated three percent (n=495) will start but not complete the questionnaire. Thus, 20,116 respondents are needed to obtain 16,000 in the final sample (Figure B1).

Table B.1 Target Number of Respondents to View Each Ad
	Adults Who Smoke Cigarettes1					Adults Who Don’t Smoke Cigarettes2
	Young Adult, Low SES	Older Adult, Low SES	Young Adults, Non-Low SES	Older Adults, Non-Low SES	Total Adults Who Smoke Cigarettes	Young Adult	Older Adult	Total Adults Who Don’t Smoke Cigarettes	Total Adults Who Smoke Cigarettes and Adults Who Don’t Smoke Cigarettes
Television Ad A	200	200	200	200	800	400	400	800	1,600
Television Ad B	200	200	200	200	800	400	400	800	1,600
Television Ad C	200	200	200	200	800	400	400	800	1,600
Television Ad D	200	200	200	200	800	400	400	800	1,600
Radio Ad A	200	200	200	200	800	400	400	800	1,600
Radio Ad B	200	200	200	200	800	400	400	800	1,600
Radio Ad C	200	200	200	200	800	400	400	800	1,600
Radio Ad D	200	200	200	200	800	400	400	800	1,600
Print Ad	200	200	200	200	800	400	400	800	1,600
Digital Ad	200	200	200	200	800	400	400	800	1,600
All Ads					8,000			8,000	16,0003
¹ Adults who smoke cigarettes were defined as persons between 18-54 years of age who reported smoking > 100 traditional cigarettes during their lifetime and who, at the time of the survey, reported smoking traditional cigarettes every day or some days, and had smoked at least one cigarette in the past 30 days. 2 Adults who don’t smoke were defined as persons between 18-54 years of age who did not currently smoke and had not smoked a traditional cigarette in the past 30 days. 3 For a total final sample size of 16,000, it is estimated that a starting sample of 20,116 respondents will be needed.

Figure B.1 Flowchart of the Sampling of Respondents for Quantitative Data Collection*

*Respondents who are screened out include respondents from strata that have filled, after which recruitment is closed.

B.2 Procedures for the Collection of Information

The data collection subcontractor, Qualtrics, will be responsible for coordinating data collection activities, collecting and summarizing information, and preparing topline reports. Battelle will prepare final reports, in collaboration with Qualtrics. Information for this proposed project will be collected using quantitative methods. The testing will collect information about the respondents’ reactions to the rough-cut ads, as well as basic demographic and cigarette use information.

Recruitment and Screening

Respondents will be recruited from the Toluna online panel, which is an established, online panel system that includes more than 1.7 million people in the U.S. Toluna has profiled their panels regarding smoking behavior and can target and identify respondents who are pre-identified as being in the age range of 18-54 years for the survey. Toluna also collects demographics such as gender and ethnicity. However, additional screening will be conducted to confirm that Toluna’s profiling information is current and to assess whether any information has changed (i.e., educational status, state of residence). To protect potentially identifiable information (PII) of respondents, no comparison to the original individual profiling data will be made after assessment. The screener and online questionnaire will be hosted on Qualtrics’ server farm (Attachment 4).

Individuals who are enrolled in the online panel will be sent an invitation to participate in the proposed project using an Email Invitation (Attachment 1) provided by Qualtrics’ sample management system. The invitation will include a link behind a “Start” button, with the link going to a web page that contains the Screener (Attachment 2). In addition, the same-worded invitation will be sent at regular intervals after the original invitation is sent to those respondents who have not yet responded. If the potential respondent agrees to participate in the proposed project, he or she will click the “Start” button.

Approximately 20,116 potential respondents are anticipated to complete the screener, and 16,495 respondents in the age range of 18-54 years will then continue to the Online Questionnaire (Attachment 3). Criteria for being eligible for the questionnaire are:

1. Adult who smokes criteria: persons between 18-54 years of age who reported smoking > 100 traditional cigarettes during their lifetime and who, at the time of the survey, reported smoking traditional cigarettes every day or some days, and had smoked at least one cigarette in the past 30 days.

2. Adult who doesn't smoke criteria: persons between 18-54 years of age who did not currently smoke and had not smoked a traditional cigarette in the past 30 days.

If the respondent does not meet the eligibility criteria assessed during screening, he or she will be routed to a page that thanks the respondent but indicates that he or she does not fit the specific criteria needed for this proposed project. The page that thanks the respondent is located in the respondent’s panel system outside of the survey. It is estimated that in total 3,621 respondents will be terminated after completing the screener. Criteria for termination are:

1. Persons younger than 18 years of age or older than 54 years of age.

2. Recruitment is complete for participant subgroup (e.g. all older adults who don’t smoke needed for this project have already completed the survey).

Survey Administration

Participants who meet basic eligibility criteria will be routed to the Online Questionnaire (Attachment 3). A preamble to the questionnaire states the length of the survey and provides other information about the survey. The questionnaire will include questions regarding demographic characteristics and smoking behavior. Participants will be randomly assigned to view one of the 15 rough-cut ads. Randomization of participants to view the different rough-cut ads being tested ensures that there is a similar distribution of individuals of different characteristics (e.g., age, gender, etc.) across the different ads. Each rough-cut ad will be in a video file, audio file, or static image. Participants can click the “restart” button to view the video or listen to the audio as many times as they would like. Participants who view a static image will be allowed to view the image for as long as they would like. Then, thumbnail pictures representing the ad will accompany the aided response section.

Approximately 495 participants are expected to discontinue the questionnaire before completing it. Due to identity protection technology, it will not be possible for anyone to enter the survey who has not been recruited or for a respondent to complete the survey more than once.

B.3 Methods to Maximize Response Rates and Deal with No Response

Five methods will be used to maximize response or completion rates in this current proposed project: (1) inviting only those who have been profiled as being in the target age range of 18-54 to take the survey to reduce the proportion of “incomplete” responses due to not meeting the inclusion criteria, (2) identifying the CDC as the agency of record, since this agency is credible and serves the public good, and (3) drafting the invitations in a manner that has been shown, through prior testing, to yield optimal results. For the email invitation, this includes attending to the following: types of subject lines, topic description, survey details, incentive description, and format (html vs. text) that elicits the most favorable response rates. Additionally, (4) survey responses during the field period will be closely monitored and the invitation email will be resent, with the same content as the original invite, to eligible participants who have not yet responded. Finally, (5) to encourage participation and thus maximize the response rate, the participants who complete the questionnaire will receive a point-based incentive, which can be redeemed for other items, such as Amazon gift cards.

B.4 Test of Procedures or Methods to be Undertaken

The proposed project involves the collection of quantitative information. Similar procedures were used to conduct rough-cut testing of the ads developed for the 2020 Tips campaign under this generic clearance, specifically, Agency GenIC # 21 titled “National Tobacco Education Campaign Rough-Cut Testing of Television Advertisements (0MB No. 0920-0910).”

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Primary responsibility for methodological design, data collection, and data analysis will be performed by Carol Haney and Scott Riding from Qualtrics, and Lisa John, Abigail Evans, and Jess Wilhelm from Battelle, whose information is listed below.

Carol Sue Haney

Senior Research and Data Scientist

Qualtrics

333 West River Park Drive

Provo, UT 84604

Phone: (802) 258-0518

Email: carolh@qualtrics.com

Scott Riding, PhD

Principal Research Scientist

Qualtrics

333 West River Park Drive

Provo, UT 84604

Phone: (616) 502-4828

Email: sriding@qualtrics.com

Lisa V. John, PhD, PMP

Project/Program Manager

Battelle

505 King Ave

Columbus, OH 43201

Phone: (614) 456-5051

Email: johnl@battelle.org

Abigail Evans, PhD

Research Scientist

Battelle

6115 Falls Road, Suite 200

Baltimore, MD 21209

Phone: (410) 372-2750

Email: evansa@battelle.org

Jess Wilhelm, PhD

Principal Research Scientist

Battelle

6115 Falls Road, Suite 200

Baltimore, MD 21209

Phone: (410) 372-2726

Email: wilhelmj@battelle.org

Individuals consulted at CDC on the proposed project design are listed below.

Centers for Disease Control and Prevention Office on Smoking and Health 4770 Buford Highway, N.E MS F-79 Atlanta, GA 30341
Brian Armour	Associate Director for Science, Office of the Associate Director for Science	Phone: 404.498.3014 E-mail: bka9@cdc.gov
Diane Beistle	Chief, Health Communications Branch	Phone: 770.488.5066 E-mail:  zvg1@cdc.gov
Lauren Boyle-Estheimer	Health Communications Specialist, Health Communications Branch	Phone: 404-498-2283 E-mail: yjw7@cdc.gov
Elizabeth Courtney-Long	Health Scientist, Office of the Associate Director for Science	Phone: 404.498.0264 E-mail: gmr9@cdc.gov
Lindsey McCarter	Team Lead, Health Communications Branch	Phone: 770.488.4239 E-mail:  lpq4@cdc.gov
Michelle O’Hegarty	Health Communication Specialist, Health Communications Branch	Phone: 770.488.5582 E-mail:  mohegarty@cdc.gov
Sarah Lewis	Health Communication Specialist, Health Communications Branch	Phone: 770.488.7424 E-mail:  irr6@cdc.gov



1 Use of trade names is for identification only and does not imply endorsement by the U.S. Department of Health and Human Services.

10