Supporting Statement B

Information Collection Request

New

National Tobacco Education Campaign

Rough Cut Testing of Television Advertisements

(OMB No. 0920-0910)

Supporting Statement: Part B

Program Official/Contact

Michelle O’Hegarty, PhD

Office on Smoking and Health

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

Atlanta, Georgia 30341

770-488-5582

FAX: 770-488-5939

Email: mohegarty@cdc.gov

TABLE OF CONTENTS

B. STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

2. Procedures for the Collection of Information

3. Methods to Maximize Response Rates and Deal with No response

4. Tests of Procedures or Methods to be Undertaken

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or

Analyzing Data

LIST OF ATTACHMENTS

Attachment 1: Online Questionnaire Email Invitation to Potential Respondents

Attachment 2: Online Questionnaire Recruitment Screener

Attachment 3: Online Questionnaire

Attachment 4: Online Questionnaire Main Screen Shots

Attachment 5: Toluna Terms and Conditions

Attachment 6: Battelle Institutional Review Board Approval

Attachment 7: Toluna Privacy Policy

Notes on Excluded Attachments. In this information collection request (ICR), CDC outlines a plan to test rough cut advertisements with content that may be considered sensitive. The draft materials are not included because the near final “rough cut” advertisements have not been approved for public distribution by HHS/Assistant Secretary for Public Affairs (ASPA). To support adequate review of this Gen IC by OMB, the Centers for Disease Control and Prevention requests permission to provide OMB with a secure link to the draft materials.  

Part B: Statistical Methods

B.1 Respondent Universe and Sampling Methods

This is a request for a quantitative data collection. In this GenIC, the Centers for Disease Control and Prevention (CDC) requests OMB approval to collect information for rough cut testing of thirteen rough cut advertisements (ads) developed for the 2019 Tips From Former Smokers^® (Tips^®)¹ campaign. This testing will provide information that will inform whether changes must be made to the ads before finalizing, to help ensure that the ads are credible, persuasive, clear, and will motivate the largest number of cigarette smokers to quit smoking conventional cigarettes completely. The proposed information collection will involve testing rough cut ads among adult cigarette smokers and nonsmokers 18-54 years old. The Plowshare Group, and subcontractors, Qualtrics, and Battelle, will conduct the data collection and analysis for this proposed project.

The sample of respondents for the quantitative data collection will be drawn from Toluna’s online panel (see http://www.toluna-group.com//choose-the-people#global-reach for more detail on this panel). Toluna employs a double opt-in process for individuals to participate in a survey – they must opt-in to become panelists, and they must also opt-in to each survey. For this proposed project, Toluna will gather information for pre-screening as well as ask participants to review both a privacy policy and a terms and conditions statement that outlines the parameters for their participation. The pre-screening is designed to screen out persons < 18 and > 54 years of age for this proposed project. Although the sample will be a convenience sample, the panel sample has a reasonable degree of diversity in key demographic characteristics such as age, gender, region of residence, race/ethnicity, education, and income. The findings from this proposed project will have high internal validity, but are not expected to be widely generalizable to the universe of smokers and nonsmokers in the U.S. As this proposed project is part of formative work for campaign development and planning, these methods are not intended to generate nationally-representative samples or precise estimates of population parameters. However, the design allows for high internal validity to provide information on the perceived effectiveness of rough cut ads under test.

Power analyses were run to determine the sample sizes needed to detect statistically significant differences on key measures (e.g., PE scores for each rough cut ad). The sample size calculations were completed using the following parameters: (1) power = 80%; (2) two-tailed; (3) effect size, Cohen’s f = 0.10 or Cohen’s d = 0.20; and (4) adjustment for multiple comparisons based on a Bonferroni adjusted α error rate. To detect significant differences for the six :30s ads plus Wilma between and within subpopulations (smokers vs. nonsmokers; young adults vs. older adults; low-SES vs. non-low-SES), 2,364 respondents must view each ad. Specifically, at least 394 respondents from the following groups must view each ad:

• Young adult smokers (18-26 years of age)

• Young adult nonsmokers (18-26 years of age)

• Older adult smokers (27-54 years of age)

• Older adult nonsmokers (27-54 years of age)

• Low-socioeconomic status smokers (18-54 years of age)

• Non-low-SES smokers (18-54 years of age)

To detect statistically significant differences between the six :15s ads, a minimum of 353 low-SES smoker respondents must view each ad. Overall, a minimum of 16,548 respondents from all subpopulation groups are needed for the six :30s ads and Wilma, and an additional 2,118 low-SES smoker respondents are needed for the remaining six :15s ads, for a total sample size of 18,666 cigarette smokers and nonsmokers. This sample size will allow for tests of statistical significance between and within subpopulations (smoker vs. non-smoker; young adult vs. older adult; low-SES vs. non-low-SES) for the :30s ads and Wilma. The sample size will also allow for tests of statistical significance in ad reactions to the :15s ads among low-SES smokers.² Table B.2 provides additional information by subpopulation about the ads to be tested, the media format and length of the ad, the number of ads, and the minimum number of respondents associated with each ad to be tested.

To achieve this sample size, we conservatively anticipate screening 23,468; this estimate is based on two factors from prior experiences in the field. First, it is anticipated that roughly 18 percent of screener respondents (n=4,225) will be deemed ineligible for the proposed project because of not meeting inclusion criteria.  Second, of those deemed eligible (n=19,244),³ an estimated three percent (n=578) will start but not complete the questionnaire. Thus, 23,468 respondents are needed to obtain 18,866 in the final sample.

Table B.1 Minimum Sample Sizes from Respondent Subpopulations to View Each Ad
Ad & Media Format	Low-SES Smokers (18-54)	Non-Low SES Smokers (18-54)	Young Adult Smokers (18-26)	Older Adult Smokers (27-54)	Young Adult Nonsmokers (18-26)	Older Adult Nonsmokers (27-54)	Totals by Ad
Beatrice :30s	394	394	394	394	394	394	2,364
Beatrice :15s	353						353
Beatrice Total	2,717
Christine A :30s	394	394	394	394	394	394	2,364
Christine A :15s	353						353
Christine A Total	2,717
Christine B :30s	394	394	394	394	394	394	2,364
Christine B :15s	353						353
Christine B Total	2,717
Terrie :30s	394	394	394	394	394	394	2,364
Terrie :15	353						353
Terrie Total	2,717
Dana :30s	394	394	394	394	394	394	2,364
Dana :15s	353						353
Dana Total	2,717
Leonard Nimoy :30s	394	394	394	394	394	394	2,364
Leonard Nimoy :15s	353						353
Leonard Nimoy Total	2,717
Wilma :15s	394	394	394	394	394	394	2,364
Wilma Total	2,364
Total	4876	2,758	2,758	2,758	2,758	2,758	18,666

¹For ads with both :15s and :30s versions, respondents from all subpopulations (defined by smoking status, age, and SES) will see the :30s versions. Only low-SES smokers will see the :15s versions of these ads.

²There is no :30s version of the Wilma ad. Therefore, respondents from all subpopulations will see the :15s version of this ad.

Figure B.1 Flowchart of the Sampling of Respondents for Quantitative Data Collection

B.2 Procedures for the Collection of Information

The data collection subcontractor, Qualtrics, will be responsible for coordinating data collection activities, collecting and summarizing information, and preparing topline reports. Battelle will prepare final reports, in collaboration with Qualtrics. Information for this proposed project will be collected using quantitative methods. The testing will collect information about the respondents’ reactions to the rough cut ads, as well as basic demographic and cigarette use information.

Recruitment and Screening

Respondents will be recruited from the Toluna online panel, which is an established, online panel system that includes more than 1.7 million people in the U.S. Toluna has profiled their panels regarding smoking behavior and can target and identify respondents who are pre-identified as being in the age range of 18-54 years for the survey. Toluna also collects demographics such as gender and ethnicity. However, additional screening will be conducted to confirm that Toluna’s profiling information is current and to assess whether any information has changed (i.e., educational status, state of residence). To protect potentially identifiable information (PII) of respondents, no comparison to the original individual profiling data will be made after assessment. The screener and online questionnaire will be hosted on Qualtrics’ server farm (Attachment 4).

Individuals who are enrolled in the online panel will be sent an invitation to participate in the proposed project using an Email Invitation (Attachment 1) provided by Qualtrics’ sample management system. The invitation will include a link behind a “Start” button, with the link going to a web page that contains the Screener (Attachment 2). In addition, the same-worded invitation will be sent at regular intervals after the original invitation is sent to those respondents who have not yet responded. If the potential respondent agrees to participate in the proposed project, he or she will click the “Start” button.

Approximately 23,468 potential respondents are anticipated to complete the screener, and 19,244 respondents in the age range of 18-54 years will then continue to the Online Questionnaire (Attachment 3). Criteria for being eligible for the questionnaire are:

1. Adult cigarette smoker criteria: persons between 18-54 years of age who reported smoking > 100 traditional cigarettes during their lifetime and who, at the time of the survey, reported smoking traditional cigarettes every day or some days, and had smoked at least one cigarette in the past 30 days.

2. Adult nonsmoker criteria: persons between 18-54 years of age who did not currently smoke, and had not smoked a traditional cigarette in the past 30 days.

If the respondent does not meet the eligibility criteria assessed during screening, he or she will be routed to a page that thanks the respondent, but indicates that he or she does not fit the specific criteria needed for this proposed project. The page that thanks the respondent is located in the respondent’s panel system outside of the survey. It is estimated that in total 4,225 respondents will be terminated after completing the screener. Criteria for termination are:

1. Persons younger than 18 years of age or older than 54 years of age.

Survey Administration

Participants who meet basic eligibility criteria will be routed to the Online Questionnaire (Attachment 3). A preamble to the questionnaire states the length of the survey and provides other information about the survey. The questionnaire will include questions regarding demographic characteristics and smoking behavior. Participants will be randomly assigned to view one of the thirteen rough cut ads. Randomization of participants to view the different rough cut ads being tested ensures that there is a similar distribution of individuals of different characteristics (e.g., age, gender, etc.) across the different ads. Each rough cut ad will be in a video file. Participants can click the “restart” button to view the video as many times as they would like. Then, thumbnail pictures representing the video will accompany the aided response section.

Approximately 578 participants are expected to discontinue the questionnaire before completing it. Due to identity protection technology, it will not be possible for anyone to enter the survey who has not been recruited or for a respondent to complete the survey more than once.

B.3 Methods to Maximize Response Rates and Deal with No Response

Five methods will be used to maximize response or completion rates in this current proposed project: (1) inviting only those who have been profiled as being in the target age range of 18-54 to take the survey to reduce the proportion of “incomplete” responses due to not meeting the inclusion criteria, (2) identifying the CDC as the agency of record, since this agency is credible and serves the public good, and (3) drafting the invitations in a manner that has been shown, through prior testing, to yield optimal results. For the email invitation, this includes attending to the following: types of subject lines, topic description, survey details, incentive description, and format (html vs. text) that elicits the most favorable response rates. Additionally, (4) survey responses during the field period will be closely monitored and the invitation email will be resent, with the same content as the original invite, to eligible participants who have not yet responded. Finally, (5) to encourage participation and thus maximize the response rate, the participants who complete the questionnaire will receive a point-based incentive, which can be redeemed for other items, such as Amazon gift cards.

B.4 Test of Procedures or Methods to be Undertaken

The proposed project involves the collection of quantitative information. Similar procedures were used to conduct rough cut testing of the ads developed for the 2017 Tips campaign under this generic clearance, specifically, Agency GenIC # 19 titled titled “National Tobacco Education Campaign Rough Cut Testing of Television Advertisements (0MB No. 0920-0910).”

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Primary responsibility for methodological design, data collection, and data analysis will be performed by Carol Haney and Steven Snell from Qualtrics, and Lisa John, Amanda Berger, and Tesfa Alexander from Battelle, whose information is listed below.

Carol Sue Haney

Senior Research and Data Scientist

Qualtrics

333 West River Park Drive

Provo, UT 84604

Phone: (802) 258-0518

Email: carolh@qualtrics.com

Steven Snell, PhD

Principal Research Scientist

Qualtrics

333 West River Park Drive

Provo, UT 84604

Phone: (616) 502-4828

Email: davev@qualtrics.com

Lisa V. John, PhD, PMP

Project/Program Manager

Battelle

5712 Oakland Ave

St. Louis, MO 63110

Phone: (314) 880-3652

Email: johnl@battelle.org

Amanda Berger, PhD

Principal Research Scientist

Battelle

2111 Wilson Boulevard, Suite 1000

Arlington, VA 22201

Phone: (703) 875-2152

Email: bergera@battelle.org

Tesfa Alexander, PhD

Senior Research Scientist

Battelle

6115 Falls Rd, Suite 200

Baltimore, MD 21209

Phone: (404) 215-4405

Email: alexandert@battelle.org

Individuals consulted at CDC on the proposed project design are listed below.

Centers for Disease Control and Prevention Office on Smoking and Health 4770 Buford Highway, N.E MS F-79 Atlanta, GA 30341
Brian Armour	Associate Director for Science, Office of the Associate Director for Science	Phone: 404.498.3014 E-mail: bka9@cdc.gov
Israel Agaku	Senior Service Fellow, Office of the Associate Director for Science	Phone: 770.488.5138 E-mail:  wgn9@cdc.gov
Elizabeth Courtney-Long	Health Scientist, Office of the Associate Director for Science	Phone: 404.498.0264 E-mail: gmr9@cdc.gov
Lauren Boyle-Estheimer	Health Communications Scientist, Health Communications Branch	Phone: 404-498-2283 E-mail: yjw7@cdc.gov
Diane Beistle	Chief, Health Communications Branch	Phone: 770.488.5066 E-mail:  zvg1@cdc.gov
Michelle O’Hegarty	Health Communication Specialist, Health Communications Branch	Phone: 770.488.5582 E-mail:  mohegarty@cdc.gov
Lindsey McCarter	Team Lead, Health Communications Branch	Phone: 770.488.4239 E-mail:  lpq4@cdc.gov



1 Use of trade names is for identification only and does not imply endorsement by the U.S. Department of Health and Human Services.

2 The sample sizes for low-SES smokers will also allow for tests of statistical significance between all 13 ads seen by this subpopulation.

3 These numbers (19,244 and 4,225) do not sum to 23,468 due to rounding error.

22