Supporting Statement B

Information Collection Request

New

National Tobacco Education Campaign

Rough Cut Testing of Television Advertisements

(OMB No. 0920-0910)

Supporting Statement: Part B

Program Official/Contact

Michelle O’Hegarty, PhD

Office on Smoking and Health

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

4770 Buford Highway, NE MS F-79

Atlanta, Georgia 30341

770-488-5582

FAX: 770-488-5939

Email: mohegarty@cdc.gov

TABLE OF CONTENTS

B. STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

2. Procedures for the Collection of Information 3. Methods to Maximize Response Rates and Deal with No response

4. Tests of Procedures or Methods to be Undertaken

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or

Analyzing Data

LIST OF ATTACHMENTS

Attachment 1: Online Questionnaire Email Invitation to Potential Respondents

Attachment 2: Online Questionnaire Recruitment Screener

Attachment 3: Online Questionnaire

Attachment 4: Toluna Terms and Conditions

Attachment 5: Battelle Institutional Review Board Approval

Attachment 6: Toluna Privacy Policy

Notes on Excluded Attachments. In this information collection request (ICR), CDC outlines a plan to test rough cut advertisements with content that may be considered sensitive. The draft materials are not included because the near final “rough cut” advertisements have not been approved for public distribution by HHS/Assistant Secretary for Public Affairs (ASPA). To support adequate review of this Gen IC by OMB, the Centers for Disease Control and Prevention requests permission to provide OMB with a secure link to the draft materials.  

Part B: Statistical Methods

B.1 Respondent Universe and Sampling Methods

This is a request for a quantitative data collection. In this GenIC, the Centers for Disease Control and Prevention (CDC) requests OMB approval to collect information for rough cut testing of seven rough cut advertisements (ads) developed for the 2018 Tips From Former Smokers^® (Tips^®)¹ campaign. This testing will help ensure that the ads are credible, persuasive, clear, and will motivate the largest number of cigarette smokers to quit smoking conventional cigarettes completely. The proposed information collection will involve testing rough cut ads among adult cigarette smokers and nonsmokers 18-54 years old. The Plowshare Group, and subcontractors, Qualtrics, and Battelle, will conduct the data collection and analysis for this study.

The sample of respondents for the quantitative data collection will be drawn from Toluna’s online panel (see http://www.toluna-group.com//choose-the-people#global-reach for more detail on this panel). Toluna employs a double opt-in process for individuals to participate in a survey – they must opt-in to become panelists, and they must also opt-in to each survey. For this study, Toluna will gather information for pre-screening as well as ask participants to review both a privacy policy and a terms and conditions statement that outlines the parameters for their participation. The pre-screening is designed to screen out persons < 18 and > 54 years of age for this study. Although the sample will be a convenience sample, the panel sample has a reasonable degree of diversity in key demographic characteristics such as age, gender, region of residence, race/ethnicity, education, and income. The findings from this study will have high internal validity, but are not expected to be widely generalizable to the universe of smokers and nonsmokers in the U.S. As this study is part of formative work for campaign development and planning, these methods are not intended to generate nationally-representative samples or precise estimates of population parameters. However, the design allows for high internal validity to provide information on the perceived effectiveness of rough cut ads under test.

A power analysis was run to determine the sample size needed to detect descriptive differences on key measures (e.g., average perceived effectiveness scores for each rough cut ad). The sample size calculation was completed using the following parameters: (1) power = 80%; (2) two-tailed; (3) effect size, Cohen’s f = 0.10; and (4) adjustment for multiple comparisons based on a Bonferroni adjusted α error rate of p=0.00238 for seven “groups” (rough cut ads). Based on these assumptions, a minimum of 666 persons per ad are needed (333 cigarette smokers and 333 nonsmokers). This yields a total sample size of 4,662 (2,331 cigarette smokers and 2,331 nonsmokers). The overall sample size will allow us to assess differences descriptively between the seven ads, among both cigarette smokers and nonsmokers.

To achieve this sample size, we conservatively anticipate screening 5,863; this estimate is based on two factors from prior experiences in the field. First it is anticipated that roughly 18 percent of screener respondents (n=1,056) will be deemed ineligible for the study because of not meeting inclusion criteria.  Second, of those deemed eligible, an estimated additional approximately three percent (n=145) will start but not complete the questionnaire. Thus, 4,807 respondents are needed to obtain 4,662 in the final sample.

Figure 2. Flowchart of the Sampling of Participants for Quantitative Data Collection

B.2 Procedures for the Collection of Information

The data collection subcontractor, Qualtrics, will be responsible for coordinating data collection activities, collecting and summarizing information, and preparing topline reports. Battelle will prepare final reports, in collaboration with Qualtrics. Information for this study will be collected using quantitative methods. The testing will collect information about the respondents’ reactions to the rough cut ads, as well as basic demographic and cigarette use information.

Recruitment and Screening

Respondents will be recruited from the Toluna online panel, which is an established, online panel system that includes more than 1.7 million people in the U.S. Toluna has profiled their panels regarding smoking behavior and can target and identify respondents who are pre-identified as being in the age range of 18-54 years for the survey. Toluna also collects demographics such as gender and ethnicity. However, additional screening will be conducted to confirm that Toluna’s profiling information is current and to assess whether any information has changed (i.e., educational status, state of residence). To protect potentially identifiable information (PII) of respondents, no comparison to the original individual profiling data will be made after assessment. The screener and online questionnaire will be hosted on Qualtrics’ server farm.

Individuals who are enrolled in the online panel will be sent an invitation to participate in the study using an Email Invitation (Attachment 1) provided by Qualtrics’ sample management system. The invitation will include a link behind a “Start” button, with the link going to a web page that contains the Screener (Attachment 2). In addition, the same-worded invitation will be sent at regular intervals after the original invitation is sent to those respondents who have not yet responded. If the potential respondent agrees to participate in the study, he or she will click the “Start” button.

Approximately 5,863 potential respondents are anticipated to complete the screener, and 4,807 respondents in the age range of 18-54 years will then continue to the Online Questionnaire (Attachment 3). Criteria for being eligible for the questionnaire are:

1. Adult cigarette smoker criteria: persons between 18-54 years of age who reported smoking > 100 traditional cigarettes during their lifetime and who, at the time of the survey, reported smoking traditional cigarettes every day or some days, and had smoked at least one cigarette in the past 30 days.

2. Adult nonsmoker criteria: persons between 18-54 years of age who did not currently smoke, and had not smoked a traditional cigarette in the past 30 days.

If the respondent does not meet the eligibility criteria assessed during screening, he or she will be routed to a page that thanks the respondent, but indicates that he or she does not fit the specific criteria needed for this study. The page that thanks the respondent is located in the respondent’s panel system outside of the survey. It is estimated that in total 1,056 respondents will be terminated after completing the screener. Criteria for termination are:

1. Persons younger than 18 years of age or older than 54 years of age.

Survey Administration

Participants who meet basic eligibility criteria will be routed to the Online Questionnaire (Attachment 3). A preamble to the questionnaire states the length of the survey and provides other information about the survey. The questionnaire will include questions regarding demographic characteristics and smoking behavior. Participants will be randomly assigned to view one of the seven rough cut ads. Randomization of participants to view the different rough cut ads being tested ensures that there is a similar distribution of individuals of different characteristics (e.g., age, gender, etc.) across the different ads. The number of persons viewing each ad is approximately even to ensure descriptive comparisons. Each rough cut ad will be in a video file. Participants can click the “restart” button to view the video as many times as they would like. Then, thumbnail pictures representing the video will accompany the aided response section.

Approximately 145 participants are expected to discontinue the questionnaire before completing it. Due to identity protection technology, it will not be possible for anyone to enter the survey who has not been recruited or for a respondent to complete the survey more than once.

B.3 Methods to Maximize Response Rates and Deal with No Response

Five methods will be used to maximize response or completion rates in this current study: (1) inviting only those who have been profiled as being in the target age range of 18-54 to take the survey to reduce the proportion of “incomplete” responses due to not meeting the inclusion criteria, (2) identifying the CDC as the agency of record, since this agency is credible and serves the public good, and (3) drafting the invitations in a manner that has been shown, through prior testing, to yield optimal results. For the email invitation, this includes attending to the following: types of subject lines, topic description, survey details, incentive description, and format (html vs. text) that elicits the most favorable response rates. Additionally, (4) survey responses during the field period will be closely monitored and the invitation email will be resent, with the same content as the original invite, to eligible participants who have not yet responded. Finally, (5) to encourage participation and thus maximize the response rate, the participants who complete the questionnaire will receive a point-based incentive, which can be redeemed for other items, such as Amazon gift cards.

B.4 Test of Procedures or Methods to be Undertaken

The proposed project involves the collection of quantitative information. Similar procedures were used to conduct rough cut testing of the ads developed for the 2016 Tips^® campaign under this same generic clearance, specifically, GenIC #15 titled “National Tobacco Prevention and Control Public Education Campaign: Rough Cut Testing of Television, Radio, Print, and Digital Advertisements for the 2016 Tips^® Campaign.”

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Primary responsibility for methodological design, data collection, and data analysis will be performed by Carol Haney and Steven Snell from Qualtrics, and Lisa John, Amanda Berger, and Robert Alexander from Battelle, whose information is listed below.

Carol Sue Haney

Senior Research and Data Scientist

Qualtrics

333 West River Park Drive

Provo, UT 84604

Phone: (802) 258-0518

Email: carolh@qualtrics.com

Steven Snell, PhD

Principal Research Scientist

Qualtrics

333 West River Park Drive

Provo, UT 84604

Phone: (616) 502-4828

Email: davev@qualtrics.com

Lisa V. John, PhD, PMP

Project/Program Manager

Battelle

5712 Oakland Ave

St. Louis, MO 63110

Phone: (314) 880-3652

Email: johnl@battelle.org

Amanda Berger, PhD

Principal Research Scientist

Battelle

6115 Falls Rd, Suite 200

Baltimore, MD 21209

Phone: (410) 372-2757

Email: bergera@battelle.org

Robert Alexander, PhD, MPH, CHES

Director, Public Health Research & Translational Science

Battelle

6115 Falls Rd, Suite 200

Baltimore, MD 21209

Phone: (404) 460-1462

Email: alexander@battelle.org

Individuals consulted at CDC on the study design are listed below.

Centers for Disease Control and Prevention Office on Smoking and Health 4770 Buford Highway, N.E MS F-79 Atlanta, GA 30341
Brian Armour	Associate Director for Science, Office of the Associate Director for Science	Phone: 404.498.3014 E-mail: bka9@cdc.gov
Israel Agaku	Senior Service Fellow, Office of the Associate Director for Science	Phone: 770.488.5138 E-mail:  wgn9@cdc.gov
Satomi Odani	Oak Ridge Institute for Science and Education Fellow, Office of the Associate Director for Science	Phone: 404.649.2586 Email: lpu7@cdc.gov
Diane Beistle	Chief, Health Communications Branch	Phone: 770.488.5066 E-mail:  zvg1@cdc.gov
Michelle O’Hegarty	Health Communication Specialist, Health Communications Branch	Phone: 770.488.5582 E-mail:  mohegarty@cdc.gov
Lindsey McCarter	Team Lead, Health Communications Branch	Phone: 770.488.4239 E-mail:  lpq4@cdc.gov



1 Use of trade names is for identification only and does not imply endorsement by the US Department of Health and Human Services.

9