Ssb

Information Collection Request

New

2020 National Tobacco Education Campaign

Rough Cut Testing of Television Advertisements

(OMB No. 0920-0910)

Supporting Statement: Part B

Program Official/Contact

Michelle O’Hegarty, PhD

Office on Smoking and Health

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

Atlanta, Georgia 30341

770-488-5582

FAX: 770-488-5939

Email: mohegarty@cdc.gov

TABLE OF CONTENTS

B. STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

2. Procedures for the Collection of Information

3. Methods to Maximize Response Rates and Deal with No response

4. Tests of Procedures or Methods to be Undertaken

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or

Analyzing Data

LIST OF ATTACHMENTS

Attachment 1: Online Questionnaire Email Invitation to Potential Respondents

Attachment 2: Online Questionnaire Recruitment Screener

Attachment 3: Online Questionnaire

Attachment 4: Online Questionnaire Main Screenshots

Attachment 5: Toluna Terms and Conditions

Attachment 6: Battelle Institutional Review Board Approval

Attachment 7: Toluna Privacy Policy

Notes on Excluded Attachments. In this information collection request (ICR), CDC outlines a plan to test rough cut advertisements (ads) with content that may be considered sensitive. The draft materials are not included because the near final, “rough cut” advertisements have not been approved for public distribution by HHS/Assistant Secretary for Public Affairs (ASPA). To support adequate review of this Gen IC by OMB, the Centers for Disease Control and Prevention requests permission to provide OMB with a secure link to the draft materials.  

Part B: Statistical Methods

B.1 Respondent Universe and Sampling Methods

This is a request for a quantitative data collection. In this GenIC, the Centers for Disease Control and Prevention (CDC) requests OMB approval to collect information for rough cut testing of 15 rough cut advertisements (ads) developed for the 2020 Tips From Former Smokers^® (Tips^®)¹ campaign. This testing will provide information that will inform whether changes must be made to the ads before finalizing, to help ensure that the ads are credible, persuasive, clear, and will motivate the largest number of cigarette smokers to quit smoking conventional cigarettes completely. The proposed information collection will involve testing rough cut ads among adult cigarette smokers and nonsmokers 18-54 years old. The Plowshare Group, and subcontractors Qualtrics and Battelle, will conduct the data collection and analysis for this proposed project.

The sample of respondents for the quantitative data collection will be drawn from Toluna’s online panel (see http://www.toluna-group.com//choose-the-people#global-reach for more detail on this panel). Toluna employs a double opt-in process for individuals to participate in a survey – they must opt-in to become panelists, and they must also opt-in to each survey. For this proposed project, Toluna will gather information for pre-screening as well as ask participants to review both a privacy policy and a terms and conditions statement that outlines the parameters for their participation. The pre-screening is designed to screen out persons < 18 and > 54 years of age for this proposed project. Although the sample will be a convenience sample, the panel sample has a reasonable degree of diversity in key demographic characteristics such as age, gender, region of residence, race/ethnicity, education, and income. The findings from this proposed project will have high internal validity, but are not expected to be widely generalizable to the universe of smokers and nonsmokers in the U.S. As this proposed project is part of formative work for campaign development and planning, these methods are not intended to generate nationally-representative samples or precise estimates of population parameters. However, the design allows for high internal validity to provide information on the perceived effectiveness of rough cut ads under test.

Power analyses were run to determine the sample sizes needed to detect statistically significant differences on key measures (e.g., PE scores for each rough cut ad). The sample size calculations were completed using the following parameters: (1) power = 80%; (2) two-tailed; (3) effect size, Cohen’s f = 0.10 or Cohen’s d = 0.20; and (4) adjustment for multiple comparisons based on a Bonferroni adjusted α error rate.

To detect significant differences in ad reactions between and within subpopulations (i.e., smokers vs. nonsmokers; young adult smokers vs. older adult smokers; low-SES smokers vs. non-low-SES smokers; and young adult nonsmokers vs. older adult nonsmokers), approximately 1,422 respondents will view each of the 15 ads. Specifically, at least 237 respondents from the following groups must view each ad:

• Young adult smokers (18-26 years of age)

• Young adult nonsmokers (18-26 years of age)

• Older adult smokers (27-54 years of age)

• Older adult nonsmokers (27-54 years of age)

• Low-SES status smokers (18-54 years of age)

• Non-low-SES smokers (18-54 years of age)

Overall, a sample size of 21,330 cigarette smokers and nonsmokers is needed. Table B.1 provides additional information by subpopulation about the ads to be tested, the format of the ad, the number of ads, and the minimum number of respondents associated with each ad to be tested.

To achieve this sample size, we conservatively anticipate screening 26,817 respondents; this estimate is based on two factors from prior experiences in the field. First, it is anticipated that roughly 18 percent of screener respondents (n=4,828) will be deemed ineligible for the proposed project because of not meeting inclusion criteria.  Second, of those deemed eligible (n=21,989), an estimated three percent (n=660) will start but not complete the questionnaire. Thus, 26,817 respondents are needed to obtain 21,330 in the final sample.²

Table B.1 Minimum Number of Respondents to View Each Ad
	Smokers1				Nonsmokers2		Total3
	Young Adult	Older Adult	Low-SES	Non-Low-SES	Young Adult	Older Adult
Brian and Denise/Husband and Wife - “Time Together”	237	237	237	237	237	237	1,422
Brian and Denise/Husband and Wife - “Uncertainty”	237	237	237	237	237	237	1,422
Leah and Assad/Mother and Son - “Things I’ve Learned”	237	237	237	237	237	237	1,422
Leah and Assad/Mother and Son - “Okay” VARIATION A	237	237	237	237	237	237	1,422
Leah and Assad/Mother and Son - “Okay” VARIATION B	237	237	237	237	237	237	1,422
Geri - “Thanksgiving”	237	237	237	237	237	237	1,422
Geri - “Texting”	237	237	237	237	237	237	1,422
Michael - “Me”	237	237	237	237	237	237	1,422
Michael - “Lies”	237	237	237	237	237	237	1,422
Tonya - “38”	237	237	237	237	237	237	1,422
Tonya - “Plugged In”	237	237	237	237	237	237	1,422
Rebecca - “It’s Real” VARIATION A	237	237	237	237	237	237	1,422
Rebecca - “It’s Real” VARIATION B	237	237	237	237	237	237	1,422
Art Card VARIATION C	237	237	237	237	237	237	1,422
Variation D	237	237	237	237	237	237	1,422
All Ads	3,555	3,555	3,555	3,555	3,555	3,555	21,3304
¹ Adult smokers were defined as persons between 18-54 years of age who reported smoking > 100 traditional cigarettes during their lifetime and who, at the time of the survey, reported smoking traditional cigarettes every day or some days, and had smoked at least one cigarette in the past 30 days. 2 Adult nonsmokers were defined as persons between 18-54 years of age who did not currently smoke and had not smoked a traditional cigarette in the past 30 days. 3 The sample sizes in the "Total" column are inflated because SES and age are not mutually exclusive categories (e.g., a respondent could be “counted” as both a young adult and a low-SES smoker). 4 For a total final sample size of 21,330, it is estimated that a starting sample of 26,817 respondents will be needed.

Figure B.1 Flowchart of the Sampling of Respondents for Quantitative Data Collection

B.2 Procedures for the Collection of Information

The data collection subcontractor, Qualtrics, will be responsible for coordinating data collection activities, collecting and summarizing information, and preparing topline reports. Battelle will prepare final reports, in collaboration with Qualtrics. Information for this proposed project will be collected using quantitative methods. The testing will collect information about the respondents’ reactions to the rough cut ads, as well as basic demographic and cigarette use information.

Recruitment and Screening

Respondents will be recruited from the Toluna online panel, which is an established, online panel system that includes more than 1.7 million people in the U.S. Toluna has profiled their panels regarding smoking behavior and can target and identify respondents who are pre-identified as being in the age range of 18-54 years for the survey. Toluna also collects demographics such as gender and ethnicity. However, additional screening will be conducted to confirm that Toluna’s profiling information is current and to assess whether any information has changed (i.e., educational status, state of residence). To protect potentially identifiable information (PII) of respondents, no comparison to the original individual profiling data will be made after assessment. The screener and online questionnaire will be hosted on Qualtrics’ server farm (Attachment 4).

Individuals who are enrolled in the online panel will be sent an invitation to participate in the proposed project using an Email Invitation (Attachment 1) provided by Qualtrics’ sample management system. The invitation will include a link behind a “Start” button, with the link going to a web page that contains the Screener (Attachment 2). In addition, the same-worded invitation will be sent at regular intervals after the original invitation is sent to those respondents who have not yet responded. If the potential respondent agrees to participate in the proposed project, he or she will click the “Start” button.

Approximately 26,817 potential respondents are anticipated to complete the screener, and 21,989 respondents in the age range of 18-54 years will then continue to the Online Questionnaire (Attachment 3). Criteria for being eligible for the questionnaire are:

1. Adult cigarette smoker criteria: persons between 18-54 years of age who reported smoking > 100 traditional cigarettes during their lifetime and who, at the time of the survey, reported smoking traditional cigarettes every day or some days, and had smoked at least one cigarette in the past 30 days.

2. Adult nonsmoker criteria: persons between 18-54 years of age who did not currently smoke and had not smoked a traditional cigarette in the past 30 days.

If the respondent does not meet the eligibility criteria assessed during screening, he or she will be routed to a page that thanks the respondent but indicates that he or she does not fit the specific criteria needed for this proposed project. The page that thanks the respondent is located in the respondent’s panel system outside of the survey. It is estimated that in total 4,828 respondents will be terminated after completing the screener. Criteria for termination are:

1. Persons younger than 18 years of age or older than 54 years of age.

Survey Administration

Participants who meet basic eligibility criteria will be routed to the Online Questionnaire (Attachment 3). A preamble to the questionnaire states the length of the survey and provides other information about the survey. The questionnaire will include questions regarding demographic characteristics and smoking behavior. Participants will be randomly assigned to view one of the 15 rough cut ads. Randomization of participants to view the different rough cut ads being tested ensures that there is a similar distribution of individuals of different characteristics (e.g., age, gender, etc.) across the different ads. Each rough cut ad will be in a video file. Participants can click the “restart” button to view the video as many times as they would like. Then, thumbnail pictures representing the video will accompany the aided response section.

Approximately 660 participants are expected to discontinue the questionnaire before completing it. Due to identity protection technology, it will not be possible for anyone to enter the survey who has not been recruited or for a respondent to complete the survey more than once.

B.3 Methods to Maximize Response Rates and Deal with No Response

Five methods will be used to maximize response or completion rates in this current proposed project: (1) inviting only those who have been profiled as being in the target age range of 18-54 to take the survey to reduce the proportion of “incomplete” responses due to not meeting the inclusion criteria, (2) identifying the CDC as the agency of record, since this agency is credible and serves the public good, and (3) drafting the invitations in a manner that has been shown, through prior testing, to yield optimal results. For the email invitation, this includes attending to the following: types of subject lines, topic description, survey details, incentive description, and format (html vs. text) that elicits the most favorable response rates. Additionally, (4) survey responses during the field period will be closely monitored and the invitation email will be resent, with the same content as the original invite, to eligible participants who have not yet responded. Finally, (5) to encourage participation and thus maximize the response rate, the participants who complete the questionnaire will receive a point-based incentive, which can be redeemed for other items, such as Amazon gift cards.

B.4 Test of Procedures or Methods to be Undertaken

The proposed project involves the collection of quantitative information. Similar procedures were used to conduct rough cut testing of the ads developed for the 2019 Tips campaign under this generic clearance, specifically, Agency GenIC #20 titled “National Tobacco Education Campaign Rough Cut Testing of Television Advertisements (0MB No. 0920-0910).”

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Primary responsibility for methodological design, data collection, and data analysis will be performed by Carol Haney and Steven Snell from Qualtrics, and Lisa John, Abigail Evans, and Tesfa Alexander from Battelle, whose information is listed below.

Carol Sue Haney

Senior Research and Data Scientist

Qualtrics

333 West River Park Drive

Provo, UT 84604

Phone: (802) 258-0518

Email: carolh@qualtrics.com

Steven Snell, PhD

Principal Research Scientist

Qualtrics

333 West River Park Drive

Provo, UT 84604

Phone: (616) 502-4828

Email: davev@qualtrics.com

Lisa V. John, PhD, PMP

Project/Program Manager

Battelle

505 King Ave

Columbus, OH 43201

Phone: (614) 456-5051

Email: johnl@battelle.org

Abigail Evans, PhD

Research Scientist

Battelle

6115 Falls Road, Suite 200

Baltimore, MD 21209

Phone: (410) 372-2750

Email: evansa@battelle.org

Tesfa Alexander, PhD

Research Leader

Battelle

1550 Crystal Drive Ste. 601

Arlington, VA, 22202

Phone: (703) 413-7245

Email: alexandert@battelle.org

Individuals consulted at CDC on the proposed project design are listed below.

Centers for Disease Control and Prevention Office on Smoking and Health 4770 Buford Highway, N.E MS F-79 Atlanta, GA 30341
Brian Armour	Associate Director for Science, Office of the Associate Director for Science	Phone: 404.498.3014 E-mail: bka9@cdc.gov
Diane Beistle	Chief, Health Communications Branch	Phone: 770.488.5066 E-mail:  zvg1@cdc.gov
Lauren Boyle-Estheimer	Health Communications Scientist, Health Communications Branch	Phone: 404-498-2283 E-mail: yjw7@cdc.gov
Elizabeth Courtney-Long	Health Scientist, Office of the Associate Director for Science	Phone: 404.498.0264 E-mail: gmr9@cdc.gov
Lindsey McCarter	Team Lead, Health Communications Branch	Phone: 770.488.4239 E-mail:  lpq4@cdc.gov
Michelle O’Hegarty	Health Communication Specialist, Health Communications Branch	Phone: 770.488.5582 E-mail:  mohegarty@cdc.gov



1 Use of trade names is for identification only and does not imply endorsement by the U.S. Department of Health and Human Services.

2 These numbers (21,330 and 660 and 4,828) do not sum to 26,817 due to rounding.

9