Supporting Statement B

SUPPORTING STATEMENT FOR THE

National Tobacco Education Campaign

Message Platform Testing for

Regular Cigarettes and Dual Use

(OMB No. 0920-0910, Exp. Date 03/31/2018)

PART B: STATISTICAL METHODS

July 11, 2016

Submitted by:

Office on Smoking and Health

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

Department of Health and Human Services

Refer questions to:

Michelle O’Hegarty

Office on Smoking and Health

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

4770 Buford Highway, NE MS F-79

Atlanta, Georgia 30341

770-488-5582

FAX: 770-488-5939

E-mail: mohegarty@cdc.gov

TABLE OF CONTENTS

B. STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

2. Procedures for the Collection of Information 3. Methods to Maximize Response Rates and Deal with No response

4. Tests of Procedures or Methods to be Undertaken

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or

Analyzing Data

LIST OF ATTACHMENTS

1A. Screen Shots of Recruitment Screener

1B. Screen Shots of the Main Questionnaire for Regular Cigarette Smokers and Dual Users

2A. Moderator’s Guide for IDI – Dual Use

2B. Additional Information About the Message Platforms for Dual Users

3A. Moderator’s Guide for IDI – Regular Cigarette Smokers

3B. Additional Information About the Message Platforms for Cigarette Users

4. Email Invitation to Potential Respondents

5. Terms and Conditions

6. Privacy Policy

7. Battelle Institutional Review Board Approval

Notes on Excluded Attachments

In this Gen IC, CDC outlines a plan to test message platforms with content that may be considered sensitive. The draft materials are not included in the attachments for this Gen IC because:

• The message platforms have not been approved for public distribution by HHS/Assistant Secretary for Public Affairs (ASPA).
  
  

• The untested message platforms could be perceived by the public as ineffective or offensive (testing is designed to identify potential problems).
  
  

To support adequate review of this Gen IC by OMB, CDC requests permission to provide OMB with a secure link to the message platforms.

Section B: Statistical Methods

B.1 Respondent Universe and Sampling Methods

This is a request for a quantitative and qualitative data collection. In this GenIC, the Centers for Disease Control and Prevention (CDC) requests OMB approval to collect information for message platform testing of 23 message platforms developed for a new National Tobacco Education Campaign. The proposed information collection will involve testing of messages among adult smokers 18-54 years old, categorized as either exclusive smokers of regular cigarettes of low socio-economic status (SES), or concurrent users of regular cigarettes and e-cigarettes who are not of low SES (dual users). Exclusive e-cigarette users will not be included in the study.

The sample of respondents in the quantitative survey will be drawn from Toluna’s panel based on the populations of interest. Qualtrics’ opt-in process for this survey is designed to screen out persons < 18 and > 54 years of age. The sample will be a convenience sample, though it will be based on demographic variables to ensure a reasonable degree of diversity in key demographic characteristics such as age, gender, region of residence, race/ethnicity, education, and income. As this study is considered part of formative work for campaign development and planning, these methods are not intended to generate nationally representative samples or precise estimates of population parameters. The sample to be drawn is designed primarily to provide information on the perceived effectiveness of message platforms under test.

It is estimated that, of the 16,000 participants invited to complete the screener (Attachment 1A), 4,800 will not meet the inclusion criteria and will discontinue their participation (“Incompletes”). The remaining 11,200 respondents will then continue to the main questionnaire (Attachment 1B) (“Completes”). Respondents who meet the inclusion criteria of a given message platform will be randomly assigned and routed to the portion of the survey to view one of the message platforms for which they qualify. For example, if a respondent is identified as a non-low SES adult dual user, he or she will be randomly assigned to one of the Dual Use message platforms that focus on one of the following three categories: “Cutting down is not enough”, “effective ways to quit”, and “guidance for parent/adults”. During the information collection period, to ensure the appropriate balance of respondents, the distribution of the qualified respondents who have participated will be closely monitored and additional panel members selected, as needed, to receive targeted email invitations.

For each of the 23 message platforms being tested, ~420-800 respondents will view each message platform; each respondent will view only one platform. Having a range of 420-800 responses per message platform will allow for optimal cell size to detect between-group differences on questions that have multiple response categories (e.g., 5 or 7-point Likert scales), with a confidence level of 95% and a margin of error of 5%. See Table B.1.A for the number of respondents in each of the two groups (i.e. non-low SES adult dual user and low SES adult exclusive regular cigarette smokers) who will view each message platform.

Table B.1.A. Distribution of Respondents Viewing Each Message Platform.

Type of Respondents / Platform	Number of Platforms	N	No. Respondents per Platform
Low SES Exclusive Regular Cigarette Smokers
Health consequences	7	2,940	420
Impact on Lifestyle	3	1,260	420
Secondhand smoke consequences	4	1,680	420
Impact on young adults	1	800	800
Non-low SES Dual Users
Cutting down is not enough	3	2,100	700
Effective ways to quit	1	420	420
Guidance for parents/adults	4	2,000	500
Total	23	11,200

After the first 100 responses per message platform are collected, approximately 50 respondents will be randomly invited to participate in a qualitative individual in-depth-interview done over video and audio software on the respondent’s computer until the target number of participants is recruited (25 low SES exclusive regular cigarette smokers and 25 non-low SES adult dual user). These 50 participants will be randomly selected based on their behavioral criteria and exposure to message platforms that are resonating well with their specific target audience in the quantitative portion of the message platform test.

B.2 Procedures for the Collection of Information

The data collection contractor, Qualtrics, will be responsible for coordinating data collection activities, collecting and summarizing information, and preparation of interim reports. Final reports will be prepared in collaboration with Qualtrics and Battelle.

Toluna has profiling (behavioral and attitudinal) and demographic information on their panel members. Screening will be conducted to confirm that Toluna’s profiling information is correct and to assess whether any information has changed (i.e., educational status, state of residence). To protect PII of the respondent, no comparison to the original individual profiling data will be made after assessment. Toluna has profiled their panels in terms of smoking behavior and can target and identify respondents who are pre-identified smokers of legal age and younger than 55 to the survey. Other profiled characteristics of Toluna include demographics such as gender, ethnicity, and parenthood. It is anticipated that the likelihood of respondents who do not qualify will be in the 15-20% range. A small percentage (1%) is anticipated to decide to opt-out of the survey once started.

Information collection will occur concurrently for both target audiences. Panelists who qualify and then are randomly selected will receive an initial invitation that indicates they have been invited to participate in a new survey (Attachment 4). The email invitations will also state the length of the survey and incentive they receive if they qualify for and complete the survey. The quantitative online screener and main questionnaire (Attachments 1A/1B) will be hosted on Qualtrics’ server farm. All IDIs will be conducted by a moderator using the Moderator’s Guide for IDI: Dual Use (Attachment 2A) and the Moderator’s Guide for IDI: Regular Cigarette Smokers (Attachment 3A) via proprietary, Web-assisted interviewing software.

The qualitative IDIs will be conducted via audio-visual software over a web browser for those respondents who have computers that have those capabilities. During the quantitative data collection process, respondents will be identified in real-time to participate in the qualitative portion based on if they are adult smokers of regular cigarettes or if they are dual users, randomly selecting a smaller subset to invite (page 93 of screen shot). Respondents, if they want to participate, will be routed through a technical qualification check (pages 95-99 of screen shots) and then to appropriate messages if their behavioral characteristics match the message platform under test (for example, only dual users will see messages about dual use). If the respondent agrees, he or she is redirected to a sign-up sheet where he or she will schedule an appointment for an IDI (page 100 of screen shot). If the respondent agrees to participate on the same day he or she completed the main questionnaire, the respondent will be redirected to a software that will begin a video conference. If the respondent schedules the IDI for another time, a calendar iCal will be available for downloading in the software as a reminder of the IDI. The video conference will use the audio and webcam of the respondent’s computer, and will be guided by a moderator. The moderator will use a structured moderating guide, based on the message selected. Each IDI is anticipated to last 30 minutes. During the IDI, the moderator will have the ability to receive messages from researchers based on the discussion, and researchers will be able to “listen in” (in mute-only fashion) to the conversation. The transition from the quantitative survey to the IDI part of the assessment is summarized in the flowchart below:

Figure 1: Summary of transitions between the different parts of the study for eligible participants.

The list of study procedures is as follows:

1. Respondents are recruited from Qualtrics’ existing panel partner Toluna, using an email invitation (Attachment 4) provided by Qualtrics’ sample management system.
   
   

2. The invitation includes a link behind a “Start” button, with the link going to a web page that contains the screener (Attachment 1A).
   
   

3. If the potential respondent consents to participating in the study, he or she clicks the “Start” button. Approximately 16,000 potential respondents are anticipated to complete the screener.
   
   

4. If the respondent passes the screener, he or she progresses to the main questionnaire and is randomly assigned to view one of the message platforms. It is estimated that 11,200 respondents in the target age range of 18-54 years along with other identifying characteristics will complete the screener and continue to complete the main questionnaire (Attachment 1B). Criteria for continuing on to the main questionnaire are:
   
   

1. Adult low SES exclusive regular cigarette smoker between 18-54 years of age criteria: persons who reported smoking > 100 regular cigarettes during their lifetime and who, at the time of the survey, reported smoking regular cigarettes every day or some days;

2. Adult non-low SES adult dual user between 18-54 years of age criteria: current smokers who reported using e-cigarettes, even one time and who, at the time of the survey, reported using e-cigarettes every day or some days.

5. If the respondent does not meet the eligibility criteria assessed during screening, he or she is routed to a page that thanks the respondent, but indicates that he or she does not fit the specific criteria needed for this particular study. The page that thanks the respondent is located in the respondent’s panel system outside of the survey. It is estimated that in total 4,800 respondents will discontinue their participation after completing the screener. Criteria for termination are:

1. Non-smokers

2. Persons younger than 18 years of age.
   
   

6. Approximately 100 participants will be identified in real-time to participate in the qualitative IDI until the target number of participants is recruited (25 adult smokers of regular cigarettes and 25 adult dual users). Criteria for continuing on to the IDI are the same as the criteria for continuing on to the main questionnaire (see study procedures 4 above). Criteria for termination are:

1. The respondent is not interested in participating in the IDI.

2. The respondent does not have a computer with a reliable internet connection.

3. The respondent does not have a webcam.

4. The respondent does not have a headset or earbuds with a microphone.

Due to identity protection technology, it will not be possible for anyone to enter the survey who has not been recruited, or for a respondent to complete the survey more than once. In addition, the same-worded invitation will be sent at regular intervals after the original invitation is sent to those respondents who have not yet responded.

B.3 Methods to Maximize Response Rates and Deal with No Response

Five methods will be used to maximize response or completion rates in this current study: (1) inviting only those who have been profiled as smokers to take the survey to reduce the proportion of “incomplete” responses because of not meeting the inclusion criteria, (2) identifying the CDC as the agency of record at the beginning of the survey, thus appealing to a respondent's sense of altruism to complete the survey, (3) drafting the initial email invitations in a manner that has been shown, through testing, to yield optimal results. This includes paying attention to the following: types of subject line, topic description, survey details, incentive description, and format (html vs. text) that elicits the most favorable response rates (Attachment 4). (4) close monitoring of responses during the field period and re-sending invitational email as a reminder, which is the same content as the original invite, to eligible individuals who have not yet responded, and (5) for the 50 participants who complete an IDI, incentives will include a gift per participant to convey appreciation for their contribution to this voluntary study. The gift will include points that can be redeemed for other items, such as Amazon gift cards. As discussed in detail in Part A, contact information is collected by the panel company Toluna. Toluna will assign points that will be able to be redeemed for Amazon gift cards. Toluna does not send PII or any contact information to the CDC or any of the contractors working on this effort.

B.4 Test of Procedures or Methods to be Undertaken

The proposed project involves the collection of quantitative and qualitative information. Similar procedures were used to conduct rough-cut testing of the ads developed for the 2016 Tips campaign under this same generic clearance, specifically, GenIC #11 titled “National Tobacco Prevention and Control Public Education Campaign: Message Platform Testing for Development of Future Advertising.”

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Primary responsibility for methodological design, data collection, and data analysis will be performed by Carol Haney and David Vannette from Qualtrics, and Erica Peters and Robert Alexander from Battelle, whose information is listed below.

Carol Sue Haney

Senior Research and Data Scientist

Qualtrics

400 Qualtrics Drive

Provo, UT 84604

Phone (802) 258-0518

Email: carolh@qualtrics.com

David L. Vannette, PhD

Principal Research Scientist

Qualtrics

400 Qualtrics Drive

Provo, UT 84604

Phone: (616) 502-4828

Email: davidv@qualtrics.com

Erica N. Peters, PhD

Principal Research Scientist

Battelle Public Health Center for Tobacco Research

6115 Falls Road, Suite 200

Baltimore, MD 21209

Office: 410.372.2708

Email: finan@battelle.org

Robert L. Alexander Jr., PhD, MPH, CHES

Director, Health Communications, Policy and Surveillance

Battelle Public Health Center for Tobacco Research

2987 Clairmont Road, Suite 450

Atlanta, GA 30329

Office: 404-460-1462

Email: alexander@battelle.org

Individuals consulted at CDC on the study design are listed below.

Centers for Disease Control and Prevention Office on Smoking and Health 4770 Buford Highway, N.E MS F-79 Atlanta, GA 30341
Israel Agaku	Deputy Associate Director for Science	Phone: 770.488.5138 E-mail:  wgn9@cdc.gov
Diane Beistle	Chief, Health Communication Branch	Phone: 770.488.5066 E-mail:  zvg1@cdc.gov
Michelle O’Hegarty	Health Communication Specialist	Phone: 770.488.5582 E-mail:  mohegarty@cdc.gov

9