Information Collection #15:
National Tobacco Prevention and Control Public Education Campaign:
Rough Cut Testing of Television, Radio, Print, and Digital Advertisements for the 2016 Tips Campaign
Submitted for approval under CDC generic approval #0920-0910
Message Testing for Tobacco Communication Activities
Submission of this GenIC has been approved by
HHS/Assistant Secretary for Planning and Evaluation (ASPE)
May 12, 2015
Supporting Statement: Part B
__________________________________________
Data Collection Instruments
Attachment 1a. Screener Questionnaire in English
Attachment 1b. Main Questionnaire in English
Attachment 1c. Screener Questionnaire in Chinese
Attachment 1d. Main Questionnaire in Chinese
Other Attachments
Notes on Excluded Attachments
To support adequate review of this GenIC by OMB, CDC requests permission to provide OMB with a secure link to the draft materials.
B.1 Respondent Universe and Sampling Methods
In this GenIC, CDC requests OMB approval to collect information for rough cut testing of the 23 draft advertisements developed for the 2016 Tips campaign. Of these ads, 22 are in English and one is in Chinese. The proposed information collection will involve testing of television, radio, print and digital ads among smokers and nonsmokers ages 18-54. The sample for respondents in this survey will be drawn from Toluna’s panel based on the populations of interest. The legal age of consent is 18 years in all states in the United States except for Alabama and Nebraska, where the age of majority is 19. Qualtrics’ opt-in process for this survey is designed to screen out minors and anyone less than 18 of age, except in Alabama and Nebraska. In Alabama and Nebraska, anyone aged less than 19 will not be able to participate. In all cases, the screeners will also screen out individuals older than 54. The sample plan is essentially a convenience sample but will be based on demographic variables to ensure a reasonable degree of diversity in key demographic characteristics, such as age, gender, region of residence, race/ethnicity, education, and income. As this study is considered part of formative research for campaign development and planning, these methods are not intended to generate nationally representative samples or precise estimates of population parameters. The sample drawn here is designed primarily to provide information on the perceived effectiveness of messages under test. The following table describes the details for each of the 23 ads:
Ad Package |
Media Format |
Number |
Total respondents |
of Ads |
|||
1. Becky, COPD |
TV (:30) |
1 |
545 |
Radio (:60) |
1 |
545 |
|
1 |
545 |
||
Digital |
1 |
500 |
|
2. Brian, Heart Attack |
TV (:30) (v.1 & v.2) |
2 |
1,100 |
Radio (:60) |
1 |
545 |
|
1 |
545 |
||
Digital |
1 |
500 |
|
3. Kristy, Dual-Use Warning |
TV (:30) |
1 |
545 |
Radio (:30, :60) |
2 |
1,100 |
|
1 |
545 |
||
Digital |
1 |
500 |
|
4. Rebecca, Depression |
*TV (:30) |
1 |
545 |
*Radio (:30, :60) |
2 |
1,100 |
|
1 |
545 |
||
*Digital |
1 |
545 |
5. Rico, Cancer |
*Print (1 English, 1 Chinese) |
2 |
650 |
*Digital |
1 |
500 |
|
6. Mark and Julia’s Tip Cancer |
TV (:30) |
1 |
600 |
|
total: |
23 |
12,000 |
* Gain-frame ads as discussed in Section A.1, page 3
CDC’s data collection contractor, Qualtrics, will recruit respondents through its panel partner, Toluna. Toluna draws respondents from its established Toluna panel system that Qualtrics (the data collection provider) uses that provides points as a reward for participation. Toluna’s role in the information collection is the provisioning of respondents to Qualtrics. All information collection activities are performed by Qualtrics. Individuals in Toluna’s panel have agreed to periodically participate in online surveys. Toluna’s panel is highly profiled with both behavioral and demographic characteristics. Panelists have joined the panels through hundreds of different sources. Many diverse methods are leveraged to acquire panelists, including co-registration offers on partners’ Websites, targeted emails sent by online partners to their audiences, graphical and text banner placement on partners’ Websites, trade show presentations, targeted postal mail invitations, TV advertisements, member referrals, and telephone recruitment of targeted populations. Since it is known that people who want to join an online panel are often similar to one another, Toluna improves the quality and representative nature of its online sample by incorporating participants from multiple online sources, such as communities, social networks, and Websites of all types. Participants are recruited to the panel via banners, invitations and messaging of all types. Then they go through rigorous quality controls before being included in any sample or on any panel. In addition, Toluna can potentially access anyone online via a network of relationships with websites, panels, communities and social media groups. For Terms of Use and Conditions for Toluna, please see Attachment 3.
Each respondent will view one of the draft ads. The sample size recommendation is based on a calculation that determines number of respondents needed to assess differences between ads. Approximately 13,650 potential respondents will be screened in order to obtain completed questionnaires from 12,000 respondents in the target age range of 18-54 years along with other identifying characteristics. Note that since 23 ads are being tested in this information collection request, an average of 520 respondents will view each ad. This ad test is a monadic ad test, that is, each respondent views only one ad; therefore each ad will receive an average of 520 views. In the copy test portion of the survey, questions are five or seven-point Likert scales. Having an average of 520 responses per ad would minimally allow for a cell size to detect differences between the groups on questions that have no more than 4 categories.
Smokers and nonsmokers will be interviewed. Approximately 75% of the total sample will be smokers and 25% will be nonsmokers. Toluna has profiled their panels in terms of smoking behavior and as such can target and identify respondents who are pre-identified smokers, of legal age and younger than 55, to the survey. Other profiled characteristics of Toluna include demographics such as gender, ethnicity, and language preference (English or Chinese). It is anticipated that the likelihood of respondents who do not qualify will be in the 5-10% range. A small percentage (1-2%) is anticipated to decide to opt-out of the survey once started.
For these ads, when a respondent is identified as meeting the criteria of the specified ad, they will be randomly assigned and routed to the portion of the survey to view one of the ads for which they qualify. For example, if a respondent is identified as a smoker and speaks English, he or she will be randomly assigned to one of the 22 English ads. If the respondent identifies as a smoker and speaks Chinese, he or she will be assigned to the 1 Chinese print ad.
During the information collection period, we will review the distribution of the qualified respondents who have participated and select additional panel members, as needed, who will receive targeted email invitations to ensure the appropriate balance of respondents.
B.2 Procedures for the Collection of Information
The survey will be hosted on Qualtrics’ server farm. All interviews will be conducted using a self-administered, online questionnaire via proprietary, Web-assisted interviewing software. The selected panelists will receive an initial invitation that indicates they have been invited to participate in a new survey (see Attachment 2). The email invitations will also state the length of the survey and incentive they receive if they qualify and complete the survey.
The list of study procedures is as follows:
Respondents are recruited from Qualtrics’ existing panel partner Toluna, using an email invitation (Attachment 2) provided by Qualtrics’ sample management system.
The invitation includes a link behind a “Start” button, with the link going to a web page that contains the screening survey. If the respondent's browser is set to default to Chinese, then information collection will be conducted in Chinese. If the respondent's browser defaults to English, then information collection will be conducted in English.
If the potential respondent consents to participating in the study, he or she clicks the “Start” button.
If the respondent passes the screening questions (Attachments 1a (English) and 1c (Chinese)), he or she progresses to the main questionnaire (Attachments 1b (English) and 1d (Chinese).
If the respondent does not pass the screening questions, he or she is routed to a page that thanks the respondent, but indicates that they do not fit the specific criteria needed for this particular study.
Due to identity protection technology, it will not be possible for anyone to enter the survey who has not been recruited, or for a respondent to complete the survey more than once. In addition, the same-worded invitation will be sent at regular intervals after the original invitation is sent to those respondents who have not yet responded.
Screen shots are provided as attachments 4 and 5. Each screenshot file is annotated with comments that explain minor differences in questionnaire routing and wording, depending on whether the ad is formatted for television, radio, print, or digital media.
B.3 Methods to Maximize Response Rates and Deal with Nonresponse
The project methodology attempts to maximize response rates at two points in the data collection process. The survey invitation is targeted at pre-identified individuals who share the screening characteristics. The survey invitation itself is developed to elicit a broad response to maximize the number of respondents that “click” on the survey link. The survey will be available over desktop Internet devices. The respondent email invites will include information regarding the survey length, the incentive for participation and a secure and protected link to the survey. Response rates are closely monitored during the field period and, if needed, multiple reminders and new replicates will be sent to potential survey respondents. The invitations currently being utilized have been developed based on continual testing of the content, including but not limited to types of subject line, topic description, survey details, incentive description, and format (html vs. text) that elicits the most favorable response rates.
B.4 Test of Procedures or Methods to be Undertaken
Similar procedures were used to conduct rough cut testing of the ads developed for the 2015 Tips campaign.
B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
Primary responsibility for statistical design, data collection, and data analysis will be performed by Carol Haney and David Vanette, whose information is listed below.
Carol Sue Haney
Qualtrics
400 Qualtrics Drive
Provo, UT 84604
Phone (802) 258-0518
Email: carolh@qualtrics.com
David L. Vannette, PhD
Qualtrics
400 Qualtrics Drive
Provo, UT 84604
Phone: (616) 502-4828
Email: davidv@qualtrics.com
Yvonne Hunt, PhD, MPH
Program Director
Tobacco Control Research Branch
Behavioral Research Program
Division of Cancer Control and Population Sciences
National Cancer Institute
9609 Medical Center Drive, Room 3E534, MSC 9761
Bethesda, MD 20892-9761
Phone: (240) 276–6975
Email: huntym@mail.nih.gov
John Bremer
Toluna USA
21 River Road
Wilton, CT 06897
Phone (203) 846-5838
Email: john.bremer@toluna.com
Sarah Alio, MS
Qualtrics
400 Qualtrics Drive
Provo, UT 84604
Phone: (914) 835-3487
Email:
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | bkf4 |
| File Modified | 0000-00-00 |
| File Created | 2021-01-26 |