Ssb

Information Collection #13:

National Tobacco Prevention and Control Public Education Campaign:

Rough Cut Testing of English Language Television, Radio, Print, and Digital Advertisements for the 2015 Tips Campaign

Submitted for approval under CDC generic approval #0920-0910

Message Testing for Tobacco Communication Activities

Submission of this GenIC has been approved by

HHS/Assistant Secretary for Planning and Evaluation (ASPE)

October 24, 2014

Supporting Statement: Part B

­­­­__________________________________________

Data Collection Instruments

• Attachment 1a. Screener Questionnaire

• Attachment 1b. Main Questionnaire

Other Attachments

Attachment 2. Email to Potential Respondents (Initial Email Invitation)

Attachment 3. Toluna Panelist Privacy Policies

Attachment 4. Toluna Panelist Terms and Conditions

Attachment 5. Screen Shots (annotated)

Notes on Excluded Attachments

In this GenIC, CDC outlines a plan to test 19 draft ads in television, radio, print and digital form with content that may be considered sensitive. The draft materials are not included in the attachments for this GenIC because:

• The advertisements have not been approved for public distribution by HHS/Assistant Secretary for Public Affairs (ASPA).

• The untested advertisements could be perceived by the public as ineffective or offensive (testing is designed to identify potential problems).

• Release of the advertisements must be coordinated with the launch of a comprehensive HHS/CDC campaign. Unauthorized release could jeopardize the evaluation strategy for the campaign.

To support adequate review of this GenIC by OMB, CDC requests permission to provide OMB with a secure link to the draft materials.

B.1 Respondent Universe and Sampling Methods

CDC’s data collection contractor, Toluna, will recruit respondents from the opt-in Toluna Panel, a multimillion-member panel of cooperative online respondents. It is one of the largest databases of individual opt-in respondents for market research in the world. Individuals in the Toluna Panel have agreed to periodically participate in online surveys. Toluna’s panel is highly profiled with both behavioral and demographic characteristics. Panelists have joined the Toluna Panel through hundreds of different sources. Many diverse methods are leveraged to acquire panelists, including co-registration offers on partners’ Websites, targeted emails sent by online partners to their audiences, graphical and text banner placement on partners’ Websites, trade show presentations, targeted postal mail invitations, TV advertisements, member referrals, and telephone recruitment of targeted populations. Toluna has developed a proprietary panel across the globe. Since it is known that people who want to join an online panel are often similar to one another, Toluna improves the quality and representative nature of its online sample by incorporating participants from online communities, social networks, and Websites of all types. Participants are recruited to the panel via banners, invitations and messaging of all types. Then they go through rigorous quality controls before being included in any sample or on any panel. In addition, Toluna can potentially access anyone online via a network of relationships with websites, panels, communities and social media groups. Toluna’s goal is to provide access to people to give their opinions wherever they are in the way that best suits the needs of the research project. For Terms of Use and Conditions for Toluna, please see Attachment 4.

The sample for respondents in this survey will be drawn from the Toluna panel based on the populations of interest. The proposed information collection will involve testing of television, radio, print and digital ads among smokers and nonsmokers ages 18-54. The legal age of consent is 18 years in all states in the United States except for Alabama and Nebraska, where the age of majority is 19. Toluna’s opt-in process for this survey is designed to screen out minors and anyone less than 18 of age, except in Alabama and Nebraska. In Alabama and Nebraska, anyone aged less than 19 will not be able to participate. In all cases, the screeners will also screen out individuals older than 54. The sample plan is essentially a convenience sample but will be based on demographic variables to ensure a reasonable degree of diversity in key demographic characteristics, such as age, gender, region of residence, race/ethnicity, education, and income. As this study is considered part of formative research for campaign development and planning, these methods are not intended to generate nationally representative samples or precise estimates of population parameters. The sample drawn here is designed primarily to provide information on the perceived effectiveness of messages under test. Nineteen draft English language ads will be tested. A separate information collection request will address the 10 print advertisements that are being developed in other languages as part of the 2015 Tips campaign.

In this GenIC, CDC requests OMB approval to collect information for rough cut testing of the 19 English language draft advertisements briefly described above. Of the draft 19 advertisements:

• Six (6) are 15-second or 30-second television advertisements;

• Four (4) are 30-second or 60-second radio advertisements;

• Four (4) are print advertisements; and

• Five (5) are digital advertisements.

The six television advertisements developed in English that will undergo rough cut testing are:

1. Advertisement entitled “Julia’s Tip”, 30 seconds, colorectal cancer

2. Advertisement entitled “Mark and Julia’s Tip”, 15 seconds, colorectal cancer

3. Advertisement entitled “Mark and Julia’s Tip”, 30 seconds, colorectal cancer

4. Advertisement entitled “Marlene’s Tip 1”, 30 seconds, macular degeneration

5. Advertisement entitled “Marlene’s Tip 2”, 30 seconds, macular degeneration

6. Advertisement entitled “Tiffany’s Tip”, 30 seconds, loss of a loved one

The four radio advertisements developed in English that will undergo rough cut testing are:

1. Advertisement entitled “Julia’s Tip”, 60 seconds, colorectal cancer

2. Advertisement entitled “Mark’s Tip”, 60 seconds, colorectal cancer

3. Advertisement entitled “Marlene’s Tip”, 60 seconds, macular degeneration

4. Advertisement entitled “Dual Use Tip”, 30 seconds, dual-use

The four print advertisements developed in English that will undergo rough cut testing are:

1. Advertisement entitled “Julia’s Tip”, colorectal cancer

2. Advertisement entitled “Mark’s Tip”, colorectal cancer

3. Advertisement entitled “Marlene’s Tip”, macular degeneration

4. Advertisement entitled “Dual Use Tip”, dual-use

The five digital advertisements developed in English that will undergo rough cut testing are:

1. Advertisement entitled “Julia’s Tip”, colorectal cancer

2. Advertisement entitled “Mark’s Tip”, colorectal cancer

3. Advertisement entitled “Marlene’s Tip”, macular degeneration

4. Advertisement entitled “Dual Use Tip 1”, dual-use

5. Advertisement entitled “Dual Use Tip 2”, dual-use

Each respondent will view one of the draft ads. The sample size recommendation is based on a calculation that determines number of respondents needed to assess differences between ads. Approximately 11,667 potential respondents will be screened in order to obtain completed questionnaires from 10,100 respondents in the target age range of 18-54 years along with other identifying characteristics. Note that since 19 ads are being tested in this information collection request, an average of 530 respondents will view each ad. This ad test is a monadic ad test, that is, each respondent views only one ad; therefore each ad will receive an average of 530 views. In the copy test portion of the survey, questions are five or seven-point Likert scales. Having an average of 530 responses per ad would minimally allow for a cell size to detect differences between the groups on questions that have no more than 4 categories.

Smokers and nonsmokers will be interviewed. Approximately 75% of the total sample will be smokers and 25% will be nonsmokers. Toluna has profiled its panel in terms of smoking behavior and as such can target and identify respondents who are pre-identified smokers, of legal age and younger than 55, to the survey. Other profiled characteristics of the Toluna panel include demographics such as gender and ethnicity. It is anticipated that the likelihood of respondents who do not qualify will be in the 5-10% range. A small percentage (1-2%) is anticipated to decide to opt-out of the survey once started.

For these ads, when a respondent is identified as meeting the criteria of the specified ad, they will be randomly assigned and routed to the portion of the survey to view one of the ads for which they qualify. For example, if a respondent is identified as a smoker and speaks English, he or she will be randomly assigned to one of the 19 English ads. In addition to smokers and nonsmokers, four specific subpopulations will be monitored for a specific quota of respondents. The quota for these additional subpopulations is listed in the following table:

Subpopulations of Interest	Number of Responses
Low SES Smokers	2,100
Older smokers 30-54 years old	450
Smokers who also use electronic vapor products	800
African-American smokers	1,600

During the data collection period, we will review the distribution of the qualified respondents who have participated and select additional panel members, as needed, who will receive targeted email invitations to ensure the appropriate balance of respondents.

B.2 Procedures for the Collection of Information

The survey will be hosted on Toluna’s server farm, managing their 4.7-million member panel. All interviews will be conducted using a self-administered, online questionnaire via proprietary, Web-assisted interviewing software. The selected panelists will receive an initial invitation that indicates they have been invited to participate in a new survey (see Attachment 2). The email invitations will also state the length of the survey and incentive they receive if they qualify and complete the survey.

The list of study procedures is as follows:

1. Respondents are recruited from Toluna’s existing online panel, using an email invitation (Attachment 2) sent from Toluna’s sample management system. The sample selection is discussed above in Section B1.

2. The invitation includes a link behind a “Start” button, with the link going to a web page containing the screening survey.

3. If the potential respondent consents to participating in the study, he or she clicks the “Start” button.

4. If the respondent passes the screening questions, he or she progresses to the main questionnaire.

5. If the respondent does not pass the screening questions, he or she is routed to a page that says “Thank you for your participation in this research study. Unfortunately, your responses indicate that you are do not fit the specific criteria needed for this particular study, or that we have already reached our required quota of responses from participants similar to you. We appreciate your enthusiasm for our research study and hope you will join us on future surveys!”

Once the invitation is opened, respondents then will link to the survey URL, with an individual, unique and secure link, and complete the Screener (Attachment 1a) and Main Questionnaire (Attachment 1b). Due to identity protection technology, it will not be possible for anyone to enter the survey who has not been recruited, or for a respondent to complete the survey more than once. In addition, the same-worded invitation will be sent at regular intervals after the original invitation is sent to those respondents who have not yet responded.

The routing described above happens automatically within the programmed survey. In this manner, the random assignment of specific ads to specific subpopulations and then directed to that ad will ensure that each ad will be viewed by an average of 530 respondents. Once a quota is met for a specific subpopulation and ad, the ad will be removed from the random assignment for that specific subpopulation. Once all quotas are filled, the survey will be completed. Screen shots are provided as attachment 5. The screenshot file is annotated with comments that explain minor differences in questionnaire routing and wording.

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

The project methodology attempts to maximize response rates at two points in the data collection process. The survey invitation is targeted at pre-identified individuals who share the screening characteristics. The survey invitation itself is developed to elicit a broad response to maximize the number of respondents that “click” on the survey link. The survey will be available over desktop Internet devices. The respondent email invites will include information regarding the survey length, the incentive for participation and a secure and protected link to the survey. Response rates are closely monitored during the field period and, if needed, multiple reminders and new replicates will be sent to potential survey respondents. The invitations currently being utilized have been developed based on continual testing of the content, including but not limited to types of subject line, topic description, survey details, incentive description, and format (html vs. text) that elicits the most favorable response rates.

B.4 Test of Procedures or Methods to be Undertaken

None.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Carol Sue Haney

Toluna USA

21 River Road

Wilton, CT 06897

Phone (203) 846-5838

Email: carol.haney@toluna.com

John Bremer

Toluna USA

21 River Road

Wilton, CT 06897

Phone (203) 846-5838

Email: john.bremer@toluna.com

David L. Vannette, PhD Candidate

Stanford University

Stanford, CA 94305

Phone: (616) 502-4828

E-mail: vannette@stanford.edu

7