Supporting Statement B

Information Collection #9:

National Tobacco Prevention and Control Public Education Campaign:

Rough Cut Testing of English Language Television, Print, Digital Print, and Radio Ads for the Tips 2014 National Campaign

Submitted for approval under CDC generic approval #0920-0910

Message Testing for Tobacco Communication Activities

Submission of this GenIC has been approved by

HHS/Assistant Secretary for Planning and Evaluation (ASPE)

March 11, 2014

Supporting Statement: Part B

­­­­__________________________________________

Data Collection Instruments

• Attachment 1a. Screener - 18-54 year olds: smokers, nonsmokers, smokers with HIV, and LGBT smokers

• Attachment 1b. Main Questionnaire

Other Attachments

• Attachment 2. Email to Potential Respondents (Initial Email Invitation)

• Attachment 3. Toluna Panelist Privacy Policies

• Attachment 4. Toluna Panelist Terms and Conditions

• Attachment 5. Screen Shots (annotated)

Notes on Excluded Attachments

In this GenIC, CDC outlines a plan to test 13 draft ads in radio, television, print and digital form with content that may be considered sensitive. The draft materials are not included in the attachments for this GenIC because:

• The ads have not been approved for public distribution by HHS/Assistant Secretary for Public Affairs (ASPA).

• The untested ads could be perceived by the public as ineffective or offensive (testing is designed to identify potential problems).

• Release of the ads must be coordinated with the launch of a comprehensive HHS/CDC campaign. Unauthorized release could jeopardize the evaluation strategy for the campaign.

To support adequate review of this GenIC by OMB, CDC requests permission to provide OMB with a secure link to the draft materials.

­­­­__________________________________________

B.1 Respondent Universe and Sampling Methods

CDC’s data collection contractor, Toluna, will recruit respondents from the opt-in Toluna Panel, a multimillion-member panel of cooperative online respondents. It is one of the largest databases of individual opt-in respondents for market research in the world. Individuals in the Toluna Panel have agreed to periodically participate in online surveys. Toluna’s panel is highly profiled with both behavioral and demographic characteristics. Panelists have joined the Toluna Panel through hundreds of different sources. Many diverse methods are leveraged to acquire panelists, including co-registration offers on partners’ Websites, targeted emails sent by online partners to their audiences, graphical and text banner placement on partners’ Websites, trade show presentations, targeted postal mail invitations, TV advertisements, member referrals, and telephone recruitment of targeted populations. Toluna has developed a proprietary panel across the globe. Since it is known that people who want to join an online panel are often similar to one another, Toluna improves the quality and representative nature of its online sample by incorporating participants from online communities, social networks, and Websites of all types. Participants are recruited to Toluna’s sample via banners, invitations and messaging of all types, and then they go through rigorous quality controls before being included in any sample or on any panel.

In addition, Toluna can potentially access anyone online via a network of relationships with Websites, panels, communities and social media groups. Toluna’s goal is to provide access to people to give their opinions wherever they are in the way that best suits the needs of the research project. For Terms of Use and Conditions for Toluna, please see Attachment 4.

The sample for respondents in this survey will be drawn from the Toluna panel based on the populations of interest – the proposed information collection will involve testing of TV ads among smokers and nonsmokers ages 18-54. The legal age of consent is 18 years in all states in the United States except for Alabama and Nebraska, where the age of majority is 19. Toluna’s opt-in process for this survey is designed to screen out minors and anyone less than 18 of age, except in Alabama and Nebraska. In Alabama and Nebraska, anyone aged less than 19 will not be able to participate. In all cases, the screeners will also screen out individuals older than 54. The sample plan is essentially a convenience sample but will be based on demographic variables to ensure a reasonable degree of diversity in key demographic characteristics, such as age, gender, and region of residence, race/ethnicity, education, and income. As this study is considered part of formative research for campaign development and planning, these methods are not intended to generate nationally representative samples or precise estimates of population parameters. The sample drawn here is designed primarily to provide information on the perceived effectiveness of messages from television ads targeted at 18-54 year old smokers and nonsmokers.

We will test 13 draft English language ads. A separate information collection request will address the testing of Spanish versions of these ads.

The five television ads that will undergo rough cut testing are:

1. Ad entitled “Amanda’s Tips”, 30 seconds, maternal health

2. Ad entitled “Brett’s Tips, 30 seconds, periodontal disease

3. Ad entitled “Rose’s Tips , 30 seconds, lung cancer

4. Ad entitled “Rose’s Tip2, 30 seconds, lung cancer

5. Ad entitled “Shawn’s Tips, 30 seconds, throat cancer

The three radio ads that will undergo rough cut testing are:

1. Ad entitled “Amanda’s Tip”, 60 seconds, maternal health

2. Ad entitled “Brett’s Tip”, 60 seconds, periodontal disease

3. Ad entitled “Rose’s Tip”, 30 seconds, lung cancer

The five print ads that will undergo rough cut testing are:

1. Ad entitled “Amanda’s Tip”, maternal health

2. Ad entitled “Brian’s Tip1”, Stroke/HIV consequences

3. Ad entitled “Brian’s Tip2”, Stroke/HIV consequences (digital print)

4. Ad entitled “Felicita’s Tip”, periodontal disease

5. Ad entitled “Rose’s Tip”, lung cancer

Each respondent will view one of the draft ads. The sample size recommendation is based on a calculation that determines number of respondents needed to assess differences between ads. We expect to screen approximately 9,464 potential respondents who are part of the Toluna panel in order to obtain completed questionnaires from 7,800 respondents in the target age range of 18-54 years along with other identifying characteristics. Note that since 13 ads are being tested in this specific package, 600 respondents will view each ad, and as this ad test is a monadic ad test, that is, each respondent views only one ad, then each ad will receive 600 views. In the copy test portion of the survey, we have questions that are five or seven-point Likert scales. Having 600 responses per ad would minimally allow for a cell size to detect differences between the groups on questions that have no more than 4 categories.

Smokers and nonsmokers will be interviewed. Approximately 75 percent of the total sample will be smokers and 25 percent will be nonsmokers. Toluna has profiled its panel in terms of smoking behavior and as such can target and identify respondents who are pre-identified smokers, of legal age and younger than 55, to the survey. Other profiled characteristics of the Toluna panel include demographics such as gender, as well as those who are LGBT. It is anticipated that the likelihood of most respondents who do not qualify will be in the 5-10% range. A small percentage (2%) is anticipated to decide to opt-out of the survey once started. This is accurate for all respondents except for those with individuals with HIV. Over 300 panelists with HIV are identified in the Toluna panel. We anticipate that not all 300 will respond to the request for the survey. As such, increased screening will be necessary for that specific audience target, a quota of 100 respondents. Toluna will use imputation techniques to optimize the selection of probable respondents with HIV, looking at other correlated characteristics that often accompany HIV, such as previously-identified immune system ailments. To reach 100 responses for this specific target audience, we anticipate approximately 40 of the 300 identified respondents will respond to the survey request. To interview the remaining 60 respondents, we anticipating identifying 1 qualified respondent from 12 likely respondents, thus requiring an additional 740 respondents that will be screened yet not qualify.

For these ads, when a respondent is identified as meeting the criteria of the specified ad, the respondent will randomly be assigned and routed to the portion of the survey where he or she will view one of the ads that he or she qualifies for. For example, if a respondent is identified as a smoker who is LGBT, he or she will randomly be assigned to Brian’s Tip1 print ad or Brian’s Tip2 print ad. Or, if the respondent is a smoker or non-smoker who does not meet one of these specific target audiences, yet is between the ages of 18-54 years old, he or she will be randomly assigned to one of the thirteen ads under test. Five of the ads are assigned specific quotas for three subpopulations:

1. Ad entitled “Brian’s Tip1”, a digital print ad, has two target audiences: smokers with HIV and smokers who are LGBT;

2. Ad entitled “Brian’s Tip2”, a print ad, is targeted at smokers who are LGBT;

3. Ad entitled “Amanda’s Tip”, a print ad, is targeted at female smokers between the ages of 18-44 years old;

4. Ad entitled “Amanda’s Tip”, a 60 second radio ad, is targeted at female smokers between the ages of 18-44 years old; and

5. Ad entitled “Amanda’s Tip”, a 30 second TV ad, is targeted at female smokers between the ages of 18-44 years old

The quota for each of these subpopulations is listed in the following table:

Ad Title	GenPop Smokers, 18-54	GenPop Nonsmokers,18-54	HIV Smokers, 18-54	LGBT Smokers, 18-54	Female Smokers, 18-44
“Brian’s Tip2”, Stroke/HIV consequences (digital print)	300	100	0	200	0
“Brian’s Tip1”, Stroke/HIV consequences (print)	200	100	100	200	0
“Amanda’s Tip”, maternal health (print)	300	100	0	0	200
“Felicita’s Tip”, periodontal disease (print)	400	200	0	0	0
“Rose’s Tip”, lung cancer (print)	400	200	0	0	0
“Rose’s Tip”, 30 seconds, lung cancer (radio)	400	200	0	0	0
“Amanda’s Tip”, 60 seconds, maternal health (radio)	300	100	0	0	200
“Brett’s Tip”, 60 seconds, periodontal disease (radio)	400	200	0	0	0
“Rose’s Tip" , 30 seconds, lung cancer (TV)	400	200	0	0	0
“Amanda’s Tip”, 30 seconds, maternal health (TV)	300	100	0	0	200
“Brett’s Tip", 30 seconds, periodontal disease (TV)	400	200	0	0	0
“Rose’s Tip2", 30 seconds, lung cancer (TV)	400	200	0	0	0
“Shawn’s Tip”, 30 seconds, throat cancer (TV)	400	200	0	0	0

During the data collection period, we will review the distribution of the qualified respondents who have participated and select additional panel members, as needed, to receive targeted email invitations to ensure the appropriate balance of respondents.

B.2 Procedures for the Collection of Information

The survey will be hosted on Toluna’s server farm, managing Toluna’s 4.7-million member panel. All interviews will be conducted using a self-administered, online questionnaire via proprietary, Web-assisted interviewing software. The selected panelists will receive an initial invitation that indicates they have been invited to participate in a new survey (see Attachment 2). The email invitations will also state the length of the survey and incentive they receive if they qualify and complete the survey.

Once the invitation is opened, respondents then will link to the survey URL, with an individual, unique and secure link, and complete the Screener (Attachment 1a) and Main Questionnaire (Attachment 1b). Due to identity protection technology, it will not be possible for anyone to enter the survey who has not been recruited, or for a respondent to complete the survey more than once. In addition, the same-worded invitation will be sent at regular intervals after the original invitation is sent to those respondents who have not yet responded.

The routing described above happens automatically within the programmed survey. In this manner, the random assignment of specific ads to specific subpopulations and then directed to that ad will ensure that each ad will be viewed by 600 respondents. Once a quota is met for that specific subpopulation and ad, the ad will be removed from the random assignment for that specific subpopulation. Once all quotas are filled, the survey will be completed.

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

The project methodology attempts to maximize response rates at two points in the data collection process. The survey invitation is targeted at pre-identified individuals who share the screening characteristics. The only screening characteristic where this is not the case is the identification of individuals with HIV. These will be screened using probabilistic scoring and general population interviewing. The survey invitation itself is developed to elicit a broad response to maximize the number of respondents that “click” on the survey link. The survey will be available over desktop Internet devices. The respondent email invites will include information regarding the survey length, the incentive for participation and a secure and protected link to the survey.

Response rates are closely monitored during the field period and, if needed, multiple reminders and new replicates will be sent to potential survey respondents.

The invitations that are currently utilized have been developed based on continual testing of the content, including but not limited to types of subject line, topic description, survey details, incentive description, and format (html vs. text) that elicits the most favorable response rates.

B.4 Test of Procedures or Methods to be Undertaken

None.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

George Terhanian, PhD

Toluna USA

21 River Road

Wilton, CT 06897

Phone (646) 430-3420

Email: george.terhanian@toluna.com

David L. Vannette, PhD Candidate

Stanford University

Stanford, CA 94305

Phone: (616) 502-4828

E-mail: vannette@stanford.edu

Carol Sue Haney

Toluna USA

21 River Road

Wilton, CT 06897

Phone (203) 846-5838

Email: carol.haney@toluna.com

John Bremer

Toluna USA

21 River Road

Wilton, CT 06897

Phone (203) 846-5838

Email: john.bremer@toluna.com

7