Part B

Information Collection #12:

National Tobacco Prevention and Control Public Education Campaign:
Copy testing of television and print ads for current smokers who are active military or veterans of the military, have anxiety or depression, or are lesbian, gay, bisexual, or transgender

Submitted for approval under CDC generic approval #0920-0910

Message Testing for Tobacco Communication Activities

Submission of this GenIC has been approved by

HHS/Assistant Secretary for Planning and Evaluation (ASPE)

August 11, 2014

Supporting Statement: Part B

­­­­__________________________________________

Data Collection Instruments

Attachment 1a - Survey Screener Questionnaire - Copy testing of television ads for current smokers who are active military or veterans, have anxiety or depression, or are one or more of the following: lesbian, gay, bisexual, or transgender

Attachment 1b - Survey Main Questionnaire - Copy testing of television ads for current smokers who are active military or veterans, have anxiety or depression, or are lesbian, gay, bisexual, or transgender

Other Attachments

Attachment 2. Email to Potential Respondents (Initial Email Invitation)

Attachment 3. Toluna Panelist Privacy Policies

Attachment 4. Toluna Panelist Terms and Conditions

Attachment 5. Screenshots of online survey (screener and main)

­­­B.1 Respondent Universe and Sampling Methods

Information will be collected about the perceived effectiveness of three TV ads and three print ads that have been previously aired. The target audiences for this information collection are adult smokers (ages 18-54 years old) who (1) are active military or veterans of the military, (2) have anxiety or depression, or (3) are lesbian, gay, bisexual, or transgender (LGBT). Eighteen years of age is the legal age to purchase tobacco cigarettes in most states, except for Alabama and Nebraska, where it is 19. The data collection effort applies screening techniques in order to detect respondents of legal age in Alabama and Nebraska.

CDC’s data collection contractor, Toluna, will recruit a convenience sample of respondents from the opt-in Toluna Panel, a multimillion-member panel of cooperative online respondents. As the proposed information collection is considered part of formative review for campaign development and planning, recruitment methods are not intended to generate nationally representative samples or precise estimates of population parameters. However, participating respondents will reflect a reasonable degree of diversity in key demographic characteristics such as age, gender, region of residence, race/ethnicity, and income. The sample drawn here is designed primarily to provide information about the perceived effectiveness of messages from TV and print ads aimed at adult smokers in the target audiences.

The Toluna panel is one of the largest databases of individual opt-in respondents for market research in the world. The panel is highly profiled with both behavioral and demographic characteristics. Individuals in the Toluna panel have agreed to periodically participate in online surveys. Panelists have joined the Toluna panel through hundreds of different sources. Many diverse methods are leveraged to acquire panelists, including:

• co-registration offers on partners’ Websites;

• targeted emails sent by online partners to their audiences;

• graphical and text banner placement on partners’ Websites;

• trade show presentations, targeted postal mail invitations;

• TV advertisements;

• Member referrals;

• And telephone recruitment of targeted populations.

Participants are recruited to Toluna’s sample via banners, invitations and messaging of all types, and then they go through rigorous quality controls before being included in any sample or on any panel.

In addition, Toluna can potentially access anyone online via a network of relationships with Websites, panels, communities and social media groups. Toluna’s goal is to provide access to people to give their opinions wherever they are in the way that best suits the needs of the research project. For Terms of Use and Conditions for Toluna, please see Attachment 4.

The three television ads that will undergo copy testing are:

1. Ad named “Roosevelt’s Tip”

2. Ad named “Nathan’s Tip: Memorial”

3. Ad named “Tiffany’s Tip”

The three print ads that will undergo copy testing are:

1. Ad named “Rose’s Tip”

2. Ad named “Nathan’s Choosing Tip”

3. Ad named “I Survived Depression” ad from “Cigarettes Are My Greatest Enemy” campaign

All of the TV and print ads can be viewed at http://www.plowsharegroup.com/omb-genic/index2.html. Each respondent will view only one of the six draft ads. The sample size recommendation is based on a calculation that determines number of respondents needed to assess differences between groups. We will need to screen approximately 3,600 individuals who are smokers from the Toluna panel in order to obtain completed questionnaires from 3,000 respondents in the target age range along with other identifying characteristics. This ad test is a monadic ad test, that is, each respondent views only one ad; therefore, each ad will receive 500 views. The ads will be randomized to the respondent.

The quota for each of these subpopulations is listed in the following table

		TV Ads			Print Ads			Sub-totals
		Roosevelt’s Tip	Nathan’s Tip: Memorial	Tiffany’s Tip	Rose’s Tip	Nathan’s Choosing Tip	“I Survived Depression”
Smokers ages 18-54 years	LGBT	170	165	165	170	165	150	985
	Active Military/ Veterans	165	170	165	165	170	150	985
	Individuals with anxiety or de-pression	165	165	170	165	165	200	1,030
	subtotal:	500	500	500	500	500	500	3,000

As this is an ad test, we need a certain number of people to view each ad to compare receptivity across ads. In order for us to test differences between ads by different behavioral and demographic groups, it is important that the smallest cell size for a specific attribute (such as ages 18-24, for example) to be at minimum of 125 to 150.  For example, if a response item has 4 categories, an average of 500 respondents viewing each ad will allow a minimum cell size of 125 (500 respondents/4 categories).   A cell size of 125 is acceptable for this specific copy test.  

Only smokers within the target audience will be interviewed. Toluna has profiled its panel in terms smoking status, military service, ailments, and sexual orientation. Therefore, Toluna can target and identify respondents who are pre-identified with these screening criteria, of legal age, to the survey. It is anticipated the likelihood of respondents who do not qualify will be in the 15-18% range. A small percentage (2%) is anticipated to decide to opt-out of the survey once started. During the data collection period, the distribution of the qualified respondents who have participated will be reviewed and additional panel members will be selected as needed. Targeted email invitations will be sent to ensure the appropriate balance of respondents.

B.2 Procedures for the Collection of Information

The survey will be hosted on Toluna’s server farm, managing Toluna’s 4.7-million member panel. All interviews will be conducted using a self-administered, online questionnaire via proprietary, Web-assisted interviewing software. The screening questions are presented in Attachment 1a and the main questionnaire is included in Attachment 1b. The selected panelists will receive an initial invitation that indicates they have been invited to participate in a new survey (see Attachment 2). The email invitations will also state the length of the survey and incentive they receive if they qualify and complete the survey. The invitations currently used have been developed based on continual testing of the content, to types of subject line, topic description, survey details, incentive description, format (html vs. text), as well as other areas. Modifications are made to elicit the most favorable response rates.

Once the invitation is opened, respondents will link to the survey URL, with an individual, unique and secure link, and complete the survey. The programmed survey is in the attached “Attachment 5” for review. Due to identity protection technology, it will not be possible for anyone to enter the survey without being recruited, or for a respondent to complete the survey more than once. In addition, the same-worded invitation will be sent at regular intervals after the original invitation is sent to those respondents who have not yet replied.

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

The project methodology attempts to maximize response rates with two different strategies during the data collection process. The first strategy is that the survey invitation is targeted at pre-identified individuals (identification collected during the Toluna panel registration and profile process) who share the targeted screening characteristics. The second strategy is that the survey invitation itself is developed to elicit a broad response to maximize the number of respondents that “click” on the survey link. The survey will be available over mobile (smartphone and tablet) and desktop Internet devices. The respondent email invites will include information regarding the survey length, the incentive for participation and a secure and protected link to the survey. Response rates are closely monitored during the field period and, if needed, multiple reminders and new replicates are also sent to potential survey respondents.

B.4 Test of Procedures or Methods to be Undertaken

None.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

George Terhanian, PhD

Toluna USA

Chief Strategy Officer

21 River Road

Wilton, CT 06897

Phone (646) 430-3420

Email: george.terhanian@toluna.com

David L. Vannette, PhD Candidate

Stanford University

Methodologist

Stanford, CA 94305

Phone: (616) 502-4828

E-mail: vannette@stanford.edu

Carol Sue Haney

Toluna USA

Survey Director

21 River Road

Wilton, CT 06897

Phone (203) 846-5838

Email: carol.haney@toluna.com

John Bremer

Toluna USA

Chief Research Officer

21 River Road

Wilton, CT 06897

Phone (203) 846-5838

Email: john.bremer@toluna.com

6