Information Collection #6:
National Tobacco Prevention and Control Public Education Campaign:
Digital Media Testing and Copy Testing for
Terrie Hall Rough Cuts
Submitted for approval under CDC generic approval #0920-0910
Message Testing for Tobacco Communication Activities
Submission of this GenIC has been approved by
HHS/Assistant Secretary for Planning and Evaluation (ASPE)
November 20, 2013
Supporting Statement: Part B
__________________________________________
Data Collection Instruments
Other Attachments
Notes on Excluded Attachments
To support adequate review of this GenIC by OMB, CDC requests permission to provide OMB with a secure link to the draft materials.
__________________________________________
B.1 Respondent Universe and Sampling Methods
CDC’s data collection contractor, Toluna, will recruit respondents from the opt-in Toluna Panel, a multimillion-member panel of cooperative online respondents. It is one of the largest databases of individual opt-in respondents for market research in the world. Individuals in the Toluna Panel have agreed to periodically participate in online surveys. Toluna’s panel is highly profiled with both behavioral and demographic characteristics. Panelists have joined the Toluna Panel through hundreds of different sources. Many diverse methods are leveraged to acquire panelists, including co-registration offers on partners’ Websites, targeted emails sent by online partners to their audiences, graphical and text banner placement on partners’ Websites, trade show presentations, targeted postal mail invitations, TV advertisements, member referrals, and telephone recruitment of targeted populations. Toluna has developed multiple proprietary panels across the globe. Since it is known that people who want to join an online panel are often similar to one another, Toluna improves the quality and representative nature of its online sample by incorporating participants from online communities, social networks, and Websites of all types. Participants are recruited to Toluna’s sample via banners, invitations and messaging of all types, and then they go through rigorous quality controls before being included in any sample or on any panel.
In addition, Toluna can potentially access anyone online via a network of relationships with Websites, panels, communities and social media groups. Toluna’s goal is to provide access to people to give their opinions wherever they are in the way that best suits the needs of the research project. For Terms of Use and Conditions for Toluna, please see Attachment 4.
The sample for respondents in this survey will be drawn from the Toluna panel based on the populations of interest – the proposed information collection will involve testing of TV ads among smokers and non-smokers ages 18-54. The legal age of consent is 18 years in all states in the United States except for Alabama and Nebraska, where the age of majority is 19. Toluna’s opt-in process for this survey is designed to screen out minors and anyone less than 18 of age, except in Alabama and Nebraska. In Alabama and Nebraska, anyone aged less than 19 will not be able to participate. In all cases, the screeners will also screen out individuals older than 54. The sample plan is essentially a convenience sample but will be based on demographic variables to ensure a reasonable degree of diversity in key demographic characteristics, such as age, gender, and region of residence, race/ethnicity, education, and income. As this study is considered part of formative research for campaign development and planning, these methods are not intended to generate nationally representative samples or precise estimates of population parameters. The sample drawn here is designed primarily to provide information on the perceived effectiveness of messages from television ads targeted at 18-54 year old smokers and non-smokers.
We will test five draft ads.
Ad named “Teenager”
Ad named “Tips Don’t Smoke”
Ad named “I Wish”
Ad named “Surgeon General”
Ad named “Dana”
Each respondent will view two of the five draft ads. The sample size recommendation is based on a calculation that determines number of respondents needed to assess differences between groups. We will need to screen approximately 4,025 individuals from the Toluna panel to obtain 3,700 complete responses. Each of the 3,700 respondents will view two ads, resulting in 7,400 total views (2 * 3,700). Dividing those 7,400 views by the 5 ads, each ad will have approximately 1,480 views. To adjust for order effects within each group, we will also balance the presentation order of the ads.
1. Ad named “Teenager” |
2. Ad named “Tips Don’t Smoke” |
3. Ad named “I Wish” |
4. Ad named “Surgeon General” |
5. Ad named “Dana” |
Number of total views |
1,480 |
1,480 |
1,480 |
1,480 |
1,480 |
7,400 |
Each ad will
be viewed by 4 groups, i.e., approximately 1,480 total respondents
(370 x 4 = 1,480). To adjust for order effects within each group, we
will also balance the presentation order of the ads.
Respondent Group N=370 per group |
Ads Viewed (50% for each order) |
1 |
1 + 2 or 2 + 1 (Teenager + Tips Don’t Smoke) |
2 |
1 + 3 or 3 + 1 (Teenager + I Wish) |
3 |
1 + 4 or 4 + 1 (etc.) |
4 |
1 + 5 or 5 + 1 |
5 |
2 + 3 or 3 + 2 |
6 |
2 + 4 or 4 + 2 |
7 |
2 + 5 or 5 + 2 |
8 |
3 + 4 or 4 + 3 |
9 |
3 + 5 or 5 + 3 |
10 |
4 + 5 or 5 + 4 |
As this is an ad test we need a certain number of people to view each ad to compare receptivity across ads. In order for us to test differences between ads by different behavioral and demographic groups, it is important that the smallest cell size for a specific attribute (low-SES smokers) to be at minimum of 125 to 150. Given the number of response categories of some items range up to 10, having 1480 respondents view each ad will allow, for specific dependent variables, a minimum base size of 140-150.
Smokers and non-smokers will be interviewed. Seventy percent of the total sample will be smokers and 30% of the total sample is non-smokers. Toluna has profiled its panel in terms of smoking behavior and as such can target and identify respondents who are pre-identified smokers, of legal age and younger than 55, to the survey. Other profiled characteristics of the Toluna panel are those who are low SES (as determined by education classification). It is anticipated that the likelihood of respondents who do not qualify will be in the 5% range. A small percentage (2%) is anticipated to decide to opt-out of the survey once started.
During the data collection period, we will review the distribution of the qualified respondents who have participated and select additional panel members, as needed, to receive targeted email invitations to ensure the appropriate balance of respondents.
B.2 Procedures for the Collection of Information
The survey will be hosted on Toluna’s server farm, managing Toluna’s 4.7-million member panel. All interviews will be conducted using a self-administered, online questionnaire via proprietary, Web-assisted interviewing software. The selected panelists will receive an initial invitation that indicates they have been invited to participate in a new survey (see Attachment 2). The email invitations will also state the length of the survey and incentive they receive if they qualify and complete the survey. The invitations that are currently utilized have been developed based on continual testing of the content, including but not limited to types of subject line, topic description, survey details, incentive description, format (html vs. text) that elicits the most favorable response rates.
Once the invitation is opened, respondents then will link to the survey URL, with an individual, unique and secure link, and complete the survey. Due to identity protection technology, it will not be possible for anyone to enter the survey who has not been recruited, or for a respondent to complete the survey more than once. In addition, the same-worded invitation will be sent at regular intervals after the original invitation is sent to those respondents who have not yet responded.
B.3 Methods to Maximize Response Rates and Deal with Nonresponse
The project methodology attempts to maximize response rates at two points in the data collection process. The survey invitation is targeted at pre-identified individuals who share the screening characteristics. The survey invitation itself is developed to elicit a broad response to maximize the number of respondents that “click” on the survey link. The survey will be available over mobile (smartphone and tablet) and desktop Internet devices. The respondent email invites will include information regarding the survey length, the incentive for participation and a secure and protected link to the survey.
Response rates are closely monitored during the field period and, if needed, multiple reminders and new replicates are also sent to potential survey respondents.
B.4 Test of Procedures or Methods to be Undertaken
None.
B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
George Terhanian, PhD
Toluna USA
21 River Road
Wilton, CT 06897
Phone (646) 430-3420
Email: george.terhanian@toluna.com
David L. Vannette, PhD Candidate
Stanford University
Stanford, CA 94305
Phone: (616) 502-4828
E-mail: vannette@stanford.edu
Carol Sue Haney
Toluna USA
21 River Road
Wilton, CT 06897
Phone (203) 846-5838
Email: carol.haney@toluna.com
John Bremer
Toluna USA
21 River Road
Wilton, CT 06897
Phone (203) 846-5838
Email: john.bremer@toluna.com
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | bkf4 |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |