IK SS-Part B_10-30_rev 12 31 14

Page 11

A Comprehensive Evaluation of a Paid Social Media and Mass Media Gynecologic Cancer Campaign

Supporting Statement Part B - Collection of Information Employing Statistical Methods

Point of Contact

Matthew Reynolds, MA

Division of Cancer Prevention and Control

Centers for Disease Control and Prevention

Telephone: 404-639-0649

Email: cdu4@cdc.gov

Revised December 31, 2014

Contents

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1. Respondent Universe and Sampling Methods

B.2. Procedures for the Collection of Information

B.3. Methods to Maximize Response Rates and Deal with Non-response

Maximizing Response Rates

Dealing with Nonresponse

Sample Weighting

B.4. Tests of Procedures or Methods to be Undertaken

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Attachments

Attachment 1a. Johanna’s Law

Attachment 1b. Public Health Service Act

Attachment 2a. Federal Register Notice

Attachment 2b. Summary of Public Comments

Attachment 3: Women’s Health Survey (.PDF file of screen shots)

Attachment 4: Email Invitation, Reminder, and Information Sheet

Attachment 5: IRB Approval Letter

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1. Respondent Universe and Sampling Methods

We will conduct a cross-sectional survey of women in four cities/media markets (Milwaukee, WI; Cincinnati, OH; Las Vegas, NV; and San Antonio, TX) that fall within the Nielsen 35 – 45 Designated Market Area (DMA) to assess the impact of the Inside Knowledge gynecologic cancer campaign on audience reactions, attitudes, intentions, and behaviors. The primary target audience for the Inside Knowledge gynecologic cancer campaign is women ages 40-65, and therefore our evaluation survey will target women ages 40-65.

We will implement a paid media campaign in two of the selected cities (Milwaukee, WI and San Antonio, TX) to augment the national Inside Knowledge campaign. The supplemental campaign will deliver a higher volume (i.e., “heavy up”) of the campaign’s existing media materials. The other two cities (Cincinnati, OH and Las Vegas, NV) will not receive any additional media and will serve as controls for this study. Information will be collected in all four cities at two points in time using cross sectional samples. The first data collection will occur prior to implementation of the campaign (pre-campaign data collection). The second data collection will occur after implementing a one-month campaign “heavy up” implementation (post-campaign data collection). The same survey will be used for both pre- and post-campaign information collection. Separate cross-sectional samples of new participants will be used at each point in time.

The study sample will consist of approximately 2,424 women, with 606 women from each sample/city. Based on data from previous studies we have conducted, we anticipate that the respondent yield from the study will be approximately 80%. Therefore, we need to contact approximately 3,030 women to yield a total sample size of 2,424.

An overview of the target number of completed surveys, by city and campaign treatment, is provided in Table 1, below. The same information collection instrument is used for all data collection.

Table 1. Target Number of Completed Surveys, by City and Campaign Treatment

Treatment	City	Number of Respondents
		Pre-Campaign	Post-Campaign
“Heavy Up,” i.e., Exposed to the National Inside Knowledge Campaign and a Paid Supplemental Media Campaign	Milwaukee, WI	303	303
	San Antonio, TX	303	303
“Control,” i.e., Exposed Only to the National Inside Knowledge Campaign	Cincinnati, OH	303	303
	Las Vegas, NV	303	303
	Subtotal	1,212	1,212
	Grand Total	2,424

The online survey will be hosted by Qualtrics. We will use Qualtrics’ partners to recruit women from existing online survey panels. Samples will be randomly selected from existing survey panels as detailed in Table 2 below.

Table 2. Survey Panels to be utilized to Recruit Eligible Women

Providers	# of responses provided
Research Now	1,082
Toluna	143
Survey Sampling International	306
Global Market Insite	174
Clearvoice	141
Sample Strategies	237
Innovate	341
Totals	2,424

Study sample sizes were estimated through power analyses that were conducted to determine the necessary number of interviews to detect differences in key outcomes of interest between the intervention and control communities. For purposes of this study, we examined existing evaluation literature and research to determine the expected effect sizes on the outcome of interest (intention to seek medical care). The study will be powered with 80% power to detect an 8% increase in the outcome of interest (e.g., intention to seek medical care) in the intervention group using a measure that starts at 50% pre-intervention and more power if the baseline prevalence is either higher or lower than 50%. This will lead to a sample size of 606 completed surveys per city (total n = 2,424). Intervention effects can be measured using several analysis methods. We will look for changes in the outcomes of interest from the pre- to post-intervention time periods in the intervention group and control group.

It should be noted that while the sample recruitment procedures are designed to approximate a nationally representative sample, the limitations associated with online data collection from survey panels require that all results from this information collection be reported with appropriate caution and interpretation. Specifically, although all participants must be invited to participate and cannot volunteer on their own, there may be systematic differences between individuals who choose to join internet surveys and the type of individuals who do not wish to participate in these types of studies over an ongoing timeframe. Therefore, evaluation results must be interpreted with appropriate caution regarding our ability to generalize the findings to the national population of women.

B.2. Procedures for the Collection of Information

All surveys, regardless of sample source, will be conducted via Qualtrics Web portal for self-administered surveys. The survey instrument is included as Attachment 3. Surveys will be accessible to respondents any time of day for a designed period. Participants can complete each survey only one time. Eligible participants will include adult women ages 40-65 in the United States. Responses to the online surveys will be automatically entered into a survey response database associated with the surveys as responses are entered and submitted.

The first survey will be fielded as soon as possible once OMB approval is received. This timing will promote accurate assessment of variables of interest, particularly for capturing respondents' awareness of and reactions to the campaign. These include knowledge, attitudes, and beliefs related to gynecologic health as well as intentions to seek care. A follow-up survey will be conducted approximately 1 month later with a new sample of participants. This timeframe for follow-up data collection correlates with the duration of the campaign “heavy-up.” This will facilitate analysis of relationships between individuals’ exposure to the campaign and changes in outcomes that are relevant to the evaluation. The same survey will be used for the baseline and follow-up survey.

Survey panel participants will be contacted by email. All respondents will be asked to screening questions at the beginning of the survey to confirm they meet the eligibility criteria. The initial email invitation will include the survey link. Nonrespondents will receive a reminder email approximately 5-7 days following the initial request. A second reminder letter will be sent to the remaining nonrespondents 5-7 days after the first reminder letter. See Attachment 4 for email notifications, including the invitation to participate and reminder notifications.

B.3. Methods to Maximize Response Rates and Deal with Non-response

To reduce the potential for both survey and item nonresponse bias, several strategies will be used and are presented below.

Maximizing Response Rates

The following procedures will be used to maximize cooperation and participation in this study:

• As members of existing survey panels, participants will receive incentives consistent with their customary bonus points systems. Points awarded will have a value of approximately $5.

• Follow-up notifications/reminders will be sent to all sampled participants who do not complete their assigned survey within a given period of time after it is assigned. A second round of reminders will be sent to nonrespondents who do not complete the survey once the initial reminder has been delivered.

• Qualtrics will provide a toll-free number to all sampled individuals and invite them to call with any questions or concerns about any aspect of the study.

• Qualtrics staff will work with Battelle project staff to address concerns that may arise. Battelle will maintain ongoing communication with the Qualtrics to identify and resolve barriers to full participation.

Dealing with Nonresponse

To investigate the impacts of survey nonresponse and determine appropriate solutions, simple descriptive statistics, such as counts and frequencies, will be tabulated for respondents and nonrespondents.

Sample Weighting

All data collected for this study will be weighted for analysis. Weighting will adjust for nonresponse and noncoverage. The weighting procedure will apply a standard poststratification adjustment based on demographic distributions from the most recent Current Population Survey (CPS). Benchmark distributions for Internet access used in this weight are obtained from the most recent (October 2009) special CPS supplemental survey measuring Internet Access.

B.4. Tests of Procedures or Methods to be Undertaken

Prior to launching the survey, we will field a 9 case pretest (including cognitive testing) of the survey instrument with members of the target audience. The purpose of the pre-test (often called “cognitive testing”) is to obtain comments and advice about the format, appropriateness, and relevance of the survey questions and response categories (e.g., to ensure that the wording of questions is understandable and the answer categories are clear and comprehensive); and to verify that any skip patterns are easy to follow. The pilot test will also allow us to assess technical aspects and functionality of the survey instrument. Following the pretest, a team member will conduct a brief interview with the respondent to discuss any problems encountered in completing the survey and ask for recommendations on changing the questionnaire to address these comments. For issues with question wording, the participant will be asked for suggestions on how to revise the question to make it easier to understand.

In addition, we will test the online survey instrument prior to its fielding to verify that instrument skip patterns are functioning properly, and all survey questions are worded correctly and to the specification of the instrument approved by OMB.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following individuals from CDC were consulted on statistical aspects and on data collection and analyses procedures:

Matthew Reynolds, MA

Division of Cancer Prevention and Control

Centers for Disease Control and Prevention

Telephone: 770-488-3054

Email: cdu4@cdc.gov

Data collection and processing will be managed by Battelle Memorial Institute.

Rob Woodruff, Biostatistician

Battelle

Phone: 919-544-3717

Email: WoodruffR@battelle.org

Judy Berkowitz, Senior Research Scientist

Battelle

Phone: 404-460-1449

E-mail: berkowitzj@battelle.org

Kendra Versendaal, Research Scientist

Battelle

Phone: 404-460-1458

E-mail: versendaalk@battelle.org

Andrew Davis, Principal Research Scientist

Battelle

Phone: 919-544-3717

E-mail: davisah@battelle.org

Betsy Payn, Data Collection Manager

Battelle

Phone: 206-528-3138

Email: payn@battelle.org

The following individuals external to CDC were consulted on statistical aspects of the survey data collection and/or will be involved with the collection and analyses of the survey data:

Leslie Snyder, PhD

Professor, Dept. of Communication

Center for Health Intervention & Prevention

2006 Hillside Road, Unit 1248

University of Connecticut

Storrs, CT 06269-1248

Office: 860 486-4383

Email: Leslie.Snyder@chip.uconn.edu

Punam Keller, PhD

Professor

Tuck School of Business at Dartmouth

100 Tuck Hall

Hanover, NH 03755

Office: 603-646-1336

Email: punam.a.keller@dartmouth.edu

Bradford W. Hesse, Ph.D.

Chief, Health Comm & Informatics Research Branch

Behavioral Research Program

Division of Cancer Control and Population Sciences

National Cancer Institute

9609 Medical Center Drive

Room 3E610, MSC 9761

Rockville, MD 20852

Phone: (240) 276-6721

Email: hesseb@mail.nih.gov

Michael Spitz

Vice President, Strategy

Klick Health

Phone: (416) 214-4977 x2252

Email: mspitz@klick.com