2425-01-icr-ptB

PART B: SURVEY METHODOLOGY

1. SURVEY OBJECTIVES, KEY VARIABLES, AND OTHER PRELIMINARIES

1(a) Survey Objectives

As part of its consideration for establishing an efficacy mark for insect repellents, EPA seeks to understand which efficacy mark approach and design would be most informative and helpful to consumers when purchasing insect repellents.

Specific objectives of the survey include:

(1) Identify the types of information that users of insect repellents want on the label of an insect repellent product and

(2) Test four versions of an efficacy mark, to determine the most helpful and comprehensible design.

1(b) Key Variables

The key information to be collected includes the following:

• Primary reason(s) consumers use or do not use insect repellents

• Frequency of using insect repellents

• Key information consumers are searching for on insect repellent labels

• Reactions to sample efficacy mark designs

1(c) Statistical Approach

A national sample of 3,000 consumers will be included in the research. This sample size will yield results that are statistically reliable within +/- 1.8 percentage points at the 95 percent confidence level.

Support in conducting the survey will be provided by the following contractor:

Shugoll Research

7475 Wisconsin Avenue
Suite 200

Bethesda, MD 20814

Shugoll Research will be assisted by Opinionology (formerly Western Wats) which maintains an Internet consumer panel of more than 1,000,000 individuals drawn from throughout the country.

Panelists for the Opinionology panel are recruited using mixed recruitment approaches consisting of multiple offline and online resources. This includes random digit dial telephone calls to residential households in the U.S., emails sent by Opinionology’s online partners/affiliates to their audiences, banner ad placement on portals/websites of partners/affiliates, search engine text ads, direct mail invitations, use of social networking sites and referral programs. This ensures a diverse sample that includes hard-to-reach segments of the population. Participants recruited through all these means receive information about and are directed to online survey panels.

Panelists recruited for the Internet panel provide some demographic and personal information to create a profile for each panel member. In this way, samples can be selected to reflect certain characteristics so that specific populations of interest are efficiently targeted. Panelist profiles are updated on a regular basis.

Panel members are validated through a rigorous process using both proprietary and commercially available solutions. This process compares panelist demographics to multiple databases and data vendors specializing in consumer information to confirm key identity data, including name, address and date of birth. The identity of each new panelist is validated against a host of quality checks including government managed postal files, web address databases, Opinionology’s own Bloodhound de-duplication software, and TrueSample. There are additional validation checks at the project level through Imperium's Verity. Additional proprietary identity checks are run when panelists redeem rewards.

Panelists are rewarded through a number of options including cash, online retail gift codes, pre-paid debit cards and charitable donations. Opinionology also offers a small guaranteed reward for survey participation, regardless of outcome, which decreases panelist attrition and increases participation by up to 25%. Opinionology optimizes panelist rewards in ways that maintain a consistent mix of panelist tenure and behavioral traits needed to provide consistent data.

Opinionology maintains their panel through close scrutiny of panelists’ participation patterns, survey response patterns, reward redemption patterns, and feedback on Facebook, Twitter, and through their panelist service team. This allows a strong source of data to understand and control the sampling frame. Opinionology maintains a behavioral scoring algorithm to measure answer consistency and panelist attention over time. The behavioral score is affected by non-differentiation, completion time, unit non-response and nonsense-response, inattention, and inconsistent answers from profiles to surveys. Panelists who, over time, do not maintain an acceptable behavioral score are culled from the panel.

In addition to panelist behavioral scoring, Opinionology works with sample source auditors at Marketing, Inc. who regularly measure important behavioral metrics across panels. These metrics include activity rates, panelist tenure, media consumption and buyer behavior, and undesirable survey taking behavior. Marketing, Inc.'s service provides important feedback to help evaluate the efficacy and quality of recruiting efforts and to maintain a consistent distribution of behavioral attributes in the panel over time.

1(d) Feasibility

A consumer who has purchased insect repellent one or more times in the past 12 months will be able to complete the questionnaire without any assistance. The survey instrument is designed for ease of completion by the respondents, and employs close-ended questions and pre-filled data where possible to limit the respondent burden in entering information. The use of an online survey, instead of a telephone survey, allows the respondent to complete the questionnaire at their convenience. Further, the online survey methodology allows for respondents to easily view the efficacy mark designs under consideration.

EPA has planned this data collection as part of its investigation of whether and what type of efficacy mark might be appropriate for insect repellent products. EPA has planned for and allocated resources for the efficient and effective management of the information to be collected, and has a contract mechanism in place to provide logistical support.

2. SURVEY DESIGN

2(a) Target Population and Coverage

The universe for this study encompasses members of the contractors’ Internet panel. The sample size of consumers for the survey (n=3,000) was determined based on several considerations, including funds available for this study, the number of efficacy design marks to be evaluated, and the number of potential sub-segments to analyze, e.g., demographic characteristics (age, gender, and education) and behavioral factors (frequency of using insect repellent and amount of time spent outdoors).

2(b) Sample Design

This study plans to use a national convenience sample drawn from members of the Opinionology Internet panel. Opinionology will pull a proportional sample from the panel based on age, gender, ethnicity, income, and region of the country based on the most recent U.S. Census figures. The sample is not a probability-based sample and cannot be considered representative of all U.S. households because not all U.S. households have access to the internet. In addition, the recruitment methods used by online panels, while broad and carefully managed, are not all probability-based. The sample is likely to under-represent minorities, lower income households and individuals who are 65 years of age or older since those populations are less likely to have access to the Internet.

Opinionology has studied the relationship between samples from its Internet panel and samples collected using more traditional probability sampling techniques, such as telephone surveys. Although samples drawn from Internet panels are not probability samples, based on its studies, Opinionology has developed proprietary techniques to ensure that differences between the results from Internet panel samples and results from probability-based samples of the nation are minimized. In particular, Opinionology uses a variety of methods, including demographic weighting, propensity score weighting, and quota sampling to obtain national survey results from its Internet panel that are as accurate as possible.

EPA believes that the sample drawn from the Opinionology Internet panel will offer valuable insights for purposes of developing efficiency mark designs for insect repellent products. The survey will be carefully designed and screening criteria will be used to define the target population. With this study, the EPA is not attempting to obtain results that are statistically representative of the entire U.S. population to accurately estimate population values. Therefore, a strict probability-based sample is not required. EPA recognizes the limitations of a convenience sample drawn from the Internet panel, and, given the objectives of this study, this type of sample is an acceptable alternative to traditional probability-based sampling for its purposes.

2(c) Precision Requirements

2(c) (i) Precision Targets

Given the objectives of the survey, the study is designed for completion of 3,000 surveys from the consumer Internet panel that have purchased insect repellent products in the past year. Although the Internet panel is not a nationally representative probability sample, it is still desirable that the sample be as representative as possible of individuals at least 18 years of age in the United States. After considering the costs and benefits of various data collection methods, EPA staff concluded that the most efficient way to collect data needed to meet the research objectives within a feasible budget is to employ an Internet panel with nationwide coverage. Thus, the EPA has contracted with Shugoll Research to design an Internet survey that, while not technically representative of the nation as a whole, nonetheless reflects the views of a broad population.

2(c) (ii) Non-Sampling Error

One potential source of non-sampling error for this study is non-response bias – i.e., that the non-respondents may differ from respondents. There are several ways to minimize non-response bias including:

1. Designing the survey so that it will minimize the incidence of non-response. Shugoll Research will work very closely with EPA staff to ensure the questionnaire instrument is easy to understand and complete in a timely manner. EPA has targeted a total response time of 15 minutes per respondent.

2. Reaching out to non-responders to encourage their participation. Consumers who have not responded will receive two separate reminder emails encouraging them to complete the survey.

3. Providing contact information so respondents may obtain help in the event of technical difficulties when attempting to complete the survey.

Another potential problem with Internet panel research is selection bias. Different sub-segments of the population respond in different proportions. One method employed to address this occurrence is click balancing or differential sampling. Essentially, Opinionology factors in estimated response rates by sub-segments of the population (e.g., 18-24 year old males) and sends more invitations to individuals in these categories so that as the data is being collected, the percentage of each sub-segment that responds is accurately represented in the collected sample.

2(d) Survey Instrument Design

The EPA questionnaire will include questions to understand how consumers shop for insect repellents, specifically what information they are seeking on product labels as well as their reactions to alternative efficacy mark designs.

3. PRE-TESTS AND PILOT TESTS

The questionnaire will be pre-tested before survey implementation using a “slow start” methodology which sends a small number of questionnaires (100) out initially in order to evaluate the questionnaire timing and completion. It also provides an indication if consumers are experiencing any difficulty completing any of the questions. Revisions will be made to the questionnaire as needed based on the results of the “slow start” pre-test.

4. COLLECTION METHODS AND FOLLOW-UP

4(a) Collection Methods

Once the questionnaire is finalized, the contractor will develop an online program that will allow respondents to view the questions and materials based on their responses to earlier questions. In this way, an Internet methodology can be an improvement over mail questionnaires, where respondents can preview questions and materials that are best asked in a controlled sequence.

Data collection will be conducted by Shugoll Research assisted by Opinionology. A total of 3,000 individuals 18 years of age or older who have purchased insect repellents one or more times in the past 12 months will complete the study; that is, a total of 3,000 completed surveys will be collected.

In order to obtain a total of 3,000 completed surveys, it is expected that a national convenience sample of 30,000 Internet panel members will be sent an e-mail message inviting them to participate in a survey about consumer product consumption. The invitations will be sent in stages. The first stage outgoing sample will be proportional based on age, gender, ethnicity, income and region of the country based on U.S. Census figures. The second state sample mailing will be more targeted based on how interviewing is proceeding to this point specifically in relation to the proportions of key demographics (age, gender, ethnicity, income, region).

The e-mail invitation sent to the outgoing sample will contain a link that takes the respondents to a website where the online questionnaire resides when they click the link. Based on typical response rates achieved from Internet panels, we expect a response rate of about 10 percent from the 30,000 panel members contacted, or a total of 3,000 completed surveys.

Respondents who visit the site will be screened at the beginning of the survey to ensure they qualify before they are allowed to continue (e.g., must have primary responsibility for making decisions on purchasing insect repellants or sharing equally in making the decisions, have purchased insect repellent one or more times in the past 12 months, and do not work for a company that manufactures, distributes or sells insect repellent products). It will also be important to screen respondents on age (18 years of age and older) and gender (half male and half female) at the beginning of the survey.

Including these five questions at the beginning of the survey will ensure that respondents are qualified to complete the rest of the questionnaire, given that data on insect repellent purchase/usage behavior and employment for companies involved in making or selling insect repellents are not kept on file for panel members. If these qualification questions were placed at the end of the questionnaire and not at the beginning, the cost of administering the survey would increase because it would not be possible to determine if respondents qualify for the survey until they have taken the time to answer all survey questions. It is much more efficient and effective to qualify respondents before they are allowed to answer any of the survey questions.

Each respondent will be assigned a numeric ID that prevents them from visiting the web site and completing the questionnaires more than once. Each respondent also will have an individualized password that prevents anyone other than the panel member from completing the survey.

Shugoll Research will provide data related to the total number of invitations sent to recruit Internet panel members, the number of those who responded to the e-mails, the number of those who qualified for the survey, the number of qualified panel members who completed all questions and the number of within quota completes. The total response rate will also be calculated. This information will be provided by Shugoll Research and Opinionology as the survey progresses to enable monitoring of the efficiency and effectiveness of the online panel and methodology. Shugoll Research will make the final data on completes and response rate available to EPA immediately after the close of the survey as well as in the final report of the study results.

4(b) Survey Response and Follow-Up

To encourage as high a response as possible, two reminder e-mails will be sent to those who have not completed a survey. In addition, contact information will be provided where respondents may obtain help in the event of technical difficulties when attempting to complete the survey.

5. ANALYZING AND REPORTING SURVEY RESULTS

5(a) Data Preparation

A project spreadsheet used to collect, store and organize the data will be reviewed, edited and validated. All data will be double key-entered to ensure accuracy, and several data processing steps will create files suitable for analysis and tabulations.

5(b) Analysis

The data will be analyzed using a statistical software package. Data analysis will include descriptive statistics (e.g., frequencies of survey variables) and relationship analysis (e.g., cross-tabulations of demographic (age, gender, education) and behavioral (light versus heavy insect repellent users) variables.

The contractor, Shugoll Research, will provide EPA with raw data, as well as tabulated data with simple significant test results.

5(c) Reporting Results

The contractor will prepare a detailed report of the study which summarizes and interprets the key findings, including conclusions and actionable recommendations. Professional-quality graphs will be developed to illustrate study findings. The report will include an executive summary as well as sections on the project objectives, research methodology and procedures, detailed findings and conclusions and recommendations that provide direction to EPA regarding insect repellent labeling.

EPA will use the information collected through the survey for analysis and decision-making regarding the need, and potential benefits to consumers, for an efficacy mark.

5