Fuel Economy Materials Testing Research_Supporting Statement Part B_KH_08042015_Final

Supporting Statement for Fuel Economy, GHG, Other Emissions, and Alternative Fuels Education Program
Quantitative Materials Testing Research Plan

Section B

June 29, 2015

Contents

B. Collections of Information Employing Statistical Methods 3

B1. Describe the potential respondent universe and any sampling or other respondent selection method to be used. Indicate the expected response rates for the collection as a whole. 4

B2. Describe the procedures for the collection of information 7

B3. Describe methods to maximize response rates and to deal with issues of non-response. 7

B4. Describe any tests of procedures or methods to be undertaken. 8

B5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency. 9

2. Collections of Information Employing Statistical Methods

NHTSA is seeking approval for one survey instrument to guide the finalization of a consumer education campaign plan and materials aimed at raising consumer awareness and understanding of certain aspects of vehicle fuel economy, alternative fuel use, and topics related to greenhouse gases and other emissions. The survey will be used to gain insights into the new vehicle purchase process and test graphical communications to guide enhancements that can help make the materials as useful as possible for consumers.

This survey will be conducted one time with a sample of 2,000 drivers from all 50 states and the District of Columbia who meet the following criteria:

• Respondents must be 18 years or older.

• Respondents must currently possess a valid driver’s license.

• Respondents must be the primary or a shared decision maker for vehicle purchases and/or vehicle maintenance in their household.

The objectives guiding this survey are as follows:

1. Gather updated data on consumer knowledge of and opinions related to fuel economy, GHG emissions and other emissions, and alternative fuels information and perceptions of information currently available.

2. Develop an understanding of the current channels drivers are using to obtain information on vehicles, fuel economy to inform communications planning.

3. Test materials and content developed to assess consumer comprehension and perceived usefulness.

4. Use quantitative data cuts to guide communications planning and targeting of materials.

Data Analysis Plan

During data analysis, we will use a variety of statistical techniques to assess the materials being developed for this campaign, including whether the materials are useful and comprehended. Summary statistics will be analyzed to evaluate each material on the following metrics: informative-ness, clarity, credibility, and likelihood to read or watch the item. The purpose of this analysis is to guide final design and content decisions on the consumer-facing materials to be leveraged throughout this campaign, as well as inform communications planning for material roll-out.

Much of the analysis will be conducted by running cross-tabulations and analyzing descriptive statistics. We will use cross-tabulations for measures such as relevance and likelihood to use the material in order to provide the communications team with guidance on which channels to use during roll-out. For example, we may find that respondents ages 18-34 rate a video higher on likelihood of use compared to respondents ages 55 and older. The latter group may in turn find more value in fact sheets. This type of finding suggests to the communications team that certain channels should feature different material types in order to maximize the likelihood that the educational information is reaching consumers through their preferred format.

During analysis, we will utilize advanced image evaluation modules so respondents can easily provide feedback on this content. For graphical treatments, such as fact sheets, heat maps will be developed based on click frequency and for the purposes of this survey will provide a visual representation of content that is either useful or confusing to respondents. This analysis will inform design edits needed before releasing materials to the public.

1. Describe the potential respondent universe and any sampling or other respondent selection method to be used. Indicate the expected response rates for the collection as a whole.

For reasons explained in Part A of this supporting statement, NHTSA is seeking approval to conduct an online survey. This methodology was chosen as it provides respondents with the opportunity to evaluate images and video storyboards, which is essential for the testing of campaign materials. Since these graphical treatments contain English-language messages, the survey will only be available in English to ensure respondents understand the messages being displayed. Issues of non-response due to this methodology are addressed in section B3 of this document.

With the understanding that this survey will need to be conducted online in order for respondents to evaluate graphical content, we will utilize a convenience sample, built using members of an online panel managed by a top-rated panel provider, which will give NHTSA access to a potential respondent universe of about 4 million U.S. consumers (approximately 2% of the total universe). These panels have been built using a passive panel sourcing process as the providers work with publishers and online service providers to embed recruitment efforts into their websites. Potential panel members will only be admitted into the panel after a validation process is complete. For quality purposes, all panel members are de-duplicated at the registration stage and a continuous system of validation checks is in place to prevent professional survey takers from entering studies. The U.S. consumer panel is built to look proportional to the general population by age, gender, ethnicity and region.

The survey will be e-mailed to a random list U.S. panel members ages 18 and older. We expect approximately 20,000 invitations to be sent to randomly selected panel members on a rolling basis throughout the fielding of the study in order to obtain the 2,000 respondents necessary to complete the study. Since we must conduct this survey online due to the natural of the communications materials, using an online survey panel to recruit respondents is cost-effective and reduces the burden on potential respondents as the process of electing to take the survey and actually completing the survey is seamless.

Before entering the questionnaire, each respondent will be screened based on the criteria listed below. If qualified, the respondent then enters the survey and begins answering questions relevant to the needs of this study.

Upon completion, the respondent will be redirected to the panel provider’s website. This survey will be a one-time implementation, and respondent identification information will be removed prior to data analysis to preserve anonymity.

The sample will be gathered through the following steps:

1. Respondents will first have to qualify as adults ages 18 and older.

2. Next, respondents must qualify as currently living in the United States.

3. Next, respondents must self-report that they currently possess a valid driver’s license.

4. Finally, respondents will be screened to ensure they are the primary or shared decision maker in their household when it comes to vehicle purchase or lease decisions.

Based on past research conducted on behalf of NHTSA, we anticipate at least 85% of general consumers in the U.S. ages 18 and older in the panel will qualify as currently possessing a valid driver’s license and being the primary or shared decision maker for vehicle purchases or maintenance.

A full text of the draft survey questionnaire, including the screening questions, is included in this package as Appendix D.

The sample will be built using quota sampling to consist of percentages of US drivers ages 18+ by age, gender and geographic location proportionate to the data provided in the Federal Highway Administration’s 2012 Highway Statistics. We will impose hard quotas for age ranges, gender, and regions and soft quotas for states within each region.

Table 4. Region Definitions

Region	Sub-Region	States
South	Deep South	Alabama, Arkansas, Florida, Georgia, Louisiana, Mississippi, South Carolina
Midwest	Farm Belt	Iowa, Kansas, Missouri, Nebraska, North Dakota, South Dakota
Midwest	Great Lakes	Illinois, Indiana, Michigan, Minnesota, Ohio, Wisconsin
Northeast	Mid Atlantic	Delaware, District of Columbia, Maryland, New Jersey, New York, Pennsylvania, West Virginia
West	Mountain	Arizona, Colorado, Idaho, Montana, Nevada, New Mexico, Utah, Wyoming
Northeast	New England	Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island, Vermont
South	Outer South	Kentucky, North Carolina, Oklahoma, Tennessee, Texas, Virginia
West	Pacific	Alaska, California, Hawaii, Oregon, Washington

Sample Size Rationale

In preparing this research plan, we conducted a power analysis utilizing SPSS SamplePower. As described on page 3 of this document, we will be evaluating cross-tabulations to guide the targeting of communications based on various demographics. As a result, we have elected a sample size that will be robust enough for us to draw conclusions by comparing key disparate groups – particularly, age groups. This analysis will guide decisions on communications planning for these materials so that we are able to effectively reach a large population with the information contained in our education program.

In our analysis we plan to compare two groups at a time on a 7-point scale for key metrics. For example, “likelihood to use” is on a 7-point scale ranging from “Not at all likely” to “Very likely.” Because these responses follow a logical sequence, we can assign a numerical score to each response and compute a mean and standard deviation for each group.

In our prior qualitative research for this effort, we found that respondents aged 45 and older were more likely to feel that fact sheets provided them with information they were able to comprehend, while respondents ages 18-34 were more likely to use infographics and videos to get this information. As a result, we expect to find differences between age groups within the material evaluations. The closer in age the groups we are evaluating, the closer the mean response we expect.

The null hypothesis for this analysis is that the mean response for Group A (respondents ages 45 and older) and Group B (respondents ages 18-34) is identical. Our intent is to disprove the null and conclude that the mean response is different for Group A than for Group B.

As a result, the computation of sample size is based on the following educated assumptions and decisions:

1. The expected pattern of response for Group A (respondents ages 45-54) evaluating their likelihood to use the infographic to obtain this information corresponds to a mean of 5.58 (on a 7-point scale) with a standard deviation of 1.52.

2. The expected pattern of response for Group B (respondents ages 18-34) on this same measure corresponds to a mean of 5.86 with a standard deviation of 1.36.

3. The alpha error level included in this computation is .05 (5%).

4. In computing the sample size we assume that there will be no missing data as respondents will be unable to skip the question.

This power analysis indicates that based on these assumptions, with a sample of 328 subjects per group the study will have power of 80%. Therefore, we have used an overall sample size so that the sub-groups we will plan to compare (i.e, ages 18-34, 35-44, 45-54, 55-64, 65 and older) have a minimum sample size of 328 per group for this analysis to allow us to conclude that the mean response differs for these subgroups evaluating the same material types. An overall sample size of 2,000 slightly exceeds that threshold, with the smallest group size being 342 based on the quota plan detailed in Table 5.

We will run a post-hoc power analysis will be run using the actual mean and standard deviation for respondents’ likelihood to use the infographic. This analysis will confirm that we have, in fact, disproved our null hypothesis.

Based on the Federal Highway Administration’s 2012 Highway Statistics, there are approximately 211.8 million licensed drivers ages 18 and older¹. Since we are using a convenience sample for this study, we plan to use the following statistics to set gender, age and region quotas for our sample of 2,000.

Table 5. Expected Population and Sample Distribution by Key Demographics

Gender	Total Population	% of Licensed Drivers 18+	Expected Distribution of Sample (n)
Male	103,348,827	49.6%	992
Female	105,215,445	50.4%	1008
Age
18-34	59,747,618	28.7%	574
35-44	36,527,225	17.5%	350
45-54	40,594,647	19.5%	390
55-64	35,750,452	17.1%	342
65+	35,944,330	17.2%	344
Region
Northeast (Mid Atlantic – 76%; New England – 24%)	42,755,592	20.5%	410
Midwest (Great Lakes – 77%; Farm Belt (23%)	45,981,511	22.0%	440
South (Outer South – 52%; Deep South – 48%)	71,053,459	34.1%	682
West (Pacific – 68%; Mountain – 32%)	48,773,710	23.4%	468

2. Describe the procedures for the collection of information

The procedure for the collection of information for this study is as follows:

• Survey will be sent via e-mail to online panel members identified as consumers ages 18+.

• The questionnaire will take approximately 20 minutes to complete. All panel members who are over the age of 18 have an equal chance of receiving the survey invitation, but only those who qualify based on screening criteria will have the opportunity to complete the survey.

• A total sample size of 2,000 will be interviewed. Respondents will complete the survey on their own time, and will have an option to contact the panel provider if they experience any technical problems with the survey.

• Data tables, including important cross-tabulations, and graphical outputs of advanced survey modules (e.g., image heatmaps) will be prepared along with a final report of the key findings and strategic recommendations.

This sampling methodology results in a study that only includes respondents who have access to computers and the Internet, and are literate. According to Pew Research Center’s Internet and American Life Project², as of May 2013, 85% of US adults access the Internet at least occasionally. Further, since the online panel we will use includes approximately 4 million U.S. consumers, thus only 2% of the total population has an opportunity to be included in the survey.

Because the communications materials are meant for online consumption and the evaluation of graphical content must be done through an online survey, we feel that the benefits of this methodology outweigh the drawbacks. We will be setting quotas so that the final participating respondents have the same proportion of characteristics based on age, gender and region as the total universe. As with other quota sample studies, there will be no controlling for additional nonrepresentativeness that may exist for variables not used to define the quotas, which may result in undercoverage of certain populations. If necessary, additional weighting may be applied to the dataset, though this is not expected.

Because the study is not a probability-based sample, there is no statistical basis to derive unbiased estimates representative of the target population, U.S. passenger vehicle owners or lessees, or to estimate sampling error. While we will not project the final data to be representative of the target population, we feel that the results will provide the direction necessary to inform final design edits and communications planning activities.

3. Describe methods to maximize response rates and to deal with issues of non-response.

As detailed in B1 and B2, we are utilizing a convenience sample originating from an online panel of U.S. consumers. In order to ensure diversity in respondents, we will set quotas to ensure the sample consists of percentages proportionate to the US driver audience by gender, age, and region based on the data provided in the Federal Highway Administration’s 2012 Highway Statistics.

NHTSA has taken a number of steps to address issues of non-response:

• The panel of U.S. consumers is built to be proportional to the national population based on age, gender, region and ethnicity.

• The survey invitations will be sent to randomly selected panel members.

• The invitation to the survey will not mention the topic of the study to help ensure that our sample is composed of general consumers rather than automotive enthusiasts or consumers interested in fuel economy, alternative fuels or other planned topics of exploration.

• Reminder e-mails will be sent to participants who have been invited to the study, but have not yet entered the study.

• The survey will be programmed using technologies through which consumers can interact directly with the images presented on screen. This will help allow for easy evaluation of the labels and reduce burden, which will help prevent respondent drop off.

During data analysis, we will analyze additional demographics of our survey sample against available demographics of U.S. drivers to determine if the survey data is reliable enough to inform this communications campaign. We do not anticipate the need for weighting and will strive to attain a sample that matches the demographic characteristics described in Table 5.

For the purposes of this study, it is sufficient that the sample be a convenience sample as long as it is diverse in terms of drivers’ gender, age, and region. NHTSA believes that the study design will provide the quantitative estimates to effectively inform the development of a consumer education campaign. In addition, this methodology is a cost- and time-effective way of conducting the research necessary to inform this program (thus reducing burden on respondents) and is the most feasible methodology that allows for the testing of graphic and video storyboard content.

Additionally, planned communications channels will heavily rely on web-based sources. We will not ask respondents to rank sources of information, as there may be an inherent bias toward web-sources with these respondents, but through an online methodology we will be obtaining evaluations from a group of consumers who are able to access these channels once the campaign is launched. Therefore, the difference between respondents and non-respondents will likely not impact the guidance of consumer information dissemination and thus we expect issues of non-response to have little effect on the results of the survey.

4. Describe any tests of procedures or methods to be undertaken.

Pilot Test & Soft Launch Period

NHTSA and its contractors will conduct a test of the online survey to ensure all functionality is working properly before the study is sent to potential respondents.

The survey will then experience a ‘soft launch’ period in which a limited number of potential respondents (approximately 5% of the sample) are sent invitations to respond. During this period, we will evaluate the demographic data, completion rate, and time-spent in the survey. Any problems discovered during this test period will be addressed before the survey is sent out to a larger population.

Data Collection and Analysis

Responses will be collected in a computerized database. Data processing will consist of tabulation of quantitative and coded open-ended responses using UNCLE software, which facilitates the manual categorization of verbatim responses so that these results can be quantified. Data analysis will be conducted by NHTSA and its contractors. Summary statistics will be analyzed to gather insights into the audiences NHTSA reaches with its communications. Additional statistical analysis, such as graphic and video storyboard evaluations, will be conducted using a software program such as SPSS, a standard statistical program that provides the ability to run a range of analyses from basic cross-tabulations to regression analysis, ANOVA and other tests.

As detailed on page 3 of this document, we plan to use a variety of statistical techniques to assess the materials being developed for this campaign, including whether the materials are useful and comprehended. Summary statistics will be analyzed to evaluate each material on the following metrics: informative-ness, clarity, credibility, and likelihood to read or watch the item.

Much of the analysis will be conducted by running cross-tabulations and analyzing descriptive statistics. We will use cross-tabulations for measures such as relevance and likelihood to use the material in order to provide the communications team with guidance on which channels to use during roll-out.

We will also utilize advanced image evaluation modules so respondents can easily provide feedback on this content. For graphical treatments, such as fact sheets, heat maps will be developed based on click frequency and for the purposes of this survey will provide a visual representation of content that is either useful or confusing to respondents. This analysis will inform design edits needed before releasing materials to the public.

A final report will be prepared in PowerPoint that summarizes this analysis through a mix of text, charts and graphical depictions of results. The report will be used as the basis for final decisions related to communications planning

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The company selected as a contractor for this study is Edelman Berland. Edelman Berland is the market research arm of Edelman, Inc. This team has extensive experience in both qualitative and quantitative research practices, and has consulted with NHTSA on the statistical aspects of the design. The contact information for this team is as follows:

Ellen Bartlett Vice President 202.350.6687	Edelman Berland 1875 Eye Street NW Washington, D.C. 20006
Kelsey Cohen Senior Project Manager 202.326.1740

The statistical aspects of this information collection request have also been reviewed by:

Promod Chandhok

Bureau of Transportation Statistics (BTS), Department of Transportation

202.366.2158

1 https://www.fhwa.dot.gov/policyinformation/statistics/2012/dl22.cfm

2 The Pew Research Center’s Internet & American Life Project Spring Tracking Survey, April 17-May 19, 2010. n=2,252 adults 18 and older. Interviews were conducted in English and Spanish and on landline and cellphones. Margin of error is +/- 2.3 percentage points.

14

NHTSA: Fuel Economy Materials Testing Research

Quantitative ICR Package – Supporting Statement Part B

June 29, 2015