NCAP Communications Research_Supporting Statement Part B_2019_OMB_9.4.2019_clean

Supporting Statement for New Car Assessment Program (NCAP) Communications Research
Quantitative ICR Package

OMB Control Number 2127-new

Section B

DRAFT: May 21, 2019

(Revised August 8, 2019)

Contents

B. Collections of Information Employing Statistical Methods 3

B1. Describe the potential respondent universe and any sampling or other respondent selection method to be used. Indicate the expected response rates for the collection as a whole. 4

B2. Describe the procedures for the collection of information 6

B3. Describe methods to maximize response rates and to deal with issues of non-response. 7

B4. Describe any tests of procedures or methods to be undertaken. 7

B5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency. 8

2. Collections of Information Employing Statistical Methods

NHTSA is seeking approval for one survey instrument to guide NCAP-related communications –specifically to enhance consumer understanding of NHTSA’s 5-Star Safety Ratings and of advanced crash avoidance technologies. The survey will be used to gain insights into the new vehicle purchase process and to test graphical communications that will guide the development of NHTSA communication materials that will help consumers as they factor this information into their vehicle purchase decisions.

This survey will be conducted one time with a sample of 1,500 drivers from all 50 states and the District of Columbia who meet the following criteria:

• Respondents must be 18 years or older.

• Respondents must currently possess a valid driver’s license.

• Respondents must be in the new vehicle purchase mindset. That is, they have either purchased a new vehicle in the past six months, or plan to do so within the next 12 months.

• Respondents must be the primary or a shared decision maker for vehicle purchases in their household.

The objectives guiding this survey are as follows:

1. Validate qualitative findings and prioritize factors/considerations that influence the purchase decision.

2. Measure likelihood to seek out safety information to better understand consumer interest in communications around safety ratings.

3. Evaluate the window sticker in terms of how clear and easy to understand it is overall and prioritize the importance of specific types of information displayed on the sticker.

4. Gauge familiarity and interest with driver assistance technologies.

Data Analysis Plan

During data analysis, we will use a variety of statistical techniques to assess the information that will be most useful for consumers as they purchase a new vehicle, as well as evaluate the relative usefulness of current NHTSA safety communications. The purpose of this analysis is to guide content decisions related to the Monroney label (“window sticker”) and NHTSA.gov.

Much of the analysis will be conducted by running cross-tabulations and analyzing descriptive statistics; however, to fulfill certain objectives, we will employ advanced analysis as necessary.

One of the initial objectives guiding this research is to gain a better understanding of the tradeoffs consumers make based on the information presented on the Monroney label. During the focus groups, we were able to asses a general ranking of the importance of various pieces of information, but in order to confirm the relative importance, we will utilize maximum difference scaling (MaxDiff) method. MaxDiff presents respondents with a number of trade-off exercises, where each respondent picks the most and least important items (or pieces of information), out of three or four items at a time. Respondents do this repeatedly, so that all items are evaluated against each other. This assessment will allow NHTSA to better understand the relative importance of safety ratings compared to other information on the window sticker. This information will help to guide recommendations to potential design changes to the NCAP section of the label.

1. Describe the potential respondent universe and any sampling or other respondent selection method to be used. Indicate the expected response rates for the collection as a whole.

For reasons explained in Part A of this supporting statement, NHTSA is seeking approval to conduct an online survey. This methodology was chosen as it provides respondents with the opportunity to evaluate images, which is essential for the testing of label design and web-based communications, is a low-burden way to address long batteries of attributes and is mobile-compatible. Since these graphical treatments contain English-language messages, the survey will only be available in English to ensure respondents understand the messages being displayed. Issues of non-response due to this methodology are addressed in proceeding pages of this document.

The sample will be built using members of an online panel managed by a panel provider Critical Mix, which will give NHTSA access to a large diverse potential respondent universe of U.S. consumers. These panels have been built using a passive panel sourcing process as the providers work with publishers and online service providers to embed recruitment efforts into their websites. Potential panel members will only be admitted into the panel after a validation process is complete. For quality and accuracy purposes, all panel members go through a double opt-in registration process, digital fingerprinting (using RelevantID), identity verification (using Verity and/or TrueSample), and category and past participation exclusion (using ICE™). Please note that no personally identifiable information (such as Social Security Number, Date of Birth, or home address) are requested within the survey. The continuous system of validation checks is in place to prevent professional survey takers from entering studies.

Upon panel registration, members are asked demographic and lifestyle questions so that surveys can be targeted. Using this information, the survey will be e-mailed to U.S. panel members ages 18 and older. We expect approximately 18,520 invitations to be sent to randomly selected panel members on a rolling basis throughout the fielding of the study in order to obtain the 1,500 respondents necessary to complete the study.

Before entering the questionnaire, each respondent will be screened based on the criteria listed below. If qualified, the respondent then enters the survey and begins answering questions relevant to the needs of this study. A full text of the draft survey questionnaire is included in this package as Appendix C.

Upon completion, the respondent will be redirected to the panel provider’s website. This survey will be a one-time implementation, and respondent identification information will be removed prior to data analysis to preserve anonymity.

The sample will be built to consist of percentages of US drivers ages 18+ by age, gender and geographic location proportionate to the data provided in the Federal Highway Administration’s 2017 Highway Statistics. We will impose hard quotas for age ranges, gender, and regions and soft quotas for states within each region. These quotas are based off available Census data estimates, ensuring sample selection for inclusion is not merely a convenience sample but stratified based on known populations of our audience of interest and relevant subgroups for analysis.

Table 4. Nine-Point Region Definitions (based on Census Divisions)

Region	States
New England	Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island, Vermont
Middle Atlantic	New Jersey, New York, Pennsylvania
East North Central	Illinois, Indiana, Michigan, Ohio, Wisconsin
West North Central	Iowa, Kansas, Minnesota, Missouri, Nebraska, North Dakota, South Dakota
South Atlantic	Delaware, District of Columbia, Florida, Georgia, Maryland, North Carolina, South Carolina, Virginia, West Virginia
East South Atlantic	Alabama, Kentucky, Mississippi, Tennessee
West South Atlantic	Arkansas, Louisiana, Oklahoma, Texas
Mountain	Arizona, Colorado, Idaho, Montana, Nevada, New Mexico, Utah, Wyoming
Pacific	Alaska, California, Hawaii, Oregon, Washington

The sample will be gathered through the following steps:

1. Respondents will first have to qualify as adults ages 18 and older.

2. Next, respondents must qualify as currently living in the United States and having a driver’s license.

3. Next, respondents must self-report being in the new vehicle purchase mindset, that is, purchased a new vehicle in the past six months or plan to purchase a vehicle within the next 12 months.

4. Finally, respondents will be screened to ensure they are the primary or shared decision maker in their household when it comes to vehicle purchase decisions.

Based on the Federal Highway Administration’s 2017 Highway Statistics, there are approximately 222 million licensed drivers ages 18 and older¹. Since we are using a convenience sample for this study, we plan to use the below statistics to set gender, age and region quotas for our sample of 1,500.

Table 5. Expected Population and Sample Distribution by Key Demographics

Gender	Total Population of Licensed Drivers 18+	% of Licensed Drivers 18+	Expected Distribution of Sample (n)
Male	109,819,649	49.4%	741
Female	112,360,581	50.6%	759
Age
18-24	23,288,737	10.5%	159
25-34	39,831,017	17.9%	269
35-44	37,090,912	16.7%	252
45-54	39,175,690	18.4%	277
55-64	39,178,953	17.5%	263
65+	43,614,921	18.6%	280
Region
New England	10,860,771	4.9%	74
Middle Atlantic	27,213,036	12.2%	183
East North Central	31,860,206	14.3%	215
West North Central	14,175,846	6.4%	96
South Atlantic	45,460,623	20.5%	307
East South Atlantic	14,135,951	6.4%	95
West South Atlantic	25,053,148	11.3%	170
Mountain	16,839,810	7.8%	115
Pacific	36,580,839	16.5%	245

NHTSA will not present data from this survey as representative of the general population. The intended use of this data is for internal use only to assist NHTSA in the development of communications and general public outreach. Given that, while the study results will need to be reliable, this does not require the full rigor of probability sampling that is commonly required for informing and setting policy.

2. Describe the procedures for the collection of information

The procedure for the collection of information for this study is as follows:

• Survey will be sent via e-mail to online panel members identified as consumers ages 18+.

• The questionnaire will take approximately 20 minutes to complete. All panel members who are over the age of 18 have an equal chance of receiving the survey invitation, but only those who qualify based on screening criteria will have the opportunity to complete.

• A total sample size of 1,500 will be interviewed. Respondents will complete the survey on their own time and will have an option to contact the panel provider if they experience any technical problems with the survey.

• Data tables, including important cross-tabulations, will be prepared along with a final report of the key findings and strategic recommendations.

The sampling approach described above will yield a convenience sample. This sampling methodology results in a study that only includes respondents who have access to computers and the Internet and are literate. According to Pew Research Center’s Internet and American Life Project², as of May 2013, 88% of US adults access the Internet at least occasionally. Pew released a report titled “11% of Americans don’t use the internet. Who are they?” in March of 2018 that shows this number has remained relatively stable. Because the study is not a probability-based sample, there is no statistical basis to derive unbiased estimates representative of the target population, U.S. passenger vehicle owners or lessees, or to estimate sampling error.

NHTSA believes that the study design will provide the quantitative estimates to effectively inform the development of a consumer communications and 5-Star Safety Rating recommendations. In addition, this methodology is a cost- and time-effective way of conducting the research necessary to inform this program and is the most feasible methodology that allows for the testing of graphical content.

Additionally, planned communications channels will heavily rely on web-based sources like NHTSA.gov. We will not ask respondents to rank sources, as there may be an inherent bias toward web-sources with these respondents, but through an online methodology we will be obtaining evaluations from a group of consumers who are able to access these sources once published. Therefore, the difference between respondents and non-respondents will likely inconsequentially impact the guidance of consumer information dissemination.

3. Describe methods to maximize response rates and to deal with issues of non-response.

As detailed in B1 and B2, we are utilizing a convenience sample. In order to ensure diversity in respondents, we will set quotas to ensure the sample consists of percentages proportionate to the US driver audience by gender, age, and region based on the data provided in the Federal Highway Administration’s 2017 Highway Statistics.

4. Describe any tests of procedures or methods to be undertaken.

Pilot Test

NHTSA and its contractors will conduct a test of the online survey to ensure all functionality is working properly before the study is sent to potential respondents.

Once proper functionality of the survey program is confirmed, NHTSA will conduct cognitive testing of the survey instrument with consumers who qualify based on the screening criteria of the survey. In this portion of the research engagement, approximately 600 respondents will go through a short screening phase to identify 12 suitable participants representing a mix of demographic characteristics (such as age, gender, and vehicle purchase decision-maker). From these 12, eight will be selected to participate in one-on-one in-person cognitive interviews reviewing the survey instrument for clarity and comprehension. Each interview will be conducted separately in a professional interviewing facility and is expected to last approximately 55 minutes. Testing will take place in the Richmond, VA metro area. The purpose of this testing is to ensure the questionnaire wording is clear and comprehendible for consumers and will help ensure that we are collecting quality data in order to meet our objectives. Cognitive testing will be conducted by trained research moderators and will provide the participant the ability to take the survey on a computer and discuss how they are interpreting questions and whether or not they understand what is being asked.

Based on these results, minor edits to the survey instrument may be made to improve the clarity and comprehension of survey question language.

Soft Launch Period

The survey will experience a ‘soft launch’ period in which a limited number of potential respondents (approximately 5% of the sample) are sent invitations to respond. During this period, we will evaluate the demographic data, completion rate, and time-spent in the survey. Any problems discovered during this test period will be addressed before the survey is sent out to a larger population.

Data Collection and Analysis

Responses will be collected in a computerized database. An electronic data collection design employing modern information technology was chosen because of its versatility in employment; providing increased accessibility to the survey through a mobile-friendly platform. Data processing will consist of tabulation of quantitative and coded open-ended responses using Quantum tabulation software. Data analysis will be conducted by NHTSA and its contractors. Summary statistics will be analyzed to gather insights into the audiences NHTSA reaches with its communications. Additional statistical analysis, such as maximum difference scaling analysis, will be conducted using a software program such as Sawtooth or SPSS.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Dee Allsop, Ph.D., CEO and Managing Partner

Heart+Mind Strategies

Phone: 571-926-8852 ext.307

Andrew Cober, Senior Solutions Consultant

Heart+Mind Strategies

Phone: 571-926-8852 ext.305

Ben Custer, Senior Solutions Director

Heart+Mind Strategies

Phone: 919-803-7901

Tyler Ashe, Solutions Assistant

Heart+Mind Strategies

Phone: 571-926-8852 ext.324

Amanda McColl, Director, Research Information and Application

Heart+Mind Strategies

Phone: 919-481-1196

Carlos Elordi, Ph.D., Director, Analytics and Applied Methodologies

Heart+Mind Strategies

Phone: 540-463-4746

David Phife, President

Directions in Research

Phone: 858-707-6050

1 https:// https://www.fhwa.dot.gov/policyinformation/statistics/2017/

2 The Pew Research Center’s Internet & American Life Project Spring Tracking Survey, Surveys conducted 2000-2016. Data for each year based on a pooled analysis of all surveys conducted during that year.

NHTSA: NCAP Communications Research – OMB Control Number 2127-new

Quantitative ICR Package – Section B

DRAFT: May 21, 2019

9