ADDITIONAL METHOD INFORMATION
TITLE OF INFORMATION COLLECTION: Driver Interactions with Driver Assistance Technologies
OMB CONTROL NUMBER: 2127-NEW
NHTSA has proposed to perform driving behavior research involving the collection of information from the public as part of a multi-year effort to learn about drivers’ use of and behavior while interacting with Advanced Driver-Assistance System (ADAS) technologies. This research will help inform NHTSA regarding the current state of implementation of ADAS technologies in passenger cars, how drivers interact with them, how differences in human-machine interfaces may impact driver behavior and driving safety, and how ADAS technologies may be useful for improving driving safety. The research is not being performed to support any current rulemaking activity.
The research will involve on-road, semi-naturalistic observational driving experimentation in which participants who are members of the general public will drive government-owned instrumented production passenger vehicles while using ADAS features. Participants’ eye glance and driving behavior will be observed and recorded, as well as actions to engage the assistance features and responses to unrequested disengagements. Participants will include individuals who own particular ADAS-equipped vehicle models and individuals without ADAS experience.
The research will be conducted in two equal-sized parts. The first part of the data collection will begin upon receipt of PRA clearance and Initial research will focus on Adaptive Cruise Control (ACC) and either Lane Keeping Assistance (LKA ) or Lane Centering Assistance (LCA). Part two of the research will use the same approach to examine other ADAS features available at the time of completion of the first part of data collection.
More details regarding the research approach are given in the following sections.
Advanced Driver Assistance Systems (ADAS) are systems designed to help drivers with vehicle control tasks such as steering the vehicle to keep it within the marked lane, parking, avoiding collisions, reducing blind spots, and maintaining a safe headway (USDOT, 2018). These systems are increasingly being introduced into vehicles and have the potential to improve safety and reduce driver workload. In 2018, at least one ADAS feature was available on 92.7% of new vehicles available in the United States (American Automobile Association, 2019). As a greater proportion of drivers are exposed to ADAS, research is needed to understand how drivers interact with such systems, and subsequently how these interactions affect driving performance and driving safety.
Current ADAS differ in their human-machine interfaces (HMI) and degree of assistance provided to the driver, ranging from solely providing passive visual, auditory, or tactile alerts to the driver to automating specific aspects of the driving task, such as steering, acceleration, and deceleration. However, it is important to keep in mind that these systems are not capable of fully automating the driving task and so the driver must remain engaged in the driving task.
The ADAS technologies Adaptive Cruise Control (ACC), Lane Keeping Assistance (LKA), and Lane Centering Assistance (LCA) are the focus of the first part of data collection. The second part of data collection will focus on ADAS technologies available at that time. Currently, ACC, LKA, and LCA are among the most widely accessible ADAS features, available in 25.7% of vehicle models with ADAS features in the United States in 2018 (American Automobile Association, 2019). Adaptive Cruise Control builds on the function of traditional cruise control by adding braking capability. Lane Keeping Assist monitors the vehicle’s position within the road lane and will provide steering inputs as needed to maintain the vehicle within the marked lane. Whereas LKA is a reactive system that intervenes once a vehicle begins to drift, Lane Centering Assistance is a proactive system that continuously provides steering inputs to maintain the vehicle in the center of the lane. In most vehicles, these features require that the driver have at least one hand on the steering wheel in order for them to function. GM Super Cruise, however, uses a camera focused on the driver’s face to ensure that his or her eyes are looking at the forward roadway as a condition of its operation.
ACC, LKA, and LCA technologies may aid in mitigating crashes. However, these assistance systems have limitations and there are situations for which the technologies are not designed to handle and, thus, require the driver to be alert and attentive to the driving situation. Dense traffic, complex road characteristics, or adverse weather may require the driver to quickly resume manual control. Because these circumstances are not consistently defined and known in advance, drivers must actively monitor the environment and must be ready to resume full vehicle manual control immediately upon notification by the system. How a driver reacts to such notifications will have a substantial impact on road safety. Therefore, the design of the human-machine interface, driver monitoring, related alerts, status information, and the degree to which these are easy for the average driver to understand will determine the systems’ impact on safety. A poorly designed human-machine interface (HMI) may lead to issues for drivers such as mode confusion, function misuse/abuse, and inefficient driver response to transfer-of-control alerts. This research will aid NHTSA in understanding ADAS HMIs and the issues involved, how notification of disengagements are communicated, and how drivers understand and respond to such notifications.
Currently available ADAS functions have different methods of a) presenting system status information and b) notifying the driver when they must resume manual operation of the vehicle. The HMI of the vehicle must fully and succinctly communicate ADAS status to the driver. If the interface does not provide sufficient feedback regarding ADAS status, the driver may not know, for example, whether the vehicle will stop under the appropriate conditions or when manual steering is required (Brinkley, Dunbar, Smith, & Gilbert, 2017; Lee & Eom, 2015). Known as mode confusion, this has serious implications for safety. A lack of understanding of what mode the vehicle is operating under may lead to incorrect expectations for how the vehicle will perform.
In addition to communicating system status, ADAS technologies must be able to adequately notify the driver that resumption of manual control is required. Because ADAS are assistance systems with limited capabilities, it is necessary for drivers to understand ADAS limitations and when they may need to resume manual control of the driving task. This can be affected by many factors, a major one being whether the need to resume manual operation was expected. One study, Ruscio, Ciceri, and Biassoni 2015, found that when a collision warning device provided misleading, incomplete, or no information regarding a need to resume manual operation, the driver’s behavior was negatively affected. Brake response times (BRTs) were slower, gaze response times were slower, and drivers took longer to respond to hazards. In contrast, when a system disengagement is expected and within system guidelines, drivers exhibit better monitoring behaviors and are better able to resume stable manual operation of the vehicle (Dogan et al., 2017; Merat et al., 2014).
However, ADAS performance depends on road conditions and other factors. Not every event that necessitates a driver to resume manual operation can be predicted. Therefore, it is important that ADAS provide timely notifications about the status of the system and, when applicable, upcoming disengagements. The safety of the ADAS system may also be enhanced by providing alerts and notifications that remind the driver to remain engaged in the driving task.
With multi-vehicle households and the increased availability of car sharing programs, drivers encountering multiple vehicle models on a regular basis is becoming increasingly common. ADAS HMI differences between vehicle models can lead to driver confusion and errors, and, for a system having the ability to control vehicle steering, acceleration, and braking, may lead to compromised safety. Experienced drivers are expected to have some understanding of the limitations of technologies like ACC, such as use in adverse weather or dense traffic conditions, and can anticipate when they will need to resume manual control. The knowledge of these types of situations increases with time spent driving the system (Larsson, et al., 2012; Strand et al., 2011). A driver’s previous experience driving vehicles with ADAS has a major effect on take-over performance. When drivers are more familiar and have had training using ADAS technologies, they resume manual control faster and have higher trust in the system (Hergeth, Lorenz, & Krems, 2017). Compared to inexperienced drivers, these faster response times become more consistent over time (Zeeb, Buchner, & Schrauf, 2016). Not only do experienced drivers react quicker, they are more likely to recognize changes in the system’s behavior before a warning even occurs. The review by Larsson, Kircher, and Hultgren (2014) found that drivers with experience using ACC were better able to respond to system behavior instead of a warning signal itself.
Different vehicle manufacturers have different strategies for HMI design. For ADAS technologies that are intended to support drivers in safely controlling their vehicles, effective communication of ADAS mode (active or not active) and state (e.g., malfunction) to the driver will ensure that the systems are used correctly and as intended. NHTSA seeks to assess the degree to which available ADAS HMIs present information to drivers in a clear and easy to understand way that facilitates appropriate system use and avoids driver frustration and confusion.
This research is intended to gather information regarding how drivers interact with ADAS, how disengagement notifications are communicated and understood, and how drivers gain experience with and learn to use a new system.
A large percentage of the research presented above was conducted using driving simulators. While driving simulators provide a good first investigation into research questions concerning ADAS, they do not provide a full picture as to how drivers interact with these systems in real-world driving. On-road, semi-naturalistic driving research will be conducted that examines ADAS HMI effectiveness and disengagement notification response behavior using late-model production passenger vehicles equipped with ADAS features.
Study objectives were selected to be consistent with the NHTSA Human Factors Research Plan that describes human factors research opportunities associated with advanced vehicle technologies. Specific objectives derive from three of the core areas.
HMI Effectiveness. Focusing on the HMI within the vehicle, the first objectives address the effectiveness of the HMI design for transition of control efficiency and drivers’ understanding of system state/mode. The specific objectives include:
Examine effectiveness of disengagement notifications for selected current production systems
Examine drivers’ understanding of status displays and mode confusion for selected current production systems.
Driver Readiness. The second core area is Driver Readiness, which involves assessing drivers’ readiness to react to indications, warnings, or requests from the vehicle. Driver engagement strategies include methods used to help drivers maintain readiness to resume manual control, such as requiring drivers to have at least one hand on the steering wheel or to have their eyes focused on the forward roadway. The specific objective is to:
Examine the effectiveness of hands-on-wheel versus eyes-on-road engagement strategies.
Adaptation to Advanced Technologies. The third core area considers the behavioral changes that occur over time as drivers/users learn and adapt to vehicle technologies. Focusing on changes that occur after weeks and months of system use, the specific objective is to:
Determine the effects of prior experience with driver assistance features using experienced drivers.
For the first part of data collection, this will include Advanced Cruise Control, Lake Keeping Assistance, and Lane Centering Assistance
For the second data collection, specific ADAS technologies are still to be determined.
These specific objectives will be addressed over a two-part data collection looking at different vehicle models equipped with ADAS technology. Within the context of these specific objectives, the following questions will be addressed:
How effective are ADAS Status Displays in Communicating System State?
How easy/difficult is it to activate the ADAS features? For the first part of data collection, how easy/difficult is to activate Adaptive Cruise Control and either Lane Keeping Assistance or Lane Centering Assistance?
How is system status conveyed to the driver?
Mode confusion:
How often are drivers confused or uncertain about system status?
What behaviors do drivers exhibit when confused about mode?
How effective are ADAS HMIs in prompting efficient transfers of control?
How well do drivers understand the sequence of system alerts intended to inform the driver of possible disengagement?
How much time does it take the driver to:
Respond to disengagement notifications?
Regain stable manual vehicle control?
What behaviors do drivers exhibit in response to disengagement notifications?
What is the first response?
How does the driver retake vehicle control?
How does familiarity with one ADAS “brand” affect drivers’ understanding of ADAS status information and efficiency of response to take-over requests? (Transfer of Learning)
How does prior experience with the same vehicle affect drivers’ efficiency of response to disengagement notifications?
How does prior experience with a different “brand” ADAS affect drivers’ efficiency of response to disengagement notifications?
How does ADAS driver engagement strategy affect drivers’ proper and appropriate use?
What is the rate of compliance with readiness prompts as a function of ADAS engagement strategy?
How does the frequency of ADAS feature use differ as a function of system engagement strategy?
How does the incidence of driver engagement in secondary tasks differ as a function of system engagement strategy?
Both parts of this data collection will employ the same approach. The study objectives will be addressed in on-road, semi-naturalistic driving experimentation. The research will compare performance between inexperienced drivers, who have no prior experience with advanced driver-assistance technologies like those used in the study, and experienced drivers, who own a vehicle that is the same as one of the study vehicles and have experience using the advanced driver-assistance technologies of their vehicle. Comparisons between inexperienced drivers and experienced vehicle owners provide a unique opportunity to examine the effects of long-term adaptation on the use of specific ADAS systems.
Data collection will be conducted on local limited-access highways. Drivers selected for participation will drive to a specified destination along a set route. To promote naturalistic behavior, they will drive unaccompanied. The drive will last approximately 3 hours. Drivers will receive training and practice on system use before their experimental drive.
Vehicle models will be selected for this research based on the ADAS features offered and the degree of performance of those ADAS features. New technologies tend to be offered on more expensive vehicles first. More capable systems will be selected over less capable ones.
The experimental design for initial research is summarized in Table 1.
Table 1. Summary of Experimental Design
Experience |
Driving Assignment |
Group |
Inexperienced Drivers |
Model A |
1 |
Model B |
2 |
|
Model A Owners |
Model A |
3 |
Model B |
4 |
|
Model B Owners |
Model A |
5 |
Model B |
6 |
Owners of the ADAS-equipped vehicle models of interest will be considered experienced drivers. Their knowledge of how to use the ADAS features being studied and proficiency will be confirmed by an experimenter during a training drive.
The initial experiment will focus on two specific ADAS systems: GM Super Cruise system on a 2018 Cadillac CT6 (Model A) and a Honda Sensing system on a 2019 Honda Odyssey (Model B). These systems differ in the complexity of the interface and the level of information detail presented to the drivers. The Honda Sensing system provides ACC and LKA information within one icon and a small number of messages to the driver. The GM Super Cruise system provides ACC and lane position information across four icons and a large number of possible messages to the driver. Additionally, and the system engagement strategy, which refers to the way in which the system ensures that drivers are actively monitoring the driving situation whenever the system is engaged differs between the two systems. The Honda Sensing system uses a more common engagement strategy of requiring periodic steering inputs to verify driver attentiveness. In contrast, the GM Super Cruise system actively monitors driver head position as a surrogate for eye fixation point and provides warnings if drivers appear not to be consistently looking at the forward roadway scene.
Subsequent research will focus on production vehicles and ADAS features that are offered in commercially available U.S. model vehicles at the time the study is run. The features to be studied are not known yet as the technology that will be available in upcoming model years is not yet known.
The sample population will be middle-aged drivers (25-65 years old) recruited from the central Ohio area who either own an ADAS-equipped vehicle or have no experience with ADAS technology.
Specific inclusion/exclusion criteria include:
Be aged 25-65 years of age
Drive at least 11,000 miles annually
Hold a valid U.S. driver license
Have no uncorrected vision or hearing problems
Be in good general health, able to drive a production passenger vehicle continuously and safely for a period of 3 hours, and without the need for assistive devices
Currently have no more than 2 points on their driving record
Have no recent criminal convictions
Self-report that they are able to read, write, speak, and understand English
Be willing to drive to NHTSA’s Vehicle Research and Test Center and spend up to approximately 4 hours participating in a research study
Not have anyone in their household who works in or is retired from any automotive manufacturer, which may constitute a conflict of interest with the research.
Support for selection of the first two listed criteria follows.
This research will examine driver behavior for the largest portion of the U.S. driver population relevant to the ADAS technologies and the vehicle models selected. Therefore, we intend to focus on drivers aged 25 to 65 years old. Examining differences amongst age groups (e.g., younger and older drivers) and observing behaviors of individuals that are not likely buyers of these vehicle models and technologies is not of interest. The 25-65 year age group represents the largest proportion of U.S. drivers, encompassing over 69% of licensed drivers in 2017 (FHWA 2017), with the most homogenous driving behavior. Research has found that middle-aged drivers are less likely to be involved in crashes and, when they are involved in a crash, they are less likely to be at-fault compared to younger and older drivers (Stutts, Martell, Staplin, & TransAnalytics, 2009). While older drivers are becoming more numerous, they still represent a smaller proportion of drivers and on average drive less than 8,218 miles annually (McGukin & Fucci, 2018).
Rationale for Initial Research Participant Age Range
Focusing on a middle age range for research participants is appropriate for this work for multiple reasons:
Initial research will use a high-end, technology laden sedan, the Cadillac CT6, and a popular minivan, Honda’s Odyssey. These models tend to be more expensive, luxury vehicles. Given that the research will examine ADAS technology familiarity effects (owner vs. inexperienced), we are requiring that a portion of participants be owners of a vehicle model equipped with the ADAS technologies being studied. The typical owner and primary driver of these models is believed to be middle aged. The pool of younger and older driver owners is likely to be extremely small and including these age groups in the research would make participant recruitment very challenging. Further, a study has reported that older adults (older than 65) have more difficulty and require more time to learn ADAS technologies (Jenness et al., 2008), which may warrant special training for older drivers that would not be comparable to that provided to younger aged participants. Lastly, including younger and older drivers would drastically increase the time needed to complete the experiment and ultimately limit the usefulness of the research. Focusing on a middle age range for participants will help the research to be completed in a more efficient manner and ensure its timeliness and usefulness for impacting the state of the technology. Subsequent research will consider examining other age groups with vehicle models appropriate for those age ranges.
Minivans are known as family vehicles used to transport multiple children. An August 2018 article by Carmax stated that the average age of Honda buyers at Carmax was 39.3 years (accessed April 6, 2020 at https://www.carmax.com/articles/which-car-brands-have-oldest-youngest-buyers).
While Cadillacs in the past have had some reputation as being popular with older drivers, Cadillac buyer age is trending downward in recent years. An August 2018 article by Carmax stated that the average age of Cadillac buyers at Carmax was 47.1 years (accessed October 21, 2019 at https://www.carmax.com/articles/which-car-brands-have-oldest-youngest-buyers). A January 2018 news article stated that “Through a new advertising strategy aimed at younger buyers, Cadillac is now selling nearly half its vehicles in the U.S. to customers younger than 56.” (Accessed October 21, 2019 at https://www.freep.com/story/money/cars/general-motors/2018/01/16/cadillac-xt-4-larger-third-row-crossover-early-2019/1037805001/.)
This research will employ a minimum annual mileage criterion of 11,000 miles for study participants. This is to ensure that participants drive an amount comparable to most U.S. drivers. Inexperienced driving on highways may divert attention and result in abnormal behavior when using the ADAS technologies, which would hinder the ability to generalize results to the greater population. This criterion is based on the 2017 National Household Travel Survey (McGuckin & Fucci, 2018), which found that the average annual mileage driven on public roads by U.S. drivers is approximately 11,621 miles. More specifically, the average annual mileage driven on public roads by middle age drivers is approximately 13,000 miles.
The first part of the data collection will involve 96 participants or 16 in each group based on the power analysis summarized below. The second part of the data collection involving other ADAS features will use the same approach and is estimated to involve up to 200 additional participants to permit the possibility of including three ADAS-equipped vehicle models.
Power and Sample Size Determination
The objective of the analyses presented herein is to determine whether our selected group size provides sufficient statistical power to detect a mean difference between groups. Based on past research from the Virginia Tech Transportation Institute Naturalistic Driving Study (VTTI NDS) (Russell et al., 2018), we estimate that the means for time to intervene will differ by 1.0 seconds. Our best estimate for population standard deviation is 1.0 seconds and is also based upon this past research. This indicates that the effect size when comparing means will be large using Cohen’s (1988) index of effect size. Accordingly, a distribution of means, each computed using data points sampled from the original distribution, will have the standard error (SE) of the original distribution as its standard deviation (SD). Using the estimated SD of the overall distribution from the VTTI study, we computed SE values using different numbers of data points in each sample and use the estimate of SE as the SD in the power computation formulas.
Each participant will respond to many take-over requests over the course of the drive. Because our “samples” are repeated trials performed by a single participant, we also estimated the extent to which the correlation of these events will affect our ability to detect a significant effect. Two analyses using GLMPOWER were performed, one with 5 trials (events) for each participant and one with 10 trials (events). For simplicity, the mean values were set to be the same across each repeated trial. The SAS GLMPOWER procedure was used with different assumed levels of correlation to estimate the number of participants required to find the target 1.0-second difference with power = 0.8 using 5 repeated trials per participant. These results are presented in Table 3.
Table 3: Sample Sizes for Main Effect with Different Levels of Correlation due to Repeated Measures (5 events; 1-second difference)
Correlation due to Repeated Measures |
TOTAL N: SD = 1.0 |
N per Group |
0.0 |
10 |
5 |
0.2 |
14 |
7 |
0.4 |
20 |
10 |
0.6 |
24 |
12 |
0.8 |
30 |
15 |
These results indicate that the sample size required to detect a difference between means of 1 second with power equal to 0.8 is at most 15 participants. The advantage of having 10 repeated measures per participant was explored using the same procedure with the large effect size (ES) and estimated sample sizes are shown in Table 4.
Table 4: Sample Sizes for Main Effect with different Levels of Correlation due to Repeated Measures (10 events; 1-second difference)
Correlation due to Repeated Measures |
TOTAL N: SD = 1.0 |
N per Group |
0.0 |
6 |
3 |
0.2 |
12 |
6 |
0.4 |
18 |
9 |
0.6 |
24 |
12 |
0.8 |
28 |
14 |
With relatively small sample sizes required to detect the large ES, the gains associated with increasing the number of repeated trials from 5 to 10 are small as can be seen by comparing the results in Table 3 with those in Table 4.
Conclusion
In our design, comparisons will be made between groups that have 16 participants. The present results indicate that this group size should be sufficient for detecting a difference between group means of 1.0 seconds with power = 0.8, assuming a minimum of 5 samples per participant and independent of the correlation among the participants’ repeated responses. In the context of Cohen’s d, this represents a large ES.
We are not able to estimate the expected correlation among repeated measures that will be present in our data; however, we expect the correlation to be smaller than it would be in carefully controlled laboratory studies because most responses in our study will come from different types of situations under different traffic conditions. Thus, while the correlation among the repeated measures may be low due to situational factors, the variability due to the differences among situations is expected to elevate variability in the overall distribution.
These estimates rely heavily on the variability in the overall distribution of response time values. The analyses used a standard deviation of 1.0 seconds, which was computed from the SE and N values presented by VTTI in the NDS report. This distribution has response times from 83 events and represents the best available information about response time to intervene in response to ADAS take-over events.
Two approaches to recruiting participants will be used. Inexperienced participants will be recruited through advertisements placed in local newspapers, including those in Marysville, Bellefontaine, Delaware, and Kenton, Ohio. Ads will be placed in physical newspapers, on newspaper websites, and on Facebook. Experienced drivers who own vehicles with one of the two selected ADAS will be recruited via direct contact. Vehicle registration data will be obtained from the State of Ohio’s Bureau of Motor Vehicles. Owners of the selected vehicles will be reached by mail via postcard with the study description and instructions on how to apply to participate.
The ads and postcard will contain a link to a website that describes the study in detail including the general participation requirements. A link from the first website will direct interested respondents to a second, secure website, which contains two sets of screening questions to be answered online. The first set of questions addresses the general requirements plus contact information and more detailed questions about driving license and record. Responses to these initial questions are used to assess general eligibility for participation. A second set of questions will ask more detailed questions, including details about health that might affect drivers’ ability to complete a long drive.
To facilitate recruitment, an online application procedure will be used, which allows potential participants to complete the screening questionnaire online. Information obtained in this manner will be downloaded routinely for use in assessing the eligibility of respondents and for scheduling participation. A phone number will also be provided in the advertisements as an alternative to the online application, in case there is anyone who is unable to respond using the online method.
For Question Sets 1 and 2, no payment or gift will be provided to respondents.
For the observational driving study and the post-drive questionnaire, NHTSA plans to provide monetary payment at rate of $50 per hour for study participation. Such compensation is consistent with normal experimental practice to compensate participants for their time and encourage participation in research studies.
The compensation rate is set using a calculation method approved by NHTSA’s Office of Acquisition Management and will be reviewed by an independent Institutional Review Board. The compensation amount calculation begins with an hourly rate corresponding to a nonprofessional federal government employee (GS‐8, Step 1, $23.76) in the locality (Columbus, OH) in which the study is conducted. Additional amounts are added to this rate to compensate for things such as special participant criteria (e.g., technology experience). In addition, study participants will be reimbursed at the current GSA mileage rate for miles traveled to and from the test site located approximately 30 miles outside of Columbus, Ohio. The current government travel allowance is 57.5 cents per mile (beginning January 1, 2020).
A PC-based data acquisition system will record vehicle control data extracted primarily from the vehicle’s On-Board Diagnostics (OBD-II) data available on the Controller Area Network (CAN) bus and video data from multiple cameras. Video and vehicle dynamics data will be recorded continuously during the entire drive in order to gather information regarding driving performance and eye glance behavior. Camera locations and a description of the data they are intended to capture are listed in Table 2. Additionally, a Z ZAFFIRO Lavalier USB microphone will be located above the steering column wheel to capture audio during the experimental session. Data will be captured at a rate of 30 Hz.
Nine AXIS pinhole cameras will be installed throughout the vehicle in order to record participants’ eye glance and hand position locations. These cameras have either a 2.8 mm or 3.7 mm fixed lens with a resolution of 1280 x 720 and an 81-degree horizontal field of view. The following are descriptions of camera locations in Table 2.
Table 2. Camera Locations
Camera Name |
Location |
What is Captured |
Forward Center Road |
front of the vehicle |
lane position and the number of forward vehicles in front of the participant |
Steering Column 1 |
above the steering column |
left face for eye glance behavior |
Steering Column 2 |
above the steering column |
right face for eye glance behavior |
Steering Column 3 |
above the steering column |
instrument panel to assess the advanced driver-assistance system status |
Passenger A-Pillar |
passenger a-pillar |
eye glance behavior |
Over Center Console |
above the center console |
hand positions on the steering wheel |
Top of Instrument Panel |
above the instrument panel |
eye glance behavior |
Driver Right Shoulder |
Looking from behind over the driver’s right shoulder |
hand positions on the steering wheel |
Rear Centered Road |
rear of the vehicle |
rearward roadway and traffic to capture the number of rearward vehicles |
Observational data collection will take place during the course of on-road driving on central Ohio highways. Highway driving is appropriate for this research as the ADAS functions being studied are intended for use on roadways for which cruise control type functions are appropriate. Furthermore, GM’s Super Cruise system operates only on “compatible highways”1 that have been mapped by General Motors.
Participants will begin their experimental drive at the Transportation Research Center (TRC) proving ground and travel along public highways for approximately 80 miles. The route will involve multi-lane divided highways, exiting and merging onto different highways, and continuing onto new highways. Participants will be instructed a specific turn-around point and a similar route to return.
The test procedure will consist of six main steps: pre-brief, instruction/training, practice drive, experimental drive, post-drive questionnaire, and payment.
Pre-brief: Upon arrival at the research site, participants will first come into the laboratory for an introductory briefing. A member of the research staff will meet with the participant and have them read and sign the informed consent document.
Instruction/Training: Next, the participant will be given an overview of the experiment and general descriptions of the ADAS technologies he/she will be using while driving. Participants will be shown the specific vehicle and will view instructional videos describing the functions of and how to use the ADAS features. These videos are materials developed by the vehicle manufacturers to be provided to vehicle owners. Participants will be provided directions for the route he/she will be driving and given instructions on what to do in case of an adverse event.
Practice Drive: Participants will then perform a practice drive with the experimenter in the vehicle, while following a lead vehicle, along public roads. During the practice, the experimenter will instruct the participant to engage and disengage the assistive technologies and allow the participant to practice using the technologies as they wish. Once the participant feels confident in using the technologies, the participant will take an exit to a pre-determined stop where the experimenter will leave the participant’s vehicle.
Experimental Drive: The participant will then continue along the full specified route. Participants will be told to use the ADAS features as much as possible, but that their priority should be driving safely. Additionally, participants will be told that they are not allowed to use a cell phone while driving. However, a phone will be present in the vehicle in a secure location not reachable from the driver’s seat. The participant will be instructed that in the event of an emergency, he/she should maneuver the vehicle off of the active roadway, access the provided cell phone, and contact emergency personnel or the research staff for assistance, as appropriate.
Post-drive Questionnaire: Once participants complete the full route and return to the research site, participants will fill out a post-drive questionnaire. Completion of the questionnaire will signal the end of the experiment.
Payment: Upon completion of the post-drive questionnaire, the participant will be given a short debrief description of the research, allowed to ask any additional questions they have, and then will be provided payment. The entire experimental session will last approximately 4 hours in duration.
Information will be collected during the course of the research during participant recruitment, observation of driving behavior, and post-drive questionnaires. Questions addressed to individuals will serve to assess individuals’ suitability for study participation, to obtain feedback regarding participants’ use of the ADAS technologies, and to gauge individuals’ level of comfort with and confidence in the technologies’ performance and safety.
The information collection components for initial research and the information desired are listed below. Information collection tools for subsequent research will be of the same format, but will refer to other ADAS features to be determined (see Footnote 1).
Question Set 1, Interest Response Form (NHTSA Form 1522) – Necessary for determining individuals’ willingness to participate in the study and confirming basic qualification and vehicle ownership.
Question Set 2, Recruitment Screening Questions Form (NHTSA Form 1523) – Necessary for determining individuals’ suitability for study performance based on driving experience and history and their general health and ability to drive for approximately 3 hours without assistive equipment or health concerns.
Informed Consent Form (NHTSA Form 1524) – Used to ensure that participants understand the protocol they will be participating in and for documenting their agreement to participate.
Passive observation of driving behavior – Necessary for gathering driving behavior and advanced driver-assistance technology use data to answer research questions such as those related to assessing drivers’ compliance with system prompts and understanding of system status.
Question Set 3, Post-Drive Questionnaire (NHTSA Forms 1525) – Necessary for understanding drivers’ opinions regarding advanced driver-assistance technology performance, degree of comfort with system use, and perceptions of safety associated with the use of these features.
Question Set 4, Post-Drive Questionnaire, Owner (NHTSA Forms 1527) – Necessary for understanding experienced ADS users’ opinions regarding advanced driver-assistance technology performance, degree of comfort with system use, and perceptions of safety associated with the use of these features. The Interest Response Form and Recruitment Screening Form data will solely be used to determine individuals’ suitability for study participation and will not be analyzed in any way. Driving behavior and post-drive questionnaire responses will be combined for analysis.
Research staff will perform analyses on dependent measures of driving performance, driver behavior in response to disengagement notifications, and driver use of the ADAS system, and subjective impressions of the ADAS features assessed via the post-drive questionnaires. Vehicle control metrics such as speed, headway, steering reversal rate, accelerator use, brake use, and lane-position variability will be compared across conditions.
Research staff will also compare driver behavior in response to disengagement notifications across conditions. Response time will be measured as the time from disengagement until the driver either: presses the brake pedal, presses the accelerator pedal, or moves the steering wheel beyond a set threshold. Additionally, research staff will quantify behaviors in response to a notificationsuch as hand positions and glance behavior . These may include the time from disengagement to 1) placing both hands on the steering wheel, 2) glancing to the instrument panel, 3) glancing to the forward roadway, and 4) glancing to the center rearview mirror.
Research staff will also compare dependent measures regarding ADAS system use across conditions. These will include the frequency and duration of ADAS system use, as well as the conditions associated with any driver-initiated retake of manual control.
Beyond comparing across conditions, analyses may include examining differences in ADAS use with different road conditions. This may include comparing driver performance, response to disengagement, and ADAS system use during curves compared to straight roadways and differences in dense traffic compared to light traffic.
Research staff will assess subjective impressions of the ADAS systems following the drive by using a post-drive questionnaire to gather subjective ratings of ease of use, trust, and safety.
Data obtained in this experiment will be stored on networked file servers with shared access at NHTSA’s Vehicle Research and Test Center in East Liberty, Ohio. To access the data, one would need authorization to use the computer network of the National Highway Traffic Safety Administration and be authorized to access the specific network drive on which the data files are stored.
Research staff will inform potential participants of the opportunity for study participation via online and print advertisements as well as letters mailed to Ohio registered owners of vehicles with relevant study-related features. The advertisements and letters will direct interested individuals to either complete a short, printed set of questions or to go to a web site address and complete the questions online (via a secure website: Formsite or Qualtrics).
Interested candidates will be asked to provide the following information for use in determining their suitability for participation:
• Sex
• Date of birth (mm/dd/yy)
• Driving frequency
• Personally-owned vehicle model and characteristics
• Advanced driver-assistance system feature use experience
• Contact information:
Residential address
• E-mail address
• Phone number.
Participants must meet the following requirements in order to participate:
• Be aged 25-65 years
• Drive at least 11,000 miles annually
• Hold a valid U.S. driver’s license
• Have no uncorrected vision or hearing problems
• Be in good general health, able to drive a production passenger vehicle continuously and safely for a period of 3 hours and without the need for assistive devices
• Drive at least 11,000 miles annually
• Currently have no more than 2 points on their driving record
• Have no recent criminal convictions
• Self-report that they are able to read, write, speak, and understand English
• Be willing to drive to NHTSA’s Vehicle Research and Test Center and spend up to approximately 4 hours participating in a research study
• Not have anyone in their household who works in or is retired from any automotive manufacturer, which may constitute a conflict of interest with the research.
Research staff will handle this participant recruitment information as described in the following steps:
Recruitment screening questions will be administered via one of two means:
Electronically: Accessed via a secure web site, individuals interested in study participation complete the web-hosted question sets using their own computers or other means. Question sets will be hosted on a secure website (either Formsite or Qualtrics).
On Paper: Completed by hand and submitted via US mail. Individuals receiving mailed participation invitations can fill out the enclosed question form and mail it in a provided envelope to the research site. Research staff receiving these forms entered the responses into a recruitment data file that also contains the electronically entered screening question responses.
The candidate participant file is maintained on a limited access file server network drive for access by individuals working on the research project.
Research staff review the candidate data to determine which individuals meet the initial study participation criteria. If the participant is eligible based on their answers to the initial screening questions, research staff will send a screening form with additional screening questions electronically.
Research staff review the second form’s candidate data to determine which individuals meet the study participation criteria. Qualifying individuals are contacted by e-mail or phone to schedule a participation appointment.
Once a person is scheduled to participate, they are assigned a participant number and their contact and demographic information along with their participant number are stored in a “participant info” file. This file is password protected and only accessible by the few individuals that need to access it to schedule participants. The participant information file is maintained on the limited access network file server.
Participants’ eye glance and driving behavior will be recorded as follows:
• Eye glance behavior via face image recording from cameras. Camera image data will be collected and recorded as binary ROS .bag files from which data will be extracted. Final eye glance data will be saved in three formats:
Video data
Binary pickle file for neural network input
Text data (neural network outputs eye glance location determination).
• Hand location behavior via upper body image recording from cameras. Again, camera image data will be collected and recorded as binary ROS .bag files from which data will be extracted:
Video data
Binary pickle file for neural network input
Text data (neural network outputs hand location determination).
• Any subjective comments about ADAS feature performance or driver frustration with the system will be captured via audio recording.
Surrounding traffic situation via video recording using cameras pointed out of vehicle windows.
• Vehicle control inputs obtained via the vehicle CAN bus.
• Vehicle motion data via electronic sensors.
The vehicle will have a data acquisition system (DAS) that includes sensors that read vehicle dynamics information from the vehicle and save it to a computer hard drive. Upon completion of the study drive, driving observation data are handled as described in the following steps:
Binary ROS .bag files data are copied from the DAS to a portable hard drive.
Those binary ROS .bag files are copied from the portable hard drive to a limited access, password-protected, and bitlocked Local Area Network (LAN) workstation. On this LAN workstation, data is extracted from the binary ROS .bag files. Camera image data is extracted as jpeg images, video files, and binary pickle files. Jpeg images and video files which contain face images remain on the LAN workstation for protection. The pickle files save the image data as compressed images with noise in a binary format and are not readable as face images. Vehicle control inputs and motion data are extracted to .csv files.
The binary pickle files will be copied to an additional computer for neural network processing to determine eye glance location and hand position. For neural network processing, the files will always remain in the binary pickle format. Following neural network processing, these binary files will be moved back to the bitlocked LAN workstation. Neural network results of eye glance location and hand position are saved as .csv files. These .csv files only contain location information, no face images.
A subset of the experimental data, .csv files containing vehicle control input and motion data, along with neural network results will be copied to a limited access file server network drive for access by individuals working on the research project. These data contain no face images and are only identified via participant number.
Eye glance data confirmation and correction will take place only on the LAN workstation.
Research staff will work with the subset of files that do not contain Personal Identifying Information (PII) by accessing them on the network file server and saving any related analysis files on the same server.
After completing experimental driving, each participant will complete a post-drive questionnaire. The questionnaire will be administered via a computer and responses will be stored electronically on a local, non-networked drive. The questionnaire data will only be identified by participant number and will contain observations and opinions regarding their experience in driving the study vehicle and using the advanced driver-assistance features. No personal information will be requested in the questionnaire.
Questionnaire data are handled as described in the following steps:
Questionnaires are administered electronically using a personal computer with no network file server access and responses are saved on that computer.
Data are copied from the computer to a limited access file server network drive for access by individuals working on the research project. These subjective questionnaire data are only identified via participant number.
Research staff work with the subset of files that do not contain PII by accessing them on the network file server and saving any related analysis files on the same server.
To maintain confidentiality, participants will be identified by subject number. The Personal Identifying Information (PII) obtained during recruitment (name, address, email address, phone number) will always be separated from the physiological and driving performance data obtained during participation in the experiment. Participant gender and participant age may be used as variables for data analysis.
To prevent unauthorized access, the PII obtained during recruiting will be maintained in a single Microsoft Excel file with password protection for encryption. Answers to several medical questions, included to ensure that prospective participants meet the basic study criteria and will be able to safely operate a vehicle and drive continuously for three hours, will be maintained separately from the PII. Once eligibility decisions have been made and recruitment is complete, such information will be destroyed. All remaining PII will be destroyed at the end of data collection.
The experimental data will be retained indefinitely and may be used in public research presentations or made available to other research organizations for scientific purposes. Any released data would only contain participants’ subject number, sex and age.
The Informed Consent Form contains the following statement regarding information disclosure:
By signing the information disclosure statement contained in this document, you agree that the National Highway Traffic Safety Administration (NHTSA) and its authorized contractors and agents will have the right to use the NHTSA engineering data and the NHTSA video and audio data for scientific, educational, research, or outreach purposes, including dissemination or publication of your likeness in video or still photo format, but that neither NHTSA nor its authorized contractors or agents shall release your name; and you have been told that, in the event of court action, NHTSA may not be able to prevent release of your name or other personal identifying information. NHTSA will not release any information collected regarding your health and driving record, either by questionnaire or medical examination. Your permission to disclose this information will not expire on a specific date.
American Automobile Association. (January 2019). Advanced Driver Assistance Technology Names. Retrieved from https://www.aaa.com/AAA/common/AAR/files/ADAS-Technology-Names-Research-Report.pdf
Blanco, M., Atwood, J., Vasquez, H. M., Trimble, T. E., Fitchett, V. L., Radlbeck, J., ... & Morgan, J. F. (2015, August). Human factors evaluation of level 2 and level 3 automated driving concepts. (Report No. DOT HS 812 182). Washington, DC: National Highway Traffic Safety Administration.
Brinkley, J., Dunbar, J., Smith, J. D., & Gilbert, J. E. (2017). A Usability Evaluation of the BMW Active Cruise Control System With “Stop and Go” Function. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 61, 1536-1540.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates.
Dogan, E., Rahal, M.-C., Deborne, R., Delhomme, P., Kemeny, A., & Perrin, J. (2017). Transition of control in a partially automated vehicle: Effects of anticipation and non-driving-related task involvement. Transportation Research Part F: Traffic Psychology and Behaviour, 46, 205–215.
Eriksson, A., & Stanton, N. A. (2017). Takeover Time in Highly Automated Vehicles: Noncritical Transitions to and From Manual Control. Human Factors, 59(4), 689–705.
Federal Highway Administration (2017). Highway Statistics 2017. Washington, D.C.: Federal Highway Administration, US Department of Transportation. https://www.fhwa.dot.gov/policyinformation/statistics/2017/dl220.cfm
Gold, C., Berisha, I., & Bengler, K. (2015). Utilization of Drivetime – Performing Non-Driving Related Tasks While Driving Highly Automated. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 59(1), 1666–1670.
Gold, C., Körber, M., Lechner, D., & Bengler, K. (2016). Taking Over Control from Highly Automated Vehicles in Complex Traffic Situations: The Role of Traffic Density. Human Factors, 58(4), 642–652.
Hergeth, S., Lorenz, L., & Krems, J. F. (2017). Prior Familiarization with Takeover Requests Affects Drivers’ Takeover Performance and Automation Trust. Human Factors, 59(3), 457–470.
Jamson, A. H., Merat, N., Carsten, O. M. J., & Lai, F. C. H. (2013). Behavioural changes in drivers experiencing highly-automated vehicle control in varying traffic conditions. Transportation Research Part C: Emerging Technologies, 30, 116–125.
Jenness, J. W., Lerner, N. D., Mazor, S. D., Osberg, J. S., & Tefft, B. C. (2008). Use of Advanced In-Vehicle Technology by Younger and Older Early Adopters. Selected Results From Five Technology Surveys (No. HS-811 004).
Körber, M., Gold, C., Lechner, D., & Bengler, K. (2016). The influence of age on the take-over of vehicle control in highly automated driving. Transportation Research Part F: Traffic Psychology and Behaviour, 39, 19–32.
Larsson, A. F. (2012). Driver usage and understanding of adaptive cruise control. Applied ergonomics, 43(3), 501-506.
Larsson, A. F. L., Kircher, K., & Andersson Hultgren, J. (2014). Learning from experience: Familiarity with ACC and responding to a cut-in situation in automated driving. Transportation Research Part F: Traffic Psychology and Behaviour, 27, 229–237. https://doi.org/10.1016/j.trf.2014.05.008
Lee, S. H. & Eom, H. (2015). Design of driver-vehicle interface to reduce mode confusion for adaptive cruise control systems (pp. 61-71). ACM Press. https://doi.org/10.1145/2809730.2809757
McGuckin, N., & Fucci, A. (2018). Summary of travel trends: 2017 National household travel survey (Report FHWA-PL-18-019). Washington, DC: Federal Highway Administration, US Department of Transportation.
Merat, N., Jamson, A. H., Lai, F. C. H., Daly, M., & Carsten, O. M. J. (2014). Transition to manual: Driver behaviour when resuming control from a highly automated vehicle. Transportation Research Part F: Traffic Psychology and Behaviour, 27, 274–282. https://doi.org/10.1016/j.trf.2014.09.005.
Naujoks, F., Mai, C., & Neukum, A. (2014). The effect of urgency of take-over requests during highly automated driving under distraction conditions. Advances in Human Aspects of Transportation, 7(Part I), 431.
Radlmayr, J., Gold, C., Lorenz, L., Farid, M., & Bengler, K. (2014). How Traffic Situations and Non-Driving Related Tasks Affect the Take-Over Quality in Highly Automated Driving. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 58(1), 2063–2067.
Ruscio, D., Ciceri, M. R., & Biassoni, F. (2015). How does a collision warning system shape driver’s brake response time? The influence of expectancy and automation complacency on real-life emergency braking. Accident Analysis & Prevention, 77, 72–81.
Russell, S.M., Blanco, M., Atwood, J., Schaudt, W.A., Fitchett, V.L., and Tidwell, S. (2018, May). Naturalistic Study of Level 2 Driving Automation Functions. Unpublished report. Washington, DC: National Highway Traffic Safety Administration.
SAE International. (2018). Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. SAE Standard J3016.
Strand, N., Nilsson, J., Karlsson, I. C. M., & Nilsson, L. (2011, October). Interaction with and use of driver assistance systems: a study of end-user experiences. In Proceedings of the 18th ITS World Congress. Washington, DC: ITS America.
Stutts, J., Martell, C., Staplin, L., & TransAnalytics, L. L. C. (2009). Identifying behaviors and situations associated with increased crash risk for older drivers (No. DOT HS 811 093). United States. National Highway Traffic Safety Administration. Office of Behavioral Safety Research.
U.S. Department of Transportation (October, 2018). Preparing for the Future of Transportation; Automated Vehicles 3.0. Accessed on October 16, 2019 at https://www.transportation.gov/av/3/preparing-future-transportation-automated-vehicles-3.
Zeeb, K., Buchner, A., & Schrauf, M. (2016). Is take-over time all that matters? The impact of visual-cognitive load on driver take-over quality after conditionally automated driving. Accident Analysis & Prevention, 92, 230–239.
1 https://www.cadillac.com/world-of-cadillac/innovation/super-cruise
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Inez Finley |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |