Improving Safety of Human-Robot Interaction
OMB Control No. 0920-xxxx
NEW
Supporting Statement Part B –
Collections of Information Employing Statistical Methods
Project Officer: Justin Haney, PhD
Title: Associate Service Fellow
Phone: 304-285-6179
Email: poe5@cdc.gov
Fax: 304-285-5866
Date: December 11, 20202
B.1. Respondent Universe and Sampling Methods 3
B.2. Procedures for the Collection of Information 3
B.3. Methods to Maximize Response Rates and Deal with No Response 6
B.4. Test of Procedures or Methods to be Undertaken 6
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 6
Part B. Collections of Information Employing Statistical Methods
RESPONDENT UNIVERSE: The target study population is workers who currently work or had worked in the manufacturing industry, with varying job experiences. Study participants will be recruited in Morgantown, Pittsburgh and surrounding areas. Currently, there are 3,900 employees in Morgantown and 84,600 workers in Pittsburgh area in the manufacturing industry, based on the US Bureau of Labor Statistics report for November 2019. Efforts will be made to include balanced samples of males and females and diverse racial and ethnic groups to have a representative population of the manufacturing industry. Currently, females represent 30% of the manufacturing worker population. Black, Asian, and Hispanic/Latino represent 10%, 7%, and 17% respectively
SAMPLING METHODS: The adequate sample sizes are estimated to be 78 using a repeated measures ANOVA assuming small effect size of .10, a power of .80, and a significance level .05. Small effect size has been selected given that the adequate variance is unknown for this study. There is no dropout rate associated with simulator sickness available in the literature, but a dropout rate of 15% is estimated based on prior research involving a driving simulator (Balk, Bertola, & Inman, 2013). To be conservative, we will estimate the dropout rate to be 40% for this study, since participants may also dropout (or stop responding) between testing sessions if they opt to do the experiments on separate days. Therefore, 111 participants will be recruited.
ENROLLMENT
Research participants will be recruited from Morgantown and Pittsburgh areas, among current or former manufacturing workers who have varying job experience. Researchers will contact manufacturing employers and labor organizations and ask them to post the flyers in their workspace or to email them to their employees. We will also post the flyers in public spaces such as public bus stops, and mall bulletin boards. After confirming interest and eligibility with potential participants, they will receive email confirmation of participation. When a prospective participant contacts study personnel to inquire about the study, a brief description of the study will be given. If after hearing a description of the study they are interested in participating, they will be asked (1) if they are over the age of 18 (2) if they have experience working in the manufacturing industry (3) if they have history of motion sickness or simulator sickness (Attachment D - Instrument A).
Initially all potential participants who meet the inclusion and exclusion criteria will be enrolled.
If our target number of participants for certain demographics groups are enrolled before other target groups, the enrollment of the group will stop, and the focus will switch to enrolling the target number of other group participants. If a participant chooses to withdraw from the study, the information collected to that time may still be used to preserve the scientific integrity of the study, and it will be documented in the final report for the project.
The participants will be sent instructions via email regarding the expectations of the study and the consent form if the participant cares to review it prior to the study visit. If the participant does not have an email address or prefers not to receive email, with the participant's permission, the instructions and visit confirmation message will be mailed as a letter and will include the consent form. The enrollment information will be securely stored under password protection and destroyed after the research project is completed.
CONSENT PROCESS
Participants who decide to participate in the data collection for the study will be required to sign the Informed Consent Form (Attachment E - Instrument B) after having the study explained to them. The reading level of the information in the Informed Consent Form has been checked (MS Word) and corresponds to Flesch-Kincaid grade level 8. The participants will be encouraged to ask questions on any items that they may not understand.
The participants will be compensated $30/hour for their time. Partial hour payments will also be issued at the rate of $10 per each 20 minutes, and shorter periods will be rounded up to a full 20 minute period. Participants recruited for Experiments 1 and 2 will make one visit to the laboratory, which can take approximately 5 – 6 hours. They may choose to make two short visits (2 – 2.5 hours per visit) instead of one long visit. They will participate in Experiment 1 for their first visit and Experiment 2 for their next visit. All compensation will be paid with approved gift cards per the most recent government practice.
DATA COLLECTION
A training session will be provided to all investigators involved in the data-collection process at the NIOSH Morgantown facility. Prior to participant recruitment, investigators will practice the data collection procedure with two research team members to help assure the quality of the final data set.
When the participant arrives at the data collection site, the investigator will describe the study and the tasks that the participant is required to perform. If the participant agrees to participate then he/she will be asked to read the consent form approved by the NIOSH Institutional Review Board (IRB) (human subject review board) (Attachment E - Instrument B). After the informed consent process and introduction, participants’ demographic information will be collected (Attachment F - Instrument C). The Virtual Reality Sickness Questionnaire (VSRQ) (Attachment G - Instrument D) will also be conducted to assure wellness of participants at the beginning of the experiment and before releasing participants from the data collection site. The VRSQ questionnaire consists of 9 questions and computes a value between 0 (none) – 100 (severe), by assigning point values (0-3) to the responses. If at any point during the experiment the score is higher than 50, we will pause the test for 5 minutes, and then we will re-administer the questionnaire. Additionally, throughout the experiment researchers will encourage participants to be vocal about their symptoms. Participants will also complete the robot experience questionnaire (Attachment H - Instrument E). Prior to testing, researchers will apply motion capture markers to the participant using elastic straps, in order to track body segment locations during the experiment trials.
For Experiments 1 and 2 (Attachment I - Instrument F), a 10-minute practice will be provided before the actual tasks. During the practice, participants will be instructed to experience VR environment and get familiar with VR equipment. The VR environment will be same as it would be for the actual test, except no physical robot will be presented in the scene. Participants will be asked to practice picking up and releasing objects, making responses to questions, and making an emergency stop using the controller.
In Experiment 1 (Attachment I - Instrument F), each of the 18 conditions (3 robot sizes X 3 moving speed levels X 2 movement path types) will be tested once in a total of 36 experiment trials. In each trial, participants will complete simulated warehouse load/unload tasks in the virtual environment, which will require the participant to walk to the load point among the racks, pick a box from the rack, place it on the empty mobile robot, and then return to the starting point. They will complete 1 loading task and will encounter robots with the selected height (mobile base only, base with 1-level cart, or base with 2-level cart), moving speed, and movement path type (either move along straight lines and make 90 degree turns or move along the shortest routes in curved lines) while moving to the loading sections. The loading points will be at pre-selected locations. After completion of each trial, participants will be provided a list of questions asking their mental workload (Attachment J - Instrument G), perceived safety (Attachment K - Instrument H), and trust (Attachment L - Instrument I). They will make their responses using the controller, then return to the start position. Participant can take a short break before starting the next trial.
Experiment 2 will consist of 27 trials – one trial for each of 27 conditions (3 sizes X 3 moving speed levels X 3 numbers of axes). Participants will complete 27 trials of simulated pick and place tasks. For each trial, a participant and a collaborative robot with different size, moving speed, and number of axes will be in opposite sides of a working station. There will be a total of 10 objects in two different shapes (square or round, five each) on the table, and a participant and robot will pick up the target objects assigned to each and place on a tray left in front of them. Objects will be dispersed at pre-selected locations within the working area to make arms of a participant and the virtual robot cross paths. After completion of each trial, participants will answer to a list of questions asking their mental workload (Attachment J - Instrument G), perceived safety (Attachment K - Instrument H), and trust (Attachment L -Instrument I). They will make their responses using the controller, then move to the next table for the next trial. After completing every three trials, participants will return to the start position for a short break.
ANALYSIS
Repeated measures analyses of variance will be used to examine effects of different characteristics and aspects of robots on human behaviors, task performance, and self-reported safety measures. Tables and figures will be generated to illustrate the differences between the conditions and any interaction effects.
Researchers will contact manufacturing employers and labor organizations and ask them to post recruitment flyers in their workspace or to email them to their employees. We will also post the flyers in public spaces such as public bus stops, mall bulletin boards, etc. Each participant will receive Email Confirmation of Participation (Instrument A). In the event an appointment is cancelled, another respondent will be substituted into that time slot. It is estimated that 15% may not complete the data collection due to motion sickness. In this event, they will be replaced with substitutes.
The data collection procedures and equipment accuracy will be evaluated through 3 pilot test subjects. During data collection and subsequent analysis, the quality of the data will be ensured by constant control from the participating researchers. The instrumentation will be tested and the researchers will be trained in a series of preliminary trials of 3 subjects. Data accuracy will be assessed during the collection period to ensure the quality of the data and to identify unintended changes or possible errors. Further data quality control will be conducted during the final data analysis.
Individuals Consulted on Statistical Aspects
Dr. Justin Haney, NIOSH
Individuals Collecting and/or Analyzing Data
Dr. Justin Haney, NIOSH
Dr. Hongwei Hsiao, NIOSH
Dr. Marvin Cheng, NIOSH
Mr. Douglas Ammons, NIOSH
Mrs. Darlene Weaver, NIOSH
Balk, S. A., Bertola, M. A., & Inman, V. W. (2013). Simulator sickness questionnaire: Twenty years later.
Attachment A - Applicable Laws and Regulations
Attachment B - 60-day Federal Register Notice
Attachment C – IRB Approval Letter
Attachment D - Instrument A: Simulator Sickness Questionnaire
Attachment E - Instrument B: Informed Consent
Attachment F - Instrument C: Participant t Data Collection Form
Attachment G - Instrument D: Virtual Reality Sickness Questionnaire
Attachment H - Instrument E: Robot Experience Questionnaire
Attachment I - Instrument F: Experiment 1 and 2 Tasks
Attachment J - Instrument G: NASA Task Load Index
Attachment K - Instrument H: Perceived Safety Questionnaire
Attachment L - Instrument I: Robot Trust Questionnaire
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Richardson, Tony (CDC/OD/OADS) |
| File Modified | 0000-00-00 |
| File Created | 2021-02-04 |