Supporting Statement

OMB Control #0693-0043

Expiration Date: 06/30/2025

NIST Generic Clearance for Usability Data Collections

Digital Twins for Public Safety Training: HazMat Use Case

FOUR STANDARD SURVEY QUESTIONS

1. Explain who will be surveyed and why the group is appropriate to survey.

The User Interface/ User Experience (UI/UX) portfolio in the Public Safety Communications Research (PSCR) Division of the Communications Technology Laboratory (CTL) intends to survey firefighters on their subjective experiences about the effectiveness of digital twin configurations when learning hazardous materials tasks in Virtual Reality (VR) and collect voluntary demographic information.

We will identify participants via email to fire stations in Boulder County and adjacent areas to come to our site for an in-person user study. To the best of our abilities, we seek to conduct the study on a representative sample of the local population.

We will distribute the survey during the VR study, allowing participants to take breaks from wearing the VR headset and answer subjective questions on a tablet. By gathering responses from the same group, we can complement objective metrics, such as time to completion and accuracy, with subjective user feedback about their experience.

2. Explain how the survey was developed including consultation with interested parties, pre-testing, and responses to suggestions for improvement.

The surveys were designed based on the group’s prior experience with VR usability studies and factors that could impact performance. We are using standardized questionnaires, including the Reduced Instructional Materials Motivation Survey (RIMMS), Virtual Reality Sickness Questionnaire (VRSQ), Spatial Presence Questionnaire (SPQ), and the NASA Task Load Index (TLX).

The pre/post-training quiz was informed through consultation with the Boulder County HazMat Authority and firefighter subject matter experts. The quiz was iterated on based on SME feedback to improve clarity and alignment with the training use case.

3. Explain how the survey will be conducted, how customers will be sampled if fewer than all customers will be surveyed, expected response rate, and actions your agency plans to take to improve the response rate.

All surveys will be administered at various points throughout the in-person VR study. Each participant will complete a pre-training quiz on the training use case. They will then don a VR headset and experience an immersive training simulation. After the training in VR, the participant will complete questionnaires on cybersickness, workload, spatial presence, and a post-training quiz.

A practical test on the training use case will be conducted prior to a training quality, open feedback and demographics questionnaire. As the survey administration is integrated into the study procedure, our response rate should be 100% of the people who participate in the study.

The VR and practical test tasks will be performed standing and walking around the lab environment, while questionnaire tasks will be performed seated.

Total: 54 (respondents) * 25 Minutes total for all questions / 60 (Minutes per hour) = 22.5 burden hours

Participant information is anonymous and no personally identifiable information is retained.

4. Describe how the results of the survey will be analyzed and used to generalize the results to the entire customer population.

We will administer surveys electronically throughout the study and perform post-hoc analyses once all participants have completed them. Participant data will be anonymized, retaining only a participant number to link survey responses with objective metrics collected during the VR sessions.

Our analysis will utilize ANOVA tests to determine whether there are significant differences between the means of multiple groups. This method allows us to establish if the observed variations in the data result from actual differences between groups or if they are simply due to random chance.

We intend to examine the effects of sex, age, and prior experience in the fire service on both objective and subjective dependent variables. Participants will be randomly assigned to various digital twin configuration conditions.

The significant findings from these surveys will enhance the objective data collected from the VR-based sessions. By integrating these two data sources, we will assess how objective performance correlates with subjective perception across different conditions