Supporting Statement

OMB Control #0693-0043

Expiration Date: 06/30/2025

NIST Generic Clearance for Usability Data Collections

Questionnaire Exploring Perceptions of Artificial Intelligence

FOUR STANDARD SURVEY QUESTIONS

1. Explain who will be surveyed and why the group is appropriate to survey.

The Information Access Division (IAD) of the Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) is leading this information collection.

The purpose of this project is to better understand users’ perceptions (e.g., trust) of artificial intelligence (AI) systems by identifying existing psychometric scales for trust in automation that may be suitable for evaluation of user trust in AI systems. Psychometric scales are useful for understanding automation operator behavior and improving system interfaces accordingly. However, there is currently limited work on measuring the trust of individuals interacting with AI-based systems.

Because humans are increasingly interacting with technological systems that are AI-based (i.e., have AI and/or Machine Learning (ML) components), this study will assess the validity of two trust scales (Scale Jian and Scale Malle) among members of the general public. The study incorporates a series of vignettes representing feasible scenarios where the general public may interact with AI systems in the near future. English-speaking individuals in the U.S. aged 18 or older will be recruited for the online questionnaire study via Amazon’s Mechanical Turk (MTurk) online crowdsourcing platform. We will recruit MTurk Masters, a specialized group of MTurk Workers who have consistently demonstrated accuracy across a wide range of MTurk tasks.

2. Explain how the survey was developed including consultation with interested parties, pre-testing, and responses to suggestions for improvement.

The study instrument was developed and refined based on prior research on trust in automation and human-AI interaction. The study instrument will have six variations (see attached Collection Instruments): (1) for Scale Jian, there are 3 versions with slight differences in describing the AI vignettes (i.e., no accuracy info provided; info on 75% accuracy provided; and info on 99% accuracy provided); (2) for Scale Malle, there are also 3 versions with slight differences in describing the AI vignettes (i.e., no accuracy info provided; info on 75% accuracy provided; and info on 99% accuracy provided). In all six variations of the study instrument, we included attention check questions to improve data quality.

It was reviewed by two survey experts and two SMEs to ensure the language and questions were accurate and appropriate. An in-depth walkthrough of the study instruments was conducted with two individuals and pilot testing was conducted with 6 individuals who are representative of the target study population to receive feedback on the study instrument’s timing, usability, and question comprehension. Feedback was incorporated into the final instrument.

3. Explain how the survey will be conducted, how customers will be sampled if fewer than all customers will be surveyed, expected response rate, and actions your agency plans to take to improve the response rate.

Participants will be recruited and compensated by a NIST IDIQ contractor, Media Barn/Fors Marsh Group via Amazon’s MTurk crowdsourcing platform. The study will be posted as a Human Intelligence Task (HIT) on Amazon MTurk, available to workers who are 18 years and older and are English-speakers, using NIST-provided recruitment text (see attached Recruitment Text). We will collect demographic characteristics believed to be important for perceptions of AI based on existing research in the field (e.g., age, sex, race, ethnicity, education level, occupation, level of knowledge about technology and AI). To obtain the target sample of 1,200 participants, the study will be conducted in a series of smaller batches over the administration period to monitor sample demographics.

Various mechanisms to improve data quality will be used. We will recruit MTurk Masters, a specialized group of MTurk Workers who have consistently demonstrated accuracy across a wide range of MTurk tasks. Additionally, attention check questions are included in the questionnaire.

If participants choose to respond, they will click a link in the HIT posting to be directed to the study instructions and instrument (see attached Collection Instruments and Information Sheet). Each participant will be randomly assigned to one of 6 survey variations (see attached Collection Instruments). The study will take about 30 minutes to complete.

We determined that a sample of 1,200 participants who complete surveys would be sufficient to observe expected effects, with 200 participants in each of the six study variations.

1,200 MTurk workers x 30 minutes per response = 600 burden hours.

There will be no collection, storage, access, use, or dissemination of personally identifiable information from the survey. As stated in the provided information sheet, participants will be assigned a participant reference code that will be associated with their responses. Data will not be linked back to a respondent. NIST will not create or keep a list that links the participant reference code to a participant.

4. Describe how the results of the survey will be analyzed and used to generalize the results to the entire customer population.

NIST researchers will perform data analysis examining both descriptive and inferential statistics to understand participants’ perceptions of the AI systems in the vignettes. From the analysis results, we will plan any subsequent phases of our research efforts. No generalization will be made beyond the study participants and their demographics. For any publication resulting from this study, we will describe our recruitment strategy and sample demographics in detail in order to transparently communicate the limitations of our method and findings.