Supporting Statement

NIST Generic Clearance for Usability Data Collections

OMB Control #0693-0043

Expiration Date: 03/31/2022

Exploring AI Expert and Non-Expert Perceptions of Artificial Intelligence

FOUR STANDARD SURVEY QUESTIONS

1. Explain who will be surveyed and why the group is appropriate to survey.

For an in-depth exploratory Artificial Intelligence (AI) interview study, the Visualization and Usability Group (VUG), of the Information Technology Laboratory (ITL), of the National Institute of Standards and Technology (NIST) intends to recruit a total of 45 individuals who will participate in the virtual exploratory interviews, consisting of 25 non-experts and 20 AI subject matter experts, referred to simply as AI experts.

The purpose of this interview project is to explore non-expert versus AI expert perceptions and experiences with AI, therefore individual members of the general public from those two groups are appropriate to interview: individuals without particular AI-relevant knowledge and individuals with AI subject matter expertise (AI researchers, developers, or scientists with particular AI-relevant knowledge). Given the pervasiveness of AI and machine learning algorithms across many facets of daily life, it is vital to explore what non-experts and AI experts understand and believe about AI, in particular where their knowledge and beliefs may differ. This is a necessary exploratory first step in a larger research program seeking to provide usable guidance and data-driven recommendations for AI trustworthiness.

2. Explain how the survey was developed including consultation with interested parties, pre-testing, and responses to suggestions for improvement.

The demographic screener questions and the interview questions were developed and refined based on the following:

1. Review of literature, seminars/webinars, and conference proceedings in the AI field,

2. Participation in multiple NIST AI Workshops with leading AI experts in the field

   1. NIST Workshop Exploring AI Trustworthiness, August 2020

   2. NIST Workshop on Bias in AI, August 2020

   3. NIST Workshop on Explainable AI, January 2021

   4. NIST Workshop on AI Measurement and Evaluation, June 2021

   5. Kicking off NIST AI Risk Management Framework: Workshop #1, October 2021

3. National Academy of Science, Engineering and Medicine (NASEM) Workshop on Assessing and Improving AI Trustworthiness: Current Contexts, Potential Paths, March 2021

4. Discussions with AI experts

5. Discussions with methodologists: survey and qualitative methodologists

The demographic screener and interview questions were reviewed by six AI experts, two survey experts, and two qualitative methodologists, to ensure the language and questions were technically accurate and appropriately tailored for the study population (i.e., non-experts and AI experts). Questions were further refined based on pilot feedback from two individuals who were representative of the non-expert segment of the target study population, and two individuals who were representative of the AI expert segment of the target study population.

3. Explain how the survey will be conducted, how customers will be sampled if fewer than all customers will be surveyed, expected response rate, and actions your agency plans to take to improve the response rate.

NIST is working with a consumer research firm to host the virtual interviews and recruit participants. The research firm has access to a national panel of participants who are registered, have provided demographic information, and have agreed to receive email invitations to participate in virtual research interviews via their dashboards. The research firm’s national panel will be used to recruit non-expert participants.

Non-expert interview participants will be recruited from the DC metro area as well as two additional regions: the Northeast (CT, ME, MA, NH, RI, VT, NJ, NY, and PA), and South (DE, FL, GA, MD, NC, SC, VA, WV, AL, KY, MS, TN, AR, LA, OK, and TX) census regions. These regions vary in terms of demographic and socioeconomic characteristics believed to be important for perceptions and opinions of AI based on existing research in the field.

For the AI expert interviews, an AI Community of Interest mailing list of individual AI subject matter experts will be used as the sampling frame. Those individuals on the AI Community of Interest list meet the AI subject matter expertise eligibility requirements. Eligibility criteria for experts includes AI experts who have AI-relevant experience such as developing AI algorithms, testing AI algorithms, evaluating AI algorithms, or deploying AI algorithms. Individual AI Experts must actively work in the AI space, have a self-rated AI knowledge between moderate to high and expert, and be at least 18 years of old.

The consumer research firm will handle all logistics related to participant outreach, recruitment, reminder and scheduling emails, and providing the virtual platform link to complete the demographic screener and participate in the virtual interview session. NIST researchers will conduct the interviews. The consumer research firm will capture the demographic screener information and audio recording of the interviews with permission of participants, and will transcribe the interviews. They will deliver de-identified demographic data to NIST. Demographics will be associated with the interview recordings and transcripts by a participant reference code. No key will be maintained that links the demographic data to individual participants. No key will be maintained that links participant reference codes to participants. All data will be stored and transferred in accordance with NIST policies and security requirements. NIST researchers will keep data (transcriptions, notes, and demographic data) after the study concludes. However, audio files will be deleted after the transcriptions have been checked for accuracy and completeness. At the conclusion of the contract, the consumer research firm will be instructed to purge NIST data from their systems.

Supplementary supporting documentation is being uploaded: information sheet (which will be sent by the consumer research firm to participants, and verbally reviewed by the NIST researcher at the start of each virtual interview session), email invitation (sent by the consumer research firm), and email confirmation with Zoom instructions (sent by the consumer research firm).

This study has been approved by the NIST Research Protections Office. Although we are collecting PII in the form of demographic information, all collected data is anonymized and identifiers are not stored. As stated in the provided Information Sheet, to maintain participant confidentiality, participants will be assigned a participant reference code that will be associated with their responses. Data will not be linked back to a respondent or their personally identifiable information. NIST will not create or keep a list that links the participant reference code to a participant.

AI Expert Interview will comprise of 20 participants * 60 minutes per response = 1200 minutes / 60 minutes per hour = 20 burden hours

Non-Expert Interview will comprise of 25 participants * 60 minutes = 1500 minutes / 60 minutes per hour = 25 burden hours

Although demographic information is collected, data will not be stored in a System of Records which retrieves by a personal identifier. Therefore, Privacy Act is not triggered.

4. Describe how the results of the survey will be analyzed and used to generalize the results to the entire customer population.

Data analysis will be conducted by NIST researchers. In this exploratory in-depth interview study, we intend to use the qualitative data analysis technique of grounded theory to conduct data coding and analysis to create a list of participant’s perceptions, concerns, and expectations surrounding AI. We will compare the qualitative interview responses based on the sub-groups of non-experts and AI experts. From these results, we will plan any subsequent phases of our research efforts. As this is a qualitative exploratory interview study with a relatively small sample size, the intent is not to generalize to the entire population, but rather to inform subsequent research. Given the consistency of responses during pilot testing of the interview questions, we expect data saturation for non-experts and experts, leading to clearly identified themes that will guide future research.