Artificial Intelligence (AI) Customer Usability/Acceptance

Experiment 1

Start of Block: Default Question Block

Q10 OMB CONTROL NUMBER:  0704-0553 OMB EXPIRATION DATE: 03/31/2022  

AGENCY DISCLOSURE NOTICE The public reporting burden for this collection of information, 0704-0553, is estimated to average 5 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding the burden estimate or burden reduction suggestions to the Department of Defense, Washington Headquarters Services, at whs.mc-alex.esd.mbx.dd-dod-information-collections@mail.mil. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.
Please proceed to the next page to begin the survey.

Page Break

Q1 Introduction   The study is expected to take about 5 minutes in total, so please finish it in one sitting.   Your responses will remain anonymous and will be published only in an aggregate fashion. Your participation is voluntary (you may choose to stop at any time).   Please read the instructions carefully and answer the questions frankly based on your feelings and opinions.   Please proceed to the next screen to start.

Page Break

Q2 After reading the consent statement, please proceed to the questionnaire/survey on the next page.  Click “NEXT” to get started with the questionnaire/survey.  If you’d like to leave the questionnaire/survey at any time, just close the browser window.   Informed Consent   Purpose.  The purpose of this questionnaire/survey is to assess your choices to either produce or cancel production of movies based on (fake) film posters and a recommendation.  You are being asked to volunteer because MTurkers provide access to a large data pool. You will be asked to read a short set of task instructions, make choices on a few practice problems, and then complete the actual task. The questionnaire/survey may take about 5 minutes to complete. If interested click on Consent Form to view the full consent form.   Risks and Benefits.  There are no known risks involved in completing the questionnaire/survey.   There are no tangible benefits for completing the questionnaire/survey, but your answers may contribute to an improved understanding of how people make decisions.   Confidentiality.  Participation in entirely voluntary [Government Only: and you may not be a military servicemember]; you may withdraw at any time from participation.  All data obtained will be anonymous. 
Consent Statement. 

• I consent (1)

• I DO NOT consent (2)

Skip To: End of Survey If After reading the consent statement, please proceed to the questionnaire/survey on the next page.... = I DO NOT consent

Page Break

Q3 Instructions:   In this experiment we ask you to play the part of film studio executives. We will show you a series of (fake) film posters and reviews by an Artificial Intelligent agent (Artemis) and ask you to either “BEGIN PRODUCTION” or “CANCEL PRODUCTION.”   Pay close attention to the posters as you make your decisions. As you answer you will receive feedback revealing how the majority of respondents in a previous survey answered. As an added incentive you will receive an additional $0.02 for every decision you make that is shared by the majority of respondents from a previous survey. However, for each decision you make that is not shared by the majority of respondents we will deduct $0.02. Thus, as advertised you may earn or lose $0.10 in addition to the base task payment of $0.25. We have excluded prior participants from this survey. Since you don’t know how others responded it is in your best interest to make a choice based on your own assessment. A calculator at the top of the screen will keep track of your total. We will now give you practice problems as an example prior to the actual task. Note that the outcomes during the practice set are not based on prior respondents’ decisions, and this will not affect your earnings. This is for illustrative purposes only.
Proceed to the next page to begin.

End of Block: Default Question Block

Start of Block: Practice Set: Experienced (Verbal)

Q60 Using the slider below please indicate the approximate probability you would assign to the term “probable.”

0

10

20

30

40

50

60

70

80

90

100

Page Break

Q61 Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

Q62

Q63 Based on the film poster alone would you choose to:

• BEGIN PRODUCTION (1)

• CANCEL PRODUCTION (2)

Page Break

Q8 Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

Q5

Q6  Artemis (AI) suggests you will probably enjoy this film.

• BEGIN PRODUCTION (1)

• CANCEL PRODUCTION (2)

Page Break

Q11 Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

Q12

Q13  Artemis (AI) suggests you will probably enjoy this film.

• BEGIN PRODUCTION (1)

• CANCEL PRODUCTION (2)

End of Block: Practice Set: Experienced (Verbal)

Start of Block: Actual Set: Verbal

Q14 Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

Q55 [Calculated Earnings]

Q15

Q16  Artemis (AI) suggests you will probably enjoy this film.

• BEGIN PRODUCTION (1)

• CANCEL PRODUCTION (2)

Page Break

Q17 Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

Q56 [Calculated Earnings]

Q18

Q19  Artemis (AI) suggests you will probably enjoy this film.

• BEGIN PRODUCTION (1)

• CANCEL PRODUCTION (2)

Page Break

Q20 Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

Q57 [Calculated Earnings]

Q21

Q22  Artemis (AI) suggests you will probably enjoy this film.

• BEGIN PRODUCTION (1)

• CANCEL PRODUCTION (2)

Page Break

Q23 Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

Q58 [Calculated Earnings]

Q24

Q25  Artemis (AI) suggests you will probably enjoy this film.

• BEGIN PRODUCTION (1)

• CANCEL PRODUCTION (2)

End of Block: Actual Set: Verbal

Start of Block: Behavioral Questions

Q27 Movie recommendations are usually presented in terms of words (e.g., “you will probably like this movie.”). 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q29 Movie recommendations are usually presented in terms of numbers (e.g., “80% of respondents liked this movie.”) 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q30 The source of the provided reviews was Artemis (AI). 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q31 My selections were based on my own assessment.

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q32 My selections were based on what I thought others would choose. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Page Break

Q33 My selections were influenced by the provided review.

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q34 I was good at this task. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q35 My choices were well-informed. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q36 I knew what I was doing. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q37 The provided recommendation gave me an idea whether to produce the movie. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Page Break

Q38 The provided recommendation informed my choice. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q39 The provided recommendation was trustworthy. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q40 I accepted the provided recommendation. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q41 I was confident about the decisions that I made. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q42 I knew which choices to make.

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Page Break

Q43 The choice task was easy. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q44 What is your gender?

• Male (1)

• Female (2)

• Other (4)

Q45 What is your age?

• 18-25 (1)

• 26-35 (2)

• 36-45 (3)

• 46-55 (4)

• 56-65 (5)

• 65 and above (6)

Q47 I enjoy watching movies. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q48 I consider myself a film expert. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Page Break

Q49 I watch a lot of films. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q50 I currently work in film production. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q51 I have worked in film production. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

Q52 I am in a position to control film production. 

	Disagree (1)	(2)	(3)	(4)	(5)	(6)	Agree (7)
(1)

End of Block: Behavioral Questions

Start of Block: Conclusion

Q53 Thank you for participating in this task. As an added appreciation for your work we have entered you into a lottery for an additional bonus. Please click the button below to learn if you won and how much:

• CHECK LOTTERY TICKET (4)

Page Break

Q54 Congratulations! You won our lottery. You will be awarded an additional $0.25 + [Calculated earnings] + $0.XX. [Experimenter note: Lottery ticket adjusted so all participants will receive a total of $0.35 irrespective of selections and earnings to date]. 
Your survey completion code is 1ZaS053

End of Block: Conclusion

Page 27 of 27