Attachment D - Cognitive Testing Report

ATTACHMENT D:
COGNITIVE TESTING REPORT

1

Table of Contents

Survey of Artists Participating in International Exchanges: Web Survey Instrument Cognitive
Testing Report ................................................................................................................................. 3
Purpose........................................................................................................................................ 3
Sample......................................................................................................................................... 3
Description of Data Collection Process ...................................................................................... 3
Part 1: Modifications to the Web Survey Instrument ..................................................................... 5
Modifications Based on General Comments on the Survey ....................................................... 5
Modifications Based on Comments on Specific Survey Questions and Items ........................... 5
Part 2: Revised Web Survey Instrument ........................................Error! Bookmark not defined.
Appendix: Detailed Respondent Comments ..................................Error! Bookmark not defined.
Table 1. Cognitive Testing Respondent Sample ............................Error! Bookmark not defined.
Table 2. General Comments on the Web Survey............................................................................ 5
Table 3. Specific Comments and Revisions to the Descriptive Information .................................. 5
Table 4. Specific Comments and Revisions to the Professional Opportunities .............................. 6
Table 5. Specific Comments and Revisions to the Professional Network ...................................... 7
Table 6. Specific Comments and Revisions to the Professional Skills and Learning .................... 7
Table 7. Specific Comments and Revisions to the Reputation as an Artist .................................... 8
Table 8. Specific Comments and Revisions to Creativity .............................................................. 9
Table 9. Specific Comments and Revisions to Overall Survey ...................................................... 9

2

Survey of Artists Participating in International Exchanges:
Web Survey Instrument Cognitive Testing Report
Purpose
From November 20, 2019, to November 27, 2019, 2M Research (2M) performed cognitive
testing of the survey of artists who participated in the USArtists International (USAI) program
with nine respondents. The purpose of the cognitive testing was to detect issues with the clarity
and readability of the items in the web survey, as well as to estimate the average time to
complete the survey. This report summarizes the comments of the nine grantees and the
suggested revisions to the survey based on the comments. The report includes the summarized
respondent comments and associated suggested revisions.
Sample
To identify cognitive testing respondents, the Arts Endowment provided 2M with a list of
grantees with considerations for diversity in terms of the following criteria:
•
•
•
•

•

First-time or repeat awardee
Artistic discipline as defined by the USAI program (i.e., dance, music, opera/music
theatre, theatre, folklife/traditional arts, and multidisciplinary)
Festival location (country)
Representation from race/ethnicity groups: White – Non-Hispanic, Black/African
American – Non-Hispanic, Hispanic/Latino, no single race or ethnicity, and other – NonHispanic (Asian, American Indian/Alaska Native, or Native American/Pacific Islander)
Artists from rural or smaller city context

Using these guiding criteria, 2M intended to collect cognitive testing data from a diverse sample
of grantees to gain input from a variety of perspectives. These criteria were used to purposefully
sample diverse grantees to help to identify any gaps in the survey responses or other areas for
improvement and will help to ensure the revised survey is appropriate for a range of
respondents. 1
Description of Data Collection Process
2M programmed the draft web survey by using the survey software Qualtrics. To recruit
participants, 2M selected 20 past participants of the USAI program as potential respondents for
the cognitive testing, based on the sampling criteria. Then, the Arts Endowment sent an initial
invitation email inviting the artists to participate in testing the survey. If needed, 2M followed up
with a reminder email to encourage participation.
For the artists who indicated interest in testing the survey, we first scheduled a 30-minute
telephone interview with each respondent. Then, we sent each respondent a unique link to take
1

Upon request, additional details on the sample characteristics may be provided.

3

the survey in Qualtrics and asked the respondents to write notes as they took the survey to record
any issues with comprehension, clarity, and usefulness, as well as other thoughts. In addition to
respondent notes, 2M assessed the amount of time the respondents spent on the entire survey.
2M conducted the interview by telephone to review the completed survey and respondents’ notes
and to discuss any issues that arose with specific questions during the survey. The interview also
included general questions about the clarity of the survey questions and interpretation of the
phrasing. 2M emailed a PDF version of the completed survey to the respondents prior to the
cognitive interview so that respondents were able to re-read specific survey questions and their
responses, as needed. 2M asked respondents to elaborate on specific items (i.e., questions,
format, or responses) they found unclear so that 2M could compare respondents’ perceptions to
the intent of the questions and discuss any suggestions for improved clarity.

4

Modifications to the Web Survey Instrument
The cognitive testing respondents generally agreed the survey posed relevant, clear, and
important questions to measure the impact of the USAI program on artists’ careers. Respondents
felt that the survey could be clarified in terms of the causal relationship between the program and
outcomes (e.g., number of bookings, changes in artistic practice). Respondent comments are
summarized below and will be used to revise the updated web survey.
Modifications Based on General Comments on the Survey
Table 2 summarizes the general comments on the web survey instrument 2M received from the
nine cognitive testing participants.
Table 1. General Comments on the Web Survey
Comments
 Most respondents felt the survey was an appropriate length.
 Average time spent on the survey was approximately 3 to 13 minutes, with an average of
6 minutes.
 All respondents felt positive about the survey overall and felt it was capturing salient
information about the program’s impacts on artists’ careers.
 All respondents agreed that the survey is very relevant and easily understood.
Modifications Based on Comments on Specific Survey Questions and Items
The web survey is organized into one descriptive information section on the grantee and five
constructs measuring professional outcomes artists experience as a result of the USAI program.
The following tables summarize the comments and proposed revisions to the descriptive
information section (Table 2), the respective construct questions, and the overall survey (Table
8). The constructs include Professional Opportunities (Table 3), Professional Networking (Table
4), Professional Skills and Learning (Table 5), Reputation as an Artist (Table 6), and Creativity
(Table 7).
Table 2. Specific Comments and Revisions to the Descriptive Information
Comments




Suggested Revisions

One respondent considered himself an

artist who was sponsored through the
USAI program to travel; however, he
did not identify as a performing artist.
One respondent answered “yes” to QA- 
2; however, the respondent also
received a newer grant through the
USAI program. It was difficult for the
respondent to answer the survey
thinking about the older grant that
occurred several years ago.

Revise QA-1 to include technical
directors, producers, or any artist.
This issue is specific to the survey testing
because we are drawing on older grants to
cognitively test and pilot test the survey.
In the pilot test, ensure that the survey
corresponds to the most recent USAI
experience from 2014 to 2018 and

5



Many respondents felt that artists
would have differing answers to survey
questions based on the status of their
career (i.e., younger artist versus
established artist). One respondent
suggested including a self-selection
question for artists to indicate their
career status.



exclude 2019 USAI grantees to avoid this
issue.
No revisions suggested. The current
descriptive information is sufficient to
support the Arts Endowment’s analysis
plan.

Table 3. Specific Comments and Revisions to the Professional Opportunities
Comments 2

2

Suggested Revisions



In Q1-1, many respondents believed
the word “invitation” to mean written
communication about an upcoming
booking. Some respondents did not
believe there was a significant
difference between invitation and
booking.



Remove Q1-1 due to respondents’ similar
understanding of invitation and booking.
To focus on the outcomes of the program,
the question series will solely inquire
about the bookings rather than more
general interest.



In Q1-2, most respondents understood
“following this USAI-sponsored
project” to mean afterward rather than
resulting from. Respondents did not
understand “USAI-sponsored project”
to include the larger experience of
meeting other presenters and artists or
witnessing their work.



Revise Q1-2 (Q1-1 in revised survey)
wording to specify bookings resulting
from the USAI-sponsored experience.
Revise Q1-2a (Q1-1a in revised survey)
to specify bookings resulting from USAIsponsored experience.



In Q1-2a, most respondents preferred
to enter an estimate for the number of
bookings they secured. Respondents
felt this was easier to tally because the
bookings resulted in easily trackable
performances.



None.



In Q1-2b, all respondents agreed the
open text box is the preferred and an
easier format to enter country names.



None.



Comments are organized by the numbering in the tested version of the web survey. Part 2 includes the revised and
renumbered version of the web survey.

6

Table 4. Specific Comments and Revisions to the Professional Network
Comments

Suggested Revisions



In Q2-1 and Q2-2, respondents

understood “new professional contact”
to primarily mean other presenters,
people who witnessed their work, or
booking agents. Respondents suggested
additional clarification of who to
include in this group.

Add “new professional contacts may
include industry contacts such as booking
agents, managers, festival contacts or
liaisons, or other artists or presenters who
may provide an opportunity for new
artistic projects in the future” to Q2-1 and
Q2-2.



In Q2-2, one respondent suggested
adding an open text box to indicate
what type of contacts were made and
why relationships were/were not
maintained.
In Q2-2a, many respondents felt a
range would be easier to answer with
versus using an open text box for
number of professional contacts
maintained.
In Q2-2b, some respondents felt that an
additional option should be available to
indicate collaborations that have not
yet been completed.
In Q2-3, respondents believed
“interact” to mean engaging the
Embassy in matters related or unrelated
to the performance, such as contacting
the Embassy when artists lost visas or
passports, casually meeting with
someone from the Embassy, or inviting
Embassy contacts to the performance.
Several respondents indicated
Embassies are typically invited to
performances.



No revisions suggested to ensure survey
brevity and low respondent burden.



Revise Q2-2a to present numeric ranges.



Revise Q2-2b to remove the word
“finished” to include projects which may
be in process.



Add “Interactions may include contacting
the Embassy to invite individuals to
attend performances or other
performance-related communications.
May also include the Embassy contacting
artists regarding their performances.” to
Q2-3 to specify formal interactions
related to artwork or performance.







Table 5. Specific Comments and Revisions to the Professional Skills and Learning
Comments


One respondent suggested adding an
open-ended question for respondents to
describe how they learned about travel
logistics, as indicated under Q3-1.

Suggested Revisions


No revisions suggested to ensure survey
brevity and low respondent burden.

7









Respondents did not understand “after
your USAI experience” to have a direct
relationship to the respondent’s USAI
experience and requested clarification.
One respondent believed
“understanding” was too broad and
suggested Q3-1 be rephrased similar to
Q3-2.
Some respondents indicated that some
artists may be unable to answer Q3-1
and Q3-2 based on their role in their
organization.
In Q3-2, one respondent understood
strategies for engaging international
audiences to include non-performance
strategies.



Revise Q3-1 to indicate relationship (i.e.,
resulting from the USAI experience)
similar to Q3-2.



Revise Q3-1 for consistency with Q3-2.



Add “non-applicable” to response options
in Q3-1 and Q3-2.



Revise Q3-2 language to specify artistic
performance strategies.

Table 6. Specific Comments and Revisions to the Reputation as an Artist
Comments




In Q4-1, respondents had robust
discussions on the appropriateness of
the term “reputation,” and some
determined the term was judgmental or
may elicit negative connotations.
Despite this, respondents generally
believed it was an appropriate term that
encompassed behavioral and artistic
merit and integrity, while “image,”
“profile,” and “visibility” did not.
Respondents recommended against
“image” and “profile.” Respondents
indicated that the broader definition of
“reputation” may be more applicable to
established artists, whereas “visibility”
is more specific and may be important
to younger artists.
In Q4-1, one respondent believed not
all artists have a reputation as
international artists, despite USAI
funding. This respondent suggested
adding a “non-applicable” option.
Similarly, one respondent was unsure
whether all artists participate in social
media as noted in Q4-2.

Suggested Revisions


Add “visibility” to Q4-1 to increase
inclusivity of statement.”



Add “non-applicable” response option to
Q4-1 for artists who do not believe they
have an international reputation and in
Q4-2 for artists who do not have a social
media presence.

8



In Q4-1, some respondents felt that it
would be difficult to track a change in
the number of social media followers
before and after a specific performance
or festival.



Add “non-applicable” response option in
Q4-1 for artists who do not track social
media following.

Table 7. Specific Comments and Revisions to Creativity
Comments






Suggested Revisions

In Q5-1 and Q5-2, respondents
discussed the impact of the cumulative
experience on their artwork and artistic
process. Respondents reported
difficulty answering questions
specifically about their project.
In Q5-2, one respondent believed
“diverse” would confuse other
respondents and connote unintended
meaning. This respondent suggested
replacing with “innovative.”
One respondent suggested adding a
question asking artists whether the
program did/did not impact artists’
work, prior to asking the specific ways
in which their work changed.



Revise “USAI-sponsored project” to
“USAI-sponsored experience” to reflect
the experiential aspect of program in Q51 and Q5-2.



No revision suggested. Respondents
generally believed “diverse” ideas was a
clear term, and “innovative” is already
noted in the question.



Add new question: “I benefitted
creatively in other ways as a result of the
USAI-sponsored experience.”

Table 8. Specific Comments and Revisions to Overall Survey
Comments






Respondents asked for more openended questions, particularly at the
close of the survey, to be able to reflect
on their experiences in their own
words.
Some respondents thought artists
would be hesitant to indicate
“disagree” or “strongly disagree” on
Likert-type questions. Respondents felt
those responses would indicate they
had a negative experience or that they
were ungrateful for the experience.
Respondents understood “USAIsponsored project” to mean only the
performance, not the experience of

Suggested Revisions


Add optional open-ended question at the
end of the survey with word limit to
restrict respondent burden.



To ensure brevity of the survey,
assurances will be presented on the
survey introduction only. Add language
to the introduction ensuring the
confidentiality of survey responses.



Replace all references to “USAIsponsored project” with “USAI-

9

meeting and observing other artists and
their work.

sponsored experience” to encompass the
entire festival experience.

10

11