2 Supporting Statement Part B-9.10.20 Full study

Supporting Statement for the Survey of American Artists Participating in
International Exchanges, Part B
Last Updated: August 19, 2020

Table of Contents

B.1 Respondent universe and sampling methods ........................................................................... 3
B.2 Procedures for the collection of information............................................................................ 4
B.3 Methods to maximize the response rates and to deal with nonresponse .................................. 4
B.4 Test of procedures or methods to be undertaken...................................................................... 8
B.5 Individuals consulted on statistical aspects & individuals collecting and/or analyzing data . 12

Table of Attachments
Attachment A: Literature Scan and Focus Group Reports
Attachment B: Communication Materials
Attachment C: Web Survey Instrument
Attachment D: Cognitive Testing Report

2

B.1 Respondent universe and sampling methods
Describe (including a numerical estimate) the potential respondent universe and any
sampling or other respondent selection method to be used. Data on the number of entities
(e.g., establishments, State and local government units, households, or persons) in the
universe covered by the collection and in the corresponding sample are to be provided in
tabular form for the universe as a whole and for each of the strata in the proposed sample.
Indicate expected response rates for the collection as a whole. If the collection had been
conducted previously, include the actual response rate achieved during the last collection.
The potential respondent universe for the USArtists International (USAI) web survey are
all grantees in a given year of the program (approximately 75 a year), starting with grantees that
received awards in 2019. Grantees of the USAI program include individual artists and ensembles
that receive funding to travel and perform abroad. The survey is an annual census; thus, the
“sample” is the same as the respondent universe. The expected response rate for the collection is
60 percent. The Arts Endowment worked with a contractor to implement a pilot test of the USAI
web survey with artists that received support from the program in 2014 to 2018. The contractor
implemented phase 1 of the pilot test from March 9, 2020, to March 20, 2020 and received a
response rate of 20 percent; however, the implementation occurred during the initial wave of
event cancellations as a result of the coronavirus disease 2019 (COVID-19) pandemic, which had
an outsized impact on the performing arts community (both artists and administrators). Potential
respondents were likely tied up with responding to the emergency, communicating with
stakeholders, and managing other issues related to social isolation measures. The contractor
implemented phase 2 of the pilot test from June 4, 2020, to July 10, 2020 and incorporated
weekly email reminders to respondents and two rounds of phone calls to push respondents to
take the web survey. The contractor received a 60 percent response rate in phase 2 of the pilot
test. The Arts Endowment will implement the survey using the same methodology as phase 2 of
the pilot test and expects a similar response rate.

3

Table 1. Web Survey Respondents
Data
Source
2021 Web
Survey
2022 Web
Survey
2023 Web
Survey
TOTAL

Respondents
USAI grantees traveling between October
1, 2019 and September 30, 2020
USAI grantees traveling between October
1, 2020 and September 30, 2021
USAI grantees traveling between October
1, 2021 and September 30, 2022

Response
Universe
75

Estimated
Response Rate
60%

75

60%

75

60%

225

60%

B.2 Procedures for the collection of information
Describe the procedures for the collection of information, including
•
•
•
•
•

statistical methodology for stratification and sample selection,
estimation procedure,
degree of accuracy needed for the purpose described in the justification,
unusual problems requiring specialized sampling procedures, and
any use of periodic (less frequent than annual) data collection cycles to reduce
burden.
The information collection for Survey of American Artists Participating in International

Exchanges will consist of an annual web survey census of all USAI grantees from the prior fiscal
year. No special sampling, stratification, or estimation procedures are anticipated.
B.3 Methods to maximize the response rates and to deal with nonresponse
Describe methods to maximize response rates and to deal with issues of non-response. The
accuracy and reliability of information collected must be shown to be adequate for
intended uses. For collections based on sampling, a special justification must be provided
for any collection that will not yield "reliable" data that can be generalized to the universe
studied.
The Arts Endowment will administer the annual web survey to USAI grantees who
received funding in the prior fiscal year. For example, the planned 2021 data collection will be

4

administered to grantees that received funding from October 1, 2019 to September 30, 2020. 1 To
implement the annual survey, the Arts Endowment will first send a survey invitation by email
that contains a unique link to the survey. The Arts Endowment will continue to send weekly
email reminders to working email addresses to any respondents with an incomplete survey. 2
days before the survey closes, the Arts Endowment will send a final email reminder reminding
respondents with an incomplete survey to complete the survey by the close of data collection. In
addition, a few days after the Arts Endowment sends the survey invitation, the Arts Endowment
will start to call the survey respondents to inform them of the survey and its importance for the
continuation of the grant program. The calls may also help as the Arts Endowment update email
addresses and identifies the best contact for the survey. The Arts Endowment expects to
implement the annual web survey over a four-week data collection period. Figure 1 provides an
overview of the survey implementation process and expected response rates.

1

Due to the COVID-19 pandemic, all USAI-supported travel was canceled after March 2020. Thus, the

Arts Endowment expects to administer the 2021 data collection to grantees that traveled from October 1, 2019 to
March 31, 2020. It is unclear how future years may be impacted by the pandemic; however, it is possible that some
artists may participate virtually rather than traveling abroad in-person. The web survey currently includes a question
that asks respondents whether they are an artist that traveled abroad with USAI support. The Arts Endowment can
modify this question to include an option for virtual participants, if necessary. The Arts Endowment does not foresee
any need to update any other questions in the survey in the event of virtual participation.

5

Figure 1. Survey Implementation Flowchart
Arts Endowment
Sends Survey Link
and Calls Grantee
Contacts
(n = 75)
No Response

Added to
Response
Sample

n = 20
(27%)

Arts Endowment
Sends First Email
Reminder, Second
Week of Data
Collection
(n = 55)
No Response
Arts Endowment
Sends Second Email
Reminder, Third
Week of Data
Collection
(n = 44)
No Response
Arts Endowment
Sends Third Email
Reminder, Fourth
Week of Data
Collection
(n = 41)
No Response
Arts Endowment
Sends Final Email
Reminder 2 Days
Prior To Survey
Closing
(n = 37)

N = 30
40% Nonresponse
Rate

n = 11
(20%)

n=3
(7%)

n=4
(10%)

n=7
(19%)

N = 45
60% Response
Rate

6

The Arts Endowment will continue to conduct data collection using approved methods
until an appropriate response rate is reached. Additional methods the Arts Endowment may use
to increase the response rate include:
1. increasing the data collection period from four to six weeks,
2. incorporating a second round of phone calls toward the end of data collection, and
3. sending a mailed survey invitation to respondents with an incomplete survey.
We provide all planned communication materials in Attachment B.
Based on the pilot test, the Arts Endowment anticipates that some of the contacts
provided by Mid Atlantic Arts Foundation will be administrative staff or other individuals who
are not artists that traveled abroad. To address this issue, email communications will contain
explicit instructions asking the contact to forward the survey link to one artist who traveled
abroad as part of the USAI-sponsored project. The Arts Endowment will allow only one
response per grantee. Any individuals who click on the link of an already-completed survey will
see a message that states the survey has been completed by another representative from their
grantee organization. We provide the text of all planned communications in Attachment B.
To maximize response rates, the survey invitation email will contain the Arts
Endowment logo. In addition, all communications with grantees will explain the importance of
the survey to the Arts Endowment. All communications will also note that responses to the
survey will be kept confidential and analyzed in the aggregate and will in no way affect current
or future grants or cooperative agreements with the Arts Endowment. During the pilot test, the
contractor found that weekly email reminders and the two rounds of phone calls were helpful in
increasing the response rate and ensuring that the survey was sent to the correct individual. The
Arts Endowment will employ these methods in the survey implementation to maximize the

7

response rate.
The Arts Endowment understands that the expected response rate of 60 percent may
result in survey data that is not representative of all USAI grantees. The Arts Endowment will
assess the degree to which the survey data is representative by conducting a nonresponse bias
analysis of the survey data. The nonresponse bias analysis will analyze the response rate based
on the following grantee characteristics from administrative data provided by the Mid Atlantic
Arts Foundation: (1) whether the artist is a first-time or repeat-participant in the USAI program,
(2) artistic discipline as defined by the USAI program (dance, music, opera/musical theatre,
theatre, folklife/traditional arts, and multidisciplinary), (3) festival location (continent), (4)
grantee race/ethnicity (White, Non-White [includes individuals identifying as Asian,
Black/African American, American Indian/Alaska Native, Hispanic/Latino, and Native
American/Pacific Islander], and no single group), and (5) grantee urbanicity (urban or small
city/rural). The nonresponse bias analysis can identify any difference between the total
population and survey response sample in terms of these characteristics. If there is a statistically
significant difference (at the 95 percent confidence level), the Arts Endowment will then weight
the survey data to be more representative of the total population before reporting any information
in the Annual Performance Report. For example, if the nonresponse analysis finds that the
proportion of first-time grantees in the survey sample is half of what the proportion is in the total
population, the Arts Endowment would weight the responses of first-time grantees twice as much
as repeat grantees in the survey sample to compensate. The survey weights will correct for any
difference in the characteristics between the grantee population and the survey response sample,
as well as reduce bias in the survey.
B.4 Test of procedures or methods to be undertaken
Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an
8

effective means of refining collections of information to minimize burden and improve
utility. Tests must be approved if they call for answers to identical questions from 10 or
more respondents. A proposed test or set of tests may be submitted for approval separately
or in combination with the main collection of information.
The contractor tested the web survey with nine USAI grantees in November 2019. All
nine grantees completed a 30-minute cognitive interview after testing the survey. The objectives
were to detect issues of usability, clarity, and readability in the survey instrument, and to
estimate the average time necessary to complete the survey. Changes were made to the survey
instrument following the completion of cognitive testing. The Cognitive Testing Report can be
found in Attachment D.
In addition, the contractor received OMB clearance to conduct a pilot test of the survey
with 183 grantees that received USAI support from 2014 to 2018. The purpose of the pilot test
was to determine any issues with survey implementation and to assess the feasibility and
reliability of the composite scores the Arts Endowment will use to report the survey data in the
Annual Performance Report.
The pilot test of the USAI web survey had several implications for the survey data
collection. First, the Arts Endowment will take steps to ensure a higher response rate. The
response rate for phase 1 of the pilot test was lower than expected. The phase 1 data collection
overlapped with the most disruptive period for event cancellations, postponements, and business
closures related to the Covid-19 crisis. These disruptions heavily impacted the performing artists
who comprised the survey sample. Many respondents were likely focused on the pandemic and
did not have the ability to respond to the survey during the 2-week data collection window.
During phase 1, the contractor and Arts Endowment limited communication with survey
respondents to email. The Arts Endowment first sent a survey invitation by email introducing the

9

study and the contractor to respondents. The contractor then sent an additional email that
contained a unique link to the survey that respondents could use to complete the survey. A week
after the initial Arts Endowment email, the contractor sent a reminder email to any respondents
with an incomplete survey. The overall response rate from this phase of data collection was 20
percent. Due to this low response rate, a second phase of data collection was deemed necessary.
In phase 2, the contractor extended the time dedicated to data collection from 2 weeks to 5 weeks
and incorporated more reminder emails and phone calls. During this phase of data collection, the
Arts Endowment again sent a survey invitation to respondents who did not complete the phase 1
of the pilot test by email, reintroducing the study and reiterating the importance of responding to
the survey. The contractor then sent an additional email that contained a unique link to the
survey that respondents could use to complete the survey. The contractor also telephoned all
respondents to ensure respondents received the emails and were aware of the survey. The
contractor sent out weekly email reminders to respondents during this phase of data collection. In
the final week of data collection, the contractor sent out a final email and telephoned respondents
who had not completed the survey 2 days before closing the data collection. The response rate
for phase 2 was significantly higher than phase 1 (60 percent compared to 20 percent). For the
full survey implementation, the Arts Endowment estimates using at minimum a 4-week data
collection window that includes weekly email reminders. The Arts Endowment will also call
individuals who do not respond to emails. The contractor found that phone calls were helpful in
identifying the most appropriate contact for the survey and ensuring that respondents who missed
the emails were aware of the survey’s importance.
Second, the Arts Endowment is working with Mid Atlantic Arts Foundation to ask
grantees to provide contact emails for artists that have traveled abroad. This may lessen the issue

10

of undeliverable emails and also help to identify the correct respondent for the survey. There was
some evidence from the pilot test that individuals who were not artists that traveled abroad
responded to the survey, despite instructions that they forward the survey link to an artist who
did travel abroad.
Third, respondents tended to answer question 2-3, which asks respondents about
interaction with the host country’s embassy, differently than other questions in the Professional
Networks section of the survey. When the Arts Endowment reports information on this question,
the Arts Endowment will do so separately from the Professional Networks construct. Similarly,
only 15 artists checked the “other” option for question 3-3. The Arts Endowment will exclude
this question from reporting, unless future surveys have a larger percentage of respondents
providing a response.
Finally, the results of the contractor’s reliability analysis and exploratory factor analysis
indicated that composite scores will produce reliable information that captures the career benefits
to artists that result from the USAI program. The Arts Endowment will report the overall career
benefits to artists as well as specific benefits, including professional opportunities, professional
networks, professional skills and learning, visibility as an artist, and creativity. The contractor’s
factor analysis also identified a sixth construct, self-promotion and audience building, that was
not identified in the prior literature review or focus groups.
The contractor did not find any evidence in the pilot test that indicated the need to
remove survey questions or alter the order of the survey. The contractor received no questions
during the pilot test, and most respondents were able to complete the survey in less than 10
minutes. The pilot test results indicate the USAI survey accurately estimated the burden on
respondents and that the survey provides data that can be easily reported as composite scores in

11

the Arts Endowment’s annual performance report.
B.5 Individuals consulted on statistical aspects & individuals collecting and/or analyzing
data
Provide the name and telephone number of individuals consulted on statistical aspects of
the design and the name of the agency unit, contractor(s), grantee(s), or other person(s)
who will actually collect and/or analyze the information for the agency.
Table 2. Individuals Consulted
Name

Title (Project Role)

Persons conducting the pilot testing
James
Project Manager
Murdoch
Mary Ann
Latter
Guiomar
Ochoa

Organizational Affiliation and
Address

Phone
Number

2M Research, 1521 North Cooper
Street, Suite 600, Arlington, TX
76011
2M Research, 901 N Stuart Street,
Suite 402, Arlington, VA 22203
National Endowment for the Arts
400 7th Street SW, Washington, DC
20506

817-856-0869

Subject Matter
Expert
International
Specialist | Office of
Presenting &
Multidisciplinary
Works + Artist
Communities
Arts Endowment personnel implementing the web survey data collection
Patricia
Deputy Director |
National Endowment for the Arts
Moore Shaffer Office of Research
400 7th Street SW, Washington, DC
& Analysis
20506
Melissa
Program Analyst |
National Endowment for the Arts
Menzer
Office of Research
400 7th Street SW, Washington, DC
& Analysis
20506
Guiomar
International
National Endowment for the Arts
Ochoa
Specialist | Office of 400 7th Street SW, Washington, DC
Presenting &
20506
Multidisciplinary
Works + Artist
Communities
Persons consulted
Patricia
Administrative
National Endowment for the Arts
Germann
Officer | Office of
400 7th Street SW, Washington, DC
Research & Analysis 20506
Robyn Busch Program Officer,
Mid Atlantic Arts Foundation 201 N.
International
Charles Street, Suite 401,
Baltimore, MD 2120

703-214-1200
202-682-5766

202-682-5535
202-682-5548
202-682-5766

202-682-5793
410-539-6656
ext. 113

12