Cognitive Interviews associated with the NSF Survey of Nonprofit Research Activities

Attachment B
Survey of Nonprofit Research Activities
Cognitive Interview Protocol and Recruitment Materials

Re:
 National
 Science
 Foundation
 Study
 of
 Nonprofits

Dear
 ,
One
 of
 the
 responsibilities
 of
 the
 National
 Science
 Foundation
 (NSF)
 is
 to
 provide
 a
 clearinghouse

of
 data
 on
 the
 U.S.
 research
 enterprise.
 I
 am
 writing
 to
 request
 your
 assistance
 in
 testing
 a

questionnaire
 to
 collect
 data
 on
 research
 activities
 from
 the
 nonprofit
 sector,
 a
 sector
 that
 has

not
 been
 measured
 for
 some
 time.
 Our
 questionnaire
 was
 developed
 following
 a
 series
 of

interviews
 and
 meetings
 with
 other
 nonprofits,
 researchers,
 potential
 data
 users,
 and
 others.

We
 are
 interested
 in
 visiting
 a
 variety
 of
 nonprofit
 organizations
 to
 discuss
 the
 questionnaire,
 find

out
 whether
 the
 questions
 it
 contains
 are
 easy
 to
 understand
 and
 answer,
 and
 assess
 the
 relative

ease
 or
 difficulty
 in
 reporting
 the
 requested
 data.

NSF
 has
 partnered
 with
 ICF
 International
 to
 carry
 out
 this
 study.
 Within
 the
 next
 few
 days,
 a

member
 of
 the
 ICF
 project
 team
 will
 call
 you
 to
 discuss
 the
 study
 and
 schedule
 a
 time
 for
 a

meeting.
 We
 will
 be
 in
 the

 area
 during
 the
 week
 of
 .
 If
 you
 would

prefer
 to
 contact
 ICF
 directly,
 please
 use
 the
 information
 below.

recruiter’s
 name@icfi.com

If
 you
 have
 any
 questions
 about
 this
 study,
 please
 contact
 me
 at
 rbritt@nsf.gov
 or
 at
 (703)
 292-­‐
7765.

Sincerely,

Ronda
 Britt

Project
 Officer,
 Survey
 of
 Nonprofit
 Research
 Activities

National
 Center
 for
 Science
 and
 Engineering
 Statistics

National
 Science
 Foundation

NPRA
 Cognitive
 Interview
 Recruitment
 Script

NPO
 Name:
 [insert
 name]

Address:
 [insert
 address]

City
 and
 State:
 [insert
 city/state]

Contact
 Person:
 [insert
 point
 of
 contact]

Phone
 number:
 [insert
 phone]

E-­‐mail
 address:
 [insert
 address]

NPO
 Web
 site:
 [insert
 website]

Hello,
 my
 name
 is
 .
 I
 am
 calling
 from
 ICF
 International
 for
 the
 National
 Science

Foundation.
 May
 I
 speak
 to
 .

If
 a
 receptionist/gatekeeper
 asks,
 “What
 is
 this
 in
 regards
 to?”
 say:

 Within
 the
 last
 week,
 NSF
 sent
 an

e-­‐mail
 to

 requesting
 a
 meeting
 to
 discuss
 the
 Survey
 of
 Nonprofit
 Research
 Activities.

 I
 am

following
 up
 on
 this
 e-­‐mail.

□
 Yes
 [Continue]

□
 No
 [If
 the
 person
 is
 unavailable.
 Do
 not
 leave
 a
 message.
 If
 someone
 other
 than
 the
 person
 answers,

ask
 when
 can
 I
 call
 back?
 ___________________
 ]

If
 you
 are
 transferred
 to
 the
 person
 by
 a
 gatekeeper/receptionist,
 say:

Hello,
 my
 name
 is
 .
 I
 am
 calling
 from
 ICF
 International
 for
 the
 National
 Science

Foundation.
 Thank
 you
 for
 taking
 my
 call.

 In
 the
 past
 week,
 you
 should
 have
 received
 an
 e-­‐mail
 from

Ronda
 Britt
 at
 NSF
 about
 a
 new
 survey
 of
 Nonprofit
 Research
 Activities.

As
 mentioned
 in
 the
 e-­‐mail,
 NSF
 has
 partnered
 with
 ICF
 International
 to
 develop
 and
 administer
 the

survey.
 After
 conducting
 preliminary
 research,
 we
 have
 drafted
 the
 survey
 and
 would
 like
 to
 test
 it
 with

your
 organization
 to
 make
 sure
 that
 the
 questions
 are
 understandable
 and
 easy
 to
 answer.
 Based
 on
 a

review
 of
 your
 organization’s
 website,
 it
 appears
 that
 [NPO
 name]
 [conducts/funds]
 activities
 that
 NSF

would
 like
 to
 know
 more
 about.
 [Mention
 relevant
 specific
 activities
 of
 NPO
 as
 needed].

We
 would
 like
 to
 set
 up
 a
 more
 detailed
 interview
 with
 you
 or
 someone
 you
 designate
 at
 your
 office
 to

go
 over
 the
 draft
 survey
 with
 you
 and
 get
 your
 opinions
 about
 the
 questions.
 For
 example,
 we
 are

interested
 in:

•
•
•

How
 you
 interpret
 the
 questions,

Your
 understanding
 of
 certain
 words
 and
 phrases,

How
 easy
 or
 difficult
 it
 is
 to
 answer
 the
 questions,

Draft Recruitment Script
May 14, 2015

Page 1

•

Whether
 the
 information
 asked
 for
 and
 the
 information
 your
 organization
 maintains

allows
 you
 to
 answer
 the
 questions.

We
 will
 have
 a
 small
 team
 of
 researchers
 in
 your
 area
 between
 .

 The

interview
 should
 take
 about
 90
 minutes,
 and
 you
 will
 not
 need
 to
 prepare
 anything
 ahead
 of
 time.

[One/Two]
 people
 will
 be
 coming
 with
 me.
 [As
 appropriate,
 say:

 from
 the
 National

Science
 Foundation
 is
 coming
 as
 well
 as
 another
 person
 on
 my
 staff
 who
 will
 be
 taking
 notes
 for
 me.

 is
 coming
 to
 listen
 first-­‐hand
 to
 what
 you
 have
 to
 say
 and
 will
 be
 happy
 to

answer
 questions
 you
 may
 have
 about
 NSF’s
 interest
 in
 this
 study.]
 Can
 we
 schedule
 an
 interview
 with

you
 or
 other
 representatives
 from
 your
 organization?

□
 Yes
 [Continue
 to
 the
 end
 to
 schedule
 meeting
 and
 verify
 contact
 information.]

□
 No

 [Say:
 “Thank
 you
 for
 your
 time.
 Good-­‐bye.”]

Great!
 I
 would
 like
 to
 set
 up
 a
 date
 and
 time
 for
 our
 meeting.

Draft Recruitment Script
May 14, 2015

Page 2

Dear
 [point
 of
 contact],

This
 e-­‐mail
 serves
 as
 confirmation
 of
 the
 time
 and
 location
 for
 our
 interview
 to
 discuss
 the
 National

Science
 Foundation’s
 (NSF)
 Survey
 of
 Nonprofit
 Research
 Activities.

Interview
 Date
 and
 Time:

[day
 of
 week],
 [month]
 [date],
 2015,
 [time]
 a.m./p.m.

Name/Title

[Name/title
 of
 Interviewee]

Location:

[NPO
 name,
 address,
 city,
 state
 zip]

Phone
 Number:

[phone
 number]

It
 is
 not
 necessary
 for
 you
 to
 prepare
 anything
 prior
 to
 our
 meeting,
 which
 will
 last
 about
 90
 minutes.

We
 will
 walk
 through
 the
 survey
 together,
 and
 I
 will
 ask
 how
 you
 would
 respond
 to
 the
 questions.
 For

the
 most
 part,
 we
 are
 interested
 in:

•

Your
 interpretation
 of
 the
 questions,

•

Your
 understanding
 of
 certain
 words
 and
 phrases,

•

How
 easy
 or
 difficult
 it
 is
 to
 answer
 the
 questions,

•

Whether
 the
 information
 asked
 for
 and
 the
 information
 your
 organization
 maintains

allows
 you
 to
 answer
 the
 questions.

Two
 members
 of
 the
 ICF
 team
 and
 possibly
 a
 representative
 from
 NSF
 will
 be
 traveling
 to
 your
 area
 to

conduct
 this
 interview.
 If
 you
 need
 to
 reschedule,
 please
 contact
 me
 as
 soon
 as
 possible
 using
 the

contact
 information
 below.

Thank
 you
 for
 your
 time
 and
 we
 look
 forward
 to
 meeting
 you
 [if
 necessary
 add
 other
 name].

[Insert
 your
 name
 name]

NSF
 Survey
 of
 Nonprofit
 Research
 Activities

ICF
 International

[Your
 phone]

[Your
 cell
 phone]

[Your
 email]

Date:
 [insert
 date]

Time:
 [insert
 time]

Confirm
 the
 following
 information:

Complete
 address:
 [insert
 address]

Phone
 number:
 [insert
 phone]

E-­‐mail
 address:
 [insert
 email]

Name
 of
 individual(s)
 we
 will
 meet:
 [insert
 name
 of
 interviewee]

I
 will
 send
 you
 an
 e-­‐mail
 confirming
 the
 date
 and
 time
 of
 our
 meeting.
 When
 you
 receive
 this

confirmation,
 if
 you
 feel
 that
 another
 individual
 in
 the
 organization
 could
 add
 to
 our
 discussion,
 please

feel
 free
 to
 invite
 him/her.

Finally,
 just
 a
 reminder
 this
 research
 is
 being
 conducted
 by
 the
 National
 Science
 Foundation
 to
 test
 a

new
 survey
 of
 Nonprofit
 Research
 Activities.
 Your
 participation
 is
 voluntary,
 and
 the
 names
 of
 the

individual(s)
 who
 participate
 will
 remain
 confidential.

Our
 discussion
 should
 last
 no
 more
 than
 90
 minutes.
 If
 you
 have
 any
 questions,
 please
 contact
 me
 at:

[State
 your
 name
 again,
 phone
 number,
 and
 e-­‐mail
 address.]

Again,
 thank
 you
 very
 much
 for
 your
 time
 and
 for
 agreeing
 to
 participate
 in
 this
 research.
 We
 will
 see

you
 on
 [insert
 date,
 time,
 and
 location].

Draft Recruitment Script
May 14, 2015

Page 3

Draft
 Cognitive
 Interview
 Protocol:
 NPRA
 Survey

NPO:

Interviewer’s Name:

Date:

Note taker’s Name:

Person interviewed:

Thank you for agreeing to help us out today. My name is Arlen/Bryan and I work for ICF, a
research company based in the Washington, DC area. will be taking notes for
me. I’d also like to introduce you to from the National Science Foundation’s
National Center for Science and Engineering Statistics (NCSES). [NCSES staff to provide brief
introduction or Arlen/Bryan to say: As one of 13 federal statistical agencies, NCSES designs
and administers national surveys on the U.S. science and engineering enterprise.]
ICF is working with NCSES to develop a survey to collect information from nonprofit
organizations such as yours. In addition to asking for some general information about your
organization, this survey will collect information about research performed and/or funded by
nonprofit organizations.
Currently we are nearing the final stages of the design phase, and our purpose today and over
the next few weeks will be to better understand if the survey and its questions elicit the
information from nonprofits that we are seeking. Today, I will share with you a copy of a letter
from NSF announcing the survey as well as a draft of the survey. I’ll be asking you for your
opinions about the letter and survey. While we are testing the survey using paper forms, the
finalized survey will be conducted online.
Because our goal is to test the instrument, I’m more interested in your understanding of the
content of the letter and the survey questions than I am in your answers. After we go through
the letter and sections of the survey, I’ll ask your opinions about some of the survey items—
about how the questions are worded, if they are easy to understand, whether or not certain
words and response categories are clear and appropriate, and if the information asked for and
the information your organization maintains allows you to answer the question. Your input will
help us make improvements to the letter and survey before we launch it in early 2016.
Before we begin I want to review some information with you before we start.
•

Your participation in this discussion is voluntary.

•

The information you give us today will not be associated with your name or the name of
your organization.

•

The interview should last about 90 minutes.

•

We like to audio record this interview so that we have all your comments. Is that okay
with you/everyone?

[If Yes:] I have a consent form for you to read and sign. [Distribute form]

1

Draft
 Cognitive
 Interview
 Protocol:
 NPRA
 Survey

If anyone does not give permission to audio record the interview, please do not tape the
meeting. Verify that it’s okay for Haley/Codi to take notes.
Do you have any questions before we begin?

I.

Icebreaker

Before we begin the interview, I’d like to learn a little more about your organization.
1. Briefly, how would be describe the type of work/activities your organization does?
2. How would you describe your role?
II. Initial Letter from NSF
Hand participant a copy of the initial letter and attachment.
Please take a couple of minutes to read the letter and attachment. Once you’ve finished I’m
going to talk to you about how easy or difficult it was to understand, its relevance, and so forth.
3. Let’s talk for a few minutes about the letter. This is something that would have come in
the mail. I’m interested in your overall thoughts about the letter. Is this something you
would have read had you received it?
a. In your own words, what do you think the purpose of this survey is?
b. In your own words, what does the third paragraph convey to you? Would this
information make you more or less likely to consider completing the survey?
Would you be interested in seeing survey findings for individual nonprofits like
yours in the future?
c. Mention was made of a handout. Is this something you would likely read? Scan?
or Would not look at it?
d. If you were uncertain about the validity of this survey, what would you do?
e. If you had questions, how likely would you be to contact us? Which would you do
first – phone or email? Who do you think would be answering your questions –
someone at NSF or at ICF? Does it matter?
4. Is there any information that is missing from the letter? OR is there any information that
is not needed? If so what is missing/needs to be added?
5. Who in your organization should the letter be sent to? What would happen to this letter if
it was sent to the wrong person in your organization?
6. After reading the letter and attachment, how likely would you be (or someone else be) to
complete the survey?
7. Let’s take a quick look at the attachment that was included with the letter. Is it helpful to
see the list of topics that will be covered in the survey, or is it not helpful? Why/why not?
Looking at the list of topics, are there any that you are unclear or that leave you
wondering?
2

Draft
 Cognitive
 Interview
 Protocol:
 NPRA
 Survey

8. On the back of the page is titled, Research Definition and Guidance. Right now I’m
interested in your initial reactions. – (initial thoughts on clarity, comprehensiveness.
We’re going to talk about this in more detail in a few minutes.
9. One final question about the letter. This letter is important and we want to make sure it is
opened and read. What suggestions do you have that would make you/your assistant
more likely to open and read this letter (e.g., with NSF return address, business
envelope, legal envelope, envelope in a different color).
Now, I’d like to go through the questions on the survey. Give participant a copy of the survey.
The survey is organized into different sections. I’d like you to read each question and tell me
your answer. Some questions may not be relevant. If you come to one of those, please briefly
explain why. If a question is relevant, but one that you cannot answer without more information,
I’ll ask you to explain how you would go about getting the information. When we get to the end
of a section, I’m going to ask you a few follow up questions.
Cover Page
Please take a moment to read through the first page. When you’re finished, I’ll ask you a few
questions.
1. What did you think of starting the letter with, “Dear Nonprofit Executive/Research
Director?”
2. Is there enough information on the cover page of the survey for you to understand:
a. Who is conducting the survey?
b. The purpose of the survey?
c. Is there anything else you would add?

General Information
Now, please complete page 2. Please read each question out loud, and talk your way through
the answer. When you’re finished with this page, I’ll ask you a few questions.
1. Instructions: What do you think is meant by “all units” in the second sentence: “For the
purposes of this survey, your organization is defined as all units in your IRS Form 990
filing.”
2. Question #2: What was your total operating budget (excluding capital construction costs)
for FY 2014.

a. What does “total operating budget” mean to you? Does your organization include
or exclude capital construction costs when reports its total operating budget?
i. How would you go about finding this information?
ii. How easy or difficult is it to get this information?
3. Question #3: In what month and year did your organization’s FY 2014 begin and end?
3

Draft
 Cognitive
 Interview
 Protocol:
 NPRA
 Survey

How do you interpret “Fiscal Year 2014?”
(Interviewer should note how respondent marks their answer. Specifically, do they
provide a 2-digit month, or do something else (and if so, what)?)
4. Question #4: How many FTEs are currently employed by your organization?
What is your reaction to the definition of Full Time Equivalent (FTE)?
a. Is it helpful to have a definition?
b. Was the example helpful or not helpful?
c. Where would this information come from? What records would you look at,
and/or what people would you talk to in order to get it?
d. How easy or difficult is it to provide this information?
e. Would you be able to provide an exact figure or would you provide an estimate?
i. Are there any types of employees that you would not include in your FTE
count? (e.g., consultants, employees from a temp company, volunteers,
students)
ii. How do you count temporary employees? (Would temporary employees
get counted when you are responding to this question? Student
assistants?)
f.

Which data is more easily available in your records: the number of FTEs, or a
headcount of employees?

5. Looking back at the four questions in this section, are these questions you can:
a. Answer immediately – off the top of your head?
b. Need to check with someone else?
c. Need to check files/database(s)
d. Wouldn’t know where to get the answer?
Research Definition and Guidance
Let’s look at page 3 next. Again, please tell me what you’re reading and thinking. I’ll ask you a
few questions when you’re done.
1. Instruction: Let’s talk about the first two sentences at the top of the page. Did you
happen to read them before you looked at the rest of the page?
2. Let’s take a closer look what’s in the white box.
a. By research we mean: This survey is about research your organization
performs and/or funds.
i.

In your own words, what do you think research is?

ii.

What does experimental development mean to you?
4

Draft
 Cognitive
 Interview
 Protocol:
 NPRA
 Survey

iii.

What’s your reaction to the definition provided?

b. Research activities must be … take a look at the list.
i.

What do you think of the bolded words and their explanation? Are they
clear or confusing? If confusing, what suggestions can you offer to make
the information clearer?

ii.

Of the activities your organization identifies as research, do all of these
criteria (e.g., novel, creative, uncertain …) apply? If they say no ask,
Which criteria do not apply? Why?

iii.

Is anything missing?

iv.

Can anything be removed or combined with another?

c. Research can include projects that … take a look at the list
i.

Are there any bullets that you find confusing? If yes, which one(s)?
Please explain what is unclear.

ii.

Are there any bullets that you do not feel are not examples of what
research might include? If so, which one(s)? Why?

iii.

Is there anything you would add to the list?

d. Example of project that may meet the criteria or most likely do not meet the
criteria for research
i.

What is your reaction to the examples provided? Is this helpful?

ii.

Are there any that you think need to be added to either list (may meet or
most likely do not meet)? Anything that you would remove from either
list?

iii.

Are there any listed as most likely do not meet the criteria for research
that you think belong on the may meet the criteria for research?

3. Question 5: Considering the guidance above, did your organization perform any research
in FY 2014?

a. In your own words, what is question 5 asking?
i.

If needed ask: What does the term subrecipients mean to you?

b. How easy or difficult is it to answer this question?
c. Is this something you can answer off the top of your head? If no, please explain
why not.
4. Question 6: Considering the guidance above, did your organization fund research
activities performed outside your organization in FY 2014?

a. In your own words, what is question 6 asking?
b. How easy or difficult is it to answer this question?

5

Draft
 Cognitive
 Interview
 Protocol:
 NPRA
 Survey

c. Is this something you can answer off the top of your head? If no, please explain
why not.
5. Before we leave these two questions, did you refer to the definition of research at the top
of the page before you answered:
a. Q5 – why/why not?
b. Q6 – why/why not?
c. How helpful was it to have the definition of research and examples above these
two questions?
Research Performance and Funding Graphic
6. At this time, I will be sharing a graphic that we have created that we are considering
including with the survey materials. What do you feel like this graphic is telling you?
7. What are your overall reactions to this page?
8. How do you feel about seeing a graphic like this on a government survey?
9. What does the title “Your Organization’s Total R&D” mean to you?
10. (Repeat the following questions for each of the icons of interest:)
a. How do you interpret this image/icon?
b. Do you find these icons/images confusing?
c. Would a text-based diagram be more or less confusing? How so?
11. What is the word “internal” indicating to you? How about “external”?
o

Where would be the best location for these terms?

12. Where would you expect to see an image like this?

Interviewer/Notetaker – for respondents who answered “No” to Q5 and “Yes” to Q6, observe if
the person had any difficulty going to Q 15. Also, when respondent goes to page 4, observe and
note any seeming confusion about the instructions at the top of the page. Finally, note if the
respondent referred to page 3 – the definition when answering any of the questions (Q7-Q14).
Research Performance
Please read each question and let me know your answer. Some of these questions will ask you
to provide a dollar amount. It’s not necessary to provide the figure itself, but please tell me
how/where you would go about finding the figure.

6

Draft
 Cognitive
 Interview
 Protocol:
 NPRA
 Survey

For each question, ask if he/she could:
a. Answer immediately – off the top of your head?
b. Need to check with someone else?
c. Need to check files/database(s)
d. Need to compile or calculate a response?
e. Wouldn’t know where to get the answer?

1. Let’s look at the instructions at the top of page 4.
a. Are these instructions clear?
b. Is it useful to include a reminder to refer to page 3 for guidance on what is
included in research?
c. Is there anything we could do to make the instructions more helpful?
2. Question 7: In FY 2014, how much did your organization spend on research performed by
your employees or subrecipients?

a. In your own words, what do you think this question is asking for?
i. (Only ask if you respondent has not shared this information in Q5). What
does the term “subrecipient” mean to you?
b. What does the term “expenditures” mean to you?
c. What does the phrase “related indirect costs” mean to you?
d. Does the information asked for and the information your organization maintains
allow you to answer the question? (Note if the respondent seems confident in
their response)
i. How easy or difficult would it be to answer this question?

7

Draft
 Cognitive
 Interview
 Protocol:
 NPRA
 Survey

3. Question 8: Of the amount reported in question 7 how much was paid to
subrecipients/subcontractors in FY 2014?

a. What does the term “subrecipients” mean to you? Is this the same as
subcontractors? Why/why not?
b. Does the information your organization maintains allow you to answer the
question?
i. How easy or difficult would it be to answer this question?
4. Question 9: What were the sources of funding for research performed in FY 2014?
a. How easy or difficult is it to separately report the amounts provided by each
source of funding for research performed?
i. How easy or difficult is it to provide these amounts for FY 2014?
ii. Is this information you have or would you provide an estimate?
b. Let’s look at the descriptions of the different sources of funding. (Note – ask if
descriptions are clear, relevant to their organization)
i. (a) Federal government: What does the statement “include federal funds
passed through from another organization or institution mean to you?
ii. (e) Internal funds, including endowment. What does the term “internal
funds” mean to you? Would you generally include endowments,
donations, or other unrestricted sources? Why/why not?
iii. (f) Individual donors – how easy or difficult would it be for you to report
the amount a donation specifically designated for research? Is this
information you have readily available?
iv. Looking at the lists of sources, are these any other major sources of
funding for research that you think we should include? If yes, what are
they?
v. What are some examples of what fits the “Other” category?
vi. Does this organization receive research funding from any foreign entities?
If so…
1. Did you report them in this question? (If so, where?)
2. If we had asked for foreign sources of research funding
separately, how easy or difficult would it be for you to provide that
information?
5. Question 10: If your organization has individual donation for research of $1M or more in
FY 2014, please list the donor names and the amount donated.

NSF is interested in learning more about individual donations for research that are $1
million dollars or more in FY 2014.
a. How easy or difficult is it to separately report individual donations of $1M or more
in FY 2014?
b. How easy or difficult is it to identify donations of $1M or more that were
earmarked for research activities only?

8

Draft
 Cognitive
 Interview
 Protocol:
 NPRA
 Survey

c. Would you include unrestricted donations greater than $1M for which at least
$1M was spent on research activities in FY 2014?
d. Would your organization provide the names of the individual donors? If no, would
you be willing to provide the individual amounts over 1 million donated, listing the
donation as anonymous?
i. Is this information you have or would you provide estimates?
6. Question 11: In your own words, briefly describe the types of research performed within
your organization.

a. How do you interpret the phrase “types of research performed?”
b. How much information would you provide?
i. We put bullets in. What does this indicate to you?
7. Question 12: Approximately how many (paid) FTEs worked on research activities in FY
2014?
a. In your own words, what is this question asking for? Specifically, what does the

phrase “research activities” mean to you? What does this include (or not
include)?
b. How helpful or not helpful was the example of an FTE who works on research

activities?
c.

How easy or difficult would it be to provide the number of FTEs who worked on
research activities?
i. Is this information you have or would you provide an estimate?

8. Question 13: Did any volunteers support your research activities in FY 2014?
Interviewer/Notetaker: Note whether or not the respondent followed the skip instruction.
9. Question 14: Approximately how many volunteer FTEs worked on your research activities
during FY 2014?
a. How easy or difficult would it be to provide the number of volunteer FTEs who

worked on research activities?
i. Is this information you have or would you provide an estimate?
ii. (Depending on response) Is there a different way that your organization
tracks volunteer time on research activities?
10. This concludes the questions we have about Research Performance. Do you think these
questions capture enough information to provide NSF with a good overview of the
research activities and expenditures that occur within your organization?
a. What, if anything, do you think is missing?
b. Are there any questions that you would eliminate? If so which ones? Why? Is
there an alternative way we should ask for this information?

9

Draft
 Cognitive
 Interview
 Protocol:
 NPRA
 Survey

Research Funding
Interviewer/Notetaker: please note if respondent answered No to Q6 – if they skip Questions 1519 after reading the instructions. Note if they appear confused. If they answer these questions,
please note if they refer to page 3 at any point.
The next section of the survey is about Research Funding. Go ahead and read each question
and let me know your answers. Some of these questions ask you to provide a dollar amount. It’s
not necessary to provide the figure – after you finish the section I’ll ask some questions and ask
how/where you would go to find out what to enter.
1. Let’s look at the instructions at the top of page 6.
a. Are these instructions clear?
b. Is it helpful to include a reminder to refer to page 3 for guidance on what is
included in research?
c. Is there anything we could do to make the instructions more helpful?

For each question, ask if he/she could:
a. Answer immediately – off the top of your head?
b. Need to check with someone else?
c. Need to check files/database(s)?
d. Need to compile or calculate a response?
e. Wouldn’t know where to get the answer?

2. Question 15: In FY 2014, how much funding did your organization provide to others to
perform research activities?
a. In your own words, what does it mean to include all grants and contracts awarded by
your organization to external recipients to perform independent research? (Focus on
external recipients and independent research).
i. Is the wording of the question and subsequent include/exclude instructions clear
or confusing? If confusing, ask for suggestions in rewording the question and/or
descriptions).

c. Does the information your organization maintains allow you to answer the
question?
i.

How easy or difficult would it be to answer this question?

3. Question 16: Of the total research funded in FY 2014, how much was provided to the
following types of organizations?

10

Draft
 Cognitive
 Interview
 Protocol:
 NPRA
 Survey

a. Looking at the three types of organizations listed in question 16—

universities/educational entities, other NPOs, and businesses – are there others
that should be included in the list? If so, what are they?
b. What are some examples of what might fit in the “Other” category?
c. Does the information your organization maintains allow you to answer the
question?
i. How easy or difficult would it be to answer this question?
4. Question 17: In your own words, briefly describe the types of research funded outside your
organization.

a. How do you interpret the phrase “types of research funded outside your
organization?”
b. How much information would you provide?
i. We put bullets in. What does this indicate to you?
5. Question 18: Of the research funding your organization provided to others in FY 2014, how
much came from each of the following sources?

a. In your own words, please restate what this question is asking.
i. How do you interpret the word “provided” in the question?
b. Let’s say your organization received $30M from the Federal government, and
awarded a $5M grant to a university, to be paid out over 5 years, starting in FY
2014. How would you report that in this question?”
c. How easy or difficult is it to separately report the sources of the research funding
your organization provided others in FY 2014?
i. Is this information you have or would you provide an estimate?
d. Let’s look at the descriptions of the different sources of funding. (Note – ask if
descriptions are clear, relevant to their organization and items listed below)
i. (a) Federal government: What does the statement “include federal funds
passed through from another organization or institution mean to you?
ii. (f) Individual donors – how easy or difficult would it be for you to report
the total amount of gifts designated by donors for research? Is this
information you have readily available?
iii. Looking at the lists of sources, are these any others that you think we
should include? If yes, what are they?
iv. What are some examples of what fits the “Other” category?
6. Question 19: If any of the funding provided to others in FY 2014 came from individual
research donations of $1,000,000.00 or more, please list the donor names and the amount
donated.

a. How easy or difficult is it to separately report each individual research donation of
$1M or more?
b. How easy or difficult is it to identify donations of $1M or more that were
earmarked for research activities only?
11

Draft
 Cognitive
 Interview
 Protocol:
 NPRA
 Survey

c. Would you include unrestricted donations greater than $1M for which at least
$1M was spent on research activities in FY 2014?
d. Would your organization provide the names of the individual donors? If no, would
you be willing to provide the individual amounts over 1 million donated, listing the
donation as anonymous?
i. Is this information you have or would you provide estimates?
7. This concludes the questions we have about Research Funding. Do you think these
questions capture enough information to understand the amount of research funding
your organization provides and type of research activities your organization supports?
a. What, if anything, do you think is missing?
b. Are there any questions that you would eliminate? If so which ones? Why? Is
there an alternative way we should ask this information?
Contact Information and Survey Time
1. Based on your knowledge of the survey, is there any individual at your organization who
could better complete the survey? Or would it be a joint effort between several staff
members? Ask for the person’s(s’) title(s).
a. This survey will be conducted on the web. Have you completed surveys or
reports like this on the web in the past? If so, what do you particularly like about
those surveys or reports? What features should we be sure to include? What
features do you hate (so that we can exclude them)?
b. Respondents will be able to log in and out to complete survey as they have time.
We also are planning to allow organizations to have multiple log-ins so that
several members can complete the survey. Will this be helpful to your
organization?
2. Based on the questions you answered, approximately how long do you think it would
take you or someone else in your organization to complete this survey?
3. How likely would someone in your organization be to complete this survey?
This completes our interview. On behalf of the National Science Foundation we thank you
for your time.

12

CONSENT FORM
The National Science Foundation designs, conducts, and sponsors surveys on science,
engineering, and technology. In order to produce the best information possible, the
National Science Foundation routinely evaluates its surveys.
You have volunteered to take part in a study to improve one of the National Science
Foundation’s surveys. In order to have a complete record of your comments, your
interview session will be audio taped (and/or videotaped). We plan to use the tapes to
improve the survey. Staff directly involved in this research project will have access to
the tapes. In addition, a transcription service may have access to the audiotapes, to
provide us with a written version of your comments. The tapes may also be used for
training others to conduct this type of research, and in presentations to professional
audiences.
This study is authorized by law (42 U.S.C. 1862 Section 3.a.6.). The OMB control
number for this study is 3145-0174.
I have volunteered to participate in this study, and I give permission for my tapes to be
used for the purposes stated above.

_________________________
Researcher’s Signature

____________________________
Participant’s Signature

_________________________
Printed Name

____________________________
Printed Name

_________________________
Date

____________________________
Date

[month]
 [day],2015

Dear
 [insert
 name],

We
 greatly
 appreciate
 your
 participation
 in
 the
 interview
 on
 [month]
 [day],
 2015.
 It
 was
 very
 interesting

to
 learn
 more
 about
 the
 [name
 of
 organization]
 and
 its
 [research/funding]
 activities.
 Your
 thoughtful

feedback
 will
 definitely
 help
 move
 us
 forward
 as
 we
 finalize
 the
 Nonprofit
 Research
 Activities
 Survey.

We
 will
 be
 continuing
 with
 our
 interview
 trips
 this
 summer
 and
 anticipate
 piloting
 the
 survey
 with
 a

small
 number
 of
 nonprofit
 organizations
 in
 early
 2016.

Again,
 on
 behalf
 of
 NSF,
 I
 thank
 you
 and
 [list
 any
 other
 people
 in
 the
 interview]
 for
 talking
 with
 us.

 If

you
 have
 any
 questions
 or
 would
 like
 to
 be
 updated
 on
 the
 status
 of
 this
 project
 at
 any
 point
 in
 the

future,
 please
 do
 not
 hesitate
 to
 contact
 me.

Sincerely,

Ronda
 Britt

Project
 Officer

Survey
 of
 Nonprofit
 Research
 Activities

National
 Center
 for
 Science
 and
 Engineering
 Statistics

National
 Science
 Foundation

4201
 Wilson
 Blvd.
 Room
 965

Arlington,
 VA
 22230

(703)
 292-­‐7765

rbritt@nsf.gov