NCSES Generic Clearance Package: HERD COVID-19 Cognitive Interviews and Webinar Poll

Attachment B: Cognitive Interview Protocol
Introduction

[Interviewer introduces herself.] Thank you very much for talking to me today. As you know from our
initial contact, we’re considering adding some questions related to the COVID-19 pandemic and how this
is causing disruptions to R&D activities at colleges and universities.
Before we begin our interview, I want to review some information with you.
• The interview should last approximately 30 minutes.
• There are other people on the phone. Some work with me at ICF on the HERD team and [if
appropriate] others are from NCSES at NSF. They are listening to our discussion today and you
may hear from them towards the end of the interview. Is that okay with you?
• I received your informed consent earlier. Any questions?
• [If informed consent was not received, review form and obtain verbal consent.]
1. For the record, can you tell me your title(s) and your role(s) in the HERD Survey?
You should have received the proposed new questions in an e-mail. If you aren’t viewing them already,
please refer to that attachment for our discussion. If you need me to resend it, let me know.

Effects of the Pandemic

Let’s start broadly.
2. Can you give us a brief sense of how research at your institution has changed over the past few
months (since mid-March)?
3. What words are your institution using to describe the effects of the pandemic on research?
a. [If needed, suggest words] halted, shut down, disrupted, uncertain, decreasing,
impacted, increasing, thriving?

Question A (degree R&D disrupted)

Now let’s take a look at Question A.
4. In your own words, what do you think this question is asking?
5. How useful are the response options?
a. Do you think it will be easy or difficult to generalize across your entire institution? Can
you tell me more about that?
6. What were you thinking of as “R&D activities” when reading this question?
7. What time period were you thinking of?
b. Did you have any trouble deciding between any of the specific options such as fully shut
down, almost fully shut down, partially shut down or minimally shut down?
i. What do each of the categories mean to you?
1. Shut down for a week, a month, some other time period? Currently still
shut down?
2. Or do the categories reflect a percentage of researchers continuing to
work?
ii. What would have needed to happen to move to the higher or lower category?
8. How easy or difficult do you think it would be to report about FY 2020 R&D activity only?

12

Question B (R&D funds diverted)

Now let’s move to Question B and start by looking at Question B1.
9. What R&D funds do you think of when reading this question?
a. Did you think about externally sponsored projects? Funds from unrestricted gifts?
Departmental funding or other institutional funds.
10. How would you define “diverted funds” in your own words?
11. Would you find this question easy or difficult to answer? Please explain.
12. Are diverted funds tracked in your institution’s systems?
Now let’s look at Question B2.
13. How easy or difficult would it be to answer this question?
14. Did you see the instruction “Your best estimate is fine”? Would this help you answer the
question?
a. If we did not have that instruction, would you assume to use your best estimate?
15. How would you go about answering this question?
a. Probe on whether or not records are available, who has access to them…
16. Did the ranges provided make sense to you?
a. Could you have answered the question with smaller ranges of percentages? Or some
other response categories?

Question C (new R&D funds)

Now let’s move to Question C and start by looking at Question C1.
17. What specific funds or types of funds are you thinking about when reading this question?
a. Are you thinking about external funds only? Are you thinking of any unrestricted funds
or just sponsored R&D?
18. What kind of R&D projects were you thinking about when reading this question?
19. What does “R&D related to COVID-19” mean to you?
a. Do you think your institution received any funding that would meet these criteria? [If
yes] Can you tell me more about that?
20. Are these new funds tracked in your institution’s systems?
Let’s look at Question C2.
21. In your own words, what do you think this question is asking?
22. How would you answer this question for your institution?
23. (If respondent selected “other sources”) What sources were you thinking of?
24. Are there any sources of funds that you think should be added to the list?
25. How easy or difficult do you think this question is to answer?

Wrap Up

22. Are there any questions that you think would be more appropriate to ask in our attempt to
capture disruptions to R&D related to the COVID-19 pandemic?
Those are all of the questions I had for you today. Did you have any other comments for us, or things we
should keep in mind?
Thank you for taking the time to meet with us to discuss the effects of the pandemic on research at your
college/university. We will let you know as soon as we can if these questions (or versions of these
questions) are added to the FY 2020 HERD Survey.
13