Memo for NCSES generic clearance package: HERD COVID-19 Cognitive Interviews and Webinar Poll

MEMORANDUM
Date:

August 26, 2020

To:

Margo Schwab, Desk Officer
Office of Management and Budget

From:

Emilda B. Rivers, Division Director
National Center for Science and Engineering Statistics

Via:

Suzanne Plimpton, Reports Clearance Officer
National Science Foundation

Subject:

Notification of cognitive interview research under generic clearance

This memorandum is to inform you of the National Center for Science and Engineering
Statistics’ plans to conduct cognitive interviews under the generic clearance for survey
improvement projects (OMB number 3145-0174). NCSES also intends to conduct a brief poll of
HERD respondents as part of an annual informational webinar. This research will focus on
obtaining information to measure the disruption to higher education R&D in FY 2020 due to the
COVID-19 pandemic through the Higher Education Research and Development Survey.
Background
The National Center for Science and Engineering Statistics (NCSES), within the National Science
Foundation (NSF) collects, maintains, and disseminates information on science and engineering
resources in the United States’ economy. A component of this activity is its conduct of the
Higher Education Research and Development (HERD) Survey which is conducted annually. The
HERD Survey collects information on the financial resources allocated to R&D by universities
and colleges. Institutions reporting $1 million or more in R&D expenditures in the previous
fiscal year are asked to respond to the standard HERD survey form, whereas institutions
reporting between $150,000 and $1 million are sent a short form version of the survey
containing only a few questions. Approximately 950 institutions are in the HERD survey
universe, although only about 650 are in the standard survey form population in a given fiscal
year. The other 300 are part of the short form population.
There is a general impression that the COVID-19 pandemic disrupted R&D at universities and
may have caused the reassignment of R&D personnel and funds to help address relevant
pandemic-related projects. NCSES would like to gather information on the impact of the
disruption to R&D related activities caused by the pandemic by adding three qualitative
assessment questions to the FY 2020 survey (e.g. To what degree were your institution’s FY
2020 R&D activities disrupted due to the COVID-19 pandemic on a scale of 1-5?). Cognitive
interviews are needed to understand whether the questions are relevant and answerable since
it is unclear what sort of information colleges and universities are tracking related to the
pandemic. These questions would allow NCSES to supplement data on R&D expenditure totals.
1

The resulting information will provide data users and policy makers with a more comprehensive
picture of FY 2020 trends. NCSES anticipates that the additional questions would be on the
survey for a few cycles, similar to the question regarding American Recovery and Reinvestment
Act funding that appeared on HERD from 2009 to 2014.
Research Plan
NCSES plans to conduct up to 20 interviews with HERD standard form institutions from the
three highest quartiles by total R&D expenditures reported for FY 2019. These institutions have
higher levels of research activity and thus have a greater impact on total R&D. They also have
greater insight into issues related to the R&D interruptions. NCSES anticipates placing the final
questions on both the standard and short form HERD surveys. Short form institutions will be
able to provide feedback on the answerability of the questions through the HERD respondent
Webinar noted below. Institutions that were recently interviewed as part of the R&D
researcher full-time equivalent question development will not be contacted (see the OMB
generic clearance memo from NCSES dated March 9, 2020). Based on these factors, NCSES
expects approximately 500 universities and colleges to be eligible for contact.
NCSES will recruit from a sample of 120 institutions in order to obtain interviews with 20
institutions. The sample of 120 institutions will be randomly selected from the pool of 500
eligible institutions. The sample will be divided into 6 proportional classes from the standard
form survey defined by type of control (public or private) and the top three quartiles of total
R&D expenditures reported on the FY 2019 survey (see below).
Sample of Eligible Institutions to be Recruited for Interviews by Type of Control and Quartile
Type of
Control
Private
Public
Total

Quartile of Total R&D Expenditures
1st
2nd
3rd
4th Total
0
0
0

20
20
40

14
26
40

11
29
40

48
72
120

NCSES’s goal is to interview 8 private institutions and 12 public institutions across the range of
R&D expenditure totals as shown below.
Number of Anticipated Institutions to be Interviewed by Type of Control and Quartile
Type of
Control
Private
Public
Total

Quartile of Total R&D Expenditures
1st
2nd
3rd
4th Total
0
0
0

2
4
6

2
4
6

3
4
7

8
12
20

To recruit respondents, NCSES will send e-mail invitations to the FY 2019 survey respondents at
selected institutions. The e-mail invitation will provide detailed information about the nature of
2

the interview, including the anticipated time requirement and the process for providing
feedback. If NCSES has not scheduled a representative sample within a week, a follow-up e-mail
will be sent. After respondents volunteer to participate, an e-mail will be sent to thank them for
their willingness to participate, offer to answer any questions, and confirm the date and time of
the meeting. All contact materials can be found in Attachment A.
Each institution that participates in the interviews will be asked to respond to questions
developed to gauge the disruption to R&D activities caused by the COVID-19 pandemic. The
interviews are anticipated to last approximately 30 minutes and will be conducted via video
conference or phone (at the respondent’s preference) (Attachment B). The interviews will be
structured to obtain feedback on the wording and intent of the three test questions
(Attachment C). Respondents will also be asked to what extent they are able to track R&D
funding and expenditure changes resulting from the pandemic.
To ensure accurate note-taking, discussions will be recorded with the consent of the
participants. Before the interviews, the participants will be informed of the intent of the
discussion, privacy protections, and voluntary nature of the discussions. They will be asked to
verbally confirm their approval to record the discussion, and will also be asked to return a
signed copy of the consent form (Attachment D). Participants can decline to be recorded, at
which point the recorder will be turned off and note-taking will serve as the only means of
documentation.
NCSES also plans to poll approximately 400 HERD institutions during its annual HERD survey
respondent Webinar in September. This annual Webinar helps HERD respondents from both
the standard form and short form populations prepare for upcoming survey changes and allows
NCSES to respond to persistent questions from the previous data collection cycle. The HERD
Webinar poll will display the three COVID-19 R&D disruption questions after they are refined
through the cognitive interviews. All webinar participants can voluntarily participate in the poll.
Poll questions will be asked in a pop-up box that will be introduced during the part of the
Webinar addressing new questions to the FY 2020 survey. Webinar participants will be given
around 2 minutes to indicate their ability to respond to each of the questions if they are
included on the FY 2020 HERD survey. Webinar participants will also be able to submit written
questions and comments about the COVID-19 R&D disruption questions, which will be
addressed by NCSES. NCSES will inform the Webinar participants that their responses to the
poll are voluntary and will only be used in the aggregate by NCSES staff for question
development purposes.
NCSES will review the results of the Webinar poll and Q&A to refine the questions for
presentation on the survey. Further testing will be conducted if the questions need additional
editing. NCSES will also analyze the responses by institution characteristics such as standard
form vs short form and quartiles of total R&D. If responses to the poll indicate that institutions
will be unable answer the questions for FY 2020, the questions will not be included on the
survey.
Burden Information
3

For the cognitive interviews, NCSES estimates that 120 institutions will be contacted in the
recruiting phase of the project, for approximately 3 minutes per institution for a recruiting
burden of 6 total hours. NCSES expects to conduct up to 20 interviews with survey respondents
which will require approximately 5 minutes for scheduling per institution (approximately 1.75
total hours). Each interview will last approximately 30 minutes with up to 2 people per
institution (20 total hours).
For the webinar poll, NCSES estimates that 400 institutions will participate. Each poll response
and time to submit any questions should average no more than 3 minutes per institution for
approximately 1,200 total minutes (20 total hours).
Thus, the estimated total burden for this project is about 48 hours (6 recruiting hours + 1.75
scheduling hours + 20 respondent interview hours + 20 Webinar poll/Q&A hours).
Incentive Payments
NCSES will not be offering incentive payments.
Contact Information
The contact persons for questions regarding this research are:
Michael Gibbons
Project Officer
Research and Development Statistics Program
National Center for Science and Engineering Statistics
National Science Foundation
703-292-4590
mgibbons@nsf.gov
John E. Jankowski
Program Director
Research and Development Statistics Program
National Center for Science and Engineering Statistics
National Science Foundation
703-292-7781
jjankows@nsf.gov
Rebecca L. Morrison
Survey Methodologist
Statistics and Methods Program
National Center for Science and Engineering Statistics
National Science Foundation
703-292-7794
rlmorris@nsf.gov
4

Attachments
Attachment A: Contact Materials
Attachment B: Cognitive Interview Protocol
Attachment C: Interview Questions on Disruption to R&D Activities Due to the COVID-19
Pandemic
Attachment D: Consent form

5