Date: March 9, 2020
To: Margo Schwab, Desk Officer
Office of Management and Budget
From: Emilda B. Rivers, Division Director
National Center for Science and Engineering Statistics
Via: Suzanne Plimpton, Reports Clearance Officer
National Science Foundation
Subject: Notification of cognitive interview research under generic clearance
This memorandum is to inform you of the National Center for Science and Engineering Statistics’ plans to conduct cognitive research under the generic clearance for survey improvement projects (OMB number 3145-0174). This research will focus on testing and obtaining feedback regarding proposed new content for the Higher Education Research and Development Survey.
Background
The National Center for Science and Engineering Statistics (NCSES), within the National Science Foundation (NSF) collects, maintains, and disseminates information on science and engineering resources in the United States’ economy. A component of this activity is its conduct of the Higher Education Research and Development (HERD) Survey which is conducted annually. The HERD Survey collects information on the financial resources allocated to R&D by universities and colleges. Institutions reporting $1 million or more in R&D expenditures in the previous fiscal year are asked to respond to the full HERD survey form, whereas institutions reporting between $150,000 and $1 million are sent a short form version of the survey containing only a few questions. Approximately 950 institutions are in the HERD survey universe, although only about 650 are in the full survey form population in a given fiscal year. The other 300 are part of the Short Form population.
The HERD Survey is one of several surveys at NCSES that collect comparable information on R&D from different sectors of the economy (e.g., businesses, colleges and universities, and government). However, it does not currently collect as much information about the personnel carrying out R&D activities as some of the other NCSES surveys, making it less useful for measurements of the R&D workforce in the United States. NCSES intends to address this shortcoming by adding questions regarding the number of researchers and technicians involved in R&D, and the number of full-time equivalents for those personnel. The addition of these variables will allow NCSES to provide information not available elsewhere to data users interested in science policy, the nature of the science and engineering workforce, and U.S. R&D competitiveness.
Before adding those questions to the survey, NCSES intends to conduct cognitive interviews with several higher education institutions to get their feedback on proposed questions. The information requested in these questions was discussed during site visits with HERD institutions in 2017 and 2018, as well as a HERD respondent workshop in September 2019. The wording for the proposed new questions are based on information gathered at the site visits and workshop and is similar to questions currently found on other NCSES surveys of other R&D performers.
Research Plan
NCSES plans to conduct two rounds of testing with 16 institutions (8 institutions per round; 6 private schools and 10 public schools total) that reported 10 or more R&D personnel during the FY 2019 survey cycle. The FY 2019 survey is currently in progress. Based on previous collections, NCSES expects approximately 580 universities and colleges to meet that requirement. In order to obtain interviews with 16 institutions, NCSES will recruit from a sample of 100 institutions. The sample of 100 institutions will be randomly selected from the eligible institutions in 8 classes defined by type of control (public or private) and quartile of total R&D personnel reported on the FY 2019 survey in the following proportions:
Type of Control |
Quartile of Total R&D Personnel |
||||
1st |
2nd |
3rd |
4th |
Total |
|
Private |
12 |
10 |
7 |
7 |
36 |
Public |
13 |
14 |
18 |
19 |
64 |
Total |
25 |
24 |
25 |
26 |
100 |
NCSES’s goal is to pretest with 6 private schools and 10 public schools across the range of R&D workforce sizes as shown below.
Type of Control |
Quartile of Total R&D Personnel |
||||
1st |
2nd |
3rd |
4th |
Total |
|
Private |
2 |
2 |
1 |
1 |
6 |
Public |
2 |
2 |
3 |
3 |
10 |
Total |
4 |
4 |
4 |
4 |
16 |
To recruit respondents, NCSES will send e-mail invitations to the FY 2019 survey respondents at selected institutions. The e-mail invitation will provide detailed information about the nature of the investigation, including the proposed revisions and the process for providing feedback. If NCSES has not scheduled a representative sample within a week, a follow-up e-mail will be sent. After respondents volunteer to participate, an e-mail will be sent to thank them for their willingness to participate, offer to answer any questions, and confirm the date and time of the meeting. All contact materials can be found in Appendix A.
Each institution that participates in the testing will undertake three tasks: a 30-minute cognitive interview, completing the test questions, and a 30-minute respondent debriefing. The interviews will be structured to obtain feedback about the proposed question changes for the FY 2020 survey cycle. Appendices C and D contain the protocols for the cognitive interview and respondent debriefing, respectively. The institutions interviewed will have just recently completed the 2019 cycle of the survey. In the cognitive interview, the FY 2019 version of the instrument (Appendix B) will be used to set the context for the proposed new questions, which appear on pages 29 and 30. The discussions will focus on the wording of the questions, the additional instructions or examples provided, and the ability of the respondent to answer the new questions. In the next phase of testing, each institution will also be asked to complete the proposed questions by gathering the required information from others in their institution. The respondent debriefing will gather feedback on the ease or difficulty of completing the questions as well as the time it takes to gather the information and complete the new questions. The results of this study will be used internally for questionnaire development by NCSES staff and ICF, the data collection agent for the HERD Survey.
To ensure accurate note-taking, discussions will be recorded with the consent of the participants. Before both interviews, the participants will be informed of the intent of the discussion, privacy protections, and voluntary nature of the discussions. They will be asked to verbally confirm their approval to record the discussion. Participants can decline to be recorded, at which point the recorder will be turned off and note-taking will serve as the only means of documentation.
Burden Information
NCSES estimates that, in the recruiting phase of the project, approximately 100 institutions will be contacted, for approximately 3 minutes per institution, for a recruiting burden of approximately 5 hours. NCSES expects to conduct a total of 16 interviews with survey respondents, each lasting about 30 minutes (8 hours). NCSES also expects to conduct 16 respondent debriefings with the same respondents, also lasting about 30 minutes each (8 hours). NCSES anticipates it will take institutions 6 hours to gather the information requested in the new questions, for a total of 96 hours (16 institutions * 6 hours). Thus, the estimated total burden for this project is about 117 hours (5 recruiting burden hours + 8 cognitive interview hours + 8 respondent debriefing hours + 96 completing the new questions hours).
Incentive Payments
NCSES will not be offering incentive payments.
Contact Information
The contact persons for questions regarding this research are:
Michael Gibbons
Project Officer
Research and Development Statistics Program
National Center for Science and Engineering Statistics
National Science Foundation
703-292-4590
John E. Jankowski
Program Director
Research and Development Statistics Program
National Center for Science and Engineering Statistics
National Science Foundation
703-292-7781
Rebecca L. Morrison
Survey Methodologist
Statistics and Methods Program
National Center for Science and Engineering Statistics
National Science Foundation
703-292-7794
Attachments
Appendix A: Contact E-mails
Appendix B: FY 2019 Higher Education R&D Survey with proposed test questions
Appendix C: Cognitive Interview Protocol
Appendix D: Respondent Debriefing Protocol
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | MEMORANDUM |
| Author | Sherri Mamon |
| File Modified | 0000-00-00 |
| File Created | 2021-01-14 |