HERD 2020-2021 OMB SS Section A

The National Center for Science and Engineering Statistics (NCSES) requests approval for a revision to the OMB clearance for the Higher Education R&D and FFRDC R&D surveys. This request specifically concerns the FY 2020 Higher Education R&D (HERD) survey questionnaire; it does not apply to the FFRDC questionnaire. NCSES requests a revision to the current question on R&D personnel to collect demographic headcounts for three R&D personnel functions: researchers, R&D technicians, and R&D support staff. Total headcount data on researchers will be collected by sex, citizenship, and highest level of education completed. Total headcount data on R&D technicians and R&D support staff will only be collected by sex and citizenship. Furthermore, NCSES requests the addition of a new question on the number of full-time equivalents for the three personnel functions. These questions were developed over a four-year period through site visits, a workshop, and cognitive testing. Iterations of these questions were also presented at seven meetings and conferences that included many research administrators from HERD survey institutions over this period. As noted below under “uses of information,” these data are currently lacking for international comparisons and they will prove valuable for NCSES and National Science Board publications. They will also provide value for researchers, analysts, and others seeking to address R&D personnel effort and demographic issues. The proposed research personnel questions will only appear on the HERD standard form that is provided to approximately 650 institutions reporting at least $1 million in R&D.

NCSES also requests the addition of three qualitative questions that would provide more insight into the impacts of the COVID-19 pandemic on higher education R&D activity beyond the reported R&D expenditure totals. The closures and social distancing requirements resulting from the COVID-19 pandemic disrupted R&D at colleges and universities beginning in March 2020. At the same time, additional funding to address pandemic-related topics became available and some researchers pivoted their work to these topics. However, questions asking for detailed expenditure amounts would likely take too long to develop and would be too burdensome for survey respondents to complete. For these reasons, NCSES proposes the three new COVID-19 impact questions. These additions are designed to inform the following research questions:

To what degree were institutions’ expected R&D activities disrupted as a result of the pandemic?
How many institutions had R&D funds diverted from their originally-intended research focus as a result of the pandemic?
How many institutions received new funds for R&D as a result of the pandemic and what were the external source of those funds (i.e., federal government, state or local government, businesses, nonprofit organizations, other sources).

These questions are intended to provide insights unavailable through the financial and personnel data collected on the HERD survey.

The HERD and the FFRDC R&D surveys are annual surveys that were last conducted in fall 2019 for FY 2019. The OMB clearance for the surveys will expire on August 31, 2022 (Attachment 1).

A.1 Necessity for Information Collection

In 2010, the America COMPETES Reauthorization Act of 2010^¹ established the previously named Science Resources Statistics division as NCSES and directed NCSES to “...collect, acquire, analyze, report, and disseminate statistical data related to the science and engineering enterprise in the United States and other nations that is relevant and useful to practitioners, researchers, policymakers, and the public...” Information obtained through the HERD and FFRDC surveys is critically important to NCSES’s ability to measure science and engineering resources in the United States. Furthermore, the HERD survey (Attachments 2 and 3) provides essential data on the resources devoted to R&D in the higher education sector where over one‑half of the Nation's basic research is conducted.

A.1.1 HERD and FFRDC Background

Conducted annually since FY 1972, the HERD survey provides both summary data on R&D resources, by source and discipline, and data on individual institutions. Between 2007 and 2010, the survey underwent a full-scale redesign of both content and methodology. Beginning in FY 2012, institutions reporting $1 million or more in R&D expenditures in the previous fiscal year were asked to respond to the full HERD survey in the current survey fiscal year, whereas institutions reporting under $1 million were sent a short form version of the survey containing only a few questions. A population screener is sent each year to institutions not currently in the survey to determine eligibility (Attachment 5 and Attachment 6).

Results of this survey are combined with other NCSES data for the federal and business sectors to arrive at national levels of R&D spending, as required by the law cited above. Without information from the HERD Survey, NSF policymakers and planners, as well as other policymakers in the Executive Branch, Congress and the states, would have an incomplete and inaccurate understanding of the Nation’s R&D activities. The data from this survey are used in conjunction with information from other surveys of academic science and engineering (S&E)—such as the Survey of Graduate Students and Postdoctorates in Science and Engineering and the Survey of Science and Engineering Research Facilities—to provide the background statistics that are critical for obtaining a meaningful understanding of research activities in the academic sector.

Additionally, the population of academic institutions surveyed in the HERD survey serves as the universe for a related survey effort mandated by the United States Congress: the previously-mentioned Survey of Science and Engineering Research Facilities (Section 108, Public Law 99‑159 [1986]).

NCSES utilizes a subset of questions from the HERD survey to collect R&D performance data and the funding sources from all FFRDCs (42 FFRDCs in FY 2019) (Attachment 7). According to responsibilities assigned to the NSF in 1990 under the Federal Acquisition Regulations as recorded in the Federal Register (vol. 55, no. 24, February 5, 1990), the NCSES “maintains a list of FFRDCs… and information on each FFRDC, including sponsoring agreements, mission statements, funding data, the type of R&D being performed…” The data collected through this FFRDC R&D survey are used to inform the public on individual FFRDC R&D expenditures and to provide information on this sector’s contribution to the national R&D total.

A.1.2 Addition of New Questions

The HERD survey is one of several surveys at NCSES that collects comparable information on R&D from different sectors of the economy (e.g., businesses, nonprofits, government). However, the HERD survey does not currently collect much information about the personnel carrying out R&D activities compared, for example with the NCSES Business R&D survey, making it less useful for measurements of the R&D workforce in the United States. The Frascati Manual, which is published by the OECD and is the internationally recognized methodology for collecting and using R&D statistics, has a full chapter with recommendations on measuring R&D personnel in establishment R&D surveys.² Further, the United States is the only OECD member country that does not report higher education FTE R&D personnel as published in the OECD’s Main Science and Technology Indicators report.

NCSES addresses these shortcomings through the proposed revisions to the personnel questions in the FY 2020 HERD survey. The survey will collect data on the number of R&D personnel by function (researchers, R&D technicians, R&D support staff), by sex, citizenship (for all functions), and highest level of education completed (for researchers), and by the number of full-time equivalents (FTEs) of each R&D function. The demographic data for R&D personnel paid specifically from R&D accounts at higher education institutions are not available in other surveys. These new R&D personnel variables will, in combination with the R&D expenditure data, allow NCSES to provide new analyses of national data as well as internationally comparable information not available elsewhere to data users interested in science policy, the nature of the science and engineering workforce, and U.S. R&D competitiveness.

The previous version of R&D personnel question collected headcounts for two categories: principal investigators and all other personnel, without demographics. NCSES has been using principal investigator headcounts as a proxy for “researchers” and FTEs were not collected. The proposed new questions will only appear on the HERD standard form that is provided to approximately 650 institutions reporting at least $1 million in R&D in the previous year. The HERD short form respondents, those reporting at least $150 thousand but less than $1 million, will not receive these questions. Question development was approved under the generic clearance (OMB Control Number 3145-0174, ICR Ref. Number – 201905-3145-003) on April 8, 2020 (see Attachment 8).

The coronavirus pandemic is having a substantial impact on colleges, universities, and the nation’s workforce. In response, NCSES is adding short modules to some surveys to assess the pandemic’s disruption and impact in their specific area of the nation’s science and engineering enterprise.

The COVID-19 impact questions being requested for the FY 2020 HERD survey will contribute to this effort. As noted elsewhere in this submission, these questions will specifically provide NCSES with insights into national totals and trends from the FY 2020 HERD survey that would not be available strictly through the reporting of R&D financial and personnel data. Question development was approved under the generic clearance (OMB Control Number 3145-0174, ICR Ref. Number – 201905-3145-003) on September 03, 2020 (see Attachment 9).

A.2 Uses of Information

The data from the proposed new R&D personnel questions will be used by several audiences, including NCSES. The data will be incorporated into the annual HERD data release InfoBrief and data tables as noted in more detail below. Since the demographic data are confidential, only aggregated totals will be used in any published tables or analyses. The data will also contribute to the U.S. R&D national totals published in the Organisation for Economic Co-operation and Development’s (OECD) Main Science and Technology Indicators report. The United States is the only OECD member country that does not report higher education sector R&D FTEs in this report. Collecting both headcounts and FTE data are critical because headcounts alone create an overestimation of R&D effort, and FTEs alone do not provide an understanding of personnel characteristics. The FTE R&D personnel and demographic data will enable previously unavailable analyses for the Academic Research and Development and Science and Engineering Labor Force thematic reports in the biennial, Congressionally-mandated, Science and Engineering Indicators report published by NCSES for the National Science Board. NCSES believes these added benefits to the data user community provide a reasonable justification for the additional survey burden.

As noted earlier, the disruptions to R&D at colleges and universities in FY 2020 from the COVID-19 pandemic were potentially very impactful. The FY 2020 HERD survey data will provide some insight into these impacts, but financial totals have limitations. The data from the proposed new COVID-19 impact questions will enable NCSES to provide greater insights into the national totals and trends. NCSES intends to supplement the annual HERD data release InfoBrief with results from these three questions. NCSES does not plan to publish separate tables as part of its annual series of detailed tables. Data from the COVID-19 impact questions will only be presented in the aggregate.

These proposed new data will also create more opportunities for analyses for the following audiences, in addition to the established usage of the HERD and FFRDC survey data as outlined below.

A.2.1 Use by the Federal Government

The HERD and FFRDC surveys meet many information needs for federal policy makers. The data are used in policy formulation, implementation and evaluation, budget analyses, congressional hearings, program planning, and annual publications mandated by Congress. The information is provided to Congress, the Office of Management and Budget, and the Office of Science and Technology Policy through published reports, briefings by the NSF Director and staff, and in special tabulations.

The National Science Board, the Director of NSF, and NSF program directors and managers use the HERD and FFRDC survey data for long‑range planning and policy formulation. Specific uses include the following:

(1) The NSF Office of Integrative Activities uses HERD data to help assess the need for and the impact of special NSF programs in the Office of Experimental Programs to Stimulate Competitive Research.

(2) Data from the HERD and FFRDC surveys are incorporated into NCSES's periodic analytical report, National Patterns of R&D Resources, and the National Science Board’s biennial report, Science and Engineering Indicators (SEI). The SEI report is mandated by Congress (42 U.S.C. 1863, Section 4(j)), as follows:

"The Board shall render to the President and Congress, no later than January 15 of each even numbered year, a report on indicators of the state of science and engineering in the United States.”

(3) Data on HERD and FFRDC R&D expenditures are used in conjunction with other data sources for maintaining current information on funding, staffing, and impacts of the Nation's scientific, engineering, and technological activities. NCSES produces annual and periodic analytical reports highlighting these data. Detailed data tables for both surveys ahre also produced annually in addition to public use files. NCSES enables users to run reports for HERD data, and other surveys, from 2010 to 2018 through its Interactive Data Tool. These resources are available on the NCSES website at http://www.nsf.gov/statistics/.

(4) The Bureau of Economic Analysis (BEA/DOC) uses data from the HERD and FFRDC surveys for the development of R&D investment in the core accounts of U.S. gross domestic product (GDP) and other National Income and Product Accounts (NIPAs).

A.2.2 Use by Academic Institutions

Universities extensively utilize the HERD data for their own purposes. Requests for the data are received from hundreds of individual institutions, as well as from national academic organizations. Specifically, NCSES has an agreement with the Association of American Universities’ Data Exchange to provide them with more timely and comprehensive data from the HERD survey. Institutional Profiles (summary reports containing institution-specific trend data on key data elements from several NCSES surveys) are available electronically on the web (https://ncses.nsf.gov/profiles/).

Public universities and colleges often use R&D expenditures data in studies demonstrating the economic benefits of instruction and research to state legislatures.

A.2.3 Use by Professional Societies

Representative data users in this category include: the American Association for the Advancement of Science, the Association of Public and Land-grant Universities, the Association for Institutional Research, the National Research Council, the Council on Governmental Relations, the Association of American Universities, and the National Council of University Research Administrators.

A.2.4 Use by State Governments

State governments frequently request R&D expenditures statistics that are unavailable from state records for cross‑state comparisons. The data are requested regularly by individual state government agencies (such as state boards of higher education in Florida, Maryland, Ohio, and Texas) and by national and regional state government organizations (such as the National Governors Association and the Southern Governors Association).

The data are also used in the compilation of the annual Science and Engineering State Profiles published by NCSES.

A.2.5 Use by the Media

HERD expenditures data are well reported by the press, including the Washington Post, the New York Times, the Chronicle of Higher Education, Science, USA Today, and the Wall Street Journal.

A.2.6 Use by International Organizations

The Organization for Economic Cooperation and Development (OECD) has requested that NCSES provide HERD and FFRDC survey data annually for use in their periodic publications and for international comparisons of total R&D efforts. Other foreign users have included the Association of Universities and Colleges of Canada, the Canadian Institute for Public Policy and Public Administration, King Abdullah University of Science and Technology, and the National Institute of Science and Technology Policy in Tokyo, Japan.

A.3 Consideration of Using Improved Technology

The HERD and FFRDC surveys are web-based data collection efforts. Respondents can also download a PDF or MS Excel version of the form (meant primarily to be used as data gathering worksheets), which can be completed and submitted through email. All respondents submitted through the web version of the survey in FY 2019 for the ease of submission and error resolution capabilities. Within the Web-based version of the HERD survey standard form, respondents had the option of uploading data to the Web form using an MS Excel version of the survey; 185 institutions used this upload feature. There is also an option for uploading revisions to the previous year’s data using an MS Excel form. Respondents are electronically sent the survey package, including a letter of introduction, survey instructions and related materials.

Reporting burden is stable or potentially reduced when the survey population is constant and institutions are accustomed to providing the data requested. Consequently, most respondents have established automated systems for assembling the requested data. In addition, the survey questions are intended to be as consistent as possible with the principles of financial accounting followed by institutions of higher education and FFRDCs. Generally, these data are readily available from year‑end financial records and other records maintained regularly by most institutions. To obtain the full set of data requested in the survey, business officers of some institutions must sometimes consult with multiple colleagues, including heads of departments, research administrators, and other academic officials of the institution.

The web versions of the surveys have a real-time monitoring system, which allows NCSES to monitor data, response status, and comments from respondents. From the perspective of the respondents, the web versions are more convenient and simplify the survey (e.g., by automatically calculating totals). NCSES benefits from the use of the web versions by receiving improved data quality.

A.4 Efforts to Identify Duplication

The HERD survey collects essential information on the financial resources allocated to R&D by universities and colleges. There are no other statistical sources of comprehensive national data for this information.

As noted in section A.1.2, data on higher education FTE R&D personnel are unavailable through other data sources and therefore the U.S. in unable to report these data for international comparisons with other OECD countries. The same is true for demographic data on R&D personnel paid specifically from R&D accounts at higher education institutions. The metrics provided from these survey questions will enable comparisons and more valuable insights into R&D productivity and workforce issues.

Several organizations have produced analyses or discussions of the impacts of COVID-19 on the research enterprise in general or specifically the higher education research enterprise. The HERD survey’s unique position as the most comprehensive collection tool for higher education R&D expenditures means that the addition of the three COVID-19 impact questions can best be used by NCSES (and others) as an analytical metric in publications. This output will enhance, rather than duplicate, any of the previous work in this area.³^,⁴^,⁵

The U.S. Department of Education/National Center for Education Statistics’ (NCES) Integrated Postsecondary Education Data System (IPEDS) finance survey series is related to the HERD in that it collects data on a full range of financial resources and expenses in institutions of higher education including research expenses, while the HERD Survey requests data on R&D expenditures. However, the IPEDS survey does not collect the following information requested by the NCSES survey: (1) separately accounted for R&D expenditures by field, source of funding, and type of R&D; and (2) current fund expenditures for research equipment by field. NCSES regularly consults with the NCES to ensure that the information sought by the HERD survey is unavailable from other sources.

The Association of University Technology Managers (AUTM) collects annual data on university technology transfer activities such as patents filed and licensing revenues. The AUTM survey also asks for total R&D expenditures to be reported. However, the survey is only administered to approximately 200 AUTM member institutions and does not cover the full population of research-performing universities and colleges. It also does not collect any detailed data on the fields or types of R&D expenditures.

The FFRDC R&D survey also collects information that cannot be obtained from any other existing statistical data source. Although NCSES’s Federal Funds for R&D survey collect data on R&D obligations from the Federal agencies that obligate those funds, there are no other known sources of total FFRDC expenditure data.

A.5 Efforts to Minimize Burden on Small Business

The survey universe consists entirely of universities and colleges that perform R&D and of FFRDCs. There is no small business involvement.

A.6 Consequences of Less Frequent Data Collection

Academic R&D expenditures data were collected on a biennial basis for the period 1964 through 1972. The NSF Director and the National Science Board subsequently determined that annual information about academic R&D resources was necessary to support informed programmatic and policy analysis.

The availability of national totals of R&D resources on an annual basis provides a current and timely overview of the status of R&D activity in each sector of the economy. Given the sophistication and pace of science and technological development worldwide, it is anticipated that the need for annual data on national R&D expenditures will continue.

The experience of NCSES staff, academic advisory group members, and workshop participants indicate that survey respondents prefer to report a consistent set of data items on an annual basis. Many universities and colleges and FFRDCs have automated their record keeping systems, facilitating their ability to respond to NCSES on an annual cycle. These automated record systems considerably reduce the time required to assemble and report information needed for NCSES concerning sources of R&D support, R&D expenditures by field, etc. Thus, collecting consistent data annually considerably reduces respondent burden for academic institutions with automated data systems, since the database and software are retained and kept current. Many responding institutions have indicated that if the data were to be collected on a less frequent basis, the database and related software might not be maintained, resulting in increased burden.

Furthermore, federal, institutional, and major data users have strongly expressed their need for R&D data on no less than an annual basis. As a specific example, annual HERD and FFRDC data are needed by the Bureau of Economic Analyses to use in updating the National Income and Product Accounts. Further, because NCSES policies have a national impact, the timeliness of the data used to formulate policy, budget, and planning decisions is critical.

A.7 Special Circumstances

No special circumstances.

A.8 Federal Register Announcement and Consultations Outside the Agency

The Federal Register notice was published on October 2, 2020 (85 FR 62331) (see Attachment 10). NCSES received no comments in response to the announcement.

As described in the next sections, NCSES consulted a total of 56 HERD respondent institutions over the past four years through on-campus interviews, cognitive tele-interviews, and a topic exploration workshop. Further outreach was conducted through webinars, conference presentations, and discussions with other higher education stakeholder organizations. These dialogs provided useful information on question development and the impact of the survey’s planned data requests upon the respondents.

A.8.1 Site Visits and Workshop

The information requested in the R&D personnel questions was first discussed during site visits with eight HERD institutions in 2017 and 2018. A two-day HERD respondent workshop with ten participants was held in September 2019 to explore details of the R&D personnel questions in addition to examining the reporting of capital expenditures for R&D (Attachment 11). The wording for the proposed revision to Question 15 and the new Question 16 is based on the Frascati Manual guidelines and on information gathered at the site visits and workshop. It is also similar to questions currently found on other NCSES surveys of R&D performers.

A.8.2 Cognitive Interviews

A.8.2.1 Personnel Question Interviews

NCSES tested the proposed personnel questions through three rounds of interviews with a total of twenty-three higher education institutions (Attachment 12). All interviews were conducted remotely by video or phone, depending on the respondent’s preference. NCSES first conducted a preliminary round of interviews with four institutions to test initial understanding of the questions. Recommendations from those interviews further informed question development. In the subsequent rounds, NCSES conducted 30-minute initial interviews with institutions before they completed the test questions on their own. The initial interview focused on the wording of the questions, instructions and examples provided, and the ability of the respondent to answer the questions. In the next phase of testing, each institution completed the proposed questions by gathering the required information from others in their institution. During the 30-minute debrief interviews, the respondent provided feedback on the ease or difficulty of completing the questions as well as the time it took to gather the information and complete the questions. Four institutions participated in the preliminary interviews, eight institutions participated in round 1, and 11 institutions participated in round 2.

As part of the iterative process for the interviews, NCSES updated the interview protocols, communications and test questions between rounds. This entailed adjusting the guidance for how to report students on both questions and clarifying that respondents should only include research and development personnel paid from R&D accounts. NCSES also provided greater detail under the R&D functions examples and the examples for calculating FTEs. NCSES decided to make the revised Question 15 confidential and only publish these data in the aggregate because some institutions expressed reluctance to publish detailed demographics on employees and students.

The FY 2019 HERD survey was in the field when the sample was created. The sample was limited to institutions that had submitted a completed survey and had their data approved by the data review team. The sample was then divided into four quartiles of total R&D personnel for FY 2019. The first quartile was not included in this preliminary round to ensure that NCSES obtained information from institutions that had at least several R&D personnel. For rounds 1 and 2 of the interviews, NCSES selected an initial sample of 100 institutions with the intention of sending 50 invitations in each round.

Table A-8.2.1. Number of Institutions Sampled, Invited, and Interviewed in Each Round of Personnel Question Development

Round	Quartile	Sampled	Invited for Each Round	Interviewed
Preliminary	1st	0	0	0
	2nd	7	3	1
	3rd	8	8	1
	4th	5	3	2
	Total	20	14	4
Round 1	1st	12	12	1
	2nd	13	13	2
	3rd	12	12	3
	4th	13	13	2
	Total	50	50	8
Round 2	1st	12	21*	4
	2nd	13	22*	3
	3rd	13	14*	2
	4th	12	5	2
	Total	50	62	11
Total		120	126	23

* Eighteen institutions sampled from quartiles 1 and 2 in round 1 were sent invitations to participate in round 2. Three institutions from quartiles 3 and 4 that expressed interest during round 1 but were not selected for interviews were invited to participate in round 2 before institutions sampled for round 2 were invited. This is why quartiles 1–3 have more invitations than sampled and quartile 4 has fewer invitations than sampled.

All participants generally understood what was being asked in both questions. Participants expressed more concern over their ability to respond to the FTE question than to headcounts question. Most participants thought that responding to the questions was possible but would be more time consuming than the personnel question on the FY 2019 survey. At the debrief interview, two institutions were unable to provide some data due to time constraints at their institution that coincided with testing, but in most cases thought that they would be able to respond completely in the future. Four participants did not think the data would be available for future surveys. All institutions were able to provide headcounts, but two institutions said that citizenship would not be available. Another two institutions said that information about the highest degree of education would not be available. In all cases, it was because this information was not tracked at their institution. Only one institution said that FTEs would not be available because of the complexity of the calculation across units and employment statuses at the institution.

All of the participants said that they would need to ask the human resources departments and/or individual colleges for some of the information requested in the questions. However, for this effort a few of the smaller institutions were able to complete demographic information on their own since they know most of the research personnel. Most participants believed that while greater effort would be required to complete these questions the first year due to setting up new reports and establishing contacts with their human resources departments, much less effort would be needed in subsequent survey cycles. Recognizing this and the other challenges posed by the COVID-19 pandemic, NCSES will display the following language above the questions on the web instrument:

If you cannot provide information this year, "Unavailable" will be entered in blank fields. If you can provide some of the information, you can replace "Unavailable" with the correct numbers. You can submit your survey without explaining why values in this question are unavailable.

NCSES will impute for nonresponses to arrive at national totals that can be used for the purposes stated earlier.

A.8.2.2 COVID-19 Impact Question Interviews

NCSES tested the proposed COVID-19 impact questions through two rounds of interviews with fifteen higher education institutions. Each interview lasted approximately 30 minutes (Attachment 13). Participants were sent the COVID-19 questions prior to scheduling the interview. They were also instructed to invite staff members who were most able to answer the questions, if they themselves were not the appropriate person. Participants were offered a video conference option during scheduling, but all chose phone interviews. The interviewer initially asked the respondents to characterize how research has changed at their institution since mid-March. Most of the participants said the institution completely shut down or paused on-campus research activities for at least a few weeks beginning in mid-March. Participants mentioned that many researchers were able to pivot their work to things that could be done remotely such as analysis and writing during the closure. By June institutions were beginning limited on-campus research again.

Participants were asked about their interpretation of the language of the three questions and response options, as well as their ability to answer the questions. Initially, Question A (rating the degree of disruption to institutional R&D activities) only referred to FY 2020 R&D activities, with no other clarifications about reference periods or types of R&D. Most of the participants in round 1 thought about on-campus or field research only. Participants were asked what timeframe they thought of when reading this question. Answers varied from March through June, spring and fall, and the height of the shutdown at their institution. The initial draft response options asked respondents to gauge the level of disruption (e.g., partially shut down). While some participants thought the response options were understandable, four participants believed the categories needed clarification.

For Question B (whether funds were diverted from their originally intended research focus), most participants understood that diverted funds meant the R&D focus of the original funds changed to another purpose. Several participants were not sure how they would divert funds from the originally intended purpose in the case of sponsored research, but they did understand the question and thought it would be easy to answer. Six of the participants over the two rounds said that they did not divert funds for this purpose and therefore the question would not be difficult to answer. Five other participants who track these data found the question to be easy or not difficult. Four participants said that Question B2 (the percentage of R&D funds diverted) would be easy to answer if they diverted funds, while three participants said it would be difficult either because they don’t track those data or knowing the exact amount diverted is difficult. Most participants said that the instruction that “Your best estimate is acceptable” would help in answering the question.

For Question C (whether institutions received new funds for R&D as a result of the pandemic), participants understood this question to be asking about external funds received. While not all participants said that their institution received R&D funding as a result of the COVID-19 pandemic, those that did said this would be tracked in their systems. All participants understood Question C2 was asking for the funding agency or sponsor of the new funds received. They also said that this question would be easy to answer if they received new funds, because this is how their institution tracks new funds.

After the interviews were completed, all notes were compiled and analyzed to suggest possible areas for improvement. Revisions were made after each round of interviews. A summary of the revisions appears below:

Change the focus from FY 2020 to June 2020. Participants felt adding this specific timeframe was appropriate and made it easier to answer the question.

Add an instruction to consider on-campus and off-campus activities. Most of the participants in round 2 said they would think primarily of on-campus R&D activities or would need to ask for more clarification, if this instruction was not present.
Change the response options in Question A to reflect performance of R&D activities versus disruption/shutdown of R&D. During round 2, participants said that the new response options (e.g., “Could not perform any R&D,” “Could perform almost all R&D”) would make it easier to respond to this question.
Add an instruction to consider all R&D funds including sponsored research, unrestricted gifts, and institutionally financed R&D to Question B1. Before the instruction was added, five participants said they only thought about sponsored research. During round 2, all noted this instruction was important to include.

Change the wording of Question C1 to clarify new funds for R&D could be added as result of COVID-19 and add the instruction “This could include R&D that is not specifically related to COVID-19.” This phrasing was added due to confusion during round 1 on what type of R&D should be included and if funds other than R&D that were received because of COVID-19 should be included. An additional change was made to add specific examples: “This could include funding for R&D that is indirectly related to COVID-19 (e.g., effects of virtual learning on student performance, economic impacts of community job loss) or supplemental funding for ongoing R&D (e.g., modifications to support extended timelines or purchase additional safety equipment).”
Add “including foreign governments, other universities and gifts” to the “Other sources” option in Question C2. During round 2, participants said there were no sources missing from the list.

The sample of interviewed institutions was limited to HERD standard form institutions from the three highest quartiles by total R&D expenditures reported for FY 2019. These institutions have higher levels of research activity and thus have a greater impact on total R&D. They also have greater insight into issues related to the R&D interruptions. Institutions that were part of other recent methodological work were excluded from the sample. Eligible institutions were randomly selected from the three quartile groups. NCSES selected a sample of 60 institutions.

Table A-8.2.2. Number of Institutions Sampled, Invited, and Interviewed for COVID-19 Impact Questions

Total R&D Quartile	Type of Control	Sampled	Invited	Interviewed
2nd	Public	12	12	3
2nd	Private	8	8	2
3rd	Public	12	12	3
3rd	Private	8	8	3
4th	Public	12	12	2
4th	Private	8	8	2
Total		60	60	15

After the question language was revised, respondents in the interviews generally found the questions to be clear and they indicated they would be able to complete them without great difficulty. The revisions accounted for above and in the summary report improved the clarity and intent of the questions based on respondent input.

A.8.3 Other Consultations and Presentations

NCSES staff have engaged the higher education R&D community on question development through conference presentations and other discussions over the last three years. NCSES has had several discussions with the Council on Government Relations on both the research personnel and COVID-19 impacts questions in 2020.

NCSES staff included slides and a discussion of the potential question on FTE R&D personnel in presentations at the following conferences:

January 2018 - Federal Demonstration Partnership meeting
March 2018 - National Council of University Research Administrators (NCURA) Financial Research Administration conference
August 2018 - NCURA annual meeting
October 2018 - NCURA National Conference on College Cost Accounting (NACCA)
May 2019 - Federal Demonstration Partnership meeting

NCSES staff presented final versions of both the R&D personnel questions and the COVID-19 impact questions in slides for the following meetings:

October 2020 – NCSES HERD respondent webinar
November 2020 – NCURA NACCA (virtual presentation)

Many of the attendees at these meetings are HERD survey contacts or contribute to their institution’s reporting. The presentation for the 2020 NCURA NACCA, which typically has a high percentage of HERD survey contacts, included over 400 attendees. These presentations provided NCSES with the opportunity to engage the community on the content of the new questions. Attendees were engaged to provide questions and comments at the time of the presentation and through later exchanges. See Attachment 14 for an example of the 2018-19 presentations and Attachment 15 for an example of the 2020 presentations.

During the annual HERD survey respondent webinar conducted on October 7, 2020, NCSES presented the three COVID-19 impact questions and conducted a poll asking attendees how easy or difficult it would be to answer each question should they appear on the FY 2020 HERD survey. Each question was presented separately in a PowerPoint presentation. Attendees were given 1 to 2 minutes to read each COVID-19 question before a poll question appeared in a separate window on the screen. The poll question asked, “How easy or difficult would it be to answer Question [A] for the FY 2020 HERD survey?” Poll respondents were given another minute to respond. The five response categories were: very difficult, difficult, neither difficult nor easy, easy, very easy. Attendees were asked to provide feedback about both parts of questions B and C, which both included two parts.

There were 666 attendees from 483 institutions (390 from institutions that report on the standard form and 93 that report on the short form). Not all attendees responded to each poll question, but some institutions were represented more than once per poll. Information on the respondent webinar is included in the HERD Proposed COVID-19 Impact Questions Testing (Attachment 13).

There were 565 attendees (480 from institutions that report on the standard form and 85 that report on the short form) who responded to Question A. Most poll respondents said Question A would be very easy (7.8%), easy (38.2%) or neither difficult nor easy (37.9%) to answer on the FY 2020 survey. The remaining poll respondents thought Question A would be difficult (15.2%) or very difficult (0.9%). Over 84% of standard form institutions thought Question A would be very easy, easy or neither difficult nor easy. Less than 1% anticipated the question to be very difficult. Over 81% of the short form institutions said Question A would be very easy, easy, or neither difficult nor easy to answer, while only 1.2% anticipated the question to be very difficult.

There were 543 attendees (464 from institutions that report on the standard form and 79 that report on the short form) who responded to Question B. About 35% of poll respondents said responding to B1 would be very easy (6.1%) or easy (29.3%) to respond to. More than a quarter (27.1%) of the poll respondents selected the “neither easy nor difficult” option. Another 37% said it would be difficult (30.2%) or very difficult (7.4%). Poll respondents from short form institutions were more likely to indicate that it would be very easy or easy (45.6%) to respond to Question B1 than respondents from standard form institutions (33.6%). Respondents generally anticipated Question B2 to be more difficult to answer than Question B1, with 57.2% indicating B2 to be difficult or very difficult compared to 37.6% indicating the same view for Question B1.

There were 550 attendees (471 from institutions that report on the standard form and 79 that report on the short form) who responded to Question C. Over half of the poll respondents, regardless of whether they were from institutions that report on the standard or short form, chose the very easy or easy categories for Questions C1 and C2. In total, 60.3% of poll respondents anticipated Question C1 to be easy or very easy, and 54.5% anticipated Question C2 to be easy or very easy to complete if asked on the FY 2020 survey. About 23% of poll respondents anticipated Questions C1 to be neither difficult nor easy and about 29% had the same anticipation for Question C2. Between 16-17% believed Questions C1 and C2 would be difficult or very difficult to respond to.

As part of the cognitive interview process HERD survey institutional contacts were sent the COVID-19 questions prior to scheduling the interview. NCSES believes this may have had a substantial impact on the greater anticipated difficulty by the webinar poll participants versus those who were interviewed. When given more time than provided in the webinar for reflection and internal discussion, cognitive interview participants generally understood the questions and did not find them too difficult. Results from the webinar poll corroborate the view that questions A and C will not be difficult for the large majority of HERD respondents. The webinar poll respondents anticipated more difficulty with Question B. While most respondents in the cognitive interviews believed Question B would not be difficult to answer, 6 of the 15 interviewees did not have any diverted funds. Two other respondents did not have a way of tracking diverted funds. Of the three COVID-19 questions, Question B required the most time to digest in the cognitive interviews. NCSES believes the webinar poll was a useful tool, but it is not meant to be a replacement for cognitive interviews.

A.9 Payment or Gifts to Respondents

There are no payments or gifts to respondents.

A.10 Assurance of Confidentiality

No items on the FFRDC R&D survey are deemed confidential. All items on the HERD survey are reported at the institutional level except for the following:

the breakdown of institution funds within question 1 (institutionally financed research, cost sharing, and unrecovered indirect costs),
the amount of recovered vs. unrecovered indirect costs in question 12.
the demographic information on R&D personnel headcounts in revised question 15,
the three new COVID-19 impact questions: questions A, B and C.

These items are or will be (in the case of the new questions) presented only as aggregate totals in resulting publications. No institution-level data will be publicly available. The following confidentiality statement, covering these excepted survey items, is included on the questionnaire:

"Information from confidential items is not published or released for individual institutions; only aggregate totals will appear in publications. In accordance with the National Science Foundation Act of 1950, as amended, and other applicable federal laws, your responses will not be disclosed in identifiable form to anyone other than agency employees or authorized persons. Per the Federal Cybersecurity Enhancement Act of 2015, your data are protected from cybersecurity risks through screening of the federal information systems that transmit your data."

A.11 Justification for Sensitive Questions

The survey does not contain any questions of a sensitive nature.

A.12 Estimate of Respondent Burden

Data for the FY 2017 survey was collected from 903 institutions (644 standard HERD survey and 259 short form survey). The FY 2018 survey included 915 institutions (650 standard HERD survey and 265 short form survey). The FY 2019 survey included 916 institutions (647 standard HERD survey and 269 short form survey). NCSES expects minimal increases in both the standard HERD and the short form populations each year as new institutions meeting the threshold are added. The previous estimate in the HERD ICR approved on August 15, 2019 estimated the HERD short form population at 300. This is being revised to 275 due to slower than expected growth in the population. For purposes of estimating total burden during this clearance period, NCSES assumes a total HERD population of 925 (650 in the full survey and 275 in the short form).

The FY 2017-19 FFRDC R&D surveys have included the full population of FFRDCs each year (42 in each year). The size of the FFRDC population has been highly stable over time.

High response rates have consistently been obtained: in FY 2017 the response rate for the HERD survey was 96.2%, in FY 2018 it was 96.9% and in FY 2019 the response rate was 96.7%. The FFRDC R&D survey response rate has been 100% each year.

Based on cognitive interviews with 16 respondents regarding the new personnel questions and another 15 respondents regarding the new COVID-19 impact questions, NCSES expects the increased burden to be approximately 10 hours per HERD standard form institutions for the personnel questions for a total of 6,500 hours (650 institutions * 10 hours). The personnel questions only appear on the HERD standard form questionnaire. NCSES estimates that institutions will spend no more than 15 minutes on the new COVID-19 impact questions, which will appear on both the HERD standard form and short form questionnaires. This represents an increase in burden of 163 hours for standard form respondents (650 institutions * 15 minutes) and 69 hours for short form respondents (275 * 15) for 232 total hours of burden related to the COVID-19 impact questions.

Implementing the questionnaire changes as well as revising downward the short form population estimate from 300 to 275 requires updating the annual burden estimate for the standard and short forms. The previous annual burden estimate for the HERD standard form survey was 35,100 hours; this would be increased to 41,763. The previous annual estimate for the HERD short form survey was 2,400 hours; this would be decreased to 2,269.

The total burden estimate for the FFRDC R&D survey remains unchanged at 462. The HERD population review burden estimate remains unchanged at 125. The new total annual burden is estimated to be 44,619 hours.

For FY 2020-21, NCSES is estimating an average annual burden of 1 hour for the HERD population screener, 64 hours for the standard HERD survey, 8 hours for the HERD short form, and 11 hours for the FFRDC survey.

A summary of the annual burden estimates is included in the table below. At an estimated cost of $34.31 per hour (based on the Bureau of Labor Statistics May 2019 average hourly wages for “Budget Analysts,” within NAICS 611300 - Colleges, Universities, and Professional Schools, accessed on November 17, 2020 at https://www.bls.gov/oes/current/naics4_611300.htm), the total annual cost to respondent institutions is $1,530,878 ($1,402 per respondent).

Table A-12. Annual Burden Estimates for FY 2020-21 Surveys

Category	Estimated # of Responses	Respondent Burden (hours)	Total Burden Hours	Total Cost Burden
HERD population review	125	1	125	$4,289
HERD standard survey	650	64	41,763	$1,432,889
Previously approved burden	650	54	35,100	$1,204,281
New COVID-19 questions	650	0.25	163	$5,593
New personnel questions	650	10	6,500	$223,015
HERD short form	275	8	2,269	$77,849
Previously approved burden*	300	8	2,400	$82,344
New burden estimate	275	8	2,200	$75,482
New COVID-19 questions	275	0.25	69	$2,367
FFRDC survey	42	11	462	$15,851
Total annual burden	1,092	-	44,619	$1,530,878

*HERD short form calculation excludes the “previously approved burden” since the “new burden estimate” replaces it in the subtotals. The HERD standard form “previously approved burden” is included in that calculation.

A.13 Cost Burden to Respondents

Not applicable. There are no capital or startup costs for the respondents to the HERD or FFRDC surveys.

A.14 Cost to the Federal Government

The estimated total cost to the Federal Government for the FY 2020 and FY 2021 HERD and FFRDC surveys is approximately $2.8 million over a period of 24 months, for an annualized cost of $1.4 million. The estimate includes labor costs for NCSES staff of approximately $280,000 (project manager (part time), mathematical statistician (part time), program director (part time) and other staff) and the survey management contractual cost of $2.52 million.

A.15 Program Changes or Adjustments in Burden

NCSES provided HERD population estimates of 650 for the standard form and 300 for the short form in the ICR approved on August 15, 2019. Population growth for the HERD short form population has been less than expected. For purposes of burden calculation, NCSES is revising the HERD short form population estimate to 275 institutions per year for the FY 2020 and FY 2021 data collections. As noted above in Table A-12.1, this reduces the previously approved burden hours for the short form institutions from 2,400 to 2,200. With the proposed addition of the new COVID-19 questions (69 total burden hours), the proposed new burden for the short form institutions is 2,269. See Table A-12.1 for full details of burden calculations.

Table A-15. HERD and FFRDC Response Rates, FY 2017-2019

Survey and form	FY 2017		FY 2018		FY 2019		FY 2020-21 Estimates
Survey and form	Survey universe	Response rate (%)	Survey universe	Response rate (%)	Survey universe	Response rate (%)	Survey universe	Response rate (%)

HERD	903	96.2	915	96.9	916	96.7	925	96.0
Standard form	644	96.4	646	97.5	647	97.7	650	97.0
Short form	259	95.8	269	95.5	269	94.4	275	94.0

FFRDC	42	100.0	42	100.0	42	100.0	42	100.0

A.16 Publication Plan and Project Schedule

The FY 2020 survey began with a population review and screening in late summer 2020. The HERD and FFRDC surveys will be sent electronically to all institutions in the FY 2020 survey population and meeting the R&D expenditures threshold of $150,000 upon resolution of the OMB the revision request. The typical start date is in mid-November. The due date was planned for January 29, 2021, but this will be extended by the same number of weeks as the delay of the survey start date. The anticipated closeout of the surveys will be in June 2021 in order to allow time for late responses, corrections, and updating of previous years' data.

The contractor is responsible for all data collection and processing activities, including editing data submissions to resolve errors. For FY 2020, the same procedures will be used as those used for FY 2019 survey. For the FY 2020 survey, following the closeout of data collection in June 2021 the contractor will generate inflator/deflator factors to impute for non-response, based on data reported by responding institutions. After closeout, data for non-respondent institutions will be machine-imputed using an imputation plan developed and approved by NCSES.

The data from the FY 2020 survey will be analyzed in an NCSES InfoBrief to be published by January 2022. A report containing detailed tables and public use files showing institution-level data will also be available on the web and data will be available through the NCSES Interactive Data Tool. However, the R&D personnel demographic data and the COVID-19 question results will not be published in any institutional-level data tables.

Shape2

Data Collection TBD – June 2021

Shape3

Coding and Data Editing June 2021 – August 2021

Shape4

Final Edited/Imputed Data Files October 2021

Shape5

HERD and FFRDC InfoBrief January 2022 Shape6

HERD and FFRDC Data Tables January 2022

Shape7

HERD and FFRDC Public Use File January 2022

Shape8

A.17 Exceptions to Displaying of OMB Expiration Date

The OMB number and expiration date will appear on all survey forms.

A.18 Exceptions to the Certification Statement

No exceptions to the certification statement are being sought.

1 Section 505, Pub. L. No. 111-358. See Attachment 4

2 OECD (2015), Frascati Manual 2015: Guidelines for Collecting and Reporting Data on Research and Experimental Development, The Measurement of Scientific, Technological and Innovation Activities, OECD Publishing, Paris, https://doi.org/10.1787/9789264239012-en.

3 Morgan, Daniel and Sargent, John, Effects of COVID-19 on the Federal Research and Development Enterprise. Congressional Research Service Report #R46309. Washington DC. April 10, 2020.

4 Council on Government Relations, COGR’s Webinar Series on COVID-19. Washington DC. 2020.

5 Federal Demonstration Partnership, The Impacts of COVID-19 on Research Activities: The University Perspective. Washington DC. May 28, 2020.

Text Box 1_0

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Yamaner, Michael I
File Modified	0000-00-00
File Created	2021-01-13

HERD 2020-2021 OMB SS Section A