Supporting Statement for Testing Coronavirus Pandemic Questions for the 2022 Survey of Earned Doctorates (SED)

Memorandum

Date: December 16, 2020

To: Margo Schwab, Desk Officer

Office of Management and Budget

From: Emilda Rivers, Director

National Center for Science and Engineering Statistics

National Science Foundation

Via: Suzanne Plimpton, Reports Clearance Officer

National Science Foundation

Subject: Request for approval to test the coronavirus pandemic questions for the 2022 Survey of Earned Doctorates (SED)

The National Center for Science and Engineering Statistics (NCSES) requests approval to conduct cognitive interviews under the generic clearance for improving survey projects (OMB control number 3145-0174). NCSES plans to use the cognitive interviews to further test a module of coronavirus pandemic impact questions and revise them as necessary for inclusion in the 2022 SED. These questions are exploring the impact of the pandemic on degree completion and postgraduation plans of research doctorates at U.S. educational institutions.

Background

The SED is an annual census survey of new recipients of research doctoral degree from U.S. institutions, sponsored by the NCSES within the National Science Foundation, the National Institutes of Health, the Department of Education, and the National Endowment for the Humanities. Research doctoral degrees are oriented toward preparing students to make original intellectual contributions in a field of study.

The SED collects educational history, education-related debt, financial support during graduate studies, information about postgraduation plans, and personal demographic information for each research doctorate recipient, primarily using the web survey mode.

The COVID-19 pandemic has disrupted many aspects of higher education and the training of the next generation of the scientific and technical workforce. To measure the immediate challenges encountered by the doctoral students on their doctorate completion, the NCSES had requested and obtained approval from OMB to collect information on the impact of the COVID-19 pandemic. These questions were approved by OMB on June 9, 2020 for inclusion in the 2021 SED, which started collecting data with summer 2020 graduates.

This request is to test the coronavirus pandemic impact questions for inclusion in the 2022 SED. The questions being testing are modified versions of the pandemic impact questions from the 2021 SED. The modifications are based on the review of early responses to the 2021 SED. The open-ended verbatim that elaborated on each of the seven main impact questions provided sufficient information to validate that the impacts were reported correctly under appropriate questions and helped identify the impact area that needs further details. The verbatim data also allowed us to create close-ended items to obtain details on four main impact questions for the 2022 SED. This will reduce the response burden in the SED while providing analytically useful data to examine the pandemic’s impact on the graduating doctoral students. The questions being tested are designed to increase the understanding of the various circumstances faced by doctoral students in their last year of graduate studies.

Cognitive Interview Plan

The cognitive testing will be conducted to inform the areas needing improvements in the coronavirus pandemic impact questions. The cognitive testing will enable NCSES to:

• understand respondents’ thought processes when answering survey questions,

• understand respondents’ comprehension of terms used in the questions,

• evaluate the memory demands of the questions,

• determine appropriate presentations of response categories,

• consider development of closed-ended options for the follow-up questions,

• assess the question navigational problems users face, and

• identify sources of burden and respondent stress.

Current research doctoral students who are expected to earn a doctorate from July 2021 to June 2022 will be eligible to participate. They will be recruited through SED-participating doctorate institutions. See Attachment 1 for details on the recruitment plan and Attachment 2 for the eligibility screener survey that will be used within the recruitment process. Interviews will be held between January and February 2021.

Cognitive interviews will be conducted remotely via video conferencing. Participants will share their screen during the interview while completing and discussing the web instrument screens of the coronavirus pandemic impact questions with the interviewer.

The cognitive interview questionnaire will first ask three existing SED questions from Part B (Postgraduation Plans) of the full survey questionnaire with slight modifications appropriate for the cognitive test participants one or two years away from graduation. Thses contextual questions are included below. It will be followed by the revised coronavirus pandemic questions. The interview will then lead the participants through the questions a second time, asking what the questions meant to them, how they interpreted specific terms, and what they thought about in formulating their responses. See Attachment 3 for more details on these probe questions.

Revised COVID-19 Questions to be Tested

The first three questions below are included to provide context to the cognitive interview participants for the coronavirus pandemic questions, which are the main questions of interest.

Contextual Questions:

Q1. Where do you intend to live in the year after graduation?

1 Inside the United States or U.S. territory

2 Outside the United States

Q2. What best describes your postgraduation plan?

1. I plan to seek a postdoc, residency, or other training position (A “postdoc” is a temporary position primarily for gaining additional education and training in research, awarded in academe, industry, government, or a nonprofit organization.)

2. I plan to seek employment other than a postdoc or training position (including self-employment)

3. I plan to enroll in a full-time degree program (e.g., PhD, MD, DDS, JD, MBA)

4. I do not plan to work or study (e.g., family commitments)

5. Other - Specify:___________________________________________

{If Q2=1 or 2}

Q3. What type of employer do you plan seek postgraduate employment with? Is it with…

1. Education

2. Government, that is, other than educational institution

3. Private or nonprofit organization (other than educational institution)

4. Other – Specify: ___________________________________________

Coronavirus Pandemic Impact Questions:

The revised pandemic questions for the cognitive testing are provided below. In the full SED questionnaire, they will be asked at the end of the Part B (Postgraduation Plans) as in the current 2021 SED. The 2021 pandemic questions can be found in Attachment 4.

Q4. The next questions focus on how the coronavirus pandemic may have affected your graduate experiences and career plans.

Did you experience any of the following as a result of the coronavirus pandemic?

Select Yes or No for each

		Yes	No
COVA	The pandemic changed the timeline for completing my doctoral degree.	1	2
COVB	The pandemic led to a reduction or suspension of funding for my doctoral studies.	1	2
COVC	The pandemic affected my research (e.g., limited access to resources, changed research plan).	1	2
COVD	The pandemic changed my immediate postgraduate employment plans (e.g., limited job opportunities, less desirable employment, work visa status).	1	2
COVE	The pandemic changed my longer term career plans (e.g., pursuit of different type of job or employer).	1	2
COVF	The pandemic affected my plans about where to live in the year after graduation.	1	2
COVG	The pandemic changed my graduate experience or career plans in other ways.	1	2

If respondent says Yes to COVC, COVD, COVE or COVG, corresponding follow-up questions appear on next screens with text box for Other-specify option.

{IF COVC=Yes} Previously, you said: “The pandemic affected my research.” How was your research affected as a result of the coronavirus pandemic?

Select Yes or No for each

	Yes	No
As a result of the pandemic, I have limited or no access to resources I need (e.g., lab, data, hardware, software, human subjects, archives).	1	2
I had to make changes to my research plan (e.g., goals, topic, focus, approach, scope) as a result of the pandemic.	1	2
The pandemic disrupted my research in other ways - Specify: _________________________________________________________________________	1	2

{IF COVD=Yes} Previously, you said: “The pandemic changed my immediate postgraduate employment plans.” How did your immediate postgraduate employment plans change as a result of the coronavirus pandemic?

Select Yes or No for each

	Yes	No
As a result of the pandemic, there are limited job opportunities in the employment I desire.	1	2
I may have to accept a less desirable job in terms of the type of position, employer, and/or location as a result of the pandemic.	1	2
I encountered difficulty with my visa status or international travel as a result of the pandemic.	1	2
The pandemic changed my immediate postgraduate employment plans in other ways - Specify: ____________________________________________________________	1	2

{IF COVE=Yes} Previously, you said: “The pandemic changed my longer term career plans.” How did your longer term career plans change as a result of the coronavirus pandemic?

Select Yes or No for each

	Yes	No
As a result of the pandemic, I plan to pursue my career in a different type of job or field than I had considered before.	1	2
I plan to pursue my career with a different type of employer (e.g., from academe to industry) than I had considered before as a result of the pandemic.	1	2
The pandemic opened new opportunities in the areas I had not considered before.	1	2
The pandemic changed my longer term career plans in other ways -Specify: ________________________________________________________________________	1	2

{IF COVG=Yes} Previously, you said: “The coronavirus pandemic changed my graduate experience or career plans in other ways.” In what other ways did your graduate experience or plans change as a result of the coronavirus pandemic?

Specify:______________________________________________________________________________

Burden Hours

A maximum of 30 SED institution coordinators (ICs) will be contacted to assist with recruitment of doctoral students at their institutions. The IC’s time to forward the recruitment emails to their doctoral students is estimated to be up to 2 hours per IC. A maximum of 5,000 doctoral students will be contacted to complete a short eligibility screening survey at approximately 3 minutes per student. The cognitive interviews will be conducted with approximately 60 participants for 45 minutes per participant on average. The total burden hours, including sending of the recruitment emails (30 x 2 hours = 60 hours), screener completions (5,000 x .05 hours = 250 hours), and cognitive interviews (60 x .75 hours = 45 hours) are estimated to be up to 355 hours.

Payment to Participants

Participants who complete the cognitive interviews will receive $30 in the form of an e-gift card or direct digital payment (via Zelle or Venmo etc.). These amounts are industry standards and have been used by NCSES in the past. With 60 participants, the total payment amount is estimated to be $1,800.

Informed Consent

At the beginning of the screening survey and cognitive interviews, participants will be informed of the OMB control number, the expected completion time, and the voluntary nature of the study. Participants will be informed that the data they provide in this study will reside on a server outside of the NCSES domain and that NCSES cannot guarantee the protection of survey responses. The text of the informed consent can be found in Attachment 4.

Testing Schedule

The tentative schedule for the cognitive testing is given below.

Date	Activity/Deliverable
12/16/2020	OMB submission for approval
12/31/2020	OMB clearance
01/11/2021	Begin cognitive interviews
02/26/2021	Complete cognitive interviews
03/12/2021	Final report

Contact Person

Kelly Kang

Project Officer

Survey of Earned Doctorates

Human Resources Statistics Program

National Center for Science and Engineering Statistics

National Science Foundation

kkang@nsf.gov

703-292-7796

Attachments

Attachment 1: Test Participant Recruitment Plan

Attachment 2: Eligibility Screener

Attachment 3: Cognitive Interview Protocol

Attachment 4: 2021 SED Coronavirus Pandemic Impact Questions

6