Survey of Earned Doctorates

Attachment 1. SED Coronavirus 2019 Module Cognitive Interview Protocol

Cognitive Interview Plan

The cognitive interview will be conducted with nine participants by an experienced survey methodologist via Zoom conference call. Participants will be given a $40 incentive upon completion of the interview.

Participants will be asked to log in using a Zoom web link for the interview. Participants will be asked to read the introduction by themselves for 1-2 minutes. After they finish reading, the interviewer will confirm that they agree to participate and to be recorded. Upon getting the participant’s consent, we will start recording the session and control of the web instrument will be given to the participant. The participant will be asked to answer the survey questions as they would by themselves. Upon finishing answering the filter question B26 and its corresponding follow-up question B27, the interviewer will take control of the web instrument.

At this point, the screens will display the participant’s response to question B26 (if B26 response is “No” or missing) or the participant’s response to the follow-up question B27 (if B26 response is “Yes”). Retrospective probes will then be used to ask participants what they are thinking of and how they understood a certain word when answering the questions.

Upon completion of the cognitive interview, participants will be asked if they would like their incentive given via Zelle or Amazon GC and to confirm the email address for their incentive. They will also be reminded that they need to send an email confirmation of receipt upon receiving the incentive.

Questions Tested

The next questions focus on how the Coronavirus Disease 2019 (COVID-19) may have affected your graduate experiences and career plans.

B26. Did you experience any of the following as a result of the COVID-19 pandemic?

Select Yes or No for each

a. The timeline for completing my doctoral degree changed. Yes No

b. My research was disrupted. Yes No

c. Funding for my doctoral studies was reduced or suspended. Yes No

d. My immediate postgraduate employment or education plans changed. Yes No

e. I changed my career plans or goals (e.g., type of employer, research focus). Yes No

f. I changed my decision on where to live in the year after graduation. Yes No

g. My graduate experience or career plans changed in other ways. Yes No

B27a. (If B26a=Yes) How did the timeline for completing your doctoral degree change?

B27b. (If B26b=Yes) How was your research disrupted?

B27c. (If B26c=Yes) How was your funding reduced or suspended?

B27d. (If B26d=Yes) How did your employment or education plans change?

B27e. (If B26e=Yes) How did your career plans or goals change?

B27f. (If B26f=Yes) How did it impact your decision on where to live in the year after graduation?

B27g. (If B26g=Yes) In what other ways did your graduate experience or plans change?

Interviewer Instructions

WEB INSTRUMENT, FORM: INTRO

ASK PARTICIPANTS TO READ THE INTRODUCTION FOR 1-2 MINUTES. ANSWER ANY QUESTIONS THE PARTICIPANTS MAY HAVE. ONCE INTRODUCTION IS READ BY THE PARTICIPANT, CONFIRM THAT THEY AGREE TO PARTICIPANT AND BE RECORDED. START RECORDING ONCE CONSENT IS GIVEN.

|---------------------------------------------------------------------------------------------------------------------------------------------------|

WEB INSTRUMENT, FORMS: COVID, COVA_OTH-- COVG_OTH

COVID AND COVA_OTH to COVG_OTH SCREENS ARE TO BE SELF-ADMINISTERED BY THE PARTICIPANT. THEY DO NOT NEED TO BE INSTRUCTED TO THINK ALOUD.

TAKE NOTE OF THE RESPONDENT’S ANSWER FOR EACH SUB-QUESTION AND RECORD ANY GENERAL OBSERVATIONS.

|---------------------------------------------------------------------------------------------------------------------------------------------------|

WEB INSTRUMENT, FORM: BRK

THIS IS WHERE YOU WILL START TO GUIDE THE FLOW OF THE INTERVIEW. TELL RESPONDENTS WHAT TO EXPECT FOR THE RETROSPECTIVE PROBING. REFER TO THE INTERVIEWER GUIDE FOR THE TEXT.

|---------------------------------------------------------------------------------------------------------------------------------------------------|

WEB INSTRUMENT, FORMS: STATA - STATG

STATIC SCREENS THAT WILL HELP YOU IN YOUR RETROSPECTIVE PROBING. REFER TO THE PROBES IN THIS DOCUMENT.

|---------------------------------------------------------------------------------------------------------------------------------------------------|

WEB INSTRUMENT, FORM: EMAIL

CONFIRM IF RESPONDENTS WOULD LIKE AMAZON OR ZELLE INCENTIVE. CONFIRM THE EMAIL ADDRESS WHERE THE PARTICIPANT WANTS THEIR INCENTIVE TO BE SENT. ALSO REMIND THEM TO SEND AN EMAIL CONFIRMATION OF INCENTIVE RECEIPT.

|---------------------------------------------------------------------------------------------------------------------------------------------------|

WEB INSTRUMENT, FORM: END

THIS IS END OF INTERVIEW. MAKE SURE THE RECORDING IS SAVED IN THE PROJECT FOLDER.

|---------------------------------------------------------------------------------------------------------------------------------------------------|

FORM: INTRO

INTRODUCTION

The National Center for Science and Engineering (NCSES) within the National Science Foundation designs, conducts, and sponsors surveys on science, engineering, and technology. NCSES is considering the addition of new questions on the impact of the novel coronavirus 2019 (COVID-19) on doctoral students nearing their graduate studies. In order to produce the best information possible, the NCSES is conducting the evaluation to ensure that the new questions and the associated response options adequately capture the experiences of doctoral students.

You have volunteered to take part in a study on the NCSES’ Survey of Earned Doctorates (SED). This study is being conducted by RTI International, a not-for-profit research firm, serving as the survey contractor on behalf of the NCSES.

This interview will take approximately one hour to complete. During the interview we will ask you to complete a web survey asking about how COVID-19 affected your doctoral degree completion and postgraduation plans. We will then have a few follow-up questions about the survey questions and your responses.

Your participation in this interview is very important because it will help NCSES capture how COVID-19 has impacted graduating doctoral students. You will receive $40 Zelle or Amazon gift card and I will ask you to email me back confirming receipt to document that you have received it.

Your interview session will be recorded to ensure we have a complete record of your comments. We plan to only use the recording to improve the survey questions. Staff directly involved in this research project will have access to the recording.

Your responses will be used for research purposes only and will be kept strictly confidential and safeguarded in accordance with the National Science Foundation Act of 1950, as amended, and the Privacy Act of 1974. Any information publicly released will be in a form that does not personally identify you or your institution. The SED is authorized by law (42 U.S.C. 1862 Section 3.a.6). The OMB control number is 3145-0019.

If you have any questions about the study you may contact Ms. Jamie Friedman of RTI, at 1-877-256-8167. If you have any questions about your rights as a research participant in this study, you may contact the Office of Research Protection at RTI International at 1-866-214-2043, a toll-free number.

If you agree to participate in this study and consent to be recorded for the purpose stated above, please click NEXT to begin.

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: COVID

VAR: COVA, COVB, COVC, COVD, COVE, COVF, COVG

The next questions focus on how the Coronavirus Disease 2019 (COVID-19) may have affected your graduate experiences and career plans.

Did you experience any of the following as a result of the COVID-19 pandemic?

Select Yes or No for each

	Yes	No
The timeline for completing my doctoral degree changed.	1	2	(COVA)
My research was disrupted.	1	2	(COVB)
Funding for my doctoral studies was reduced or suspended.	1	2	(COVC)
My immediate postgraduate employment or education plans changed.	1	2	(COVD)
I changed my career plans or goals (e.g., type of employer, research focus).	1	2	(COVE)
I changed my decision on where to live in the year after graduation.	1	2	(COVF)
My graduate experience or career plans changed in other ways.	1	2	(COVG)

QUESTION CHARACTERISTICS: COVA-COVG are all displayed in a grid with radio buttons.

Administered to: ALL

Post-logic: GO TO COVA_OTH

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: COVA_OTH

VAR: COVA_OTH

PrE-LOGIC: SHOW IF COVA=1

IF FIRST FOLLOW-UP, SHOW {FILL_2}

ELSE IF NOT FIRST FOLLOW-UP, SHOW {FILL_1}

{FILL_1}=“Impact of COVID-19 on your graduate experience”

{FILL_2}=“The next questions ask you to provide more detail on the ways COVID-19 affected your graduate experiences and career plans.“

{FILL_1}

{FILL_2}

Previously, you said: “The timeline for completing my doctoral degree changed.”

How did the timeline for completing your doctoral degree change?

________________________________________________________________ [COVA_OTH]

QUESTION CHARACTERISTICS: COVA_OTH is an open-ended long-essay textbox.

Post-logic: GO TO COVB_OTH

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: COVB_OTH

VAR: COVB_OTH

PrE-LOGIC: SHOW IF COVB=1

IF FIRST FOLLOW-UP, SHOW {FILL_2}

ELSE IF NOT FIRST FOLLOW-UP, SHOW {FILL_1}

{FILL_1}=“Impact of COVID-19 on your graduate experience”

{FILL_2}=“The next questions ask you to provide more detail on the ways COVID-19 affected your graduate experiences and career plans.“

{FILL_1}

{FILL_2}

Previously, you said: “My research was disrupted.”

How was your research disrupted?

________________________________________________________________[COVB_OTH]

QUESTION CHARACTERISTICS: COVB_OTH is an open-ended long-essay textbox.

Post-logic: GO TO COVC_OTH

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: COVC_OTH

VAR: COVC_OTH

PrE-LOGIC: SHOW IF COVC=1

IF FIRST FOLLOW-UP, SHOW {FILL_2}

ELSE IF NOT FIRST FOLLOW-UP, SHOW {FILL_1}

{FILL_1}=“Impact of COVID-19 on your graduate experience”

{FILL_2}=“The next questions ask you to provide more detail on the ways COVID-19 affected your graduate experiences and career plans.“

{FILL_1}

{FILL_2}

Previously, you said: “Funding for my doctoral studies was reduced or suspended.”

How was your funding reduced or suspended?

________________________________________________________________[COVC_OTH]

QUESTION CHARACTERISTICS: COVC_OTH is an open-ended long-essay textbox.

Post-logic: GO TO COVD_OTH

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: COVD_OTH

VAR: COVD_OTH

PrE-LOGIC: SHOW IF COVD=1

IF FIRST FOLLOW-UP, SHOW {FILL_2}

ELSE IF NOT FIRST FOLLOW-UP, SHOW {FILL_1}

{FILL_1}=“Impact of COVID-19 on your graduate experience”

{FILL_2}=“The next questions ask you to provide more detail on the ways COVID-19 affected your graduate experiences and career plans.“

{FILL_1}

{FILL_2}

Previously, you said: “My immediate postgraduate employment or education plans changed.”

How did your employment or education plans change?

________________________________________________________________[COVD_OTH]

QUESTION CHARACTERISTICS: COVD_OTH is an open-ended long-essay textbox.

Post-logic: GO TO COVE_OTH

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: COVE_OTH

VAR: COVE_OTH

PrE-LOGIC: SHOW IF COVE=1

IF FIRST FOLLOW-UP, SHOW {FILL_2}

ELSE IF NOT FIRST FOLLOW-UP, SHOW {FILL_1}

{FILL_1}=“Impact of COVID-19 on your graduate experience”

{FILL_2}=“The next questions ask you to provide more detail on the ways COVID-19 affected your graduate experiences and career plans.“

{FILL_1}

{FILL_2}

Previously, you said: “I changed my career plans or goals (e.g., type of employer, research focus).”

How did your career plans or goals change?

________________________________________________________________[COVE_OTH]

QUESTION CHARACTERISTICS: COVE_OTH is an open-ended long-essay textbox.

Post-logic: GO TO COVF_OTH

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: COVF_OTH

VAR: COVF_OTH

PrE-LOGIC: SHOW IF COVF=1

IF FIRST FOLLOW-UP, SHOW {FILL_2}

ELSE IF NOT FIRST FOLLOW-UP, SHOW {FILL_1}

{FILL_1}=“Impact of COVID-19 on your graduate experience”

{FILL_2}=“The next questions ask you to provide more detail on the ways COVID-19 affected your graduate experiences and career plans.“

{FILL_1}

{FILL_2}

Previously, you said: “I changed my decision on where to live in the year after graduation.”

How did it impact your decision on where to live in the year after graduation?

________________________________________________________________[COVF_OTH]

QUESTION CHARACTERISTICS: COVF_OTH is an open-ended long-essay textbox.

Post-logic: GO TO COVG_OTH

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: COVG_OTH

VAR: COVG_OTH

PrE-LOGIC: SHOW IF COVG=1

IF FIRST FOLLOW-UP, SHOW {FILL_2}

ELSE IF NOT FIRST FOLLOW-UP, SHOW {FILL_1}

{FILL_1}=“Impact of COVID-19 on your graduate experience”

{FILL_2}=“The next questions ask you to provide more detail on the ways COVID-19 affected your graduate experiences and career plans.“

{FILL_1}

{FILL_2}

Previously, you said: “My graduate experience or career plans changed in other ways.”

In what other ways did your graduate experience or your plans change?

________________________________________________________________[COVG_OTH]

QUESTION CHARACTERISTICS: COVG_OTH is an open-ended long-essay textbox.

Post-logic: GO TO BRK

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: BRK

Thank you. The control of the screen will now be given back to the interviewer.

We will now ask you some additional questions about the ways you understood what was asked and how you arrived at your answers.

Sometimes our questions may sound a little strange. For example, we might ask what a certain word means to you. The reason for this is to learn about the process you went through in your mind when you answered the questions. There are no right or wrong answers. We are interviewing people with different backgrounds to help find the question wordings that will work best for everyone.

Post-logic: GO TO STATA

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: STATA

[SHOW FIRST BOX ONLY IF COVA=2 or missing]

Below is how you answered the following question:

Did you experience any of the following as a result of the COVID-19 pandemic?

	Yes	No
The timeline for completing my doctoral degree changed.	1	2	(COVA)

[SHOW SECOND BOX ONLY IF COVA=1]

Impact of COVID-19 on your graduate experience

Previously, you said: “The timeline for completing my doctoral degree changed.”

How did the timeline for completing your doctoral degree change?

________________________________________________________________[COVA_OTH]

Probe 1:

First, let us look at your answer to the first statement “The timeline for completing my doctoral degree changed.” You answered [ ].

IF ANSWER IS YES, ALSO READ ALOUD THEIR RESPONSE TO THE OPEN-ENDED QUESTION.

Probe 1a: What did you consider when you were answering this question?

PLEASE PROBE FURTHER FOR REASON OF DIFFERENCE IF EXPECTED GRADUATION DATE MENTIONED IN JANUARY 2020 DIFFER FROM DATE MENTIONED IN SCREENER

Probe 1b: [IF R ANSWERED YES AND IF RESPONSE TO OPEN-ENDED QUESTION AND PROBE 1A DO NOT PROVIDE SUFFICIENT DETAIL, ASK] Can you tell me more about how the timeline for completing your doctoral degree has changed?

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: STATB

[SHOW FIRST BOX ONLY IF COVB=2 or missing]

Below is how you answered the following question:

Did you experience any of the following as a result of the COVID-19 pandemic?

	Yes	No
My research was disrupted.	1	2	(COVB)

[SHOW SECOND BOX ONLY IF COVB=1]

Impact of COVID-19 on your graduate experience

Previously, you said: “My research was disrupted.”

How was your research disrupted?

________________________________________________________________[COVB_OTH]

Probe 2:

Next statement was, “My research was disrupted”. You answered [ ].

IF ANSWER IS YES, ALSO READ ALOUD THEIR RESPONSE TO THE OPEN-ENDED QUESTION.

Probe 2a: What does “research” mean to you in this context?

PLEASE NOTE HOW BROAD OR NARROW THEY HAVE DEFINED “RESEARCH” WHEN THEY READ AND RESPONDED TO THE STATEMENT

Probe 2b: [IF R ANSWERED YES AND IF RESPONSE TO OPEN_ENDED QUESTION AND PROBE 2A DO NOT PROVIDE SUFFICIENT DETAIL, ASK] What kind(s) of research activities were you thinking about?

Probe 2c: [IF R ANSWERED YES AND IF RESPONSE TO OPEN_ENDED QUESTION AND PROBE 2A DO NOT PROVIDE SUFFICIENT DETAIL, ASK] What kind(s) of disruptions were you thinking about?

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: STATC

[SHOW FIRST BOX ONLY IF COVC=2 or missing]

Below is how you answered the following question:

Did you experience any of the following as a result of the COVID-19 pandemic?

	Yes	No
Funding for my doctoral studies was reduced or suspended.	1	2	(COVC)

[SHOW SECOND BOX ONLY IF COVC=1]

Impact of COVID-19 on your graduate experience

Previously, you said: “Funding for my doctoral studies was reduced or suspended.”

How was your funding reduced or suspended?

________________________________________________________________[COVC_OTH]

Probe 3:

Next one is “Funding for my doctoral studies was reduced or suspended.” You answered [ ].

IF ANSWER IS YES, ALSO READ ALOUD THEIR RESPONSE TO THE OPEN-ENDED QUESTION.

Probe 3a: How do you interpret the word “funding” in this context?

[IF NEEDED, ASK] Are there multiple types or sources of support you received for your doctoral studies? If so, were you thinking only about one type or multiple types of support?

Probe 3b: [IF R ANSWERED YES AND IF RESPONSE TO OPEN-ENDED QUESTION AND PROBE 3A DO NOT PROVIDE SUFFICIENT DETAIL, ASK] What type of funding was affected, and did you ever get it back or were you able to find another source of funding?

[PROBE THE PERMANENCY OF THE FUNDING REDUCTION OR SUSPENSION, IF NOT MENTIONED ALREADY]

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: STATD

[SHOW FIRST BOX ONLY IF COVD=2 or missing]

Below is how you answered the following question:

Did you experience any of the following as a result of the COVID-19 pandemic?

	Yes	No
My immediate postgraduate employment or education plans changed.	1	2	(COVD)

[SHOW SECOND BOX ONLY IF COVD=1]

Impact of COVID-19 on your graduate experience

Previously, you said: “My immediate postgraduate employment or education plans changed.”

How did your employment or education plans change?

________________________________________________________________[COVD_OTH]

Probe 4:

Next, “My immediate postgraduate employment or education plans changed.” You answered [ ].

IF ANSWER IS YES, ALSO READ ALOUD THEIR RESPONSE TO THE OPEN-ENDED QUESTION.]

Probe 4a: When you read “immediate postgraduate employment or education plans”, what time period were you thinking of when you answered this statement?

Probe 4b: [IF R ANSWERED YES AND IF RESPONSE IN TO OPEN-ENDED QUESTION AND PROBE 4A DO NOT PROVIDE SUFFICIENT DETAIL, ASK] What types of changes to your postgraduate employment or education plans were you thinking about?

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: STATE

[SHOW FIRST BOX ONLY IF COVE=2 or missing]

Below is how you answered the following question:

Did you experience any of the following as a result of the COVID-19 pandemic?

	Yes	No
I changed my career plans or goals (e.g., type of employer, research focus).	1	2	(COVE)

[SHOW SECOND BOX ONLY IF COVE=1]

Impact of COVID-19 on your graduate experience

Previously, you said: “I changed my career plans or goals (e.g., type of employer, research focus).”

How did your career plans or goals change?

________________________________________________________________[COVE_OTH]

Probe 5:

Next, “I changed my career plans or goals (e.g., type of employer, research focus).” You answered [ ].

IF ANSWER IS YES, ALSO READ ALOUD THEIR RESPONSE TO THE OPEN-ENDED QUESTION.

Probe 5a: How far ahead into the future were you thinking as you answered this question?

Probe 5b: What came to mind when you read “career plans or goals”?

MAKE A NOTE WHETHER RESEARCH SPECIALIZATION OR EMPLOYMENT SECTOR WAS MENTIONED

Probe 5c: [IF R ANSWERED YES AND IF RESPONSE TO OPEN-ENDED QUESTION AND PROBE 5B DO NOT PROVIDE SUFFICIENT DETAIL, ASK] What type of changes you had to make to your career plans or goals?

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: STATF

[SHOW FIRST BOX ONLY IF COVF=2 or missing]

Below is how you answered the following question:

Did you experience any of the following as a result of the COVID-19 pandemic?

	Yes	No
I changed my decision on where to live in the year after graduation.	1	2	(COVF)

[SHOW SECOND BOX ONLY IF COVF=1]

Impact of COVID-19 on your graduate experience

Previously, you said: “I changed my decision on where to live in the year after graduation.”

How did it impact your decision on where to live in the year after graduation?

________________________________________________________________[COVF_OTH]

Probe 6:

Next, “I changed my decision on where to live in the year after graduation.” You answered [ ].

IF ANSWER IS YES, ALSO READ ALOUD THEIR RESPONSE TO THE OPEN-ENDED QUESTION.

Probe 6a: Prior to COVID-19, did you have a clear plan for where you would live in the year after graduation?

[PROBE IF THEY NEED A SPONSOR FOR WORK]

Probe 6b: [IF R ANSWERED YES AND IF RESPONSE TO OPEN-ENDED QUESTION AND PROBE 6A DO NOT PROVIDE SUFFICIENT DETAIL, ASK] Please you describe the way(s) your decision on where to live in the year after graduation changed.

|-----------------------------------------------------------NEXT SCREEN----------------------------------------------------------------------|

FORM: STATG

[SHOW FIRST BOX ONLY IF COVG=2 or missing]

Below is how you answered the following question:

Did you experience any of the following as a result of the COVID-19 pandemic?

	Yes	No
My graduate experience or career plans changed in other ways.	1	2	(COVG)

[SHOW SECOND BOX ONLY IF COVG=1]

Impact of COVID-19 on your graduate experience

Previously, you said: “My graduate experience or career plans changed in other ways.”

In what other ways did your graduate experience or your plans change?

________________________________________________________________[COVG_OTH]

Probe 7:

Your answer to “My graduate experience or career plans changed in other ways” was [ ].

IF ANSWER IS YES, ALSO READ ALOUD THEIR RESPONSE TO THE OPEN-ENDED QUESTION.

Probe 7a: What were you thinking about when you answered this question?

Probe 7b: [IF R ANSWERED YES AND IF NECESSARY] How prevalent would you say this other experience would be to other doctoral students?

Probe 7c: Are there any other details about COVID-19 in relation to the graduate experience you would like us to know about?

6