NCSES - Testing for Coronavirus Pandemic Questions for the 2022 Survey of Earned Doctorates (SED)

Attachment 3: Cognitive Interview Protocol

2022 SED Coronavirus Pandemic Impact Question Module Testing

The cognitive interview will be conducted with up to 60 participants by an experienced RTI survey methodologist via Zoom conference call. Participants will be asked to log in using a Zoom web link for the interview.

Participants will be asked to read the consent statement by themselves for 1-2 minutes. After they finish reading, the interviewer will confirm that they agree to participate and to be recorded. Upon getting the participant’s consent, we will start recording the session. The participant will be asked to answer the survey questions as they would by themselves. After the participant answers the survey questions and all follow up questions, the interviewer will take control of the interview flow and ask the probe questions.

Upon completion of the cognitive interview, participants will be asked if they would like their incentive given via Zelle or as an Amazon gift card and to confirm the email address for their incentive. They will also be reminded that they need to send an email confirmation of receipt upon receiving the incentive.

Cognitive Interview Guide

Interviewer: ______________

Participant ID: ___________

Date: __________

Start time: ________

Interviewer Instructions

FORM: INTRO

ASK PARTICIPANTS TO READ THE INTRODUCTION FOR 1-2 MINUTES. ANSWER ANY QUESTIONS THE PARTICIPANT HAS. ONCE THE PARTICIPANT READS THE INTRODUCTION, CONFIRM THAT HE OR SHE AGREES TO PARTICIPATE AND BE RECORDED. START RECORDING ONCE CONSENT IS GIVEN.

|------------------------------------------------------------------------------------------------------------------------|

FORMS: Q1-Q3, COVID, COVC—COVG

THESE SCREENS ARE SELF-ADMINISTERED BY THE PARTICIPANT. THE PARTICIPANT DOES NOT NEED TO THINK ALOUD.

|------------------------------------------------------------------------------------------------------------------------|

FORM: BRK

THIS IS WHERE YOU WILL START TO GUIDE THE FLOW OF THE INTERVIEW. EXPLAIN HOW THE RETROSPECTIVE PROBES WILL WORK.

|------------------------------------------------------------------------------------------------------------------------|

FORMS: STATA- STATG

STATIC SCREENS THAT WILL HELP YOU IN RETROSPECTIVE PROBING. REFER TO THE PROBES IN THIS DOCUMENT.

|------------------------------------------------------------------------------------------------------------------------|

FORM: EMAIL

CONFIRM HOW THE PARTICIPANT WOULD LIKE TO RECEIVE THE INCENTIVE. CONFIRM THE PARTICIPANT’S EMAIL ADDRESS. REMIND THE PARTICIPANT TO SEND AN EMAIL CONFIRMATION OF INCENTIVE RECEIPT.

|------------------------------------------------------------------------------------------------------------------------|

FORM: END

THIS IS END OF INTERVIEW. MAKE SURE THE RECORDING IS SAVED IN THE PROJECT FOLDER.

|------------------------------------------------------------------------------------------------------------------------|

GENERAL INFORMATION

(FORM:CONSENT)

RTI International is conducting research on behalf of the National Science Foundation (NSF) to include questions related to the coronavirus pandemic for the Survey of Earned Doctorates (SED). You will be asked for your thoughts and opinions about the survey questions on the impact of the coronavirus pandemic on your graduate experience and career plans.

This interview will take approximately 30 to 45 minutes to complete. During the interview we will ask you to complete a short web survey asking about impact of the coronavirus pandemic in different areas in your doctorate studies and post-graduation plans. After you complete the survey, I will ask you some follow-up questions such as how you arrived at your answers, or how easy or difficult it was to answer the questions.

During the interview you can help us by describing anything in the questions you find confusing or difficult to understand. Your feedback will help us to improve the final set of questions. In appreciation for your participation in this survey, you will receive $30 at the end of the interview.

Your interview session will be recorded in order to have a complete record of your comments and the recording will be used to make sure that our notes are correct. Only staff directly involved in this research project will have access to the recording. The audio file will be destroyed upon completion of our report.

Your responses will be used for research purposes only and will be kept strictly confidential and safeguarded in accordance with the National Science Foundation Act of 1950, as amended, and the Privacy Act of 1974. This study is authorized by law (42 U.S.C. 1862 Section 3.a.6). The OMB control number for this study is 3145-0174.

If you have any questions about the study you may contact Ms. Jamie Friedman of RTI, at 1-877-256-8167. If you have any questions about your rights as a research participant in this study, you may contact the Office of Research Protection at RTI International at 1-866-214-2043, a toll-free number.

If you agree to participate in this study and consent to be recorded for the purpose stated above, please click NEXT to begin.

(FORM:Q1)

Q1. Where do you intend to live in the year after graduation?

Inside the United States or U.S. territory

Outside the United States

(FORM:Q2)

Q2. What best describes your postgraduation plan?

Select one

₁ I plan to seek a postdoc, residency, or other training position (A “postdoc” is a temporary position

primarily for gaining additional education and training in research, awarded in academe, industry,

government, or a nonprofit organization.)

₂ I plan to seek employment other than a postdoc or training position (including self-employment)

₃ I plan to enroll in a full-time degree program (e.g., PhD, MD, DDS, JD, MBA)

₄ I do not plan to work or study (e.g., family commitments)

₅ Other - Specify:___________________________________________

(FORM:Q3)

(IF Q2=1 OR 2)

Q3. What type of employer do you plan seek postgraduate employment with? Is it with…

Select one

Education

Government, that is, other than educational institution

Private or nonprofit organization (other than educational institution)

Other – Specify: ___________________________________________

(FORM:COVID)

The next questions focus on how the coronavirus pandemic may have affected your graduate experiences and career plans.

Did you experience any of the following as a result of the coronavirus pandemic?

Select Yes or No for each

		Yes	No
COVA	The pandemic changed the timeline for completing my doctoral degree.	1	2
COVB	The pandemic led to a reduction or suspension of funding for my doctoral studies.	1	2
COVC	The pandemic affected my research (e.g., limited access to resources, changed research plan).	1	2
COVD	The pandemic changed my immediate postgraduate employment plans (e.g., limited job opportunities, less desirable employment, work visa status).	1	2
COVE	The pandemic changed my longer term career plans (e.g., pursuit of different type of job or employer).	1	2
COVF	The pandemic affected my plans about where to live in the year after graduation.	1	2
COVG	The pandemic changed my graduate experience or career plans in other ways.	1	2

Take note of the respondent’s answer for each sub-question and record any general observations.

The pandemic changed the timeline for completing my doctoral degree.

The pandemic led to reduction or suspension of funding for my doctoral studies.

The pandemic affected my research.

The pandemic changed my immediate postgraduate employment plans.

The pandemic changed my longer term career plans.

The pandemic affected my plans about where to live in the year after graduation.

The pandemic changed my graduate experience or career plans in other ways.

Follow Up Questions (FORMS: COVC- COVG)

NOTE: If respondent says Yes to COVC, COVD, COVE OR COVG, corresponding follow-up questions appear on next screens with a text box for Other-specify response option.

[IF COVC=1]

Previously, you said: “The pandemic affected my research.”

How was your research affected as a result of the coronavirus pandemic?

Select Yes or No for each

	Yes	No
I have limited or no access to resources (e.g., lab, data, hardware, software, human subjects, archives) as a result of the pandemic.	1	2
I had to make changes to my research plan (e.g., goals, topic, focus, approach, scope) as a result of the pandemic.	1	2
The pandemic disrupted my research in other ways – specify: _________________________________________________________________________	1	2

Take note of the respondent’s answer for each sub-question and record any general observations.

I have limited or no access to resources as a result of the pandemic.

I had to make changes to my research plan as a result of the pandemic.

The pandemic disrupted my research in other ways in other ways – specify.

[Record responses for other specify below]

[IF COVD=1]

Previously, you said: “The pandemic changed my immediate postgraduate employment plans.”

How did your immediate postgraduate employment plans change as a result of the coronavirus pandemic?

Select Yes or No for each

	Yes	No
As a result of the pandemic, there are limited job opportunities in the employment I desire.	1	2
I may have to accept a less desirable job in terms of the type of position, employer, and/or location as a result of the pandemic.	1	2
I encountered difficulty with my visa status or international travel as a result of the pandemic.	1	2
The pandemic changed my postgraduate employment plans in other ways - specify: ____________________________________________________________________________	1	2

Take note of the respondent’s answer for each sub-question and record any general observations.

As a result of the pandemic, there are limited job opportunities in the employment I desire.

I may have to accept a less desirable job in terms of the type of position, employer, and/or location as a result of the pandemic.

I encountered difficulty with my visa status or international travel as a result of the pandemic.

The pandemic changed my postgraduate employment plans in other ways – specify.

[Record responses for other specify below]

[IF COVE=1]

Previously, you said: “The pandemic changed my longer term career plans.”

How did your longer term career plans change as a result of the coronavirus pandemic?

Select Yes or No for each

	Yes	No
As a result of the pandemic, I plan to pursue my career in a different type of job or field than I had considered before.	1	2
I plan to pursue my career with a different type of employer (e.g., from academe to industry) than I had considered before as a result of the pandemic.	1	2
The pandemic opened up opportunities in areas I had not considered before.	1	2
The pandemic changed my longer term career plans in other ways - specify:_________________________________________________________________	1	2

Take note of the respondent’s answer for each sub-question and record any general observations.

As a result of the pandemic, I plan to pursue my career in a different type of job or field than I had considered before.

I plan to pursue my career with a different type of employer than I had considered before as a result of the pandemic.

The pandemic opened up opportunities in areas I had not considered before.

The pandemic changed my longer term career plans in other ways – specify.

[Record responses for other specify below]

[IF COVG=1]

Previously, you said: “The pandemic changed my graduate experience or career plans in other ways.”

In what other ways did your graduate experience or plans change as a result of the coronavirus

pandemic?

______________________________________________________________________________

Take note of the respondent’s answer for each sub-question and record any general observations.

In what other ways did your graduate experience or plans change as a result of the coronavirus pandemic?

[Record responses for other specify below]

TRANSITION (FORM:BRK)

Thank you. I will guide the interview now.

Now that you have just finished answering questions about how you may have been impacted by the coronavirus pandemic, I’d like to ask you some additional questions about the ways you understood what was asked and how you arrived at your answers.

Sometimes my questions may sound a little strange. For example, I might ask what a certain word means to you. The reason for this is to learn about the process you went through in your mind when you answered the questions. There are no right or wrong answers. We are interviewing people with different backgrounds to help find the question wordings that will work best for everyone.

Probe01 (FORM: STATA)

Below is how you answered the following question:

Did you experience any of the following as a result of the coronavirus pandemic?

	Yes	No
The pandemic changed the timeline for completing my doctoral degree.	1	2	(COVA)

First, let us look at your answer to the first statement “The pandemic changed the timeline for completing my doctoral degree.” You answered [ ].

Probe01a: What did you consider when you were answering this question?

Probe 01b: [If R answered Yes] Can you tell me more about how the timeline for completing your doctoral degree has changed?

Interviewer: Confirm that the changes are related to pandemic, and not other issues

Probe02 (FORM: STATB)

Below is how you answered the following question:

Did you experience any of the following as a result of the coronavirus pandemic?

	Yes	No
The pandemic led to reduction or suspension of funding for my doctoral studies.	1	2	(COVB)

The next statement was, “The pandemic led to reduction or suspension of funding for my doctoral studies.” You answered [ ].

Probe02a: How do you interpret the word “funding” in this context?

[if needed]: Are there multiple types or sources of support you received for your doctoral studies? If so, were you thinking only about one type or multiple types of support?

Probe02b: [If R answered Yes]: What type of funding was affected, and did you ever get it back or were you able to find another source of funding?

interviewer: please probe the permanency of the funding reduction or suspension.

Interviewer: confirm that the changes are related to pandemic, and not other issues

Probe03 (FORM: STATC)

Below is how you answered the following question:

Did you experience any of the following as a result of the coronavirus pandemic?

	Yes	No
The pandemic affected my research (e.g., limited access to resources, changed research plan).	1	2	(COVC)

Next one is “The pandemic affected my research” You answered [ ].

Probe03a: How do you interpret the word “research” in this context?

Probe03b: Were the examples helpful or unhelpful? Why?

Probe03c: [If R answered Yes] How was your research affected? What kind(s) of research activities and pandemic effects were you thinking about?

Interviewer: confirm that the changes are related to pandemic, and not other issues

Probe03 Follow Up (FORM: STATC1)

[IF COVC=1]

Previously, you said: “The pandemic affected my research.”

How was your doctorate research affected as a result of the coronavirus pandemic?

	Yes	No
I have limited or no access to resources (e.g., lab, data, hardware, software, human subjects, archives) as a result of the pandemic.	1	2	(COVC1)
I had to make changes to my research plan (e.g., goals, topic, focus, approach, scope) as a result of the pandemic.	1	2	(COVC2)
The pandemic disrupted my research in other ways - specify: ________________________________________________________________	1	2	(COVC3)

The next questions are the follow up questions to “The pandemic affected my research”

For “I have limited or no access to resources as a result of the pandemic.” You answered [ ].

Probe03d: Were the examples helpful or unhelpful? Why?

Probe03e: [If R answered Yes] What resources were you thinking about when you answered this question?

Interviewer: confirm that the changes are related to pandemic, and not other issues

For the statement “I had to make changes to my research plan as a result of the pandemic.”

You answered [ ].

Probe03f: In this context, what does “research plan” mean to you?

Probe03g: Were the examples helpful or unhelpful? Why?

Probe03h: [If R answered Yes] How did your research plan change?

Interviewer: confirm that the changes are related to pandemic, and not other issues

For “The pandemic disrupted my research in other ways.” You answered [ ].

Interviewer: probe to figure out why their answer does not fit into other categories presented.

Interviewer: if needed, ask any other clarifying questions about the answer.

Probe03i: In your own words, what does “disrupted” mean?

Probe03j: [if covc1=yes and covc3=yes] How was your research disrupted? How is it different from the other response options provided?

Interviewer: confirm that the changes are related to pandemic, and not other issues

Probe04 (FORM: STATD)

Below is how you answered the following question:

Did you experience any of the following as a result of the coronavirus pandemic?

	Yes	No
The pandemic changed my immediate postgraduate employment plans (e.g., limited job opportunities, less desirable employment, visa status).	1	2	(COVD)

Next, “The pandemic changed my immediate postgraduate employment plans.” You answered [ ].

Probe04a: When you read “immediate postgraduate employment”, what time period were you thinking of?

Probe04b: Were the examples helpful or unhelpful? Why?

Probe 04c: [If R answered Yes] What types of changes to your postgraduate employment plans were you thinking about?

Interviewer: confirm that the changes are related to pandemic, and not other issues

Probe04 Follow Up (FORM: STATD1)

[IF COVD=1]

Previously, you said: “The pandemic changed my immediate postgraduate employment plans.”

How did your immediate postgraduate employment plans change as a result of the coronavirus

pandemic?

	Yes	No
As a result of the pandemic, there are limited job opportunities in the employment I desire.	1	2	(COVD1)
I may have to accept a less desirable job in terms of the type of position, employer, and/or location as a reuslt of the pandemic.	1	2	(COVD2)
I encountered difficulty with my visa status or international travel as a result of the pandemic.	1	2	(COVD3)
The pandemic changed my postgraduate employment plans in other ways - Specify:______________________________________________________________	1	2	(COVD4)

The next questions are the follow up questions to “The pandemic changed my immediate postgraduate employment plans.”

For the statement “As a result of the pandemic, there are limited job opportunities in the employment I desire.” You answered [ ].

Probe04d: How did you arrive at your answer? What were you thinking about?

Interviewer: confirm that the changes are related to pandemic, and not other issues

Probe04e: [if unclear] How did you interpret “the employment I desire”?

For “I may have to accept a less desirable job in terms of the type of position, employer, and/or location as a result of the pandemic.” You answered [ ].

Probe04f: In this context, what does “less desirable” mean to you?

Probe04g: [If R answered Yes] Can you tell me more about the situation you were thinking about when answering this question?

Interviewer: confirm that the changes are related to pandemic, and not other issues

For the statement “I encountered difficulty with my visa status or international travel as a result of the pandemic.” You answered [ ].

Probe04h: [If R answered Yes] What situation were you thinking about when you answered this question?

Finally, for “The pandemic changed my postgraduate employment plans in other ways.” You answered [ ].

Interviewer: probe to figure out why their answer does not fit into other categories presented.

Interviewer: if needed, ask any other clarifying questions about the answer.

Probe05 (FORM: STATE)

Below is how you answered the following question:

Did you experience any of the following as a result of the coronavirus pandemic?

	Yes	No
The pandemic changed my longer term career plans (e.g., pursuit of different type of job or employer).	1	2	(COVE)

Next, “The pandemic changed my longer term career plans.” You answered [ ].

Probe05a: How far ahead into the future were you thinking as you answered this question?

Probe 05b: What came to mind when you read “career plans”?

Make a note whether research specialization or employment sector was mentioned,

Probe05c: Were the examples helpful or unhelpful? Why?

Probe 05d: [If R answered Yes] What type of changes did you have to make to your career plans or goals?

Interviewer: confirm that the changes are related to pandemic and not other issues

Probe05 Follow Up (FORM: STATE1)

[IF COVE=1]

Previously, you said: “The pandemic changed my longer term career plans.”

How did your longer term career plans change as a result of the coronavirus pandemic?

	Yes	No
As a result of the pandemic, I plan to pursue my career in a different type of job or field than I had considered before.	1	2	(COVE1)
I plan to pursue my career with a different type of employer (e.g., from academe to industry) than I had considered before as a result of the pandemic.	1	2	(COVE2)
The pandemic opened up opportunities in areas I had not considered before.	1	2	(COVE3)
The pandemic changed my longer term career plans in other ways - specify:___________________________________________________________	1	2	(COVE5)

Next questions are the follow up questions to “The pandemic changed my longer term career plans.”

For “As a result of the pandemic, I plan to pursue my career in a different type of job or field than I had considered before.” You answered [ ].

Probe05e: How did you arrive at your answer? What were you thinking about?

Interviewer: confirm that the changes are related to pandemic, and not other issues

Prove05f: [If R answered Yes and unclear] What kinds of plans have you made?

Interviewer: probe if specific concrete plans have been made.

For “I plan to pursue my career with a different type of employer than I had considered before as a result of the pandemic.” You answered [ ].

Probe05g: How did you arrive at your answer? What were you thinking about?

Interviewer: confirm that the changes are related to pandemic, and not other issues

Probe05h: How is this question [different type of employer] different from the previous question [different type of job/field]?

Prove05i: [If R answered Yes and unclear] What kinds of plans have you made?

Interviewer: probe if specific concrete plans have been made.

For “The pandemic opened up opportunities in areas I had not considered before.” You answered [ ].

Probe05j: In your own words, what is this question asking?

Probe05k: [If R answered Yes] What situation were you thinking about when you answered this question?

Interviewer: confirm that the changes are related to pandemic, and not other issues

Finally, for “The pandemic changed my longer term career in other ways”

You answered [ ].

Interviewer: probe to figure out why their answer does not fit into other categories presented.

Interviewer: if needed, ask any other clarifying questions about the answer.

Probe05l: What time period were you thinking about when you answered this question?

Probe06 (FORM: STATF)

Below is how you answered the following question:

Did you experience any of the following as a result of the coronavirus pandemic?

	Yes	No
The pandemic affected my plans about where to live in the year after graduation.	1	2	(COVF)

Next, “The pandemic affected my plans about where to live in the year after graduation.”

You answered [ ].

Probe 06a: Prior to coronavirus pandemic, did you have a clear plan for where you would live in the year after graduation?

[if they are on a temporary visa, probe if they need a sponsor for work to stay in U.S.]

Probe 06b: [if r answered yes and if probe 06a did not provide sufficient detail, ask] Please describe the way(s) your decision on where to live in the year after graduation changed.

Interviewer: confirm that the changes are related to pandemic, and not other issues

Probe07 (FORM: STATG)

Below is how you answered the following question:

Did you experience any of the following as a result of the coronavirus pandemic?

	Yes	No
The pandemic changed my graduate experience or career plans in other ways.	1	2	(COVG)

[IF COVG=1]

Previously, you said: The pandemic changed my graduate experience or career plans in other ways.”

In what other ways did your graduate experience or your plans change as a result of the COVID-19 pandemic?

______________________________________________________________________________ [COVG_OTH]

Your answer to the statement “The pandemic changed my graduate experience or career plans in other ways.” was [ ].

IF ANSWER IS YES, ALSO READ ALOUD THEIR RESPONSE TO THE OPEN-ENDED QUESTION.

Probe 07a: What were you thinking about when you answered this question?

Interviewer: probe to figure out why their answer does not fit into other categories as presented.

Probe 07b: [If R answered Yes and if necessary] How common would you say this experience would be for other doctoral students?

Probe 07c: Are there any other details about how coronavirus pandemic has impacted your graduate experience and career plans that you would like us to know about?

Interviewer Observations

Did the respondent mention experiencing any emotional difficulties related to the coronavirus pandemic?

If Yes: Was the respondent able to separate emotional impacts of the pandemic from other impacts?

If Yes: Were there any questions where the respondent’s answer was especially influenced by the emotional impacts of the pandemic?

12