Summary of SED COVID-19 Questions Cognitive Testing Results

Attachment 2. Summary of SED COVID-19 Questions Cognitive Testing Results

1. BACKGROUND AND PURPOSE

The purpose of this task is to cognitively test newly-developed question items for the National Center for Science and Engineering Statistics (NCSES)’ Survey of Earned Doctorates (SED) to determine how well the questions work (i.e., are understood and are capturing intended data) when administered to the graduating doctoral students. This report summarizes the results from cognitive testing and provides recommendations for changes where appropriate. An overview of participant recruitment strategies and interview procedures is presented first, followed by a presentation of findings and recommendations.

2. RECRUITMENT OF COGNITIVE INTERVIEW PARTICIPANTS

The participants were recruited from a list of 182 individuals who expressed their interest in testing 2021 SED questions earlier in the year. The interested participants are all from research universities and provided information in January 2020 about their expected doctorate graduation date, field of study, and doctorate institution name. From this list, we identified 60 students who are pursuing science, engineering, social science, and humanity research fields and said they expect to graduate in 2020 or 2021. We emailed 32 students, providing them with a description of the study and asking them to fill out a brief web screener. The screener asked about their expected graduation date, sex, and interview scheduling availability. Some participants had discrepancies in the expected graduation date they provided in January 2020 and May 2020. Because these changes may be related to COVID-19, they were kept in the pool of eligible participants. Fifteen students completed the screener. We selected 12 who were diverse on location and field of study. Nine interviews were then completed, based on both interviewers’ and participants’ schedules. Table 1 breaks down the participants by their characteristics.

TABLE 1. Cognitive interview participant characteristics breakdown

Participant ID	Institution State	Field of study	Sex	Expected Graduation Year from the Web Screener
P2	California	Nutritional Biology	F	2021
P3	Washington D.C.	Public Policy & Administration	F	2020
P5	Virginia	Geotechnical Engineering	M	2020
P6	Virginia	Electrical Engineering	M	2020
P7	Indiana	Information Science	M	2020
P8	Illinois	Biomedical Engineering	F	2022
P9	Mississippi	Biology	F	2023
P11	California	Nutritional Biology	F	2020
P13	Minnesota	Geography	F	2021

3. INTERVIEW PROCEDURES

All cognitive interviews were conducted using Zoom videoconferencing software. At the start of each interview, participants were asked to log in using a Zoom web link for the interview. Then, participants were asked to log in to the web instrument where they were instructed to read the introduction by themselves for 1-2 minutes. After the participants finish reading, the interviewer confirmed that the participants agree to participate and to be recorded. Participants were then asked to click the ‘Next’ button and then instructed to answer the survey questions as they would by themselves. Once they finished answering the filter question and the corresponding follow-up questions, the interviewer followed a guide (see Attachment 1 for the cognitive interview protocol) that contained a list of scripted probes and, when necessary, asked unscripted (i.e., spontaneous) probes to inquire more deeply about issues as they were raised by participants (e.g., can you tell me more about that). After interviews were completed, all notes were compiled and analyzed to identify issues and suggest possible areas for improvement. The following sections provide a summary of findings and suggested recommendations.

4. GENERAL FINDINGS

Overall, the filter question with the 7 impact items and the follow-up open-ended questions presented after the impact items worked well as intended. Participants were able to explain more about their situations when they said “yes” to experiencing an impact of COVID-19 to their graduate experience or career plans. When those with negative responses to a specific impact question were probed, we also found that they understood the meaning of the question and indeed meant to answer “no.” While the participants’ comprehension of the proposed questions are consistent with the intended design, we noticed that the framing of the two impact items can be improved further to better assist participants in their answering process. Based on the item specific findings described below, RTI proposes revised wording for these two items and one of the related open-ended follow-up questions.

5. ITEM SPECIFIC FINDINGS

Participants generally did not have issues responding to the filter question. They were able to read through the items and respond with ease and little hesitation. However, issues were identified with comprehension for some items. This section outlines findings for these questions and makes recommendations for improvement where appropriate.

FILTER QUESTION, ITEM 1

The timeline for completing my doctoral degree changed.

Findings:

Six participants answered “yes” to this question [P2, P5, P6, P9, P11, and P13]. In their open-ended response, three participants [P2, P5, and P11] described delays in completing their research due to COVID-19. When asked what the participants considered when answering this question, they thought of delays in their anticipated graduation date. Some participants had to access the lab on campus to complete research. Due to COVID-19, the campus laboratories either closed or had restricted hours. One participant [P13], discussed the disruption of the academic job market and felt she might delay graduation to allow the job market to improve. Many of the participants were in continuing communication about their doctorate research process and the timing of graduation in relation to job prospects with their advisors.

Recommendations:

Overall, the question appears to be well-understood by the participants and we do not recommend any changes.

FILTER QUESTION, ITEM 2

My research was disrupted.

Findings:

Six participants answered “yes” to this question [P2, P5, P8, P9, P11, and P13]. Four of the participants [P2, P8, P9, and P11] explained that they could not access the campus lab/were unable to conduct experiments, while one participant [P5] explained that since official travel was suspended, field work has been cancelled. Another participant [P13] described difficulties with accessing certain software applications due to a weak home internet connection. When asked what “research” meant in the context of the question, participants thought of bench or lab experiments, collecting and analyzing data, and writing papers. In general, they were thinking of research that is required as part of their doctoral coursework, dissertation, or work they need to do in their doctoral studies to get a better job market profile.

Recommendations:

Overall, the question appears to be well-understood by the participants and we do not recommend any changes.

FILTER QUESTION, ITEM 3

Funding for my doctoral studies was reduced or suspended.

Findings:

Two participants answered “yes” to this question [P7 and P9]. Both participants explained that this was related to their assistantship. One participant [P7] explained that the assistantship was temporarily cancelled due to COVID-19, but it was not associated with the dissertation research. The other participant [P9] was concerned about losing the teaching assistantship all together if classes are resumed online in the fall. When asked how participants interpreted the word “funding”, they generally thought of stipends and assistantships. This was either related to their research or funds to help support their daily needs (i.e., food, rent, etc..). One participant [P2] explained that she was not thinking of her larger grant when considering funding.

Recommendations:

Overall, this question performed as expected and no changes are recommended.

FILTER QUESTION, ITEM 4

My immediate postgraduate employment or education plans changed.

Findings:

Five participants answered “yes” to this question [P3, P5, P6, P7 and P9]. Three participants [P3, P6, and P7] have delayed their graduation due to fears of not having employment following graduation. One participant [P5] explained that the in-person interviews he was conducting as part of his research had been postponed, possibly delaying his plans to complete the dissertation and defend next semester. Another participant [P9], explained that the delay was due to lack of funding. When asked what participants thought was meant by “immediate postgraduate employment or education plans”, they generally thought of the months (i.e., 6-12 months) that followed graduation. Since the participants thought of postgraduate employment or education plans that followed in the months after graduation, we felt they clearly understood the question as worded.

Recommendations:

We do not recommend any changes to this question.

FILTER QUESTION, ITEM 5

I changed my career plans or goals (e.g., type of employer, research focus).

Findings:

Three participants answered “yes” to this question [P2, P3, and P6]. When probed, all participants, including those who said “no,” understood that we are asking about a change in employment sector or research focus. However, the participants focused on short-term immediate plans because of the COVID-19 outbreak. Two of the participants [P2 and P3] were concerned about the economy immediately following their graduation and their likelihood of finding a good job. Another participant [P6] hoped to open a consulting company after graduation and will likely have to delay plans due to the economic situation in his home country. When asked how far into the future all the participants were thinking about when answering this question, they provided near-term responses following graduation.

Recommendations:

To better assist participants’ comprehension of the question for providing information about their career plans or goals beyond the immediate next steps after graduation, we recommend changing the item to: “My longer term career plans or goals changed (e.g., type of employer, research focus)”. This recommendation will also distinguish the item from the previous item about immediate plans. Further, to be more specific and match the filter question, we also recommend changing the open-ended question to “How did your longer term career plans or goals change?”

FILTER QUESTION, ITEM 6

I changed my decision on where to live in the year after graduation.

Findings:

Only one participant responded “yes” to this question [P6]. He explained that his original plan was to move back to his home country, but the effects of COVID-19 have been devastating to the country’s economy. As a result, he decided to stay in the U.S. longer than he originally planned. When asked if participants had a clear plan for where you would live in the year after graduation, the participants provided mixed responses. Some had developed a plan and others were willing to live wherever they found a good job.

Recommendations:

To acknowledge that many graduates may not have an explicit decision or plan when completing the survey, we recommend: “My plans about where to live in the year after graduation were affected.”

FILTER QUESTION, ITEM 7

My graduate experience or career plans changed in other ways.

Findings:

Seven of the eight participants answered “yes” to this question [P2, P3, P5, P6, P7, P11, and P13]. The participants discussed changes such as:

• [P5] Not being able to teach a class or adapt to teaching the class online

• [P3] Challenges in finding a good job due to a reduction in negotiating power

• [P11] Uncertainty of getting a postdoc position following graduation

• [P13] Opportunities lost such as attending conferences and weekly seminars and the ability to network with committee members in person

• [P6] Not being able to bring family members to the U.S. for the dissertation defense and graduation

• [P7] Concern about no employment available post-graduation

• [P2] Concerns about the job market

Recommendations:

This appears to be a helpful question for participants to discuss any additional changes that were not covered in previous questions. No changes recommended.

4