Survey of Earned Doctorates - Cognitive Interviews

Attachment 5: Cognitive Interview Protocol

19

Cognitive Interview Protocol
Four rounds of cognitive interviews will be conducted. The structure of the cognitive interview
protocol will be adapted as we move through the four rounds of testing. In the early rounds we
will use a think-aloud technique to encourage respondents to verbalize their thoughts as they
work through the questionnaire. The cognitive interviewer will probe as necessary to obtain
additional information from the respondent. The full SED web instrument will be cognitively
tested in the think-aloud protocol.
As questionnaire revisions are tested and finalized over the rounds of interviewing, we will move
from the think-aloud approach to one based primarily on concurrent and retrospective probing.
As outlined earlier, several items in the SED instrument have been selected for focus in the
cognitive testing.
Think-Aloud Interviews (anticipated for Rounds 1through 3)
In the Think-Aloud interviews, all items in the SED will be explored, but we will focus on items
that have been identified as requiring evaluation and new items that are being developed. During
the Think-Aloud interview, respondents will be trained on how to “think aloud” as they complete
the questionnaire. Attachment 1 is the paper version of the SED. Exhibit 3 at the end of this
attachment lists all the current SED items. As shown in the Exhibit, items that will receive focus
in these interviews include probes for the interviewer to use as well as notes on issues with the
items that are being explored. However, note that the protocol and items will be subject to
modification during cognitive testing.
Think-Aloud Protocol
Step 1: The interviewer will explain the study and obtain informed consent (Attachment 6) for
participation in the interview and for audio-recording.
Step 2: The interviewer will explain the think-aloud process. The respondent will do a practice
activity.
“While you fill out the survey I want you to think out loud about how you are answering
each question. Hearing you talk about how you answer the questions will help me
understand how to improve the survey questions. To help you get started on thinking
aloud, we are going to do a practice activity.”
“Try to visualize the place where you live, and think about how many windows there are
in that place. As you count up the windows, tell me what you are seeing and thinking
about.”1
“Now I would like you to do the same thinking aloud while you complete the
questionnaire. While you are completing the questionnaire, I will also ask you some
questions about some of the items.”
Step 3: The interviewer will observe the respondent completing the questionnaire while thinking
aloud and note any issues that require follow-up probing:
1

Willis, G. B. (2004). Cognitive interviewing: A tool for improving questionnaires. Thousand Oaks: Sage.

20







What questions does the respondent ask while completing the questionnaire?
Does the respondent make comments to qualify the responses he/she has chosen?
Are there items that the respondent spent extra time on?
Are there items that the respondent skipped because they were unsure how to
answer?
Did the respondent provide any answers that were inconsistent with other answers
(there are specific items to check in the cognitive interview protocol)?

During the think-aloud, the interviewer will conduct concurrent probing on selected
questions, as shown in the cognitive interview protocol (see Exhibit 3 below). In addition
to these scripted probes, the interviewer will include additional probing about other issues
that were noted. Since respondents will vary in a number of characteristics, including
field of study, educational history, citizenship and type of doctoral institution, they will
likely vary in which aspects of the questionnaire they find difficult or confusing.
Therefore, it is not possible to script in advance the probes that may be asked of a
particular respondent. Interviewers will probe spontaneously as needed to explore issues
in greater depth. Interviewers will be trained to probe on the unique issues that may arise
in each interview.
Step 4: The interview will wrap up with several retrospective probes. The interviewer will ask
the respondent about what questions were the easiest or most difficult to answer, and
whether the respondent has additional suggestions for improving any of the survey items.
Step 5: At the conclusion of the interview, the interviewer will answer any questions the
respondent may have about the study. The interviewer will give the respondent $50 for
their participation and obtain a signed receipt.
Respondent Debriefing Interviews (anticipated for Round 4)
In the Respondent Debriefing interviews, the full instrument will be cognitively tested. Instead
of a think-aloud approach, these interviews will rely primarily on retrospective probing to
explore the respondent’s approach to the questionnaire. We expect to move to Respondent
Debriefing by Round 4 of the interviews.
After the respondent has completed the questionnaire independently, the interviewer will conduct
a cognitive debriefing with the respondent. On occasion the debriefing questions may be asked
concurrently or at the end of a section, should the interviewer observe an issue that requires
exploration. Probing more immediately instead of at the end of the survey may allow the
respondent to recall more fully their thought processes on any problematic items. Since most
issues with the questionnaire items are expected to have been explored and addressed in the early
rounds of testing, we do not anticipate the wide use of concurrent probing.
Respondent Debriefing Protocol
Step 1: The interviewer will explain the study and obtain informed consent (Attachment 6) for
participation in the interview and for audio-recording.

21

Step 2: The interviewer will instruct the respondent to complete the questionnaire independently.
During survey completion, the interviewer will observe and note any issues that require
follow-up probing (see list in Step 3 of the Think-Aloud Protocol above).
Step 3: After the respondent has completed the questionnaire, the interviewer will review
selected questions with the respondent. Since the specific items to be tested will depend
on the results of earlier rounds of testing, suggestions for probes will be prepared after the
data from earlier rounds have been analyzed. The interviewer will include additional
spontaneous probing about other issues that arise.
Step 4: At the conclusion of the interview, the interviewer will answer any questions the
respondent may have about the study. The interviewer will give the respondent $50 for
their participation and obtain a signed receipt.

22

Exhibit 3: Cognitive Interview Protocol for the Think-Aloud Interviews

PART A - EDUCATION
1.

2.

3.

Probes:
 How did you decide on the name of the primary field of your dissertation research?
 How did you decide which code to choose?
Notes to interviewer:
Does R select a field name off the list or describe the field in his or her own words? Is the R able to find a code that
matches his or her field?

4.

Probes:
 In your own words, what is this question asking?
 In your own words, what is interdisciplinary research?
 Tell me how you decided on your answer.

23

Notes to interviewer:
Does R select a field name off the list or does the R describe the field in his or her own words? Is the R able to find
the code that matches his or her field? Does R have difficulty deciding whether his or her research is
interdisciplinary? Does R have difficulty selecting the secondary field?

5.

Probes:
 In your own words, what is this question asking?
 What did you think about in deciding which department supervised your doctoral studies?
Notes to interviewer:
Does the respondent have any difficulty deciding on an answer to this question?

6.

24

7.

8.

25

9.

10.

[Note: Although the screen shot above displays “December 2014” as the month/year the degree was
granted, respondents will see the month and year they self-report when completing the online SED web
survey registration.]
Probes:
 In your own words, what is this question asking?
 How did you decide on the dates that you gave?
 The question asks about when your degree was granted. What does the word “granted” mean to you?
Notes to interviewer:
Does R have any difficulty determining the date he or she started or was granted the doctoral degree? Does R notice
that date granted is asked before date started the degree?

11.

Probes:
 How did you decide on the name of the primary field of study for your doctoral degree?
 How did you decide which code to choose?
Notes to interviewer:
Does R select a field name off the list or describe the field in his or her own words? Is R able to find the code that

26

matches his or her field? Does R note any redundancy between this question and A3?

12.

[Note: Although the screen shot above displays “Univ. of Chicago” as the institution name, respondents
will see the institution name they self-report when completing the online SED web survey registration.]

13.

Notes to interviewer:
Does R have any difficulty deciding how to respond?

14.

Probes:
 In your own words, what is this question asking?
 How did you decide on the dates that you gave?
Notes to interviewer:
Does R have any difficulty determining the date he or she started or was granted the degree? Does R notice that this
question concerns the most recent master’s degree? Do the dates of the master’s degree overlap with the dates of the
doctoral degree? If so, was this a joint degree program?

27

15.

Probes:
 How did you decide on the name of the primary field of your most recent master’s degree?
 How did you decide which code to choose?
Notes to interviewer:
Does R report on the field for the most recent master’s? Does R select a field name off the list or describe the field
in his or her own words? Is the respondent able to find a code that matches his or her field?

16.

17.

Notes to interviewer:
Does R have any difficulty deciding how to respond?

18.

Probes:
 In your own words, what is this question asking?
 How did you decide on the dates that you gave?
Notes to interviewer:

28

Does R have any difficulty determining the date he or she started or was granted the degree? Does R notice that this
question concerns first bachelor’s degree? Do the dates of the first bachelor’s degree overlap with the dates of the
most recent master’s degree? If so, was this a joint degree program?

19.

Probes:
 How did you decide on the name of the primary field of your first bachelor’s degree?
 How did you decide which code to choose?
Notes to interviewer:
Does R report on the field for the first bachelor’s? Does R select a field name off the list or describe the field in his
or her own words? Is the respondent able to find a code that matches his or her field?

20.

21.

Probes:
 In your own words, what is this question asking?
 What does the term “postsecondary” mean to you?
 How did you decide on your answer to this question?
Notes to interviewer:
Has R considered all degrees earned after high school? Has R considered community college degrees?

29

22.

Probe:
 How easy or difficult was it to provide this information?
Notes to interviewer:
Do the dates for any degrees reported across the educational history questions overlap? Probe the overlap. Was R in
a joint degree program? Did R misreport dates? How certain is R of the dates? If needed, ask R to step through the
timeline of degrees earned and compare to degrees reported.

23.

Notes to interviewer:
Does R have any difficulty deciding what degrees to include? Does R have difficulty deciding how to respond?

24.

Probes:
 In your own words, what is this question asking?
 What does the phrase “prerequisite for admission” mean to you?
 How did you decide on your answer?

30

Notes to interviewer:
How does R interpret this question? How did R decide whether a master’s degree was a prerequisite for admission?

25.

Probes:





In your own words, what is this question asking?
What does the phrase “any program or capacity” mean to you?
How did you decide on your answer?
How did you decide on the date that you gave?

Notes to interviewer:
How does R interpret this question? What graduate schools did R consider? What did R include?

26.

Probes:
 In your own words, what is this question asking?
 How did you decide on your answer?
 How easy or difficult was it to determine your answer?
Notes to interviewer:
How does R interpret the question? How does R arrive at an answer? Does R have difficulty separating time spent
on coursework and exams from time spent working on the dissertation? Does R have difficulty rounding?
What was R including in the answer to each part of this question?

27.

Probes:
 In your own words, what is this question asking?
 How did you decide on your answer?
Notes to interviewer:
Does R have any difficulty deciding on an answer?

31

28.

Probes:
 How did you decide on your answer?
Notes to interviewer:
Does R have any difficulty deciding on an answer? What was R doing during the time he or she was not working on
the degree? Does R have any difficulty rounding?

29.

Probes:





In your own words, what is this question asking?
In your own words, what is a community college? What is a two-year college?
How did you decide on your answer?
What did you include in your answer?

Notes to interviewer:
Does R have any difficulty deciding on an answer? Does R consider credits that transferred to a 4-year college?
Did R consider credits that did not transfer?

30.

Probes:
 In your own words, what is this question asking?
 How did you decide which option(s) to select?
 How easy or difficult was it to answer this question?
Notes to interviewer:
Does R have any difficulty deciding on an answer? Note if R asks how many radio buttons to choose.

32

PART B – POSTGRADUATION PLANS
31.

Probes:
 What did you think about to arrive at your answer?
 What period of time are you referencing to arrive at your answer?
Notes to interviewer:
Does R have any difficulty providing a response?

32.

Probes:





What did you think about to arrive at your answer?
What period of time are you referencing to arrive at your answer?
What does “postdoc” mean to you?
In your field of study, are new PhDs expected to have postdocs or not?

Notes to interviewer:
Does R have any difficulty providing a response?

33.

Probes:







What did you think about to arrive at your answer?
In your own words, what is this question asking?
What period of time are you referencing to arrive at your answer?
[If “other” chosen] Why did you pick “Other”?
What does “organizations” mean to you in the context of this question?
Can you think of other plans that should be listed here, but are not?

Notes to interviewer:

33

Does R understand what the question is asking? Does R have any issues choosing a response? Does R have any
difficulty with specific terms in the response options?

34.

Probe (If necessary):
 Could you tell me more about your answer?
Notes for interviewer:
Does R have difficulty choosing a response? What time period is R thinking about when answering this question?

35.

Probes:
 How did you decide which answer to choose?
 In your own words, what are these responses?
 [If “other” chosen] Why did you pick “Other”?
Notes to interviewer:
Does R have any difficulty finding a response that matches his or her training or study? Is R able to distinguish
between each of the response options?

36.

34

37.

Notes to interviewer:
Does R have any difficulty finding a response that matches his or her main source of financial support? Does R have
any issues selecting just one response? What time period is R thinking about when answering this question?

38.

Probes:
 How did you decide which answer to choose?
 [If “other” chosen] Why did you pick “Other”?
 Are there any types of employers you think should be on this list, but are not there? (In other words, what’s
missing?)
Notes to interviewer:
Does R have any difficulty finding a response that matches his or her principal employer? Is R able to choose just
one response? What time period is R thinking about when answering this question?

35

39.

Notes to interviewer:
Is R’s response consistent with the response provided for B6? If not, ask R to explain how he arrived at his/her
response to B6.

40.

41.

42.

36

Probes:





How did you decide which answer to choose for your primary work activity?
How did you decide which answer to choose for your secondary work activity?
Could you tell me more about the work you will be doing on a day to day basis?
[If “other” chosen] Why did you pick “Other”?

Notes to interviewer:
Does R have difficulty finding a response that matches his/her primary and/or secondary work activities? Is R able
to distinguish between the response options? What time period is R thinking about in answering this question?

PART C – BACKGROUND INFORMATION
43.

44.

Probes:
 In your own words, what does the second response option mean?
Notes to interviewer:
Does R have any difficulty finding a response that matches his or her marital status?

37

45.

Notes to interviewer:
Does R have any issues finding the “check if none” box?

46.

Notes to interviewer:
Does R have any difficulty identifying with the terms “mother” and “father”?

47.

48.

38

49.

Probes:
 [If R does not provide a response] Can you tell me why you chose not to respond?
Notes to interviewer:
Does R have any difficulty finding a response that matches his or her citizenship status?

50.

51.

Notes to interviewer:
Does R have any difficulty choosing a response?

52.

39

Probes:
 [If R does not provide a response] Can you tell me why you chose not to respond?
Notes to interviewer:
Does R have any difficulty finding a response that matches his or her racial background?

53.

54.

40

55.

56.

Probes:
 [If R does not provide a response] Can you tell me why you chose not to respond?
 If we had requested your full Social Security Number, would you have provided it?
 [If R is not willing to provide full SSN] Why would you choose not to provide your full Social Security
Number?
Notes to interviewer:
Is R hesitant to provide the last four digits or full SSN? What are R’s concerns about providing this information?

41