NCSES - Early Career Doctorates Project cognitive interviews

Attachment B.3: Protocol for Cognitive Interviews

Introduction

The National Science Foundation (NSF) is preparing for the pilot wave of a national survey of postdocs, faculty and nonfaculty researchers, the Early Career Doctorates (ECD) Survey. The purpose of the ECD Survey will be to learn more about the career paths and research opportunities for early career doctorates. As part of these preparations, the NSF would like to evaluate a number of questions that may appear on the questionnaire. RTI International is conducting these interviews for the NSF as part of this effort. I will be asking you to answer survey questions on paper and then, after each one, asking you questions such as how you interpreted the questions or how you came to your answers.

[Informed consent form – Attachment B.2]

Do you have any questions before we start?

[Begin audio recording]

[NOTE: Throughout the cognitive interview, if the ECD hesitates or appears to be confused by a question, the interviewer will administer a general probe: I noticed that you hesitated/changed your answer. Can you tell me what you were thinking about?]

ITEM 1

Please answer this question to the best of your ability. If you are unsure how to answer or choose not to answer a particular item you may skip to the next one in the series. When you are done, I will ask you some questions.

How would you rate your own knowledge, attributes and behaviors at the time you completed your doctorate or doctorate-equivalent degree?
	Very Poor	Poor	Fair	Good	Very Good
METHODOLOGY: Applying research methodologies, tools and techniques appropriately	О	О	О	О	О
INNOVATION: Developing new ideas, processes or products, which are rooted in research	О	О	О	О	О
CRITICAL-ANALYTICAL: Critically analyzing and evaluating findings and results	О	О	О	О	О
ENTHUSIASM: Being both passionate and dedicated to my tasks	О	О	О	О	О
CAREER MANAGEMENT: Taking ownership for and managing my own professional development actively	О	О	О	О	О
EMPLOYMENT CONTEXT: Understanding how organizations, institutions or businesses work	О	О	О	О	О
PROBLEM SOLVING: Formulating and applying appropriate solutions to problems and challenges	О	О	О	О	О
EFFECTIVE COMMUNICATION: Communicating information effectively and confidently to different audiences	О	О	О	О	О
CREATIVITY: Being imaginative, thinking beyond normal boundaries and developing new insights	О	О	О	О	О
FLEXIBILITY: Responding quickly to changes and adapting easily to new situations	О	О	О	О	О
RESPONSIBILITY: Working independently and taking responsibility for my actions	О	О	О	О	О
NETWORKING: Developing, maintaining and using networks or collaborations	О	О	О	О	О
SUBJECT KNOWLEDGE: Demonstrating a theoretical and practical understanding of my subject area and its wider research context	О	О	О	О	О
PROJECT MANAGEMENT: Effectively planning, managing and delivering projects in good time	О	О	О	О	О
TEAM WORKING: Working constructively with colleagues, acknowledging their contribution	О	О	О	О	О
LEADERSHIP: Influencing others, providing direction and encouraging their contribution	О	О	О	О	О
LANGUAGES: Communicating effectively in a language other than my mother tongue	О	О	О	О	О

(SOURCE: OECD Careers of Doctorate Holders questionnaire 3^rd edition, Question EDU.12)

1. First we are going to focus on the question at the top of the table. In your own words, what is this question asking? What does “at the time you completed your doctorate” mean to you?

Objective of 1: To test for consistent interpretation of the question, especially the time reference.

2. On a scale of 1 to 5 where 1=Very accurately and 5=Very inaccurately, overall how accurately do you think you remember your knowledge, attributes and behaviors at that time? How long ago was that?

Objective of 2: To evaluate recall.

3. Against what standard were you comparing yourself when you evaluated yourself?

Objective of 3: To evaluate how the respondent came up with his/her answers.

4. Were particular items harder to answer than the others? If so, which ones and why?

Objective of 4: To evaluate clarity of item wording.

5. You will notice before each definition of a skill, there is a descriptive word such as “METHODOLOGY.” When answering the question, did you read the definitions or did you just read the descriptive word? When did you read the definitions and when did you not?

Objective of 5: To determine the degree of satisficing.

Now, starting with the first row, I would like you to reread the descriptive word and the definition.

6. What does this mean to you?

7. In your opinion, is the descriptive word consistent or inconsistent with the definition provided? {If inconsistent} Did you respond in terms of the descriptive word or in terms of the definition provided?

Objective of 6 and 7: To evaluate consistency of interpretation across respondents.

Repeat 6 and 7 for each row.

ITEM 2

8. The line below represents the scale you used in the previous question. Please write a P where you would place Poor, an F where you would place Fair, a G where you would place Good, and a VG where you would place Very Good.

Very Very

Poor Good

[________________________________________________________________________________________]

Objective of 8: To evaluate the balance of response options, particularly “fair”.

ITEM 3

Now please answer this question. If you are unsure how to answer or choose not to answer a particular item you may skip to the next one in the series.

To what extent are the following knowledge, attributes and behaviors important in your CURRENT job?
	1=Not at all important	2=Of little importance	3=Moderately important	4=Important	5=Very important
INNOVATION: Developing new ideas, processes or products, which are rooted in research	О	О	О	О	О
TEAM WORKING: Working constructively with colleagues, acknowledging their contribution	О	О	О	О	О
ENTHUSIASM: Being both passionate and dedicated to my tasks	О	О	О	О	О
CAREER MANAGEMENT: Taking ownership for and managing my own professional development actively	О	О	О	О	О
PROBLEM SOLVING: Formulating and applying appropriate solutions to problems and challenges	О	О	О	О	О
EFFECTIVE COMMUNICATION: Communicating information effectively and confidently to different audiences	О	О	О	О	О
CREATIVITY: Being imaginative, thinking beyond normal boundaries and developing new insights	О	О	О	О	О
METHODOLOGY: Applying research methodologies, tools and techniques appropriately	О	О	О	О	О
FLEXIBILITY: Responding quickly to changes and adapting easily to new situations	О	О	О	О	О
RESPONSIBILITY: Working independently and taking responsibility for my actions	О	О	О	О	О
NETWORKING: Developing, maintaining and using networks or collaborations	О	О	О	О	О
EMPLOYMENT CONTEXT: Understanding how organizations, institutions or businesses work	О	О	О	О	О
SUBJECT KNOWLEDGE: Demonstrating a theoretical and practical understanding of my subject area and its wider research context	О	О	О	О	О
LANGUAGES: Communicating effectively in a language other than my mother tongue	О	О	О	О	О
PROJECT MANAGEMENT: Effectively planning, managing and delivering projects in good time	О	О	О	О	О
CRITICAL-ANALYTICAL: Critically analyzing and evaluating findings and results	О	О	О	О	О
LEADERSHIP: Influencing others, providing direction and encouraging their contribution	О	О	О	О	О
TAKING INITIATIVE	О	О	О	О	О
BEING ABLE TO DEAL WITH DIVERSITY	О	О	О	О	О
WRITTEN COMMUNICATION SKILLS	О	О	О	О	О
VERBAL COMMUNICATION SKILLS	О	О	О	О	О
PRESENTATION SKILLS	О	О	О	О	О

(SOURCE: OECD Careers of Doctorate Holders questionnaire 3^rd edition, Question EMP.7)

9. First we are going to focus on the question at the top of the table. In your own words, what is this question asking?

Objective of 9: To test for consistent interpretation of the question.

10. Are there any areas that are not listed that are that are important to your current job? If so, what are they?

Objective of 10: To test to see if important skills are not represented.

11. Now, you will notice that the last 5 rows do not have definitions. How would you define each of these?

Objective of 11: To collect qualitative information to assist with generating definitions meaningful to ECDs.

ITEM 4

12. The line below represents the scale you used in the previous question. Please write an N where you would place Not at all Important, and L where you would place Of Little Importance, an M where you would place Moderately Important, an I where you would place Important, and a VI where you would place Very Important.

Not at all Very

Important Important

[________________________________________________________________________________________]

Objective of 12: To evaluate the balance of response options.

ITEM 5

To what extent do you agree with the following statements?
	Strongly disagree	Disagree	Neutral	Agree	Strongly agree
I was well prepared for my first job after obtaining/completing my doctorate or doctorate-equivalent degree	О	О	О	О	О
My doctorate or doctorate-equivalent degree enabled me to progress towards my career aspirations	О	О	О	О	О
My doctorate or doctorate-equivalent degree enabled me to access immediate or short-term job opportunities in my chosen career	О	О	О	О	О
It was not clear to me what career opportunities I could aspire to after my doctorate or doctorate-equivalent degree was granted	О	О	О	О	О
My doctorate or doctorate-equivalent degree enabled me to make a difference in the workplace	О	О	О	О	О
If I could start all over again, I would not do a doctorate or doctorate-equivalent degree	О	О	О	О	О
My doctorate or doctorate-equivalent degree enabled me to be innovative in the workplace	О	О	О	О	О
The transition to my first job after the doctorate or doctorate equivalent degree was difficult	О	О	О	О	О
My doctorate or doctorate-equivalent degree was important to my current employer	О	О	О	О	О
I am still using the subject knowledge or methods of my doctoral or doctoral-equivalent research	О	О	О	О	О
Having a doctorate or doctorate-equivalent degree made no difference to my career path	О	О	О	О	О

(SOURCE: OECD Careers of Doctorate Holders questionnaire 3^rd edition, Question EMP.11)

13. On a scale of 1 to 5 where 1 is very easy and 5 is very difficult, how easy or difficult was it for you to answer this question? Why?

14. Were any rows harder to answer than the others? If so, which ones and why?

15. (If selected “neutral” for any rows) I noticed that you selected “neutral” for some rows. What does ‘neutral’ mean to you in this context?

Objective of 13 - 15: To evaluate consistency of interpretation of item wording and response options.

ITEM 6

Please complete the next two questions.

How many postdoctoral appointments, in total, have you had? ________

(SOURCE: Early Career Doctorates Study Academic Questionnaire methodological study version, question PA)

Since receiving your [T_DEG] degree, how many different full-time and part-time paid positions, including your current position, have you had? Please include any paid postdoctoral appointments (including ones you previously mentioned), residencies, fellowships, and internships.

Full-time positions:_____

Part-time positions:_____

(SOURCE: Early Career Doctorates Study Academic Questionnaire methodological study version, Question B2 modified from Survey of Doctorate Recipients 06 Question A38)

16. For the first question, what positions did you include? Did you include the position you currently hold? Were there any positions you have held since receiving your doctorate degree that you excluded? If so, what were they and why did you exclude them?

Objective of 16: 1) To test to see if “postdoctoral appointment” is being consistently interpreted as a postdoc or if any appointment/position after the doctorate is being counted and 2) To test to see if current postdoctoral appointments are being counted.

17. For the second question, what type of positions did you count? Were there any positions you have held since receiving your doctorate degree that you excluded? If so, what were they and why did you exclude them?

Objective of 17: To test to see if, in the context of the first question about postdocs, the second question is being interpreted as asking only about postdocs.

18. What does full-time mean to you? Part-time?

Objective of 18: to test to see if full-time and part-time are being interpreted consistently.

19. On a scale from 1 to 5 where 1 is very easy and 5 is very difficult, how difficult would you say it was to recall the number of positions you have had since receiving your [T_DEG} degree?

Objective of 19: to test recall issues related to the number of positions one has had.

ITEM 7

Now, please read this question and then think aloud as you answer it. That is, please tell me what you are thinking about as you interpret and answer the question.

How many months elapsed between the time you completed your [doctorate/doctorate-equivalent] degree and the time you accepted your first CAREER PATH JOB (if applicable)?

A "career path" job is a job that will help further your career plans or is a job in a field where you want to make your career.

NUMBER OF MONTHS _____

Have not accepted my first career path job _____

Accepted career path job either before or during my doctorate or doctorate-equivalent degree program_____

(SOURCE: OECD Careers of Doctorate Holders questionnaire 3^rd Edition, Question CAR.8)

20. In your own words, what is this question asking?

Objective of 20: To test for consistent interpretation of the question.

21. What does “completed” mean to you in the context of this question? Did you complete the requirements for your degree before you were awarded your degree? {If so} How much time elapsed between the time you completed the requirements for your degree and when your degree was awarded? {If one month or more} Did you answer in terms of when you completed the requirements for your degree or when the degree was awarded?

22. What does “accepted” mean to you in the context of this question? How much time elapsed between the time you accepted the job and when you started the job? {If one month or more} Did you answer in terms of when you accepted the job or when you started the job?

23. In your own words, how is a career path job defined?

24. Do you consider your current job a career path job as it is defined? Why or why not?

25. Since you received your degree, have you ever held a job you would not consider a career path job? If so, what was it and why would you say it is not a career path job?

Objective of 21 - 25: Follow-up probes to test for consistent interpretation of key terms.

26. On a scale of 1 to 5 where 1 = not at all difficult and 5=extremely difficult, how difficult was it to remember the number of months between the time you completed your [doctorate/doctorate equivalent] and the time you accepted your first career path job?

Objective of 26: determine the extent to which this item might suffer from recall issues.

ITEM 8

Thinking about the field in which you received your [doctorate/doctorate-equivalent] degree, is it expected that one obtains a postdoctoral appointment to advance?

О Yes

О No

(SOURCE: Early Career Doctorates Study Academic Questionnaire methodological study version, Question A5)

27. In your own words, what is this question asking? What does “advance” mean to you in this question? What timeframe were you thinking about?

28. What employment sector or sectors were you thinking about when you answered this question? Does your answer depend on the employment sector?

Objective of 27 and 28: 1) to determine if the question is consistently interpreted, and 2) to determine if the question needs to be more specific about the sector.

ITEM 9

How does your institution classify your job?

О Employee

О Student

О Other

О Don't know

(SOURCE: Early Career Doctorates Study Academic Questionnaire methodological study version, Question B12)

29. Why did you select the answer you did?

30. What type(s) of position do you think would belong in the employee category? The student category? The other category? Where would you place a visiting scholar? A fellow? Why?

Objectives of 29 and 30: 1) to determine if the response options are interpreted as intended and consistently and 2) if ECDs can answer the question and whether this varies by type of ECD

ITEM 10

Now, please read these questions and then think aloud as you answer it. That is, please tell me what you are thinking about as you interpret and answer the question.

Which of the following sources primarily funds your current job?

О Federal (U.S.) research grant or contract

О Federal fellowship or training grant

О Private foundation funds

О State or university funds

О Non-profit organization (other than a private foundation) funds

О For-profit business funds

О Foreign (non-U.S.) government funds

О Personal or other individually held funds

О Other sources:______________________

О Don't know

(SOURCE: Early Career Doctorates Study Academic Questionnaire methodological study version, Question F1 based on question from Sigma Xi National Postdoctoral Survey)

31. {If don’t know} You answered “don’t know.” Can you tell me more about that?

32. {If other} What are the other sources?

Objectives of 31 and 32: The methodological study suggests that a good number of ECDs do not know how to answer this question. The objective here is to try to identify the source of confusion so the question may be revised.

How many grants or contracts support this job? _____

(SOURCE: Early Career Doctorates Study Academic Questionnaire methodological study version, Question F2 based on question from Sigma Xi National Postdoctoral Survey)

33. {If provided an answer} On a scale of 1 to 5 where 1 is Very sure and 5 is Very Unsure, how sure are you of your answer to the second question? What were you including? What were you excluding?

Objective of 33: To evaluate certainty of responses among those who responded to evaluate whether ranges may be a better level of precision for this question.

34. {If said they did not know} Would you or would you not be able to answer if the survey provided the following ranges: “1 or 2,” “3 or 4,” or “5 or more”? {If so} What would your answer be?

Objective of 34: One third of the Academic Questionnaire respondents indicated that they did not know. This is to evaluate whether they have no idea at all or if they would be able to answer in terms of a range.

ITEM 11

Now, please read complete the question and think aloud as you answer it. That is, please tell me what you are thinking about as you interpret and answer the questions.

Listed below are common expectations and requirements of early career doctorate jobs. For each characteristic, please tell us if it is an expectation or requirement of your job.
	Yes	No
Requires a PhD or doctorate-equivalent degree (such as a ScD, MD, DO, DVM, or DDS)	О	О
Requires that the doctorate was awarded within the past 10 years	О	О
Is a temporary position	О	О
Is intended to provide training in research	О	О
Is intended to advance professional skills	О	О
Requires a full-time commitment to research	О	О
Requires that you work under the direction of a senior scholar	О	О
Is for a defined period of time (i.e., has a term limit.)	О	О
Requires publication of research in scholarly journals	О	О
Requires mentorship for professional development	О	О
Is intended to prepare you for an independent career	О	О
Requires service to the department or professional society (for those in faculty, non-faculty or administrator positions)	О	О

(SOURCE: Early Career Doctorates Study Academic Questionnaire methodological study version, Question B38 from the Postdoc Data Project)

35. In your own words, what is the question at the top of the table asking? In the context of this question, what does “expectation” mean to you in this question? What does “requirement” mean to you in this question?

Objective of 35: Evaluate question stem’s clarity.

36. Based on the question stem alone, did you expect “yes”/”no” response options or something else? {If something else} What were you expecting and why?

Objective of 36: To evaluate whether response task is clear from the question stem. (This will be especially important in CATI.)

37. What does “requires mentorship for professional development” mean to you?

Objective of 37: To evaluate whether it is clear that the ECD is the mentee in this item.

38. {For non-postdoc respondents} Are there common characteristics of early career [Fill position: e.g., faculty, non-faculty researchers] positions that are not listed above? If so, what are they?

Objective of 38: To identify new items to help differentiate non-postdoc ECDs.

ITEM 12

Now, please read these questions and then think aloud as you answer them. That is, please tell me what you are thinking about as you interpret and answer the questions.

What was the MINIMUM education level required for your current job? _____________

(SOURCE: OECD Careers of Doctorate Holders questionnaire 3^rd Edition, Question EMP.4)

What was the DESIRABLE education level required for this job?_________________

(SOURCE: OECD Careers of Doctorate Holders questionnaire 3^rd Edition, Question EMP.5)

39. In your own words, what are these questions asking? What does “required” mean to you in these questions?

40. Is a postdoc a minimum requirement for your current job?

41. {If not} If a postdoc was required for your position, would it have changed your answer? Why or why not?

Objectives of 39 - 41: 1) to determine if the question is interpreted as intended and consistently, 2) to generate appropriate response options for the U.S. context (whether to include postdoc as a level of education or rephrase the question to include experience), and 3) to determine if ECDs have the information to answer these questions.

ITEM 13

How many months does your salary cover?

О 12 months

О Less than 12 months (please specify)_____

(SOURCE: Early Career Doctorates Study Academic Questionnaire methodological study version, Question B9 based on question from Survey of Earned Doctorates 08/09)

42. In your own words, what is this question asking? What does “cover” mean to you in this question?

Objective of 42: To test that the question is interpreted as intended and consistently across respondents.

ITEM 14

Now, please read this question and then think aloud as you answer it. That is, please tell me what you are thinking about as you interpret and answer the question.

Since initially completing your [doctorate/doctorate-equivalent] degree in [field of study], have your career interests changed or stayed the same?

О Changed

О Stayed the same

(SOURCE: Early Career Doctorates Study Academic Questionnaire methodological study version, Question M9 based on questions from Sigma Xi National Postdoctoral Survey and National Postdoctoral Association)

43. {If changed} How have your career interests changed?

44. Do you have any current or former colleagues whose career interests have changed as far as you know? If so, how have their career interests changed?

45. Would you consider each of the following to be a change in career interests or not? Why or why not?

Change Not New research interest within the same broad field of research

Change Not New interest in a different field from doctoral degree

Change Not New interest in research versus teaching (or vice versa)

Change Not New desire to work in academia versus another sector (private, government)

Objectives of 43-45: 1) to determine what respondents consider to be a change and 2) to see if respondents interpret change consistently.

ITEM 15 {Note: Administer only if participant answered “changed” in ITEM 14}

Now, please read this question and then think aloud as you answer it. That is, please tell me what you are thinking about as you interpret and answer the question.

What is the primary reason for the change in your career interests since completing your [doctorate/doctorate-equivalent degree]?

О Change in research focus

О Obtain autonomy

О Job market

О Balance professional and personal life

О Pay

О Benefits

О Spouse’s or partner’s career

О Other_____________________________________________________________________________

(SOURCE: Early Career Doctorates Study Academic Questionnaire methodological study version, Question M10 based on questions from Sigma Xi National Postdoctoral Survey and National Postdoctoral Association)

46. Besides the reason you selected, were there any other reasons for your change in career interests? If so, what are they? How did you choose the answer you gave as the primary reason?

Objectives of 46: 1) to determine if there are common reasons not listed and 2) to determine if the question should allow for multiple responses

ITEM 16

Now, please read the question and think aloud as you answer it. That is, please tell me what you are thinking about as you interpret and answer the questions.

Of the following career paths, which are you interested in pursuing in the next 10 years?
	Yes	No
Tenure-track faculty position with an emphasis on teaching	О	О
Tenure-track faculty position with an emphasis on research	О	О
Clinical-track faculty position with an emphasis on teaching	О	О
Clinical-track faculty position with an emphasis on research	О	О
Non-tenure track teaching position	О	О
Research position	О	О
Administrative position	О	О
Clinical position	О	О
Independent consultant or business owner	О	О
Other	О	О

(SOURCE: Early Career Doctorates Study Academic Questionnaire methodological study version, Question M6a based on questions from Sigma Xi National Postdoctoral Survey and National Postdoctoral Association)

47. Are any of the response choices unclear or ambiguous? If so, which ones? Why?

48. What does “research position” mean to you in this question? “Administrative position”? “Clinical position”? “Tenure-track”? “Clinical-track”?

49. {If other} I noticed that you answered “other.” Can you tell me more about that?

Objective of 47 -49: To evaluate suggested revisions made to the question based on methodological study, to evaluate within question context effects.

ITEM 17

For the next two questions, think about the job you held before the one you have now.

What was the primary reason you left this job?

О Term-limited position ended

О Job terminated

О Funding not renewed

О Promotion

О Took a better job

О Family-related reasons (e.g., children, spouse's or partner's job location)

О Location of position (unrelated to family reasons)

О Visa issues

О Dissatisfaction with job

О Other, specify: ______________________________________________

(SOURCE: Early Career Doctorates Study Academic Questionnaire methodological study version, revised Question Q7 which has some response options from the Survey of Doctorate Recipients)

If you selected “dissatisfied with job”…

What is the primary reason you were dissatisfied with this job?

О Change in professional interests

О Not related to your doctorate or doctorate-equivalent degree

О Lack of job security

О Working conditions (e.g., hours, equipment, and working environment)

О Pay

О Lack of employment benefits (e.g., health insurance, vacation days)

О Not enough opportunity to conduct your own research

О Not enough opportunity to publish

О Not enough opportunity to teach

О Other, specify: ______________________________________________

(SOURCE: Early Career Doctorates Study Academic Questionnaire methodological study version, revised Question Q7 which has some response options from the Survey of Doctorate Recipients)

50. On a scale of 1 to 5 where 1 is very easy and 5 is very difficult, how easy or difficult was it for you to answer the first question? Why?

51. (if applicable) On a scale of 1 to 5 where 1 is very easy and 5 is very difficult, how easy or difficult was it for you to answer the second question? Why?

Objective of 50 and 51: To evaluate revisions to Academic Questionnaire question Q 7.

34