Cognitive interviews under the generic clearance for survey improvement projects (OMB number 3145-0174) for the Survey of Earned Doctorates (SED)

Attachment

Interview Protocol: Cognitive Interviews on Interdisciplinary Research

Protocol for Cognitive Interviews on Interdisciplinary Research

Introductory Material:

When participants arrive at the cognitive interview laboratory, they will be thanked for agreeing to help with the research. The purpose of the interview, which is to evaluate questionnaire items used in surveys conducted by the National Science Foundation, will be explained. At least one note-taker will be present for each interview in order to record the events and responses of the participant. It will be explained to participants why the note-takers are present, and then participants will be asked to read and sign a confidentiality statement, which is also signed by the interviewer. Then the process of the interviews will be explained: respondents are asked to complete a questionnaire and speak aloud as they do so. They will be asked to speak aloud everything they read, and to tell their thoughts and reactions as they respond to the questions. In order to verify that individuals understand these procedures, they will be given a one or two item practice questionnaire to illustrate the process (see example in Appendix 1).

STEP 1: Interview Questionnaire:

Respondents will be asked to complete the actual questionnaire of interest. The interview questionnaire (see Appendix 2) consists of several items from the Survey of Earned Doctorates questionnaire. It includes questions about dissertation titles, fields of dissertation research (the question of interest, numbered A2 on the questionnaire), department, institution, field(s) of degrees, and some demographic questions. The list of fields that is provided with the Survey of Earned Doctorates will be provided with the questionnaire (see Appendix 3).

Two versions of the questionnaire have been constructed. Version A contains the dissertation research fields question as it was worded in the 2001-2003 SED. Version B uses the current version of the question, adopted in 2004. Half of the respondents will be given Version A, and the remaining half will be given Version B (the distribution of these versions will be determined based on the expected likelihood that individuals will report multiple fields. Both types of respondents will be able to see both question versions).

Respondents will be told to complete the questionnaire as they would do if they were at home alone, except they will be asked to read them aloud and tell what they are thinking. As respondents proceed through the questions of interest, the interviewer will probe with additional questions as necessary to ensure full understanding of how the participant interprets and responds to the items. The interviewer will use follow-up questions to determine if and how the list of fields and the wording of the questions affect how people respond to these items. For example, as respondents answer Question A2 (the question about dissertation research fields), they might be asked:

Did you have any trouble finding your field in the list of fields and codes provided? If so, how did you reconcile this?

Would you have answered this question differently if you were not asked to rely on this list of fields?

Would you have answered differently if we asked about the department or program of your research, instead of field?

What if the question asked about your primary field of study rather than dissertation research? What if it asked about your research in general?

After respondents answer question A4 (about the field of their degree), they may be asked questions such as:

Did you have any trouble finding the field of your degree in this list? If yes, how did you deal with that?

(If the individual puts a different field than one that was listed for the dissertation question, will ask) I noticed this response is different than the field(s) you listed before. Can you tell about this?

What if this asked about the primary field of your department or program, rather than field of study? Would you have answered differently?

STEP 2: Alternative Version of Dissertation Research Question:

After respondents complete the primary questionnaire, they will be told that NSF could also ask some of these questions in different ways, and it would be useful to have them answer some different versions of a question and talk about their answers. First they will be shown the alternate SED question (either A or B, whichever they have not already seen) (see Appendix 4 for both versions). Once they have answered it they will be probed about whether they see this question as different or calling for a different answer than the first. Any differences in the way they answered the two questions will be noted.

STEP 3: Multidisciplinary Version:

Then a third version of the question that uses the term “multidisciplinary” instead of “interdisciplinary” (see Appendix 5) will be shown to them. Once they have answered it they will be probed again about whether they see this question as different or calling for a different answer than the others. Any differences in the way they answered the three questions will be noted. Additional probing will be used to understand if individuals rely on different information to answer the three versions of the question. For example, they may be asked:

Are all of these questions the same to you? Do you see any difference between asking these different versions?

(If yes), what about them is different?

STEP 4: Additional Questionnaire about Interdisciplinarity:

Next they will proceed to an additional questionnaire designed to better understand how respondents think about interdisciplinarity and multidisciplinarity. They will be told that one of the concerns prompting the previous questions is that some of the terms used in these questions may be interpreted in different ways by different people. So, it would be helpful if they answer an additional set of questions to help understand how doctoral students think about some of these terms and the terms’ relation to their work. They will be provided with this questionnaire (see Appendix 6) and probed as needed to elicit further explanation and detail about participants’ responses.

Appendix 1

Practice Questionnaire

Note: This questionnaire will be used to train participants to speak aloud as they read the questionnaire and formulate their responses.

Practice questionnaire

Please read this question aloud and tell me what you are thinking as you come up with your answer.

Q1. How many windows are in the apartment, mobile home, or house where you live?

________ Number of windows

Appendix 2

Interview Questionnaire

Note: There are two versions of this questionnaire, Version A and Version B. Half of the respondents will be given Version A, and half will be given Version B.

Version A

Select Items from the Survey of Earned Doctorates (SED)

INSTRUCTIONS

Thank you for taking the time to complete this questionnaire. Directions are provided for each question. Please print all responses; you may use either a pen or a pencil.

PART A – EDUCATION

A1. What is the title of your dissertation?

■ Please mark (X) this box if the title below refers to a performance, project report, or musical or

literary composition required instead of a dissertation.

Title

A2. Using the Specialties List (page 5), please write the name and number of the primary field of your dissertation research.

Name of Field

Number of Field

If you had a secondary field for your dissertation research, list the name and number.

Name of Field

Number of Field

A3. Please name the department (or interdisciplinary committee, center, institute, etc.) of the university that supervised your doctoral studies.

Department/Committee/Center/Institute/Program

A 4. The next few questions ask about the degrees you have received (or are about to receive). Please provide the following information for this doctoral degree, your most recent master’s degree, and your first bachelor’s degree in the appropriate columns below.

	This research doctoral degree	Most recent master’s degree (e.g. MS, MA, MBA) or equivalent	First bachelor’s degree (e.g. BA, BS, AB) or equivalent
a. Have you received a degree of this type?......................	Yes No	Yes No	Yes No
b. Month/year degree granted……………	Month Year	Month Year	Month Year
c. Month/year that you started your degree	Month Year	Month Year	Month Year
d. Primary field of study…………….....
e. Field number from list on p. 5...............
f. Institution name……
g. Branch or city……..
h. State or province…
i. Country……………..

PART B – BACKGROUND INFORMATION

B1. Are you –

■ Male ■ Female

B2. What is your date of birth?

Month Day Year

B3. What is your marital status?

Mark (X) one

■ Married

■ Living in a marriage-like relationship

■ Widowed

■ Separated

■ Divorced

■ Never married

B4. Not including yourself or your spouse/partner, how many dependents (children or adults) do you have – that is, how many others receive at least one half of their financial support from you?

Write in number of dependents

5 years of age or younger ……….

6 to 18 years ………………………

19 years or older ………………….

Mark (X) if none

B5. Are you Hispanic or Latino?

■ Yes  Go to B6

■ No  Go to B7

B6. Which of the following best describes your Hispanic origin or descent?

■ Mexican or Chicano

■ Puerto Rican

■ Cuban

■ Other Hispanic – Specify

B7. What is your racial background?

Mark (X) one or more

■ American Indian or Alaska Native

Specify tribal affiliation(s)

■ Native Hawaiian or other Pacific Islander

■ Asian

■ Black or African-American

■ White

Version B

S elect Items from the Survey of Earned Doctorates (SED)

INSTRUCTIONS

Thank you for taking the time to complete this questionnaire. Directions are provided for each question. Please print all responses; you may use either a pen or a pencil.

PART A – EDUCATION

A1. What is the title of your dissertation?

■ Please mark (X) this box if the title below refers to a performance, project report, or musical or

literary composition required instead of a dissertation.

Title

A2. Please write the name of the primary field of your dissertation research.

Name of Field

Using the list on page 5, choose the code that best describes the primary field of your dissertation research.

Number of Field

If your dissertation research was interdisciplinary, list the name and number of your secondary field.

Name of Field

Number of Field

A3. Please name the department (or interdisciplinary committee, center, institute, etc.) of the university that supervised your doctoral studies.

Department/Committee/Center/Institute/Program

A 4. The next few questions ask about the degrees you have received (or are about to receive). Please provide the following information for this doctoral degree, your most recent master’s degree, and your first bachelor’s degree in the appropriate columns below.

	This research doctoral degree	Most recent master’s degree (e.g. MS, MA, MBA) or equivalent	First bachelor’s degree (e.g. BA, BS, AB) or equivalent
a. Have you received a degree of this type?......................	Yes No	Yes No	Yes No
b. Month/year degree granted……………	Month Year	Month Year	Month Year
c. Month/year that you started your degree	Month Year	Month Year	Month Year
d. Primary field of study…………….....
e. Field number from list on p. 5...............
f. Institution name……
g. Branch or city……..
h. State or province…
i. Country……………..

PART B – BACKGROUND INFORMATION

B 1. Are you –

■ Male ■ Female

B2. What is your date of birth?

Month Day Year

B3. What is your marital status?

Mark (X) one

■ Married

■ Living in a marriage-like relationship

■ Widowed

■ Separated

■ Divorced

■ Never married

B4. Not including yourself or your spouse/partner, how many dependents (children or adults) do you have – that is, how many others receive at least one half of their financial support from you?

Write in number of dependents

5 years of age or younger ……….

6 to 18 years ………………………

19 years or older ………………….

Mark (X) if none

B5. Are you Hispanic or Latino?

■ Yes  Go to B6

■ No  Go to B7

B6. Which of the following best describes your Hispanic origin or descent?

■ Mexican or Chicano

■ Puerto Rican

■ Cuban

■ Other Hispanic – Specify

B7. What is your racial background?

Mark (X) one or more

■ American Indian or Alaska Native

Specify tribal affiliation(s)

■ Native Hawaiian or other Pacific Islander

■ Asian

■ Black or African-American

■ White

Appendix 3

List of Fields

Note: This is the list of fields and codes that is used in the SED. It will be given to respondents with the questionnaire used in STEP 1.

Appendix 4

Alternative Versions of Question A2

Note: There are 2 versions, Version A and Version B. Version A will be given to respondents who received Version A of the initial questionnaire in Step 1, and Version B will be given to those who received Version B of the initial questionnaire in Step 1.

Alternative Question Version A

A2b. Please write the name of the primary field of your dissertation research.

Name of Field

Using the list on page 5, choose the code that best describes the primary field of your dissertation research.

Number of Field

If your dissertation research was interdisciplinary, list the name and number of your secondary field.

Name of Field

Number of Field

Alternative Question Version B

A2b. Using the Specialties List (page 5), please write the name and number of the primary field of your dissertation research.

Name of Field

Number of Field

If you had a secondary field for your dissertation research, list the name and number.

Name of Field

Number of Field

Appendix 5

Multidisciplinary Question

Note: This question will be given to all respondents in Step 3.

A2c. Please write the name of the primary field of your dissertation research.

Name of Field

Using the list on page 5, choose the code that best describes the primary field of your dissertation research.

Number of Field

If your dissertation research was multidisciplinary, list the name and number of your secondary field.

Name of Field

Number of Field

Appendix 6

Additional Questionnaire about Interdisciplinary/Multidisciplinary

Note: This will be given to all respondents in Step 4, after they answer all versions of the dissertation research question.

Additional Questions

Next we would like to ask you a series of questions about the concepts of interdisciplinarity and multidisciplinary. Please provide as much information as you can; your responses are important for helping us understand how to improve the questions used in the NSF questionnaires.

Q1. Do you think the meanings of the terms “interdisciplinary” and “multidisciplinary” are:

■ Exactly the same

■ Very similar

■ Somewhat similar

■ Somewhat different

■ Very different

■ Completely different

■ Not sure

Q2. What exactly does “interdisciplinary” mean to you?

Q3. How do you think the meaning of “multidisciplinary” differs from the meaning of “interdisciplinary” (if at all)?

Q 4. Based on your definitions, how would you describe your doctoral dissertation research?

■ Neither interdisciplinary nor multidisciplinary  SKIP to Q10 on pg. 3

■ Interdisciplinary

■ Multidisciplinary

■ Both interdisciplinary or multidisciplinary

Q5. (If interdisciplinary and/or multidisciplinary) Do you consider your research to be interdisciplinary or multidisciplinary for any of the following reasons? Mark Yes or No for each item.

Yes No Does not apply

You cite research from multiple fields/disciplines ■ ■ ■

You use research from multiple fields/disciplines to formulate your ■ ■ ■

research questions

You use theories or methods from multiple fields/disciplines to conduct ■ ■ ■

your research

Your research’s implications are relevant or applicable to multiple ■ ■ ■

fields/disciplines

You have presented or plan to present your research at a ■ ■ ■

multidisciplinary or interdisciplinary conference

Your research may be publishable in a journal that speaks to multiple ■ ■ ■

disciplines or in an interdisciplinary journal

Your committee consists of faculty from multiple fields/disciplines ■ ■ ■

Q6. If there are additional reasons (not listed above) why you consider your research to be interdisciplinary or multidisciplinary, please describe them here:

Q7. Did you begin your graduate studies with the intention of conducting interdisciplinary or multidisciplinary research?

■ Yes

■ No

Q8. What factors influenced your decision to do an interdisciplinary or multidisciplinary dissertation?

Q9. How did you become involved in interdisciplinary or multidisciplinary research?

Q10. Based on your definitions of interdisciplinary and multidisciplinary, would you say that the doctoral degree you will earn (or have just earned) is:

■ Neither interdisciplinary nor multidisciplinary  SKIP to Question Q13 on pg. 4

■ Interdisciplinary

■ Multidisciplinary

■ Both interdisciplinary or multidisciplinary

Q11. (If interdisciplinary and/or multidisciplinary) Do you consider your degree to be interdisciplinary or multidisciplinary for any of the following reasons? Mark Yes or No for each item.

Yes No Does not apply

Your field of study is considered to be an interdisciplinary field ■ ■ ■

You are earning a minor in an additional field/discipline ■ ■ ■

You took courses from multiple academic departments ■ ■ ■

You took courses that were cross-listed to multiple departments ■ ■ ■

Your committee consists of faculty from multiple fields/disciplines ■ ■ ■

Your program/department has faculty from multiple fields/disciplines ■ ■ ■

You have another degree in a different field/discipline ■ ■ ■

Your research uses theories or methods from multiple fields/disciplines ■ ■ ■

Q12. If there are additional reasons (not listed above) why you consider your doctoral degree to be interdisciplinary or multidisciplinary, please describe them here:

Q13. In your opinion, is there a difference between doing interdisciplinary/ multidisciplinary research for a dissertation and getting an interdisciplinary/ multidisciplinary doctoral degree?

■ Yes

■ No

■ Not sure

Q14. In general, do you think that interdisciplinary or multidisciplinary research is typically:

■ Of higher quality than research produced within a single discipline

■ Of the same quality than research produced within a single discipline

■ Of lower quality than research produced within a single discipline

■ Not sure

Q15. Please explain why you answered Q14 the way you did.

Q 16. How supportive or unsupportive do you think WSU is of graduate students who conduct interdisciplinary or multidisciplinary research?

■ Very supportive

■ Somewhat supportive

■ Neither supportive nor unsupportive

■ Somewhat unsupportive

■ Very unsupportive

■ Not sure

Q17. To what extent do you think that doctoral students who conduct interdisciplinary or ultidisciplinary research are advantaged or disadvantaged when looking for a job, compared to students whose research is based only in one discipline?

■ Very advantaged

■ Somewhat advantaged

■ Neither advantaged nor disadvantaged

■ Somewhat disadvantaged

■ Very disadvantaged

■ Not sure

Q18. Thinking about careers in general, do you think conducting interdisciplinary or multidisciplinary research is advantageous or disadvantageous for researchers?

■ Very advantageous

■ Somewhat advantageous

■ Neither advantageous nor disadvantageous

■ Somewhat disadvantageous

■ Very disadvantageous

■ Not sure

Q 19. Some scholars define multidisciplinary research as research that uses perspectives from multiple disciplines, but does not necessarily integrate the concepts from the various disciplines.

This is distinguished from interdisciplinary research, which these scholars view as research that integrates perspectives from multiple disciplines to solve problems that are beyond the scope of a single discipline.

Based on these definitions, would you say that your dissertation research is:

■ Neither interdisciplinary nor multidisciplinary

■ Interdisciplinary

■ Multidisciplinary

■ Either interdisciplinary or multidisciplinary

Q20. Thinking of other graduate students in your program, about how many would you say conduct dissertation research that is interdisciplinary or multidisciplinary?

■ None

■ Few

■ Some

■ Most

■ All

29 | Page