Request for Approval of Non-substantive Changes to the Survey of Earned Dcotorates

MEMORANDUM

Date: March 30, 2016

To: Shelly Wilkie Martinez, Desk Officer

Office of Management and Budget

From: John R. Gawalt, Director

National Center for Science and Engineering Statistics

National Science Foundation

Via: Suzanne Plimpton, Clearance Officer

National Science Foundation

Subject: Request for Approval of Non-substantive Changes to the Survey of Earned Doctorates (SED) questionnaire

The National Science Foundation requests approval of non-substantive changes to the SED questionnaire. For the 2017 cycle, NSF proposes modifying wording and reordering items based on the results of the cognitive testing that was carried out under the generic clearance of survey improvement projects (OMB #3145-0174) and approved by OMB on 17 March 2015. The testing results identified improvements to increase consistency of item interpretation among respondents and reduce discrepancies in educational history timelines, both of which are expected to improve data quality.

The cognitive testing also identified new items that would be beneficial to add to the SED instrument to provide a more detailed picture of doctoral graduates. However, items that request “new” information are not addressed in this memo but will be presented to OMB as potential changes for the 2018 SED when the full clearance package is submitted at the beginning of 2017.

A brief description of the cognitive testing follows, along with a description of the proposed modifications to the SED instrument. Appendix A displays images of the current questionnaire items and their proposed modifications. Appendix B contains the current 2016 SED paper questionnaire as a reference.

SED Cognitive Testing

Purpose. The purpose of the cognitive testing was to identify respondent difficulty with question wording and to simplify the data editing process by clarifying questions. Part A of the SED questionnaire is intended to collect information about respondents’ educational history. The current organization of the questionnaire has evolved, in part, due to the space limitations of the paper form. Now that 90% of SED respondents complete the survey online, the instrument was reorganized to optimize the Web mode. As such, the questions comprising Part A were reordered to lead respondents sequentially through their educational history. In addition, question wording was elaborated to clarify the information being sought.

Methodology. The survey items were reordered and tested using a Web survey and think-aloud interviews with participants. The interview results were then used to revise the instrument and re-test it, following the same procedure with new participants. Once it was clear that the reordering of the items was effective in the Web instrument, an equivalent paper version was developed and similarly tested (because a few educational institutions still use paper SED questionnaires). The whole process involved four rounds of testing the Web instrument (with 65 participants) and three rounds of testing the paper instrument (with 29 participants).

Results. The cognitive testing results revealed that several minor modifications to question wording, terminology, and response options could be made to improve data quality and respondent experience. These types of modifications were not confined to Part A of the questionnaire, but pertained to a few items in Part B, as well. Moreover, the reordering and elaboration of items in Part A provided a more complete picture of respondent educational history.

One of the measures that is calculated based on information gathered from Part A of the SED questionnaire is time-to-degree. By requesting a more thorough accounting of respondents’ educational history, less data editing is expected to be required to fill in gaps and/or resolve discrepancies.

Proposed Modifications to the 2017 SED Instrument

All of the proposed changes are listed here, and images of the current and proposed forms are provided in Appendix A. Timed testing of the revised paper questionnaire demonstrated no change in the estimated respondent burden compared with the current form.

Part A of the Questionnaire.

1. In the 2016 SED, respondents are specifically asked questions about their research doctoral degree, their most recent master’s degree, and their first bachelor’s degree. Then they are to provide similar information for each “additional postsecondary degree” that they received. Because it may be unclear to respondents that they should report their associate’s degrees, in the 2017 SED we propose asking specifically about such degrees. We ask specifically for additional bachelor’s and master’s degrees, as well. Moreover, for the additional postsecondary degrees that respondents report in the 2016 SED, they are asked for the month and year that the degree was granted. In the 2017 SED, we propose asking for the month and year that the degree was started, as well. This will provide a clearer picture of a respondent’s educational timeline. (See figures A.1.Old and A.1.New in Appendix A.)

2. In the 2016 SED, respondents are asked for the primary field of study of their bachelor’s degree. For the 2017 SED, we propose allowing respondents to provide two fields of study if they had a double major for their bachelor’s degree. (See figures A.2.Old and A.2.New in Appendix A.)

3. In the 2016 SED, respondents are asked, “Was a master’s degree a prerequisite for admission to your doctoral program?” Because of the ambiguity of this question in relation to the variety of doctoral program structures that exist, we propose clarifying this question by offering additional response options. (See figures A.3.Old and A.3.New in Appendix A.)

4. In the 2016 SED, respondents are asked whether they are earning or have earned an MD or a DDS. In the 2017 SED, we propose asking whether they are earning or have earned any professional doctoral degree, such as an MD, DDS, DVM, JD, PsyD, or DDiv. (See figures A.4.Old and A.4.New in Appendix A.)

5. In asking about sources of financial support, we propose changing response option b from “Grant” to “Dissertation grant” to prevent respondents from reporting duplicate funding sources (e.g., “grant” and “research assistantship,” both referring to the same funds). (See figures A.5.Old and A.5.New in Appendix A.)

6. We propose modifying the question about starting graduate school (item A11 in the 2016 SED) from “In what month and year did you first enter any graduate school in any program or capacity?” to “In what month and year did you first enter any graduate program, even if you did not earn a degree?” for greater clarity.

Part B of the Questionnaire.

1. We propose clarifying the wording of the question stem and response options for the postgraduation plans item to help respondents select the best fit for their circumstances. In addition, the old response option pertaining to “predoctoral employment” was determined to be ambiguous, so for clarity we propose separating it into its own question, which would be asked of respondents who indicate they have accepted or are employed in a position other than a postdoc. (See figures B.1.Old, B.1a.New, and B.1b.New in Appendix A.)

2. We propose combining “postdoc fellowship” and “postdoc research associateship” into one response option when asking about the nature of one’s postdoc, because respondents have difficulty distinguishing between these two responses. (See figures B.2.Old and B.2.New in Appendix A.)

3. In the list of principal employers (item B6 in the 2016 SED), we propose changing the response option “Not for profit organization” to “Nonprofit organization (including private foundation)” to better align the terminology with another survey item that includes similar response options (B5 in the 2016 SED). (See figure B.3.Old in Appendix A; an image of the revised question is not available.)

4. We propose adding the response option of “No financial support (unpaid position)” to the question asking about the main source of financial support for a postdoc or other training. (See figures B.4.Old and B.4.New in Appendix A.)

5. We propose clarifying the response option of “professional services to individuals” in the question asking about work activities by adding the following examples “(such as healthcare, counseling, financial services, legal services, consulting).” (See figures B.5.Old and B.5.New in Appendix A.)

Appendix A

The first image (“Old”) in each pair is from the current 2016 SED paper questionnaire and the second image (“New”) is the proposed change. The format differs slightly in the Web version of the questionnaire.

Part A.

A.1.Old.

A.1.New.

A.2.Old.

A.2.New.

A.3.Old.

A.3.New.

A.4.Old.

A.4.New.

A.5.Old.

A.5.New.

Part B.

B.1.Old.

B.1a.New.

B.1b.New.

B.2.Old.

B.2.New.

B.3.Old.

B.4.Old.

B.4.New.

B.5.Old.

B.5.New.

Appendix B

Current 2016 SED Paper Questionnaire

See separate PDF file.

14