Recent Methodological Research in the Survey of Earned Doctorates

ATTACHMENT 9
Recent Methodological Research in the Survey of Earned Doctorates
Graduate School Experiences (GSE) Research: To examine the potential for expanding the
content of the SED to capture information on the GSE of doctorate recipients, a three-phased
research plan was pursued. First, focus groups were conducted with the following SED
stakeholders: graduate school deans, heads of graduate departments, and recent SED data users.
Focus group participants were asked about their institution’s current efforts to collect data on
GSE, topics that would be useful to include in a GSE section of the SED, issues/concerns related
to collecting GSE data, and how useful GSE data would be to their institution (aggregated and at
the student level). The focus group results indicated that participants from each stakeholder
group were generally interested in a GSE section for the SED. The second phase of the research
included one-on-one interviews with the same three stakeholder groups to explore the same
issues covered in the focus groups but in greater depth. The results of the interviews echoed
those of the focus groups, with participants expressing an interest in the inclusion of a GSE
section on the SED. Based on input gathered from the focus groups and interviews, a web
survey was constructed for the final phase of the research. This survey asked current graduate
school deans at SED institutions to rate the usefulness of each of eight potential broad topics
(with selected detailed subtopics) for a GSE section of the SED and to provide feedback on other
GSE-related issues. Survey results revealed that deans at SED institutions are interested in data
on GSE, would find it useful to have GSE data, particularly on the graduate’s scholarly products
and presentations, and their experience in the areas of professional development and career
opportunities. (Lee, L., Neishi, K., Zimowski, M., Gleicher, D., Hoffer, T., 2018)
A Redesign of the Instrument for the Survey of Earned Doctorates: As part of the ongoing
effort to reduce respondent burden, improve data quality, and increase data processing
efficiency, the SED instruments (Web survey and paper questionnaire) were reviewed and
revised. The revised instruments were then tested in a series of cognitive interviews. After each
wave of interviews, the instruments were further revised and cognitively tested again. This
process culminated in a redesigned Web instrument and a paper complement of that instrument.
Whereas the original instruments had been optimized for the paper mode, the redesigned
instruments were optimized for the Web. (Lee, L., Neishi, K., Gleicher, D., Groenhout, B., &
Zimowski, M., 2016)
Survey of Earned Doctorates Prompt Mode Experiment: SED nonrespondent follow-up
protocol had been to send all nonrespondents with a mailing address up to five mail prompts
(four letters and one postcard) before any other treatment, regardless of the presence of an email
address for the nonrespondent. The SED would then send up to two email prompts if an email
address was available. Best practices for nonrespondent prompt design and delivery sequence are
to tailor them to minimize (1) the total number of prompts that must be delivered to elicit a
survey response, (2) respondent burden, and (3) survey costs. The resulting design depends on
several factors, including how prompt modes differ in terms of cost, and quality of contact
information. This experiment did not measure differences in respondent burden, but rather tested
the effect of sending nonrespondents email prompts first versus sending them mail prompts in
order to compare the efficacy and efficiency of an email-first to mail-first prompt strategy.

Respondents eligible for the experiment were those for whom both email and mailing addresses
were available. The control group received the standard protocol of mail prompts first, followed
by email prompts, if needed. The treatment group received the prompts in the opposite order:
that is, email prompts followed by mail prompts, if necessary. The results demonstrated that
sending enough of a combination of mail and email prompts will eventually procure the same
response rate. However, an email-first strategy was capable of eliciting quicker survey responses
and saving costs. As a result, the email-first strategy has become standard protocol for the SED.
(Groenhout, B., Bilgen, I., Schacht, S., & Zimowski, M., 2016)
Survey of Earned Doctorates (SED) Progress Indicator Experiment: In AY 2015, an
experiment was implemented to examine if the presence of a progress indicator in the web
instrument would have an impact on overall survey completion rates, as well as survey
completion rates during a single session. The key aim of using a progress indicator is to increase
respondent motivation to proceed with the next survey question and, ultimately, complete the
survey. Upon entering the survey, nonrespondent cases were randomly assigned to either the
experiment (progress bar indicator) or control (no progress bar) group. The progress bar would
advance based on the percent of the survey completed at each screen, with each screen counting
as one in the numerator, taking into account skip logic in the survey. The denominator for the
calculation included the total number of screens in the survey, excluding the confirmation and
data check notification screens. The results showed small, positive, but statistically
nonsignificant effects of the progress indicator on the overall completion rates as well as the
single session completion rates for respondents who received the progress bar compared to
respondents who did not receive it. Consequently, a progress bar was not included in the 2016
SED web survey. However, the web survey was redesigned for the 2017 SED and a progress bar
was incorporated. Results will be monitored. (Bilgen, I., Groenhout, B., Zimowski, M., &
Schacht, S., 2016)
Defining and Measuring Interdisciplinary Research and Degree Programs: Implications
for the Survey of Earned Doctorates: This report summarizes work on a project that examined
the value and feasibility of collecting information about interdisciplinary doctorate degree (IDD)
programs and interdisciplinary dissertation research (IDR). The research team reviewed relevant
literature to gain a basic understanding of how IDD and IDR are defined and interrelated,
examined published sources on IDD currently supported by U.S. universities, synthesized the
published typologies of IDDs, and analyzed Doctorate Records File data on dissertation fields of
study to identify institution types, particular institutions, and fields of study that might be
supporting IDR. NSF is considering whether to pursue follow-up research that would examine
alternative ways to identify IDD programs at particular institutions, and would analyze the
validity and scalability of these alternatives. (Selfa, L., Neishi, K., & Hoffer, T., 2016)
An Examination of the Institution Eligibility Criteria for the Survey of Earned Doctorates:
This report was the culmination of work conducted during 2013 at the request of NSF that
examined the eligibility criteria for institution inclusion in the SED against a broader national
and international context. The report also made recommendations for an adjudication process for
determining the eligibility of institutions and programs not currently in the SED but appearing to
meet the criteria for inclusion. The report is currently under consideration by NSF. No changes
have yet been implemented to the SED eligibility review process. (Kirby et al., 2016)

Survey of Earned Doctorates Confidentiality Report: This report presents the findings from
the cognitive interviews and focus groups conducted in 2013 and 2014 with doctorate recipients,
graduate deans, institution contacts and institution researchers concerning the confidentiality
procedures employed by the SED. Additional topics covered in the interviews and focus groups
included a revised consent statement and a proposed data linking and data sharing project among
select institutions. The report included findings and recommendations concerning the data
linking and data sharing project under consideration at NSF. No changes have yet been
implemented based on the findings. (Welch et al., 2016)
Timeline Data Quality Improvements for the Survey of Earned Doctorates: This report
presents findings from a study conducted in 2014 that analyzed the current approach the SED
employs to collect, edit and report timeline data. The report provided recommendations for the
increased utility and improved quality of the data through potential questionnaire changes, edit
changes and changes to how the data are contained in the DRF and presented in reports. Findings
were used to inform a number of process revisions, including: expansion of the auto-coding
process for timeline variables; modification of rules used to flag nontraditional timeline
sequences; addition of timeline variables to the Doctorate Records File (DRF) for use in further
research; and revision of select imputation rules. In addition, revisions to timeline questions in
the instrument are included in the planned future cognitive interview activities. (Bautista et al.,
2016)