Recent Methodological Research Concerning the SED

Attachment 10

Recent Methodological Research Concerning the Survey of Earned Doctorates

Note: Reports available upon request from Lynn Milan (lmilan@nsf.gov).

A Redesign of the Instrument for the Survey of Earned Doctorates: As part of the ongoing effort to reduce respondent burden, improve data quality, and increase data processing efficiency, the SED instruments (Web survey and paper questionnaire) were reviewed and revised. The revised instruments were then tested in a series of cognitive interviews. After each wave of interviews, the instruments were further revised and cognitively tested again. This process culminated in a redesigned Web instrument and a paper complement of that instrument. Whereas the original instruments had been optimized for the paper mode, the redesigned instruments were optimized for the Web. (Lee, L., Neishi, K., Gleicher, D., Groenhout, B., & Zimowski, M., 2016)

2015 Survey of Earned Doctorates Prompt Mode Experiment: SED nonrespondent follow-up protocol had been to send all nonrespondents with a mailing address up to five mail prompts (four letters and one postcard) before any other treatment, regardless of the presence of an email address for the nonrespondent. The SED would then send up to two email prompts if an email address was available. Best practices for nonrespondent prompt design and delivery sequence are to tailor them to minimize (1) the total number of prompts that must be delivered to elicit a survey response, (2) respondent burden, and (3) survey costs. The resulting design depends on several factors, including how prompt modes differ in terms of cost, and quality of contact information. This experiment did not measure differences in respondent burden, but rather tested the effect of sending nonrespondents email prompts first versus sending them mail prompts in order to compare the efficacy and efficiency of an email-first to mail-first prompt strategy. Respondents eligible for the experiment were those for whom both email and mailing addresses were available. The control group received the standard protocol of mail prompts first, followed by email prompts, if needed. The treatment group received the prompts in the opposite order: that is, email prompts followed by mail prompts, if necessary. The results demonstrated that sending enough of a combination of mail and email prompts will eventually procure the same response rate. However, an email-first strategy was capable of eliciting quicker survey responses and saving costs. As a result, the email-first strategy has become standard protocol for the SED. (Groenhout, B., Bilgen, I., Schacht, S., & Zimowski, M., 2016)

Survey of Earned Doctorates (SED) Progress Indicator Experiment: In AY 2015, an experiment was implemented to examine if the presence of a progress indicator in the web instrument would have an impact on overall survey completion rates, as well as survey completion rates during a single session. The key aim of using a progress indicator is to increase respondent motivation to proceed with the next survey question and, ultimately, complete the survey. Upon entering the survey, nonrespondent cases were randomly assigned to either the experiment (progress bar indicator) or control (no progress bar) group. The progress bar would advance based on the percent of the survey completed at each screen, with each screen counting as one in the numerator, taking into account skip logic in the survey. The denominator for the calculation included the total number of screens in the survey, excluding the confirmation and data check notification screens. The results showed small, positive, but statistically nonsignificant effects of the progress indicator on the overall completion rates as well as the single session completion rates for respondents who received the progress bar compared to respondents who did not receive it. Consequently, a progress bar was not included in the 2016 SED web survey. However, the web survey was redesigned for the 2017 SED and a progress bar was incorporated. Results will be monitored. (Bilgen, I., Groenhout, B., Zimowski, M., & Schacht, S., 2016)

Defining and Measuring Interdisciplinary Research and Degree Programs: Implications for the Survey of Earned Doctorates: This report summarizes work on a project that examined the value and feasibility of collecting information about interdisciplinary doctorate degree (IDD) programs and interdisciplinary dissertation research (IDR). The research team reviewed relevant literature to gain a basic understanding of how IDD and IDR are defined and interrelated, examined published sources on IDD currently supported by U.S. universities, synthesized the published typologies of IDDs, and analyzed Doctorate Records File data on dissertation fields of study to identify institution types, particular institutions, and fields of study that might be supporting IDR. NSF is considering whether to pursue follow-up research that would examine alternative ways to identify IDD programs at particular institutions, and would analyze the validity and scalability of these alternatives. (Selfa, L., Neishi, K., & Hoffer, T., 2016)

SED Institution Exit Interview Report: This report represents the first step in a larger (potential) project to explore the value and feasibility of collecting information via the SED about the graduate school experiences of doctorate recipients. The report examined the use of “student exit surveys” by a sample of n=50 SED institutions. The sample was constructed to include institutions from different Carnegie categories, minority-serving status (HBCU, HSI), institutional sector (public vs. private), and size of graduating doctoral student cohort. The findings cataloged the range of content included in institutional exit surveys and provided an initial estimate of the proportion of SED institutions that currently administer exit surveys to their graduating doctoral students. At the request of NSF, the survey contractor is currently developing a research plan for a follow-up study that will collect information about the ways in which SED institutions use (or might use) information from graduate student exit surveys, and the relative utility of selected exit survey items to different types of data users. (Bartolone, J., 2015)

An Examination of the Institution Eligibility Criteria for the Survey of Earned Doctorates: This report was the culmination of work conducted during 2013 at the request of NSF that examined the eligibility criteria for institution inclusion in the SED against a broader national and international context. The report also made recommendations for an adjudication process for determining the eligibility of institutions and programs not currently in the SED but appearing to meet the criteria for inclusion. The report is currently under consideration by NSF. No changes have yet been implemented to the SED eligibility review process. (Kirby et al., 2014)

Survey of Earned Doctorates Confidentiality Report: This report presents the findings from the cognitive interviews and focus groups conducted in 2013 and 2014 with doctorate recipients, graduate deans, institution contacts and institution researchers concerning the confidentiality procedures employed by the SED. Additional topics covered in the interviews and focus groups included a revised consent statement and a proposed data linking and data sharing project among select institutions. The report included findings and recommendations concerning the data linking and data sharing project under consideration at NSF. No changes have yet been implemented based on the findings. (Welch et al., 2014)

Timeline Data Quality Improvements for the Survey of Earned Doctorates: This report presents findings from a study conducted in 2014 that analyzed the current approach the SED employs to collect, edit and report timeline data. The report provided recommendations for the increased utility and improved quality of the data through potential questionnaire changes, edit changes and changes to how the data are contained in the DRF and presented in reports. Findings were used to inform a number of process revisions, including: expansion of the auto-coding process for timeline variables; modification of rules used to flag nontraditional timeline sequences; addition of timeline variables to the Doctorate Records File (DRF) for use in further research; and revision of select imputation rules. In addition, revisions to timeline questions in the instrument are included in the planned future cognitive interview activities. (Bautista et al., 2014)

Enhancements in Auto-Coding in the Survey of Earned Doctorates: This report presents findings from a study conducted in 2014 on the feasibility of employing an automated coding application to additional SED variables that are currently coded manually by trained clerks. The report recommended additional coding rules that would allow for automated coding, reducing the need for manual coding which in turn would improve quality and lower labor costs. Findings from this study have been implemented, starting with the SED 2015 auto-coding activities. (Groenhout et al., 2014)

After the Breakoff Part 2: Converting Web Break-offs to Completes: This presentation was discussed at the 2014 International Field Directors & Technologies Conference. The aim of the presentation was to identify predictors of survey breakoff and identify the most successful prompts for converting breakoffs to completions. Conclusions from the presentation included (1) certain characteristics found in survey paradata are significant predictors of survey breakoff (2) tailored email prompts were much less successful than standard email prompts and (3) standard email prompts were more successful than mail prompts in converting breakoffs. (Groenhout et al., 2014)