B.4. Explanation

Proposed Tests and Research

Over the course of the proposed OMB cycle (April 2013 – December 2015), NSF anticipates conducting several methodological research tasks and analyses of data user needs, some involving focus groups, cognitive interviews, and/or workshops. The tasks associated with these research studies and user analyses will be conducted under the Generic Clearance of Survey Improvement Projects package.

One research study will involve a review of the institutional criteria for participation in the SED. Historically, NSF and the other Federal Sponsors have set two criteria to establish whether institutions are eligible to participate in the SED: 1) the institution must offer a research doctoral degree, and 2) the institution should be accredited by one of the six regional accreditation organizations. However, the SED does include a few examples of non-accredited institutions (e.g., Rockefeller University) and NSF faces issues of determining whether certain newer, accredited institutions (e.g., virtual institutions) meet the requirements for research doctoral programs. In response, NSF will conduct an analysis of the current eligibility criteria in order to develop clear, consistent eligibility criteria and/or eligibility review process. Implementing the new criteria/process in place will help the NSF avoid potential charges of implicit or explicit bias. The SED eligibility analysis will consider the impact of any changes on the fielding, implementation, and reporting of the SED and SDR, while addressing: 1) accreditation as an indicator of quality, 2) the rigor and comprehensiveness of the accreditation agencies’ standards, and 3) reasons why some institutions decide not to seek accreditation. This will include reviewing accreditation agency websites, interviewing agency representatives and a small sample of accredited and non-accredited institutions offering research doctoral programs, and reviewing how other nations view and implement accreditation. The results will be documented in a report and a policy brief that summarizes the major findings and issues, and which could be widely disseminated. As an ancillary step, a panel of experts from different types of institutions will be convened for a one-day meeting in Washington DC. The charge to the panel will be to: 1) develop objective SED-eligibility criteria such as a definition for a research doctoral program and the need for accreditation; and 2) outline how to address issues of reporting, should the new criteria change the universe of the SED in significant ways. All findings will inform decisions by NSF regarding the institutional eligibility criteria and review process.

Another methodological research project planned for the next clearance cycle is related to improving the accuracy of debt reports. For reporting on education-related debt, the SED does not ask respondents to provide actual values but, rather, to select a set of dollar intervals into which their undergraduate and graduate education-related debts fall. As a check on the quality of these reported data, NSF will examine whether respondents can actually enumerate these values. Using decomposition, respondents will be asked about their debt for their undergraduate, then graduate, and finally total postsecondary education. This experimental examination will not only have consequences for the SED but also broader implications for collecting financial information in other surveys. The experiment will use two versions of the survey: (V1) will not change the current survey items, but will add a final item that ask respondents to provide an estimated total debt from the combined sources; (V2) will ask respondents to enumerate values for undergraduate, graduate, and total debt owed, but will not provide ranges. Both versions will include an item asking respondents to indicate their degree of confidence in their answers for the total value. The Web-SED versions will also collect client-side paradata on response changes, elapsed time, and response order. NSF will use the responses to the degree of confidence question to examine the effect the ranges have on providing numeric values, how decomposition influences actual value estimation and confidence, and how these changes affect respondent burden. This research could uncover new ways to ask respondents to report dollar values.

NSF will continue its investigation of alternative methods of disclosure avoidance in order to increase the utility and usability of data tables based on SED data. The current means of disclosure treatment in SED tabular data, cell suppression in tabulation (cs-tabulation), can have a severe impact on the reporting of doctorates awarded to underrepresented race/ethnicity groups and women, to the extent that information released on earned doctorates by fine field of study may be very limited. This is particularly undesirable for policy analysts working toward the goal of equal opportunity. One disclosure avoidance technique under consideration would replace non-informative ‘D’ symbols with more informative, but still non-disclosive, model-based estimates for suppressed values.

NSF will continue the qualitative investigations of the communication of confidentiality assurances to institutions, data users, and survey respondents. This research will focus on how doctoral students, Graduate Deans, Institutional Contacts (IC), and Institutional Researchers understand the SED confidentiality assurances and guidelines, and how this understanding affects institutional and individual participation in the SED and the perceived utility of SED data. SED will conduct a series of focus groups and/or interviews with individuals from each of the previously mentioned stakeholder groups, and a sample of respondent comments will also be analyzed for confidentiality-related content. Findings from the focus groups and interviews will be supplemented with a textual analysis of the SED’s confidentiality-related material that is given to potential respondents and doctorate-granting institutions.

To ensure that these proposed methodological studies draw upon the collective experience and judgment of both NCSES and NCES and help align the two sets of surveys, where appropriate, NCSES commits to collaborating with NCES on the design of these studies and the evaluation of the results. For example, in the institutional eligibility study, the draft project report will be circulated to NCES for review and comment, and NCES staff will be invited to participate in the experts panel meeting held in Washington.

With respect to user needs analyses, NSF will conduct a study that will culminate in the design of data products specific to the needs of Graduate School Deans. The study will analyze the history of SED data requests made by deans, review the types of information posted on their graduate school websites, and conduct focus groups that bring Graduate Deans together to discuss their particular data needs. In another user needs analysis, NSF plans to research the utility and operational feasibility of providing SED data users with “early estimates” of selected SED data without compromising data quality. The results of the study will identify: 1) a preferred cut-off date for collecting the early-estimates data, 2) the trade-offs involved in not waiting to report the final data, 3) the best methods for addressing the situations involving missing data, and 4) scheduling implications.

Some revisions to the 2014 SED survey instrument are based on a NSF-sponsored review of the Web-SED conducted by Drs. Jolene Smyth and Kristin Olson from the University of Nebraska-Lincoln, which examined the SED web instrument in comparison to the SED hardcopy, SDR instrument and other relevant surveys. These analyses may continue during 2014 and 2015 rounds, and may include a methodology study to test the efficacy of the recommendations for Web-SED survey and paper questionnaire redesign. Interviews may be conducted with respondents to gauge their reaction to these changes, their reaction to the “Field of Study” lists on the paper survey versus the web survey, and other possible mode effects.

The draft SED 2014 questionnaire was first reviewed in December 2012, and the final questionnaire changes were reviewed and approved by the sponsors in January 2013. (See Attachment 5 for the list of persons who were consulted or who reviewed the questionnaire.) See Attachment 2 for a list detailing the changes made to the SED 2014 questionnaire from the 2011 version and the rationales for those changes.