SED Cover Memo

MEMORANDUM
Date:

April 13, 2017

To:

Margo Schwab, Desk Officer
Office of Management and Budget

From:

John R. Gawalt, Director
National Center for Science & Engineering Statistics
National Science Foundation

Via:

Suzanne Plimpton, Clearance Officer
National Science Foundation

Subject:

Notification of data collection under generic clearance (3145-0174)

The purpose of this memorandum is to inform you that the National Center for Science and
Engineering Statistics (NCSES) within the National Science Foundation (NSF) plans to conduct
qualitative research in the form of focus groups and one-on-one interviews for the Survey of
Earned Doctorates (SED) under the generic clearance of survey improvement projects (OMB
#3145-0174).
NCSES is considering expanding the content of the SED to include information on the Graduate
School Experiences (GSE) of doctorate recipients. The qualitative research described in this
memorandum is part of a more expansive plan that includes a quantitative component. The
objectives of the qualitative portion of the research are to collect data from current users and
potential users of SED data to assess the value of adding GSE items to the SED. The results of
the qualitative research will inform the design of the quantitative component, a Web survey of
SED graduate school deans, which will be addressed in a separate generic clearance request. The
results of the Web survey will help determine whether the SED should collect graduate school
experience information in the future.

Background
The SED has been conducted annually since academic year 1958. The survey collects data on the
number and characteristics of all individuals receiving research doctoral degrees from all
accredited U.S. institutions. The results of this annual survey are used to assess characteristics
and trends in doctorate education and degrees.
In previous research, the SED contractor (NORC at the University of Chicago) examined the use
of exit surveys at SED institutions. This examination focused on determining whether selected
institutions conducted an exit survey for doctorate recipients, and the content of those surveys.
NORC selected a sample of 50 SED institutions stratifying by number of doctorates granted,
Carnegie classification, public-private sector, and minority-serving status (Historically Black
Colleges and Universities, Hispanic-Serving Institutions). A Web search of the 50 selected
1

institutions suggests that at least two out of every five SED institutions currently conduct an exit
survey, either at the institution or department level that collects GSE information on doctorate
recipients. A review of approximately 20 of the existing surveys and related materials describing
the use of GSE information indicated that institutions vary in the depth in which GSE topics are
explored. However, nearly all the surveys examined were alike in measuring, in one form or
another, the perceptions of doctorate recipients on the overall quality of or their satisfaction with
their graduate program or graduate school experience. Many SED institutions that are part of the
Association of American Universities Data Exchange (AAUDE) conduct a GSE survey. A
special focus of the research is whether AAUDE institutions would respond favorably to the
inclusion of GSE items in the SED.
NSF intends to conduct the GSE qualitative research between May and October of 2017.

Purpose
The primary objective of this qualitative research is to collect information on the interest level
among SED institutions of adding GSE items to the SED and the issues surrounding such an
addition. Findings from this qualitative research will guide the development of topics to be
covered in a Web survey of all SED graduate school deans (which will be covered by a separate
generic clearance request). Results from the Web survey will ultimately be used to help guide
decision-making as to whether a GSE component should be added to the SED.
GSE topic areas of potential interest that will be explored in the qualitative research include the
following:
• Communication of Important Information
• Program/Campus Climate
• Professional Development Experiences
• Curriculum and Instruction
• Skill Development
• Faculty and Staff
• Scholarly Products and Presentations
• Financial Support/Aid
• Student Support Services
• Factors Hindering or Facilitating Academic Progress
• Campus Life and Activities
• Program and University Facilities
• Overall Satisfaction

Research Plan
General approach. The research team will conduct focus groups and one-on-one interviews.
The research team anticipates completing six focus groups with six to eight participants in each
group. The findings from the focus group discussions will guide the topics to be discussed in the
next phase of the research, 30 one-on-one interviews. The focus groups and one-on-one
2

interviews will be held with graduate school deans, heads of graduate departments,
researchers/data users, and representatives of scholarship foundations and programs. Individuals
sampled for a focus group who are not available during the scheduled focus group session may
be invited to participate in a one-on-one interview.
Focus group and one-on-one interview logistics. Focus groups and one-on-one interviews will
be conducted via Web using GoToMeeting. The online format will allow for the participation of
individuals across the country. In advance of the focus group or one-on-one interview,
participants will be queried as to whether they have the capability to participate in a video
conference from their own computer. Webcams will be provided to participants who need one.
To prevent technical difficulties, a research team member will review meeting logistics with each
participant to solve any technological issues in advance of the scheduled meeting time.
The discussion guide for the qualitative research is in Attachment 1. As the qualitative work
progresses, this guide will be updated to incorporate new findings. The focus groups and one-onone interviews will be recorded if respondent consent is granted. Video-recording will be the
preferred method if participants give their consent. In the event that participants express
reluctance to be video-recorded, audio-recording will be offered instead. A copy of the consent
form will be sent to respondents in advance for their review. Prior to the beginning of each focus
group or one-on-one interview, respondents will be asked to provide verbal consent for
participating in the research. The informed consent documents are presented in Attachment 2.
Recruitment. There are several characteristics of respondents and their doctoral institutions that
are of specific interest in the sample recruitment for this study. For both the focus groups and
one-on-one interviews, the following individuals at SED doctoral institutions and other
organizations will be identified for recruitment. Selected participants will be users and potential
users of SED data who also understand the nature and uses of GSE survey results or of other
graduate student data available at their institution or organization.
•
•
•
•
•
•

Graduate School Deans at AAUDE member institutions
Graduate School Deans at non-AAUDE institutions that conduct a GSE survey
Graduate School Deans at non-AAUDE institutions that do not conduct a GSE survey
Heads of graduate school departments
Other researchers/data users
Scholarship foundations and programs

One focus group with between six and eight participants each will be held with each of the
groups listed above. The 30 one-one-one interviews will also be conducted with individuals
representing each of these groups.
Participants will be recruited via several sources. Graduate school deans will be identified using
the list of deans that is maintained by the SED. The list includes approximately 550 deans.
Deans will be considered eligible to participate if they still hold the position of dean at an SED
institution. The deans will be selected to cover a range of institutional characteristics: both
AAUDE and non-AAUDE institutions, institutions that do and do not conduct a GSE survey,
institutions of varying size and location, and public as well as private institutions.
3

The SED does not maintain a list of heads of graduate departments, and such a list is not likely to
be available from another source. However, a question in the SED asks respondents to name the
department that supervised their doctoral studies. We will use the data from the 2015 SED to
compile responses to this question. The list will be de-duplicated so that each department is listed
only once and then departments will be selected from the list to represent the same range of
institutional characteristics that will be used to guide the selection of deans (as noted at the end
of the previous paragraph). A Web search will be conducted to identify the heads of the selected
graduate departments.
Data users will be personnel at SED institutions or other agencies or organizations that use SED
data. The SED maintains a database of requesters of the data. The list includes those who have
requested a custom data tabulation, an institution data set, a baccalaureate origins table or a
master’s origins tables. As with the selection of deans and graduate department heads, the
institutional data users selected will represent a range of doctoral institutions. Typically, the list
of data users also includes non-institutional requesters, such as newspapers and foundations. The
types of requesters will be reviewed to assure that institutional users or other users with specific
interest in research on graduate education will be included in the data users group. When data
users make a request for data, they specify the name of the individual who will receive the data.
That individual will be the person contacted regarding participation in the study.
Foundations and programs that are eligible to participate in the research will include those that
fund graduate school scholarships or that conduct research in graduate education. In compiling a
list of foundations, we will consult several sources, including the list of data users as well as
recommendations offered by NCSES and NORC. We will also ask representatives at foundations
we have identified to recommend other foundations that should be contacted about this research.
Contacting Potential Respondents. All potential respondents will be contacted via email
(Attachments 3 and 4). Within a few days of sending the email, staff will attempt to reach
potential participants via telephone to further describe the research, answer any questions
potential participants may have, and encourage their participation (Attachment 5). During the
phone contact, individuals will be presented with information about the study and will then be
asked to participate in a brief screening interview to confirm the potential participant’s eligibility
for the study (Attachment 6). Recruitment staff will determine whether the potential participant
is available during the time frame in which the focus groups are to be conducted. Those who are
not available may be invited to participate in a one-on-one interview to be scheduled for a later
date. The screener represents all potential questions that the research team anticipates asking
during the screening process. However, as the research progresses, the screener may be revised
to focus only on the characteristics that remain of interest for recruiting.
Interested participants who sign up for a focus group or one-on-one interview will receive a
packet via U.S. Postal Service (see Attachments 7 through 9). The packet will contain a cover
letter reminding them of their focus group or one-on-one interview day and time, materials to be
discussed during the session, and instructions for joining the online meeting. Although informed
consent will be obtained at the beginning of each focus group or one-on-one interview, a copy of
the consent form will be included with the packet to allow participants to review it in advance.
4

Participants who need a web camera will also be sent one. In advance of each focus group, a
research team member will step through the use of GoToMeeting with participants to assure that
any technical issues are resolved before the scheduled group meeting.

Schedule
Data collection activities will begin immediately after obtaining OMB clearance. Contacting,
recruiting, and scheduling of respondents will proceed until all focus groups and one-on-one
interviews have been completed. The focus groups will be completed within the first three
months after OMB clearance is obtained. The one-on-one interviews will follow, and will be
completed within the next three months.

Burden Information
The total estimated burden associated with this study is anticipated to be 202 hours. Exhibit 1
provides details about the burden estimates by study activity.
Exhibit 1: Estimated burden
Activity

1.

Number of
Average Time
Total Burden
Respondents
(minutes)
(minutes)
A total of 390 individuals will be selected from lists of deans, heads of graduate school departments,
members of foundations, and data users to receive a recruitment email. Estimated 5 minutes of
respondent burden per respondent.
Recruitment email

2.

3.

4.

5.

390

5

1,950

Of the 390 individuals receiving the recruitment email, NORC anticipates reaching 292 by telephone to
provide further information about the study and conduct screening. The estimated burden of the
telephone recruitment and screening process is 10 minutes.
Telephone
recruitment and
screening
292
10
2,920
Of the 100 who are expected to complete telephone screening, 78 are expected to be recruited to
participate in a focus group or one-on-one interview. These individuals will receive a follow-up
email/call to confirm their participation day/time and will receive an information packet with
instructions for joining a one-on-one interview or focus group.
Information packet
and follow-up
78
10
780
call/email
Of the 100 expected to complete telephone screening, the 22 not participating in a focus group or oneon-one interview will receive an email thanking them for their interest and informing them that focus
group and one-on-one interview slots are filled.
Thank you email
22
110
5
The 78 participants will receive a reminder email regarding the day and time of the focus group or oneon-one interview.
Reminder email

78

5

5

390

6.

7.

8.

Of the 78 participants, 48 will be selected for participation in a focus group and will be contacted in
advance of the focus group meeting to test and resolve any issues with the use of the online meeting
software.
Test of online
48
480
10
meeting logistics
Of the 78 participants, 48 are expected to participate in a focus group (6 groups of up to 8 people).
Estimated interview length is up to 75 minutes.
Focus group
48
3,600
75
Of the 78 participants, 30 are expected to participate in an individual one-on-one interview. Estimated
length of these interviews is up to 60 minutes each.
One-on-one
interviews

30

60

Total burden

1,800
12,030 minutes
(202 hours)

Confidentiality
Research participants will be informed that their participation is voluntary and their responses
will be used for research purposes only. They will be asked to give verbal consent for the focus
group or interview to be video-recorded (or audio-recorded if participants are reluctant to agree
to video-recording) and informed that only staff directly involved in the research project will
have access to the recordings. Further, participants will be advised that selected quotes may be
used in papers and presentations and researchers will not identify participants by name.
Participants in focus groups will be advised that their identities will be known to other
participants. However, all focus group participants will be asked to respect the privacy of the
other group members and not to disclose anything said within the context of the discussion.

Incentive Payments
As senior level representatives of their institutions and organizations, graduate school deans,
graduate school department heads, and foundation officials will likely not expect or require an
incentive to participate in the research. Data users, however, will be research staff working
under the direction of key staff at institutions or foundations. Since it is customary to provide
incentive payments for junior level employees, we propose to offer an incentive of $40 to
compensate data users for the time associated with participation in the study. Although data users
may receive permission from their employers to participate in the research on employer time,
NORC will not request that data users ask for permission to take part in the research during work
hours. To avoid making employer permission a barrier to participation, NORC will attempt to
schedule the data user focus group and interviews during lunch breaks or outside of work hours.
(Note: Names of any senior level administrators that appear in the lists of data users used to
construct the sample frame will be removed prior to selection of the data user sample.
Accordingly, the data user list will only include names of potential participants who would
typically receive payment for study participation, as there will be no senior level officials
represented.)
6

Contact Information
The contact persons for questions regarding this research are:
Lynn Milan
Project Officer
Survey of Earned Doctorates
National Center for Science & Engineering Statistics
National Science Foundation
703-292-2275
lmilan@nsf.gov
John Finamore
Program Director
Human Resources Statistics Program
National Center for Science & Engineering Statistics
National Science Foundation
703-292-2258
jfinamor@nsf.gov

7

Attachments
1:
2:
3:
4:
5:
6:
7:
8:
9:

Discussion Guide for Qualitative Research on GSE Module
Participant Consent Form
Recruitment Email
Frequently Asked Questions
Phone Script for Participant Recruitment
Screener
Cover Letter
Example Visuals
Discussion Topics Guide

8