NORC email to NCSES

Memorandum

To:

Mark Fiegener and Emilda Rivers, NCSES

From:

Vincent Welch, NORC

CC:

Mary Ann Latter, Tom Hoffer, and Dan Kasperzyk,
NORC

Proposed Confidentiality, IRB, and Data Security
Subject: project
Date:

September 25, 2012

In the current context of heightened concern over institutions’ abilities to maintain confidentiality assurances,
it is imperative for the SED to review on a regular basis both the way the assurances are conveyed to
universities and the doctorate recipients, and how the assurances are actually supported by the project. NORC
is well prepared to conduct that review and to prepare an analysis of the SED confidentiality related policies,
practices, and issues.
The proposed report will focus on how doctoral students, Graduate Deans, Institutional Contacts (IC), and
Institutional Researchers understand the SED confidentiality assurances and guidelines and how this
understanding affects institutional and individual participation in the SED as well as the perceived utility of
SED data. In addition, as part of the Association of American Universities (AAU) data-sharing project, NSF
and several AAU institutions are engaged in designing a data-sharing process that could return SED data to
institutions quickly and in a manner that allows each participating institution to: (1) link the returned SED data
on their graduates with data collected via their own exit survey of these same graduates; and (2) merge their
linked doctorate recipient data with similar data from other participating institutions. The AAU data-sharing
project will require a new confidentiality pledge that will be presented only to SED respondents from
participating AAU institutions and will include an “opt-out” box by which respondents can choose to disallow
their SED data from being linked with exit survey data and/or shared among AAU institutions. The wording
of the new confidentiality pledge and the design of the opt-out box are still under development; the proposed
Confidentiality, IRB, and Data Security project will provide a means to test alternative versions of the pledge
and opt-out box.

1

Memorandum
We propose conducting a series of focus groups and/or interviews with individuals from each of the
previously mentioned stakeholder groups to investigate these issues. We will also analyze a sample of
respondent comments for confidentiality-related content. We propose to supplement the findings from the
focus groups and interviews with a textual analysis of the SED’s confidentiality-related material that is given to
potential respondents and doctorate-granting institutions.
Task 1: Clearance and approval
The proposed research will require the approval of NORC’s Institutional Review Board (IRB) and clearance
from the Office of Management and Budget (OMB). NORC stands ready to prepare protocols for the focus
groups and structured interviews and other accompanying documentation that will be required to gain
approval from both oversight bodies before conducting the proposed research.
NORC will work with NCSES to compile all of the necessary materials to produce a compelling submission to
the OMB. After submission, NORC will be ready to respond quickly to any requests for additional
information or clarification that OMB might make of NCSES in order to gain clearance in a timely manner.
Task 2: Prepare Protocols
NORC will prepare first drafts of the protocols to submit to NCSES for approval. NORC recognizes that
there may be multiple iterations of protocols with each of the stake-holder groups with NCSES before the
final structure is agreed upon. After the protocols are finalized, NORC will submit the protocols and
procedures to NORC’s IRB for approval.
Task 3: Focus groups with graduate deans
NORC will conduct focus groups with graduate deans from institutions of various sizes from various regions
of the country. Additionally, the focus group participants will include both members and non-members of the
Association of American Universities (AAU).
We propose that the conversations with graduate school deans revolve around four major themes: (1) their
understanding of the confidentiality assurances we provide to respondents, including those related to the AAU
data-sharing project (if applicable); (2) if and how their understanding of our confidentiality procedures affects
their institutional cooperation with the SED; (3) their use of SED data (with examples) and their awareness of
other SED data products, including the institutional data set; and (4) their understanding of the data use
agreements they sign when they receive their institutional data. We will also ask the deans about the data
security procedures they employ with SED data both before the data (i.e., survey forms, etc.) are returned to
2

Memorandum
NORC and after the data are delivered to the deans in the form of institutional data sets (or, for schools
planning to participate in the AAU data-sharing project, data sets of recent graduates who do not choose to
opt-out of the data-sharing process).
Within these broad areas, we think that it would be useful to explore whether there is anything that NCSES
and NORC can do regarding confidentiality that will increase the deans’ comfort level with our confidentiality
procedures and lead them to encourage greater institutional cooperation with the SED among their staffs. We
would also like to know what, if any, interactions deans are having with their institutional IRBs regarding SED
confidentiality procedures and cooperation.
NORC will conduct a total of three focus groups; one group that will consist of deans of institutions that are
AAU members and two groups that are made up of deans from non-AAU members. Due to the wide
dispersion of doctorate-granting institutions, NORC does not believe that the interviews with deans can be
conducted face-to-face within budgetary constraints. Therefore, NORC will conduct the focus groups
electronically.
NORC recognizes that recruiting deans to participate in the focus groups could be a hurdle. We will use
Kristy Webber, Task Leader for institution contacting, to aid in the recruitment process. Webber has close
relationships with staff, including deans, at many participating schools and will capitalize on those relationships
to facilitate the recruitment process. During recruitment, NORC will ensure that deans from institutions of
different sizes and Carnegie classifications are represented in order to fully represent the breadth of experience
of this critical stake-holder group. NORC will offer an incentive of $55 to each group member for
participation in these groups. However, NORC recognizes that regulations at the deans’ institution may
prevent him or her from accepting the incentive.
NORC anticipates that all deans will have easy access to high-speed internet connections. We further
anticipate that most, if not all, of the deans will have ready access to webcam-equipped computers. Should any
of the deans lack webcam equipment, NORC is prepared to send webcams that will operate on a variety of
platforms to any deans that wish to participate. Groups will consist of 6-8 deans and will last 45 – 60 minutes.
Groups will be recorded to allow for review at a later date.
Task 4: Focus groups with institutional researchers

NORC will conduct two focus groups with institutional researchers. One of the groups will be conducted

3

Memorandum
with institutional researchers who are employed at AAU institutions and the second will be conducted with
researchers who are employed at non-AAU institutions.
Institution-based users of SED data are an important constituency who are greatly affected by the
confidentiality-related issues that the proposed research will focus on. The conversations with institutional
researchers will focus on three broad issues: (1) their understanding of the SED disclosure avoidance policies;
(2) the degree to which their research is limited by the disclosure avoidance procedures implemented in the
past few years of the SED; and (3) what the institutional researchers’ data security and confidentiality
procedures are. We would also like to know if institutional researchers interface with ICs or graduate deans to
better understand institutional procedures regarding SED data.
To aid in recruitment NORC will capitalize on our database of external requests from institutional researchers
and those who have requested licensed data files from NCSES. We will contact institutional researchers who
have requested institutional data sets and analyses from NORC. NORC will offer an incentive of $55 to each
group member for participation in these groups. However, NORC recognizes that regulations at the
researchers’ institution may prevent him or her from accepting the incentive.
After recruiting, NORC will send participants copies of SED’s confidentiality related material (e.g., SED
confidentiality brochure, copies of the survey instrument with the confidentiality pledge highlighted, etc.) to
facilitate the discussion of the impact of these policies on this stake-holder group.
NORC anticipates that institutional researchers will have easy access to high-speed internet connections. We
further anticipate that most, if not all, of the researchers will have ready access to webcam-equipped
computers. Should any of the researchers lack webcam equipment, NORC is prepared to send webcams that
will operate on a variety of platforms to any participants. Groups will consist of 6-8 institutional researchers
and will last 45 – 60 minutes. Groups will be recorded to allow for review at a later date.
Task 5: Interviews with doctorate recipients
NORC will conduct a series of semi-structured interviews with doctorate recipients. These interviews will be
conducted electronically, either over the telephone or via the internet. We will attempt to conduct
approximately 12 interviews with recent doctorate recipients from AAU-member institutions and 6 interviews
with recent doctorate recipients from institutions who are not a part of the AAU.
We propose that the conversations with doctorate recipients revolve around three major themes: (1) their
understanding of how their SED data are used and protected; (2) the degree to which confidentiality-related
4

Memorandum
concerns affected their responding to the SED; and (3) their understanding of and reactions to alternative
descriptions of the AAU data-sharing process (i.e., wordings of the new confidentiality pledge and designs of
the opt-out box), if applicable. Within the first two themes, we would be particularly interested in determining
if doctorate recipients have different confidentiality-related concerns depending on the SED response mode
(i.e., hard copy, web, or telephone).
After recruiting, NORC will send participants copies of SED’s confidentiality related material (e.g., SED
confidentiality brochure, copies of the survey instrument with the confidentiality pledge highlighted, etc.) to
facilitate the discussion of the impact of these policies on this stake-holder group. For the AAU group, we will
be certain to include the proposed NSF confidentiality pledge which includes the “opt out” clause regarding
linking SED data to institutional exit survey data and regarding the sharing of SED data among AAU
institutions.
We will recruit participants for these interviews from 2012 doctorate recipients. We will contact both web and
hardcopy respondents from AAU and non-AAU schools. Interviews will be recorded to allow for analysis.
We will also attempt to recruit doctorate recipients with a variety of educational foci (i.e., field of doctoral
study) and from a variety of institution types (i.e., Carnegie Class). NORC will offer an incentive of $55 to
each participant for participating in these interviews.
Task 6: Interviews with institutional contacts
NORC will conduct a series of semi-structured interviews with SED ICs. These interviews will be conducted
electronically, either over the telephone or via the internet. We will attempt to conduct approximately 6
interviews with ICs from AAU-member institutions and 4 interviews with ICs from institutions who are not a
part of the AAU.
We propose that the conversations with ICs revolve around three major themes: (1) their understanding of the
confidentiality assurances we provide to respondents, including those related to the AAU data-sharing project
(if applicable); (2) what they tell respondents about the confidentiality of their SED data; and (3)the ICs’ data
security procedures vis-à-vis SED data.
Within these broad areas, we would also want to discuss the frequency with which students express
confidentiality concerns and what, if anything, the ICs say or do to allay those concerns. We would also like to
know what, if any, instructions ICs are getting from their institutional IRBs regarding SED data security and
confidentiality procedures.

5

Memorandum
NORC recognizes that recruiting institution contacts to participate in the interviews could be a hurdle. We
will also use Webber to aid in this recruitment process. NORC will ensure that IC’s represent a variety of
program types (e.g., single-program institutions and multi-program institutions) and institution types (i.e.,
Carnegie class of institution). NORC will offer an incentive of $55 to each group member for participation in
these groups. However, NORC recognizes that regulations at the ICs’ institution may prevent him or her
from accepting the incentive.
After recruiting, NORC will send participants copies of SED’s confidentiality related material (e.g., SED
confidentiality brochure, copies of the survey instrument with the confidentiality pledge highlighted, etc.) to
facilitate the discussion of the impact of these policies on this stake-holder group. For the AAU group, we will
be certain to include the proposed NSF confidentiality pledge which includes the “opt out” clause for linking
SED data to institutional exit survey data and regarding the sharing of SED data among AAU institutions.
NORC will record each of the interviews to allow for later analysis.
Task 7: Report
NORC will compile the results of the focus group discussions and structured interviews into a comprehensive
report that will allow NCSES to assess the full range of confidentiality related concerns for each stake-holder
group. Where appropriate, the report will also include recommendations for updating or enhancing the SED’s
confidentiality-related messaging. The final report will be authored by Vincent Welch, Senior Survey
Methodologist at NORC, with oversight from Mary Ann Latter, SED Project Director at NORC, and Dan
Kasperzyk, Senior Fellow and Vice President at NORC.
NORC understands that the crafting of the final report will be an iterative process. NORC will submit an
initial draft of the confidentiality report to NCSES by March 5, 2013. NORC will submit a revised, final, draft
of the report to NCSES no later than three weeks after receiving feedback on the initial draft.
Task 8: Management and oversight

NORC will work closely with NCSES to ensure that this project stays within budget and time
parameters. The NORC team will meet weekly to address the progress of recruiting and data
collection and to ensure that any potential barriers are quickly overcome. These meetings will be
attended by Mary Ann Latter, SED Project Director at NORC, Vincent Welch, Task Leader for this
Study, and the survey support staff. Additionally, NORC will conduct monthly meetings with
NCSES to ensure that the project is proceeding as expected. These meetings will be attended by
6

Memorandum
Mary Ann Latter, Vincent Welch, and Dan Kasperzyk, Vice President and Director of CESR at
NORC, who will be providing corporate and research oversight on this task. NORC stands ready to
adjust the meeting schedule to accommodate NCSES’s schedule.
Project Timeline
Below is a draft timeline for this project. NORC envisions initiating the project in October of 2012 and
completing the final report by March 31, 2013
Task

Duration

Prepare interview and focus group
protocols

10/15/2012 – 11/2/2012

IRB Submission and approval

11/12/2012 – 11/19/2012

OMB submission and approval

11/9/2012 – 1/6/2013

Conduct focus groups with deans

2/4/2013 – 2/18/2013

Conduct focus groups with
institutional researchers

1/7/2013 – 2/4/2013

Conduct interviews with doctorate
recipients

1/7/2013 – 2/15/2013

Conduct interview with institution
contacts

2/4/2013 – 2/18/2013

7

Memorandum

Compile first draft of report

2/19/4/2013 – 3/5/2013

Complete final draft of report

3/26/2013 – 3//31/2013

Project Budget

See attached

8