MEMORANDUM
Date: September 5, 2013
To: Shelly Wilkie Martinez, Desk Officer
Office of Management and Budget
From: John R. Gawalt, Director
National Center for Science & Engineering Statistics
National Science Foundation
Via: Suzanne Plimpton, Clearance Officer
National Science Foundation
Subject: Notification of data collection under generic clearance (3145-0174)
The purpose of this memorandum is to inform you of plans by the National Center for Science and Engineering Statistics (NCSES) to conduct cognitive research for the Survey of Earned Doctorates (SED) under generic clearance for survey improvement projects (OMB #3145-0174). The purpose of this cognitive research is to understand various stakeholder groups’ perceptions of SED confidentiality procedures and how those perceptions affect their willingness to support the SED project. The results will be used to improve the confidentiality materials and protocols currently in practice, and to inform decisions about the phrasing of the confidentiality pledge.
The SED has been conducted annually since 1957. The survey collects data on the number and characteristics of all individuals receiving research doctoral degrees from accredited U.S. institutions. The results of this annual survey are used to assess characteristics and trends in doctorate education and degrees.
The research doctorate-granting institution is the main SED interface with doctorate recipients. Paper SED questionnaire forms and web-SED registration information are distributed to new doctorate recipients by Institutional Contacts, who are typically located within the office of the Graduate Dean at the doctorate-granting institution. Institutional Contacts also encourage doctorate recipients to complete the SED, monitor SED completion rates at their institution, and answer student questions about the SED, including questions about the SED’s confidentiality assurances. Institutional Researchers, a large and important category of SED data users who support the data analysis needs of Graduate Deans, are subject to the constraints imposed by NSF efforts to protect the confidentiality of data collected by the SED.
Given the heightened concern of institutions about their ability to uphold the confidentiality assurances they provide to students, it is imperative that NSF examine both the ways in which SED confidentiality assurances are communicated to universities and doctorate recipients, and how the assurances are interpreted by those participating in the SED project.
Purpose
The primary purpose of this research is to understand how individuals from four stakeholder groups involved in the SED project – Graduate Deans, Institutional Contacts, doctorate recipients, and Institutional Researchers – interpret the SED confidentiality assurances and how these interpretations affect institutional and individual participation in the SED.
In addition, the proposed research will also support the efforts of the “AAU data-sharing project,” a collaboration between NSF and the Association of American Universities (AAU) to design a process by which AAU member institutions can: (1) link SED data on their doctorate recipients with administrative records on these same graduates; and (2) merge their linked doctorate recipient data with similar data from other AAU institutions. The data-sharing process will require a new confidentiality pledge that would be presented only to SED respondents from the participating AAU institutions. This new pledge will include an “opt-out” response by which respondents can choose to disallow their SED data from being linked with administrative records and/or shared among AAU institutions. The wording of the new confidentiality pledge and the design of the opt-out response are still under development; a test version of the pledge and opt-out response is included in this package among the materials that will be provided to study participants.
The research results are intended for internal NSF use and will not be used to produce any official releases. The results will be used to improve the confidentiality materials and protocols currently in practice, and to inform decisions about the confidentiality pledge and opt-out response to be used in the AAU data-sharing project.
Methodology
The proposed research entails a series of focus groups and individual semi-structured interviews with individuals from each of four stakeholder groups: Graduate Deans, Institutional Contacts, doctorate recipients, and Institutional Researchers. The findings from the research will be supplemented with a textual analysis of the confidentiality-related materials that is provided to potential SED respondents and doctorate-granting institutions. The SED survey contractor will conduct the focus groups, interviews, and textual analysis.
The survey contractor will select and recruit study participants in each stakeholder group from AAU institutions and non-AAU institutions of various sizes and regions of the country. All participants in the doctorate recipient group will have earned doctorates in the 2012 academic year, and collectively they will represent several academic disciplines. After recruiting the study participants, the survey contractor will send SED confidentiality-related materials (e.g., SED confidentiality brochure, copies of the survey instrument with confidentiality pledge highlighted, etc.) to facilitate discussion in the focus groups and interviews. Participants from AAU institutions will also receive the proposed new confidentiality pledge with “opt-out” response.
The data collection method for each stakeholder group follows. Study participants from AAU institutions will also be asked about confidentiality-related issues relevant to that project.
Graduate Deans – 3 groups of 6-8 Graduate Deans will participate in electronic focus groups (using webcams) that explore their understanding of SED confidentiality assurances, willingness to encourage doctorate recipients to participate in the SED, and their use of SED data and awareness of SED data products. One group will consist of Deans of AAU institutions and two groups will include Deans of non-AAU institutions.
Institutional Contacts – 10 Institutional Contacts will participate in semi-structured interviews either over the telephone or via the internet. The interviews will explore their understanding of SED confidentiality assurances, what they tell SED respondents about the confidentiality of SED data, their reaction to the confidentiality materials provided by the survey contractor, and their institution’s use of SED data. Approximately one-half of the interviewees will be Institutional Contacts at AAU institutions.
Doctorate Recipients – 30 doctorate recipients will participate in semi-structured interviews either over the telephone or via the internet. The interviews will explore their understanding of SED confidentiality assurances, the degree to which confidentiality-related concerns affected their response to the SED, and concerns about the uses of their SED data. Approximately 18 of the interviewees will have earned doctorates from AAU institutions. To mitigate possible recall problems, the SED questionnaire and confidentiality statement will be provided to each doctorate recipient participant at the outset of the interview.
Institutional Researchers – 2 groups of 6-8 Institutional Researchers will participate in electronic focus groups (using webcams) that explore their understanding of SED confidentiality assurances and SED disclosure avoidance policies, the data security and confidentiality procedures they employ to protect SED data, and their institution’s use of SED data. One group will consist of Institutional Researchers from AAU institutions and the other will include Institutional Researchers from non-AAU institutions.
As a condition of recruitment to the electronic focus groups, Deans and Institutional Researchers will be asked if they have easy access to high-speed internet connections. The proposed methodology anticipates that most if not all Deans and Institutional Researchers will also have access to webcam-equipped computers; should any invitees lack this equipment, the survey contractor will send webcams that are capable of operating on a variety of platforms.
Interviews and focus groups are expected to last no longer than 45 and 75 minutes, respectively, and will be scheduled during the day based on respondent convenience. All focus groups and interviews will be recorded to allow for review at a later date.
Once all interviews and focus groups have been completed, the survey contractor will prepare a comprehensive report that will allow NSF to assess the full range of confidentiality-related concerns for each stakeholder group. The report will also include recommendations to improve the SED’s confidentiality-related materials and messaging.
Schedule
Data collection activities will be initiated immediately after obtaining OMB clearance, with interviews conducted over the subsequent 4 weeks and focus groups expected to take place over a longer period (owing to the inherent difficulties with scheduling meetings that include several participants). The tentative schedule for this research is as follows:
Proposed Completion Date |
Activity or Deliverable |
Month xx, 2013 |
OMB submission for approval |
Submission date + 4 weeks |
OMB clearance |
Clearance date + 2 weeks |
Recruit and schedule interviews and focus groups |
Clearance date + 4 weeks |
Conduct individual semi-structured interviews |
Clearance date + 6 weeks |
Conduct focus groups |
Clearance date + 10 weeks |
Preliminary report delivered to NSF |
Clearance date + 14 weeks |
Final report delivered to NSF |
The protocol for each of the four stakeholder groups is attached to this memo.
NORC will conduct a total of 5 electronic focus group sessions with 6-8 participants each and 40 individual semi-structured interviews. The estimated duration is 75 minutes per focus group and 45 minutes per interview. The estimated average time to recruit each focus group participant and each interviewee is 10 minutes. The total respondent burden is estimated at between 79 and 93 hours.
The contact persons for questions regarding this research are:
Lynn Milan
Project Officer
Survey of Earned Doctorates
National Center for Science & Engineering Statistics
National Science Foundation
703-292-2275
Emilda Rivers
Program Director
Human Resources Statistics Program
National Center for Science & Engineering Statistics
National Science Foundation
703-292-7773
Attachments
1: Graduate Dean Focus Group Protocol
2: Institutional Contact Individual Interview Protocol
3: Doctorate Recipient Individual Interview Protocol
4: Institutional Researcher Focus Group Protocol
Materials for participant review will include:
Confidentiality statement on the SED 2013 (web-)questionnaire
Opt-out language for Association of American Universities (AAU) data-sharing project (provided to Deans from AAU institutions)
Examples of SED detailed statistical tables: tables 8 and 11 from the 2009-10 S&E Doctorate Awards report
Institution data set order form with confidentiality language
Quick-turn-around data set order form
Example of Institution Profile mock-up
Please describe your understanding of the confidentiality assurances given to your students concerning the information they provide on the SED.
Do you think the extent of those confidentiality assurances is reasonable? That is, do you think the confidentiality assurances ensure adequate protection or, conversely, they promise too little regarding the protection and usage of your data.
Do you feel that the current confidentiality pledge is too long, too short, or about the right length, in terms of number of words?
Do you think that any part of the current confidentiality pledge is not essential, and could be removed without reducing the overall level of protection provided? Which part or parts of the pledge do you think are not essential?
When you compare the part of the confidentiality assurance that relates to the partial SSN to the part of the confidentiality brochure that relates to the collection of partial SSN, do you interpret those passages as saying the same or different things about the usage of the SSN data?
Data from the SED are collected, processed, and analyzed by the National Opinion Research Center at the University of Chicago (NORC) and by the National Science Foundation (NSF). Do you believe these organizations rigorously adhere to the confidentiality assurances given to SED respondents, or do you have concerns about how well the data on your students might be protected?
Is there anything that you believe should be added to the confidentiality assurances?
Is there anything in the confidentiality assurances that you think is unnecessary or overly restrictive?
Does anything in the language of the confidentiality assurances make you more likely to encourage doctorate recipients from your institution to participate in the SED? Does anything make you less likely to encourage your doctorate recipients to participate in the SED?
What changes to the confidentiality assurances would strengthen your support for your doctorate recipients to participate in the SED?
Please refer to the alternative confidentiality statement (Attachment 5). Does the alternative language in this statement convey sufficient information about the SED confidentiality assurances? That is, do you think this shorter statement provides the same level of protection as the current statement?
Do you prefer the current statement or the shorter alternative statement?
Do you believe doctorate recipients would be more likely to read the alternative statement than the current statement?
Can you suggest changes to the alternative statement that would improve it?
In what ways does your institution make use of SED data?
What SED publications or data products do you use? For what purposes?
Have you ever looked at published data tables produced from SED data such as these two sample tables, tables 8 and 11?
Are there ways that public SED data are used that make you uncomfortable? What about these uses of SED data makes you feel uncomfortable?
Have you ever ordered or used an institution data set containing SED data on your institution’s doctorate recipients?
How long ago did you order an institution data set?
In what ways did you use your institution data set?
Please describe your understanding of the data use agreement that must be signed in order for you to receive your institution’s SED data.
That is, what does the data use agreement allow you to do with your institution’s SED data, and what does the agreement not allow you to do with the data?
Do you have any concerns about the data use agreement you must sign in order to receive an institution data set?
Does anything in the language of the data use agreement make you more likely to request and use your institution’s SED data? Does anything make you less likely to request and use the SED data?
Have you ever ordered or used data from a “quick-turn-around” institution data set?
Do you have any concerns about the data use agreement you must sign in order to receive a “quick-turn-around” data set?
What security measures does your staff employ to protect the institution data set, both the physical CD and the electronic data files?
If your institution uses paper SED questionnaire forms, what physical security procedures does your staff employ to protect the forms after they have been collected from respondents?
Have you received any guidance or instructions from you institution’s institutional review board (IRB) regarding SED data security and confidentiality protection procedures?
What are your thoughts about the proposed opt-out language of the data-sharing agreement that would be presented to respondents from institutions participating in the data-sharing process?
What, if anything, would you tell prospective SED respondents about the data-sharing process?
Does the “opt-out” language affect your willingness to encourage doctorate recipients at your institution to participate in the SED?
Would you be more comfortable with the data-sharing process if the confidentiality assurances included an “opt-in” box rather than an “opt-out” box, such that they would need to signal their explicit consent to allow their data to be linked and shared? Why?
Materials for participant review will include:
Confidentiality statement on the SED 2013 (web-)questionnaire
Opt-out language for Association of American Universities (AAU) data-sharing project (provided to Institutional Contacts from AAU institutions)
Examples of SED detailed statistical tables: tables 8 and 11 from the 2009-10 S&E Doctorate Awards report
SED confidentiality brochure
Please describe your understanding of the confidentiality assurances given to your students concerning the information they provide on the SED.
Do you think the extent of those confidentiality assurances is reasonable? That is, do you think the confidentiality assurances ensure adequate protection or, conversely, they promise too little regarding the protection and usage of your data.
Do you feel that the current confidentiality pledge is too long, too short, or about the right length, in terms of number of words?
Do you think that any part of the current confidentiality pledge is not essential, and could be removed without reducing the overall level of protection provided? Which part or parts of the pledge do you think are not essential?
When you compare the part of the confidentiality assurance that relates to the partial SSN to the part of the confidentiality brochure that relates to the collection of partial SSN, do you interpret those passages as saying the same or different things about the usage of the SSN data?
Data from the SED are collected, processed, and analyzed by the National Opinion Research Center at the University of Chicago (NORC) and by the National Science Foundation (NSF). Do you believe these organizations rigorously adhere to the confidentiality assurances given to SED respondents, or do you have concerns about how well the data on your students might be protected?
Does anything in the language of the confidentiality assurances make you more likely to encourage doctorate recipients from your institution to participate in the SED? Does anything make you less likely to encourage your doctorate recipients to participate in the SED?
Is there anything that you believe should be added to the confidentiality assurances?
Is there anything in the confidentiality assurances that you think is unnecessary or overly restrictive?
Do the SED confidentiality materials sent to you by NORC and NSF give you a thorough understanding of how to respond to confidentiality-related questions from SED respondents at your institution?
How frequently do you receive confidentiality-related questions from SED respondents?
What are the most frequently occurring confidentiality questions or concerns voiced by SED respondents?
Please refer to the alternative confidentiality statement (Attachment 5). Does the alternative language in this statement convey sufficient information about the SED confidentiality assurances? That is, do you think this shorter statement provides the same level of protection as the current statement?
Do you prefer the current statement or the shorter alternative statement?
Do you believe doctorate recipients would be more likely to read the alternative statement than the current statement?
Can you suggest changes to the alternative statement that would improve it?
In what ways do you believe your institution can make use of the SED data provided by your doctorate recipients?
In what ways do you believe Federal agencies – such as the National Institutes of Health, the Department of Education, and the Department of Agriculture – can make use of the SED data provided by your doctorate recipients?
Have you received any guidance or instructions from your institution’s institutional review board (IRB) regarding SED data security and confidentiality protection procedures?
What security measures do you employ to protect the paper SED questionnaire forms after they have been collected from respondents?
Have you ever looked at published data tables produced from SED data such as these two sample tables, tables 8 and 11?
Does your institution use public SED data in ways that, if people outside of your institution used the SED data in those same ways, it would make you feel uncomfortable?
What are your thoughts about the proposed opt-out language of the data sharing agreement that would be presented to respondents from institutions participating in the data-sharing process?
What, if anything, would you tell prospective SED respondents about the data-sharing process?
Does the “opt-out” language affect your willingness to encourage doctorate recipients at your institution to participate in the SED?
How do you think SED respondents from your institution will react to the information about the data-sharing process, and the “opt-out” box?
Would you be more comfortable with the data-sharing process if the confidentiality assurances included an “opt-in” box rather than an “opt-out” box, such that they would need to signal their explicit consent to allow their data to be linked and shared? Why?
Materials for participant review will include:
Confidentiality statement on the SED 2013 (web-)questionnaire
Opt-out language for Association of American Universities (AAU) data-sharing project (provided to doctorate recipients from AAU institutions)
Examples of SED detailed statistical tables: tables 8 and 11 from the 2009-10 S&E Doctorate Awards report
SED 2013 paper questionnaire
Please describe your understanding of the confidentiality assurances given to students concerning the information they provide on the SED.
Do you recall reading the full confidentiality statement before you completed the SED?
Do you think the extent of those confidentiality assurances is reasonable? That is, do you think the confidentiality assurances ensure adequate protection or, conversely, they promise too little regarding the protection and usage of your data.
Do you feel that the current confidentiality pledge is too long, too short, or about the right length, in terms of number of words?
Do you think that any part of the current confidentiality pledge is not essential, and could be removed without reducing the overall level of protection provided? Which part or parts of the pledge do you think are not essential?
When you compare the part of the confidentiality assurance that relates to the partial SSN to the part of the confidentiality brochure that relates to the collection of partial SSN, do you interpret those passages as saying the same or different things about the usage of the SSN data?
Data from the SED are collected, processed, and analyzed by the National Opinion Research Center at the University of Chicago and by the National Science Foundation (NSF). Do you believe these organizations rigorously adhere to the confidentiality assurances given to SED respondents, or do you have concerns about how well your data might be protected?
Does anything in the language of the confidentiality assurances make you more likely to participate in the SED? Does anything make you less likely to participate in the SED?
Is there anything that you believe should be added to the confidentiality assurances?
Is there anything in the confidentiality assurances that you think is unnecessary or overly restrictive?
After reading the confidentiality assurances, can you tell me the ways you think your SED data can and cannot be used?
That is, what is your institution allowed to do with your SED data? What is your institution not allowed to do?
What about NSF and other government agencies – what are they allowed to do and not allowed to do?
How did you complete the SED: on a paper survey form that you gave to your school; on a paper survey form that you mailed to the SED contractor; on the web-survey form; some other way?
Are there any items on the SED questionnaire that you did not feel comfortable answering?
Which items?
What is it about this (these) items that make you feel uncomfortable?
Do concerns about confidentiality affect the mode you used to complete the survey?
Are there items that you would feel comfortable answering on the paper survey but not the web-survey? On the web-survey but not the paper survey?
What is it that makes you feel uncomfortable about answering the items in that particular survey mode?
Please refer to the alternative confidentiality statement (Attachment 5). Does the alternative language in this statement convey sufficient information about the SED confidentiality assurances? That is, do you think this shorter statement provides the same level of protection as the current statement?
Do you prefer the current statement or the shorter alternative statement?
Do you believe doctorate recipients would be more likely to read the alternative statement than the current statement?
Can you suggest changes to the alternative statement that would improve it?
In what ways do you think your institution makes use of the SED data of its doctorate recipients?
Have you ever looked at published data tables produced from SED data such as these two sample tables, tables 8 and 11?
Are you concerned that someone might be able to identify your particular responses from published SED data tables such as these?
Would you be concerned if someone were able to determine that you were one of a very few people with your sex and race/ethnicity who earned a doctorate in your field of study?
Why or why not?
Do you believe the data you provide on the SED can be useful to researchers and policy makers?
Do you think the restrictions imposed by the confidentiality assurances may limit the usefulness of SED data to researchers and policy makers?
What are your thoughts about the proposed opt-out language that would be presented to respondents from institutions participating in the data-sharing process?
Would the “opt-out” language affect your willingness to participate in the SED?
If you chose to allow your data to be linked and shared, are there particular survey items on the SED that you would decline to answer? Would you be inclined to answer these same survey items if, instead, you chose to opt-out of the data linking and sharing?
Would you be more comfortable participating in the data-sharing process if the confidentiality assurances included an “opt-in” box rather than an “opt-out” box, such that you would need to signal your explicit consent to allow your data to be linked and shared? Why?
Materials for participant review will include:
Confidentiality statement on the SED 2013 (web-)questionnaire
Institution data set order form with confidentiality language
Quick-turn-around data set order form
Examples of SED detailed statistical tables: tables 8 and 11 from the 2009-10 S&E Doctorate Awards report
Please describe your understanding of the confidentiality assurances given to your students concerning the information they provide on the SED.
How does your understanding of the confidentiality assurances guide the conduct of the research you conduct with SED data at your institution?
Please describe your understanding of the data use agreement that must be signed in order for you to receive your institution’s SED data.
That is, what does the data use agreement allow you to do with your institution’s SED data, and what does the agreement not allow you to do with the data?
Does anything in the language of the data use agreement make you more likely to request and use your institution’s SED data? Does anything make you less likely to request and use the SED data?
Have you ever received any specific instructions or warnings from members of administration (e.g., your graduate dean) about what you could or could not do with your institution’s SED data?
In what ways do you make use of SED data at your institution?
What SED publications or data products do you make use of?
Have you ever looked at published data tables produced from SED data such as these two sample tables, tables 8 and 11?
Have you ever ordered an institution data set containing SED data on your institution’s doctorate recipients?
How do you use the data from the institution data set?
Do you use the data from the institutional data set differently than you use the publicly-available SED data?
What security measures do you employ to protect the institution data set, both the physical CD and the electronic data files?
Do you use research assistants to help you with your institutional research using SED data or do you do analyze the SED data yourself?
Do the confidentiality assurances provided to SED respondents impose constraints on your uses of SED data? In what ways?
Have you ever ordered or used data from a “quick-turn-around” institution data set?
What types of analyses did you perform with those data?
Have you ever applied for a license to gain access to record-level SED micro-data that include data on doctorate recipients from other institutions?
What is your understanding of the policies and procedures the National Science Foundation (NSF) uses when creating data products for publication (i.e., data tables, figures, and charts) to protect against the disclosure of confidential information provided by SED respondents?
Do the NSF’s disclosure avoidance policies and procedures limit your ability to conduct research using publicly-available SED data?
What disclosure avoidance policies and procedures do you employ before publishing data from your SED institution data set?
Are the disclosure avoidance standards different if the publication’s intended audience is internal to your institution rather than external?
Are you required to get clearance from your institutional review board (IRB) before publishing research utilizing the SED data from your institutional data set?
Have you ever publicly reported the results of research that utilized SED data, such as at a conference presentation or workshop, or in a journal article, campus-wide report, or some other publication?
Must the presentation, article, or report complete a formal clearance process regarding disclosure avoidance in advance?
Do you work with the Institutional Contact for the SED at your institution concerning your use of SED data?
Do you know who your institution’s SED Institutional Contact is?
Do you work with the Dean of the Graduate School (or someone in an equivalent position) concerning your use of SED data?
Does the Graduate Dean direct your research with SED data?
This information is solicited under the authority of the National Science Foundation Act of 1950, as amended. Your responses are protected under the NSF Act and the Privacy Act of 1974, and will be kept confidential. The information you provide will be used by the survey sponsors and your doctoral institution for statistical purposes and selecting samples for carefully defined follow-up studies. Your response is voluntary and failure to provide some or all of the requested information will not in any way adversely affect you. The average time to complete this survey is about 20 minutes. Please send any comments on the time required for this survey to National Science Foundation, 4201 Wilson Blvd., Suite 295, Arlington, VA 22230, Attn: NSF Reports Clearance Officer.
| File Type | application/msword |
| File Title | MEMORANDUM |
| Author | LCHRISTO |
| Last Modified By | mark fiegener |
| File Modified | 2013-09-05 |
| File Created | 2013-09-05 |