MEMORANDUM
Date: March 12, 2015
To: Shelly Wilkie Martinez, Desk Officer
Office of Management and Budget
From: John R. Gawalt, Director
National Center for Science & Engineering Statistics
National Science Foundation
Via: Suzanne Plimpton, Clearance Officer
National Science Foundation
Subject: Notification of data collection under generic clearance (3145-0174)
The purpose of this memorandum is to inform you that the National Center for Science and Engineering Statistics (NCSES) of the National Science Foundation (NSF) plans to conduct cognitive interviews for the Survey of Earned Doctorates (SED) under the generic clearance of survey improvement projects (OMB #3145-0174).
The objectives of the cognitive interviews are to evaluate the SED content, identify potential sources of response error and make improvements for the 2017 SED. NSF plans to conduct four rounds of cognitive interviews between February and August 2015 using the 2015 SED questionnaire.
Background
The SED has been conducted annually since 1957. The survey collects data on the number and characteristics of all individuals receiving research doctoral degrees from all accredited US institutions. The results of this annual survey are used to assess characteristics and trends in doctoral education and degrees. The current SED paper survey questionnaire is provided for reference in Attachment 1. Results from previous research indicated several areas where the SED questionnaire might be improved
Purpose
The cognitive interviews will be conducted to identify issues with question content, organization, and ordering. Improvements are expected to reduce respondent burden and time spent on post-data-collection editing to resolve discrepancies. While a few new items (on topics such as joint degree programs and employment expectations of students without a firm postgraduation plan) may be developed and tested as part of this effort, item development is not the main focus. The cognitive interviews will assess:
Respondents’ understanding of terms in the survey
How confident respondents are in their responses
How they remembered the information they provided in response to factual questions
Whether they found a response option that fit their answer
How easy or difficult it is to answer a question
Issues with sensitive questions
Consistency of answers within the questionnaire and in comparison to the expected range of answers
Methodology
General Approach. Four rounds of cognitive interviews will be conducted with up to 25 participants per round for a maximum total of 100 cognitive interviews. Interviewing in rounds will allow an assessment of the effectiveness of a set of proposed instrument changes in one round. The instrument will then be revised as needed, and the revisions will be tested in the next round. The cognitive interview protocol is in Attachment 5. The interviews will be audio-recorded if participant consent is granted.
Think-Aloud Interviews. Up to three rounds of interviews will focus on improvements to question order and wording, introducing or deleting items, response categories, and instructions, as needed. For these rounds of testing, the Think-Aloud approach will be used to have participants verbalize their thoughts as they complete the survey. The interviewer will observe the participants as they think aloud during survey completion and will note any questions or problems with particular items. The interviewer will focus on the items identified to be of specific interest (see Attachment 2) such as the timeline questions, questions on field of study (interdisciplinary/joint degree programs), and questions on plans after graduation. The interviewer will also note any additional issues that arise on other items and will probe where necessary during the think-aloud. In addition, retrospective probing will be used to obtain further detail on survey questions, and to explore any issues such as evidence of response problems, confusing terms, discrepancies across questions or any other issues observed during the interview.
Cognitive Debriefing Interviews. The fourth round of interviews will focus on how well the revisions from the prior rounds worked. In this round, participants will complete the questionnaire independently with minimal interviewer probing. After the participant has finished, the interviewer will use scripted retrospective probing to ensure that revised items are working well. Interviewers will also probe spontaneously as needed to explore other issues that arise during the interview. If needed, the instrument will be revised after this final round based on the findings.
Test Instrument. To avoid confusion of the current research with the ongoing SED, we will use the name “SED Questionnaire Design Study” to refer to the study. We will explain to participants that the current study is research designed to improve the SED and the data collected in the cognitive interview will not be included in the actual SED.
Recruitment. A non-probability sample of current and former doctoral students will be recruited for this effort, specifically those who will be completing their degrees within the next few months as well as recent PhD recipients who received their degrees within the last year.
There are several characteristics of respondents and their doctoral institutions that are of specific interest in the sample recruitment for this study. Participants will be screened to ensure that each criterion listed in Exhibit 1 receives adequate coverage during the cognitive interview testing.
Exhibit 1: Sample Selection Characteristics of Respondents and Doctoral Institutions |
|
Participants will be recruited via two main sources: (1) through the SED Institutional Contacts at institutions in the Chicago and Washington, DC areas (which are near contractor facilities) to identify current doctoral students and (2) through the list of recent completers of the 2013-2014 SED web survey to identify recent doctoral recipients. If enough candidates have not been recruited through Institutional Contacts and the list of recent SED completers, cognitive interview participants will be asked if they know of other current doctoral students and recent graduates who might be interested in participating.
Students Identified through Institutional Contacts. Current doctoral students will be recruited directly from PhD-granting institutions in the Chicago and Washington, DC areas and potentially other areas as the opportunity arises (e.g., contractor interviewers traveling off site for other project commitments will attempt to obtain interviews with doctoral students in other geographic areas). The SED Institutional Contacts will be asked to identify students enrolled in a doctoral program who are expected to complete their degree within 6 to 8 months and to provide information on their characteristics as outlined in Exhibit 1.
Recent SED Respondents. Advantages of including prior SED respondents is that we are assured of having participants with the specific characteristics of interest and who have been out of school for up to a year since they completed their degrees, which is similar to typical SED respondents. For example, we will be able to select respondents whose educational history data reveal discrepancies. Interviewing such respondents would enhance our ability to understand the reasons for the discrepancies and to test potential solutions. When selecting potential respondents from the SED data set, we will extract the variables that correspond to the characteristics in Exhibit 1 and confirm the student’s information during the recruitment and screening process.
Contacting Potential Participants. Initial contact efforts will be made via email; if unsuccessful, attempts to reach them via mail and telephone contact information will follow. Contacted individuals expressing interest in participating in the cognitive interviews will be presented with information about the study (see Attachment 3—E-mail Recruitment Message, Recruitment Script, and Frequently Asked Questions) and will then be asked to participate in a brief screening interview to determine the potential participant’s eligibility for the study (see Attachment 4—Screener Questions). Some of the screening information may be obtained in advance from the Institutional Contacts or the SED survey records. In these cases, we will verify that the information we have received is correct and updated. As the research progresses, the screener may be streamlined to focus only on the characteristics that remain of interest for recruiting.
Schedule
The four rounds of cognitive interviews will be conducted starting in late February and ending in August of 2015. Contacting, recruiting, and scheduling of participants will continue until all four rounds of interviews have been completed.
The cognitive interviews will be conducted using the 2015 SED web version. Based on the literature review and data analysis currently underway, we may add a few new questions to help clarify the information we are currently collecting and also may revise, reorder, or remove selected questions in the instrument. A draft of the proposed protocol is included in Attachment 5 and the statement of informed consent is provided in Attachment 6.
The total estimated burden associated with this study is 186.7 hours. Exhibit 2 provides details about the burden estimates by study activity. The estimated burden for cognitive interview participants is 166.7 hours and for Institutional Contacts, 20 hours.
Exhibit 2: Estimated Burden
Activity |
Number of Individuals |
Average Time (Minutes) |
Total Burden (Minutes) |
Recruitment e-mail |
400 |
5 |
2,000 |
Telephone recruitment and screening |
200 |
10 |
2,000 |
Cognitive interview |
100 |
60 |
6,000 |
Subtotal for Participants |
|
|
10,000 minutes (166.7 hours) |
Initial phone call with ICs |
20 |
15 |
300 |
ICs compile and send list of students |
15 |
60 |
900 |
Subtotal for ICs |
|
|
1,200 minutes (20 hours) |
Total Burden |
|
|
11,200 minutes (186.7 hours) |
Cognitive interview participants will be informed that their participation is voluntary and their responses will be used only for statistical purposes. They will be informed that the interview will be audio-recorded and that the recording will only be used for analysis and report writing. They will also be advised that direct quotes may be used in research papers and professional presentations, but names will never be attributed to anything a participant says and no participant will be identifiable from any quotes that may be used. To protect participant confidentiality, participants will be assigned a unique identifier. The signed consent forms will be kept separately from the interview files in a locked cabinet for the duration of the study and will be destroyed after the final report is completed.
Incentive Payments
Participants will be recruited from institutions in the Chicago and Washington, DC areas. Based on prior challenges we have experienced in recruiting this population, to encourage the participation of a broad group of individuals, cognitive interview participants will receive an incentive of $40. This amount of incentive will assist in compensating participants for these 60-minute interviews and the expected transit time and transportation costs associated with participation in the study.
The contact persons for questions regarding this research are:
Lynn Milan
Project Officer
Survey of Earned Doctorates
National Center for Science & Engineering Statistics
National Science Foundation
703-292-2275
Emilda Rivers
Program Director
Human Resources Statistics Program
National Center for Science & Engineering Statistics
National Science Foundation
703-292-7773
Attachments
1: Survey of Earned Doctorates (SED) Questionnaire (paper form)
2: SED Questionnaire Item Issues
3: Recruitment Materials
4: Screener Questions
5: Cognitive Interview Protocol
6: Participant Consent Form
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Milan, Lynn M. |
| File Modified | 0000-00-00 |
| File Created | 2021-01-29 |