Memo - Notification of Information Collection Under Generic Clearance

Memorandum

June 25, 2009

To: Shelly Martinez, Desk Officer

Office of Management and Budget

From: Lynda T. Carlson, Division Director

Division of Science Resources Statistics

Via: Suzanne Plimpton, Reports Clearance Officer, National Science Foundation

Subject: Notification of information collection under generic clearance

The purpose of this memorandum is to inform you that the National Science Foundation (NSF) Division of Science Resources Statistics (SRS) plans to conduct a split panel pretest of a questionnaire under the generic clearance for survey improvement projects (OMB number 3145-0174) for the Postdoc Data Project (PDP).

Background

To better understand how postdoctoral (postdoc) experiences relate to education and labor force outcomes, NSF/SRS is conducting the Postdoc Data Project (PDP). The PDP is a multi-year multi-phased project with the overarching goal of determining the feasibility of gathering in-depth information on postdocs in the U.S and to improve data for the scientific community. SRS currently collects limited data on postdocs through the Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS), the Survey of Earned Doctorates (SED), the Survey of Doctorate Recipients (SDR), and the National Survey of College Graduates (NSCG).

During Phase I of the PDP, which is now completed, SRS identified and confirmed the need to fill several postdoc data gaps in current SRS surveys. The coverage gaps are the foreign degreed and MD/PhD degreed postdocs and postdocs in nonacademic establishments. The subject-matter content gaps are related to education and career goals/expectations, quality and experience indicators, career transitions and outcomes of postdocs.

During Phase 2, now in progress, SRS has assessed various options for filling the coverage and content gaps. To fill the coverage gaps, SRS is using quick turnaround list assessment surveys to determine which lists are the most comprehensive in identifying foreign degreed and MD/PhD postdocs. Through previous PDP activities, SRS has learned that filling these gaps requires expanding the target population beyond postdocs to early career researchers, defined as those receiving doctorates or equivalent degrees within the last ten years. The primary reason for this expansion is to reduce the difficulties that organizations have in producing a list of postdocs given the inconsistencies in postdoc definitions within and across organizations. An added benefit of the expansion is that the resulting dataset will allow analysts to examine the data according to various postdoc definitions and to compare data for postdocs and nonpostdocs. To fill the content gaps, SRS has developed a draft questionnaire based on feedback from the SRS Postdoc Advisory Group, postdocs and their funders and employers, policy stakeholders, and data users. Many of the items on the draft questionnaire are from other surveys conducted by both government agencies and nongovernmental organizations (e.g. SED, SDR, National Postdoctoral Association). These surveys furnished tested questions and valuable research information used for the development of this NSF/SRS survey.

SRS has conducted the following tasks to develop a postdoc questionnaire:

• Conducted and maintained a literature review to stay abreast of key topics and issues of importance to the postdoc community;

• Obtained feedback from the policy community, Congress, SRS staff, methodologists, statisticians, and project managers; researchers including postdocs: and the larger postdoc community to identify relevant issues for inclusion in the questionnaire and priorities for questionnaire topics; and

• Conducted focus groups with foreign-degreed and U.S. degreed postdocs to learn more about priorities and specific question design issues.

Pretest Design

The goals of the pretest are to (a) develop a questionnaire that covers topics salient to the general population of early career researchers as well as to foreign-degreed postdocs and MD/PhD postdocs; (b) include enough sample cases to conduct a split-panel test of the questions identified in early qualitative efforts to be problematic; and (c) identify areas on which to focus for the PDP Phase 3 questionnaire design work.

The questionnaire pretest is designed as a quick-turnaround web-based survey, with the core sections of the questionnaire targeted at 20 minutes in length, and the total interview targeted at 30 minutes in length. The contractor for the PDP Phase 2, the Survey Sciences Group, LLC, will conduct the survey.

Contact Strategy

SRS anticipates the need to conduct a small tracking effort to locate/update contact information. Past experiences indicated that many individuals hold onto their email addresses for longer periods of time than telephone numbers or street addresses; therefore, as needed, our initial tracking efforts will be by mail and by email (see attachments 1 and 2). Once the contractor has the appropriate contact information, each sample member will be assigned a unique identification number to enter the web-based survey. SRS will invite selected sample members to the study initially with a postal mail letter (see attachment 3) and then by email contact (see attachment 4), provided that such contact information is available for the individual sample members. Within one week of the email invitation, an email reminder (see attachment 5) will be sent to all sample members who have not yet responded. Up to two email reminders will be sent during the course of the study (see attachment 6 for the final reminder email). The administration of the pretest survey will take place during late summer 2009.

Questionnaire

The questionnaire covers a number of broad topics: educational background; first position after PhD; current employment status; professional activities; postdoc definitions; characteristics of current employer; compensation; decision to take the current position; expectations for current position; evaluation of current position; work-life balance; mentoring; future career plans; and demographics (see attachment 7). Many of the topics and associated questionnaire items come from current SRS surveys. The topics are prioritized as: (a) a set of core sections that are administered to all participants; and (b) a set of modules that touch on secondary topics of interest that will be administered to subgroups of respondents or at periodic data collection points. In the pretest, we will administer all sections to the participants to maximize the information gathered at this developmental stage and use the results to inform a core versus modular design.

Imbedded within the questionnaire are questions specifically for a split-panel test and others with randomized response options. The questions for the split-panel test are in the following sections: (1) work activities of the current position; (2) employment sector; and (3) salary. The sections containing questions with randomized response options are: (1) postdoc definitions and (2) decision to take the current position. This design will assist SRS in determining best approaches to asking questions. Further information follows.

Work Activities of the Current Position

In the SDR, there is the following question, “The next question is about activities performed in your current position. Which of the following activities occupies at least 10 percent of your time during a typical work week?” This question has 14 response options including the response option “other.” In the SED this same question has only 5 response categories. SRS is testing both versions and will evaluate whether the SED question with fewer response categories performs better than the SDR with 14 response categories for this population.

Employment Sector

Both the SED and the SDR ask the respondent what employment sector they work in but have historically asked the question in different ways. In the 2008 SDR, the employment sector question closely mirrors the SED format. However, the main difference is that the SED includes the education sector in the question whereas the SDR queries about education in a later question. Analysis of the SDR data indicates that respondents have difficulties answering how educational institutions are classified. For example, respondents who report that they work at a particular university sometimes indicate that they work at a state government agency; sometimes they indicate they are self-employed; etc. In addition, the PDP has received feedback that it is difficult to choose the sector when the academic institution serves a dual function for educating and conducting research. Based on the PDP findings and the analysis of the SDR data, SRS will test two approaches. In one approach, we combined features of the SED and SDR sector questions into one question. In a second approach, we ask about sector first and then business or primary function next.

Salary on the Current Position

For the salary question, we plan to test a recommendation from an earlier SED methodological finding that the best approach to collecting salary data is to first present an open-ended question and then follow up with a categorical question for nonrespondents to the open ended item.

Postdoc Definitions

Postdoc definitions vary widely within and across organizations that fund and employ early career researchers. To capture the widest array of information on how postdocs are defined, we will ask the respondent to indicate which requirements from a series of definitional elements pertain to their current position. This approach will provide the analyst with options for defining a postdoc and examining the resulting data.

Decision to Take the Current Position

The results from previous PDP methodological efforts indicated that early career researchers have many reasons for taking their current position. To determine which reasons really are most germane and not a factor of fatigue from having such a long list, we are randomizing the response options.

Sample

SRS will select a purposive sample of 1,000 from the target population, early career researchers. This design is to ensure that minimally we represent foreign-degreed and MD/PhD degreed postdocs and that we have sufficient sample to conduct the split-panel test for both of these populations. SRS has learned from the quick turnaround list assessment surveys that some lists have strong potential for filling one or more gaps in the target population. We plan to acquire lists that include but are not limited to the sources listed below.

• NSF administrative grant database. The previous quick turnaround list assessment surveys (LAS) conducted with a few of the early career researchers indicated that the source was viable.

• NIH administrative grant database. Based on the success of the use of the NSF grants database LAS, NIH agreed to provide list of postdocs on training grants and on individual fellowship grants.

• National Research Council (NRC) researcher placement list. The NRC recruits and places postdocs in government agencies and government-contracted laboratories. This list would expand the ability to reach postdocs in non-academic settings as well as capture foreign degreed postdocs.

• National Postdoctoral Association (NPA) membership list. The NPA list would capture foreign degreed postdocs and postdocs working in non-academic settings.

• American Societies for Experimental Biology (FASEB) membership list. In addition to foreign-PhD degreed and foreign born, this list contains postdocs working in non-academic settings.

• American Medical Colleges (AAMC) membership list. This list potentially captures all MD/PhD degreed postdocs.

At the sample design stage, the contractor will conduct an un-duplicating effort to identify and exclude any SDR respondents and individuals who may have memberships on more than one list. This administrative file matching will be conducted to remove duplicates from the pretest sample.

Confidentiality

At the outset of the interview, respondents will be informed of their privacy and confidentiality rights, including the right to decline participation altogether and the right to refuse any individual question item in the interview. The paragraphs below convey the confidentiality information that will be provided to all respondents:

This information is solicited under the authority of the National Science Foundation Act of 1950 as amended, and the Confidential Information Protection and Statistical Efficiency Act of 2002. The information that you provide will be used for statistical purposes only. Your responses will be kept confidential. Your response is voluntary and failure to provide some or all of the requested information will not in any way adversely affect you.

The average time to complete this survey is about 30 minutes. Please send any comments on the time required to complete this survey to the National Science Foundation, 4201 Wilson Blvd., Suite 295, Arlington VA 22230, Attn: NSF Reports Clearance Officer.

Burden Information

The sample includes 1,000 individuals. We expect that the interviews will be no more than 25-30 minutes in length. Thus, the estimated total maximum burden is 500 hours (1000 interviews x .5 hours per interview).

Incentive Payments

None.

Contact Information

The contact person for questions regarding this research is:

Emilda B. Rivers

Division of Science Resources Statistics

National Science Foundation

4201 Wilson Boulevard, Suite 965

Arlington, VA 22230

703-292-7773

erivers@nsf.gov

Attachments:

• Attachment 1_ First Contact — PreNotification Letter

• Attachment 2_ Second Contact –Tracking PreNotification Email

• Attachment 3_ Third Contact – Letter Invite

• Attachment 4_ Fourth Contact – Email Invite

• Attachment 5_ Fifth Contact – Reminder Email_1

• Attachment 6_ Sixth Contact – Reminder Email_2

• Attachment 7_ Survey of Early Career Researchers

5