May 17, 2013 Memo requesting approval for methodological research

Memorandum

Date: May 17, 2013

To: Shelly Martinez, Desk Officer

Office of Management and Budget

From: John R. Gawalt, Director

National Center for Science and Engineering Statistics

National Science Foundation

Via: Suzanne Plimpton, Reports Clearance Officer

National Science Foundation

Subject: Request for Approval of Methodological Research on Reporting of Non-faculty Researchers in the NSF-NIH Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS) (3145-0062)

The National Science Foundation requests approval of methodological research study to examine the quality of data on non-faculty doctorate researchers reported by the GSS institutions. This research will be carried out under the clearance for the Survey of Graduate Students and Postdoctorates in Science and Engineering (3145-0062).

Background

The GSS is an annual survey that is designed to collect data about graduate students, postdoctorate scholars (postdocs), and other non-faculty doctorate researchers (NFRs). The target population for the GSS is defined as U.S. academic institutions that offer graduate degree-credit programs in the sciences and engineering (as defined by NSF) and in health-related fields (as defined by NIH) in the U.S.

In 2010, the GSS expanded collection of the data items on postdocs and NFRs. (See Attachment 1 for copies of the 2009 and 2010 question items on NFRs). Prior to 2010, questions about NFRs were limited to a total number, and details about the gender of NFRs and the number who also had a professional degree (e.g., MD, DO, DDS, or DVM). Starting in 2010, the level of detail collected about postdocs and NFRs increased to match the level of detail collected about graduate students. As part of this postdoc data expansion effort, the NFR question was collected separately from the postdocs.

At the same time, university presidents were asked to consider appointing a second GSS coordinator who could focus on postdocs and NFRs, since NSF was concerned that school coordinators were more knowledgeable about graduate students. The letters to institution presidents also mentioned how the GSS NFR data have been used in the past, including its use as one component of the Carnegie Classification’s measure of research activity for doctorate-granting institutions.

As a result of these changes, the number of NFRs reported by GSS institutions increased by about 50%, from 14,059 in 2009 to 21,145 in 2010. The number of NFRs reported in 2011 was 21,972; the figure for 2012 is currently unavailable, as the survey is currently in the field.

Purpose

The purpose of the proposed research study is to assess the quality of the data provided by institutions on the NFR items in the survey and to better understand the changes in the NFR reporting by contacting and debriefing the GSS institutional respondents.

The postdoc section of the survey was greatly expanded in 2010, and significant effort was made to ensure that appropriate personnel at the GSS institutions were providing postdoc and NFR data. New emphasis on the importance of postdoc data and a request to the institutions to consider appointing a postdoc coordinator may have contributed to an increase of about 10% in the number of postdocs reported and an increase of about 50% in the number of NFRs reported, from 2009 to 2010.

Proposed Methodology

Via the web, GSS coordinators will be contacted and asked to complete a short questionnaire about their 2009 -2011 data. For purposes of this study, we will classify the GSS institutions into four groups:

• Institutions that reported a higher number of NFRs than postdocs in 2010

• Institutions that reported at least 15% more NFRs in 2010 compared to 2009

• All other institutions that reported NFRs in 2010

• Institutions that reported 0 NFRs in 2010

Results of the debriefing questions will be analyzed to determine the type of research staff school and postdoc coordinators are reporting as NFRs and whether there are any systematic errors in their NFR reporting. If errors are found, the study will expand to other schools in that group and will be corrected as part of the 2012 GSS data collection.

Purposive Selection of Schools

The 2010 GSS included 574 institutions. Of these, 234 are ineligible for the study, since they did not report any postdocs or NFRs. The remaining 340 institutions reported NFRs only, postdocs only, or a mix of both. Table 1 presents information on the number of institutions in each category.

Table 1. Number of GSS institutions with postdocs and/or NFRs
	Number of Institutions	Percent of Institutions
With PDs and NFRs	256	44.6
With NFRs only	14	2.4
With PDs only	70	12.2
No PDs and no NFRs (ineligible for this study)	234	40.8
Total	574	100.0

The 340 institutions eligible for the study will be grouped according to patterns of NFR reporting in 2010. The number of institutions in each group is shown below in Table 2.

Table 2. Number of institutions to be selected for study by pattern of reporting NFRs
GSS Institutions	Number Eligible Institutions	Number of Institutions to Select
Institutions reporting NFRs
Number of NFRs is greater than or equal to number of postdocs	48	30
Number of NFRs increased by more than 15% in 2010	118	30
All others	104	30
Institutions reporting no NFR but reporting postdocs	70	30
Total	340	120

Institutions will be selected from within each group purposively so that they include institutions that upload data from central databases, have postdoc coordinators who report NFR data, or have unit respondents who were delegated to report NFR data. Institutions that can be included in more than one category will be eligible for selection only once, however, they will be asked all questions across categories that are relevant to their institution.

Debriefing Survey Instrument

The proposed debriefing questions are included as Attachment 2. To limit respondent burden, items will be tailored for each group of respondents. For institutions that use unit respondents, a single unit respondent will be selected and will be asked about the NFRs in that unit; references to “institution” will be changed to “unit” for this group.

Contacts with Selected Coordinators

Selected coordinators will be sent an email from the NSF GSS Survey Manager, asking them to respond to the debriefing survey via the link provided in the email. The draft email is provided in Attachment 3. Coordinators will be given four weeks to complete the survey; a reminder email will be sent one week before the deadline, then telephone reminders and follow-ups will be made.

Schedule

The tentative schedule for this methodological work is as follows:

Proposed Date	Activity or Deliverable
May 13, 2013	OMB submission for approval
June 3, 2013	OMB clearance
June 17, 2013	Finalize debriefing survey web instrument
June 24, 2013	Send emails to selected coordinators
July 8, 2013	Begin phone prompts, answer questions
July 22, 2013	End data collection
August 12, 2013	Preliminary report delivered to NSF
August 30, 2013	Final report available to NSF

Response Burden

We estimate approximately 24 burden hours will be required for this research study:

• 2 minutes per institution for recruiting (2min x 120 institutions = 4 hours).

• 10 minute debriefing survey for approximately 30 coordinators selected for each of the 4 categories, or 120 respondents in total.

This estimate is covered by the current GSS OMB clearance (3145-0062), which includes 360 burden hours for future testing needs.   

Contact Person

Kelly Kang (kkang@nsf.gov, 703-292-7796)

Human Resources Statistics Program

National Center for Science and Engineering Statistics

National Science Foundation

5