Request for Approval of Non-Substantive Changes to the NSF-NIH Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS)

SRS MEMO

Date: September 28, 2010

To: Shelly Martinez, Office of Management and Budget

From: Kelly Kang, Human Resources Statistics, Division of Science Resource Statistics, National Science Foundation

Subject: Request for Approval of Non-Substantive Changes to the NSF-NIH Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS)

The National Science Foundation requests approval of a non-substantive change to the GSS survey. For the 2010 cycle, NSF proposes adding items regarding postdoctoral appointments that were piloted in a subsample of schools during the 2009 data collection. All but two of the proposed postdoc items are substantively similar to the items asked about graduate students, including questions on race/ethnicity, and sources and mechanisms of financial support. The exceptions are a question about the unit’s definition of a postdoc position and a question on the origin of the doctoral degree, neither of which is applicable for graduate students.

A description of the pilot study that was conducted to test these questions follows, along with a description of the proposed changes.

Results of 2009 Postdoc Pilot Study

The purpose of the Postdoc Pilot Study was to improve the procedures currently used in the GSS to collect information on postdocs. The specific objectives of the Postdoc Pilot included the following:

• Objective 1. Determine the best methods for identifying the person at a school who is most knowledgeable about postdocs and nonfaculty researchers (NFRs)

• Objective 2. Identify the common characteristics of a postdoc position across institutions and develop the best operational definition of a postdoc position

• Objective 3. Determine whether schools can reliably report additional data about postdocs beyond what is currently collected on the GSS, e.g. race/ethnicity, detailed funding, and origin of doctoral degree

• Objective 4. Implement methods to reduce underreporting of postdoc data in the GSS.

A detailed set of postdoc questions was developed during two rounds of cognitive and usability testing. A sample of 74 institutions participated in the pilot, completing a version of the GSS web survey that contained the new postdoc items. In all, there were 870 reporting units within the participating schools. (A copy of the question that has historically been asked about postdocs can be found in Appendix A; screenshots of the 2010 postdoc items can be found in Appendix B.)

The results of the pilot study suggest that, for the most part, schools can answer the new postdoc questions. The demographic items (Question A) appear to be the easiest to complete and have the least amount of item nonresponse; 96.7 percent of reporting units were able to supply these data. The financial support items (Question B) were answered by 90.8 percent of reporting units. On doctoral degree type items (Questions C1 and C2), it was difficult to distinguish between true zeros and missing values given the format of the questions in the pilot study, which was similar to the format historically used in the GSS for such questions. Respondents may leave a cell blank rather than typing a zero. These two items are being modified from the pilot format to a format similar to doctoral degree origin item (Question C3) to ensure the ability to distinguish between true zeros and missing data.

The results of the pilot also suggest that providing the option of appointing a separate postdoc coordinator was a valuable addition. Exactly half of the 74 pilot schools chose to designate a postdoc coordinator. From the debriefing interviews, Student Coordinators and Postdoc Coordinators seemed pleased with the arrangements. None of the schools that had designated a postdoc coordinator asked to change back to having only a school coordinator.

Postdoc coordinators had higher response rates (97.3%) compared to school coordinators (86.5%) to the postdoc section. Having a postdoc coordinator may also result in more comprehensive coverage of postdocs in an institution. For small schools (those with less than 10 units reporting postdocs) designating a postdoc coordinator was associated with a larger net increase in units per school reported to have postdocs (2.5) compared to a net increase of about 0.5 per school if postdocs were reported by school coordinators. Similarly, small schools with postdoc coordinators reported 30% more postdocs than in the previous year compared to a 12% increase for school coordinators.

Proposed Changes to the GSS 2010 Instrument

For the 2010 data collection NSF plans to implement the questions piloted in 2009 with the exception of two minor modifications listed below.

• For question B “Financial Support”, what had been line N in the pilot, “Unknown, not stated, or personal resources” has been split into two separate categories, line N: “Personal resources” and line O: “Unknown or not stated.”

• The format of doctoral degree type questions C1 and C2 will be changed to appear similar to C3.

In advance of the survey, RTI (the contractor conducting the GSS) will send a letter to the presidents of participating schools advising them of the additional questions regarding postdocs and asking them whether or not they would like to appoint a separate postdoc coordinator for their school.

Appendix A shows the screen shot for the information gathered on postdocs in the 2008 and 2009 GSS.

Appendix B shows the proposed screen shots for the 2010 postdoc data collection. Minor changes remain to be made to questions C1 and C2 to change the format to resemble Question C3 so that the categories will sum to the total.

Appendix A – Screenshot of postdoc questions in the 2009 GSS

Appendix B – Screenshots of proposed postdoc questions in the 2010 GSS