OMB Passback Q&A

Responses to OMB questions on the 2014-16 GSS clearance

 

Q1: Did NCSES explain in advance to OMB and others that the “data collected from newly eligible GSS institutions were not included in the 2011-2013 survey data published…?” (SS p18)

A1: We regret that we did not convey to OMB when the data collected from these new institutions would be incorporated in the final GSS data. Because this was such an extensive and major frame evaluation, we were concerned about the impact on data quality, unit eligibility, and the interpretation of overall trends. We routinely conduct quality assessments before disseminating newly collected information that might impact trends, in addition to pretesting and evaluating prior to data collection.

Upon review of the 2011 GSS data from the newly eligible schools, we decided that more years of data were needed to thoroughly assess the accuracy, eligibility, and stability of the responses. During the 2012-13 GSS cycles, we worked extensively with the new frame schools coordinators to correctly identify the GSS eligible units (e.g., research-oriented, not practitioner degree) and to report complete data on their students, postdocs, and nonfaculty researchers (NFRs). Most new coordinators were not familiar with the GSS or the data items being requested.

In addition to working with the coordinators, we recently completed an analysis which examines the impact of adding the newly eligible institution data on GSS trends (see attached for an executive summary). As can be seen in the summary, the new frame institutions are significantly different on several key data items than the core institutions that form the basis of 30 years of GSS trend data. Also, some of the new schools that were initially deemed eligible in 2011 were subsequently verified to be ineligible.

Based on the assessment over the last 3 years and on the impact study, we are now poised to incorporate data from the new institutions. Thus any new institutions meeting the eligibility requirements in the 2014 cycle will be included in the final 2014 data products. We will highlight the addition of these new frame institutions and their impact on GSS trends. The data and information will be available on the NCSES website at: http://www.nsf.gov/statistics/srvygradpostdoc/.

Q2: Is NCSES committed to publishing those results in each of the 2014-2016 rounds of data collection?

A2: Yes, as aforementioned, NCSES is committed to publishing the data collected from all eligible GSS institutions, including those from the new institutions after each of the 2014-16 cycles.

Q3: The project schedule indicates that the Info Brief and detailed data tables will be released at the end of the year, as in the past.  Does NCSES commit to those releases including the newly eligible institution data and also the NFRs (p 26)?

A3: Yes, we plan to publish the 2014 GSS Info Brief by December 2015, followed by the detailed data tables shortly after release of the Info Brief. Once OMB clearance is received, the 2014 GSS data collection will begin with the Part 1 data submission due date of December 1, 2014, followed by the Part 2 data submission due date of February 28, 2015. The final data closeout takes place on April 30, 2015.

When data editing and imputation steps are complete and the data file becomes available in July 2015, we will draft the data release InfoBrief. The Graduate Students and Postdoctorates in Science and Engineering: Fall 2014 report will include data from all institutions, including the NFR data.

Q4: If institutions report at what is essentially the department level, what is the purpose of the GSS code list? 

A4: The GSS code list is important for many reasons. The most important reason is its use to establish eligibility of the units (departments, programs, research centers, or health care facilities) – if a unit does not fit within the fields listed in the GSS code list, it is not eligible for the survey.

Additionally, the GSS code list enables the comparison of similar units across institutions and across data collection cycles. The GSS code list has been developed over the lifetime of the survey to meet the needs of NSF, NIH, and the GSS data users.

Finally, while there are a fair number of departments at the same level as the GSS code (e.g., Chemistry department with degree programs in Organic Chemistry, Polymer Chemistry, and Chemistry (GSS code 202)), some departments have degree programs that have multiple GSS codes (e.g., Engineering department with degree programs Civil Engineering (GSS code 105), Electrical Engineering (GSS code 106), etc.). The GSS collects and reports data at the GSS code level.

Q5: What evidence does NCSES have that schools use the GSS–CIP crosswalk?

A5: NCSES has substantial evidence that many schools utilize the GSS-CIP crosswalk. In 2013, 426 respondents from 309 schools (mostly coordinators but also some unit respondents) visited the GSS-CIP crosswalk page on the survey website. 104 of these respondents downloaded the GSS-CIP crosswalk. These counts likely understate the use of the GSS-CIP crosswalk, since there were no major changes that required a respondent to download a new crosswalk in 2013 and many respondents have downloaded the crosswalk in prior years.

Additionally, GSS project staff routinely interacts with school coordinators and unit respondents who reference the GSS-CIP crosswalk during the data collection process. Discussions about the GSS-CIP crosswalk are particularly common during Part 1 of the survey when coordinators are updating their lists of GSS-eligible units. Coordinators also regularly refer to the GSS-CIP crosswalk when explaining why they changed a unit’s GSS code or why a significant count change occurred.

The GSS-CIP cross walk is essential for schools that directly upload their data. Because CIP codes are used for IPEDS reporting, the CIP codes exist on many postsecondary institution databases. Institutional respondents use the crosswalk to create programs to identify GSS eligible units and then extract the enrollment counts and other information directly from their systems and upload/deliver the file to the GSS contractor.

Q6: How does the NCSES Taxonomy of Disciplines relate to STEM classification by BLS or other statistical agencies?

A6: Over the past several years, NCSES has been reviewing our internal taxonomies with the following goals: (1) to provide standard taxonomies to enable improved consistency in the collection, analysis, and reporting of data across our different surveys, and (2) to the extent possible, better align our internal taxonomies with commonly used federal statistical agency classifications such as the National Center for Education Statistic’s (NCES) Classification of Instructional Programs (CIP) and OMB statistical standards like the Standard Occupation Classification (SOC).

The Taxonomy of Disciplines (TOD) is being developed by NCSES to improve consistency in the collection, analysis and reporting of data collected in NCSES surveys on fields of study or fields of research. The NCSES TOD uses CIP as its standard reference so it is very closely aligned with CIP. The GSS/CIP crosswalk is used to identify the science and engineering fields that are eligible for the survey. Implementation of the TOD in GSS will result in reclassification of some fields at the GSS code level and a revised GSS/CIP crosswalk.

The CIP/SOC crosswalk provided by National Center for Education Statistics (NCES) could be used to identify which fields of study or research collected using the TOD relate to which SOC occupations, including the STEM occupations.

We had informed OMB that GSS would implement the TOD in the 2014 survey but that will no longer be possible due to delays in finalizing the taxonomy within NCSES. Therefore, the 2014 GSS will continue to use the same GSS code list and GSS/CIP crosswalk as was used in the 2013 survey. Once the TOD is finalized, it will be implemented in the next GSS survey cycle that provides us with enough time to revise the survey instruments and materials and to inform our institutional respondents in advance of the changes.

Q7: How do NCSES’s publications facilitate alignment of subsets of the GSS with other statistical agency definitions?

A7: GSS collects and reports data at the GSS code level. Each GSS code is comprised of one or more CIP codes. A user interested in comparing the GSS data with another statistical agency’s CIP-based data can subset the GSS data by specific science, engineering, and health fields using the GSS/CIP crosswalk.

Assessment of the Impact of Adding Newly Eligible GSS Frame Schools in GSS Estimates

Executive Summary

In 2010, NSF conducted a comprehensive frame evaluation study for the first time since 1979 in the NSF-NIH Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS). The evaluation was to identify any potentially eligible U.S. postsecondary academic institutions that had not previously been surveyed in the GSS.

In 2011, GSS conducted an eligibility screening survey and identified 165 newly eligible “frame” institutions with at least one master’s or doctorate-granting program in science, engineering, or health (SEH). After an initial assessment of the data quality, it was determined that a couple of more years of data were needed for a thorough assessment. GSS staff worked with the new frame institutions over the following two GSS survey cycles to verify the degree program eligibility and to assist them in accurately reporting the needed data.

During the 2012 data collection, 3 institutions were verified and confirmed as ineligible. During the 2013 data collection, 12 more were determined as ineligible along with all 10 for-profit institutions, leaving 140 frame institutions eligible for the 2014 survey.

To better understand the effect of new “frame” data on the GSS trends, the frame data were analyzed to examine any differences in institutional characteristics and characteristics of graduate students, postdocs and nonfaculty researchers between new frame and core institution. A primary focus of the evaluation was to determine the impact of the change on the population estimates. Therefore, we examined the percentage point difference on the GSS data when excluding the new frame institutions (GSS core data) and when including data from the new frame institutions.

A summary of the key findings are listed below. A table summarizing the GSS data for core and new frame institutions is also included.

Key Findings

• Key data differences between core and frame institutions include institutional control, degree programs offered, student enrollment status, demographic characteristics, and primary funding source.

• New frame institutions typically had only a few eligible SEH units and only one in 10 new frame institutions reports postdocs or NFRs. In 2013, the average core institution contained 24.9 eligible units, whereas the average frame institution contained 3.2 eligible units.

• The new frame institutions added 20,772 students to an overall total of 653,782 SEH graduate enrollments, an increase of 3.2% from 633,010 core institution students.

• Including graduate students in new frame institutions led to a 2.0% increase in full-time students and a 6.9% increase in part-time students. The increase in part-time enrollment will create one of the largest trend changes from the addition of the frame institutions, causing a 0.9 percentage point increase in the proportion of total SEH graduate students enrolled part-time.

• Graduate students in new frame institutions were more likely to be part-time students, women, minorities (particularly African-Americans), U.S. citizens or permanent residents, and self-supported than those in core institutions.

• New frame institution units are much more likely than core institution units to be in computer sciences (11.3% vs 4.0%) and psychology (11.3% vs 6.2%) and much less likely to be in engineering and health (6.4% vs 14.7% and 12.0% vs 19.1%, respectively).

• New frame institutions employed small numbers of postdocs and NFRs. In 2013, there were 1,048 postdocs in new frame institutions, accounting for 1.7% of the 62,990 total postdocs. The 343 NFRs in new frame institutions accounted for 1.5% of the 22,808 total NFRs.