2021 NSCG SupportingStatement_SectionA_21Sep20

This request is for a three-year renewal of the previously approved Office of Management and Budget (OMB) clearance for the National Survey of College Graduates (NSCG). The NSCG has historically served as a valuable source of information on the education and career paths of the nation’s college-educated population. The most recent NSCG was conducted in 2019 (OMB approval number 3145-0141). The current OMB clearance for the NSCG expires 28 February 2022, which does not cover the complete survey cycle for the 2021 NSCG.

For the 2021 NSCG, the following modifications will be implemented:

The production sample size will increase from 147,000 to approximately 164,000 to allow the survey’s estimates to continue to meet the reliability requirements given the increase in sample attrition (see details in Section B.1.);
A non-production bridge panel with a sample size of approximately 5,000 cases will be fielded in parallel to the production sample. This bridge panel provides a mechanism for NCSES to explore modifications to the NSCG core survey questions and quantify the potential impact on key survey estimates (see details in Section B.4);
An attempt will be made to identify accurate contact information for all sample members prior to the start of data collection (i.e., “upfront locating”);
The contact strategy will be modified in two ways:
1. A “prenotice” will be mailed to announce the imminent commencement of the 2021 NSCG, as had been done in cycles prior to the 2019 NSCG,
2. The postal mailings will include response deadlines, based on research findings from the 2019 NSCG (see Section A.8. Contact Strategies Research);
The NSCG questionnaire will include two additions:
1. A checkbox will be added allowing respondents who provide a mobile phone number to opt in to receiving text messages in future NSCG cycles (see details in Section A.8),
2. To understand the impact of the coronavirus pandemic on NSCG measures, we will modify the response options or add follow-up questions for eight existing NSCG questionnaire items; and
Two methodological experiments will be incorporated: one experiment to investigate further enhancements to the adaptive survey design techniques being used and the other to examine the effects of mailing a prenotice about the 2021 NSCG to sample members (see Section B.4. and Appendices H and I).

1. WHY THE COLLECTION IS NECESSARY

The America COMPETES Reauthorization Act of 2010¹ established the National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation (NSF) and directed NCSES to “...collect, acquire, analyze, report, and disseminate statistical data related to the science and engineering enterprise in the United States and other nations that is relevant and useful to practitioners, researchers, policymakers, and the public...” Information obtained through the NSCG is critically important to NCSES’s ability to measure the education and employment of scientists and engineers. Furthermore, the NSCG data along with the NCSES’s Survey of Doctorate Recipients (SDR)² data serve as the nation’s only source of comprehensive information about the size and characteristics of the science and engineering (S&E) workforce.³ These data are solicited under the authority of the NSF Act of 1950,⁴ as amended, and are central to the analysis presented in a pair of congressionally mandated reports published by NSF:

Science and Engineering Indicators⁵
Women, Minorities, and Persons with Disabilities in Science and Engineering.⁶

In addition, the Science and Engineering Equal Opportunities Act of 1980 directs NSF to provide to Congress and the Executive Branch an “accounting and comparison by sex, race, and ethnic group and by discipline, of the participation of women and men in scientific and engineering positions.”⁷ The NSCG and SDR provide much of the information to meet this mandate. The combined data from these two surveys, initially created for the 1993 survey cycle and developed throughout the past two decades, are based on recommendations of the National Research Council’s Committee on National Statistics (CNSTAT) report to NSF.⁸

NSCG Background

The NSCG is a repeated cross-sectional survey conducted to provide data on the nation’s college graduates, particularly those in the S&E workforce. The NSCG samples individuals who are living in the United States, have at least a bachelor’s degree, and are less than 76 years of age. As of 2017, the NSCG fully implemented a four-panel rotating panel design, in which every new panel receives a baseline questionnaire, followed by three biennial follow-up questionnaires before rotating out of the survey. (See Supporting Statement B, Section 1, for further details about the panel design and sampling methods.) Sample members are invited to complete the NSCG questionnaire online, on paper, or by phone.

The NSCG is a unique source for examining various characteristics of college-educated individuals, including occupation, work activities, salary, the relationship of degree field to occupation, and demographic information. This survey provides information on individuals residing in the United States with at least a bachelor’s degree, including those who received degrees only from foreign institutions. The SDR complements these data with information on the population of U.S.-degreed doctoral level scientists and engineers. Collectively, the NSCG and SDR provide comprehensive information on the education and employment of the entire U.S. population of scientists and engineers with at least a bachelor’s degree. The NSCG and SDR are the only available sources of detailed information that support a broad range of policy and research topics on the dynamics of the S&E workforce over time.

The NSCG has a history of seeking survey improvements through methodological experiments, and the 2021 NSCG continues that trend with the inclusion of two studies. (1) Based on the success of past adaptive survey design experiments, the 2021 NSCG will continue to innovate methods for identifying cases for intervention as well as automating those interventions. (2) Due to lower than expected response to the 2019 NSCG, the prenotice mailing, which was eliminated in 2019, will be used again. To investigate the impact of the prenotice, it will be withheld from a small subset of respondents, thus allowing us to measure its effect on response, cost, and locating. (See Supporting Statement A, Section 3, for further information on the adaptive design study and Supporting Statement B, Section 4, for more details about both studies.)

USERS OF THE INFORMATION

The data from the NSCG provide valuable information on careers, training, and educational development of the nation’s college graduate population. These data enable government agencies to assess the scientific and engineering resources available in the United States to business, industry, and academia, and provide a basis for the formulation of the nation's S&E workforce policies. For example, educational institutions can use the NSCG data in establishing and modifying scientific and technical curricula, while various industries can use the information to develop recruitment and remuneration policies.

Policymakers, researchers, and other data users use information from the NSCG and SDR to answer questions about the number, employment, education, and characteristics of the S&E workforce. Because the NSCG and SDR provide up-to-date and nationally representative data, policymakers and researchers use these datasets to address questions on topics such as employment of foreign-born or foreign-degreed scientists and engineers, the transition from higher education to the workforce, diversity in both education and employment, the implications of an aging cohort of scientists and engineers as baby boomers reach retirement age, and long-term trends in the S&E workforce.

Uses for Policy Discussion

Data from NCSES’s surveys are used in policy discussions of the executive and legislative branches of government, the National Science Board, NSF management, the National Academy of Sciences, Engineering, and Medicine, professional associations, and other private and public organizations. Some recent specific examples of the use of NSCG data are:

The National Science Board (NSB) used NSCG and SDR data in its investigation to develop national policies for the S&E workforce;⁹
The U.S. Small Business Administration used NSCG data to investigate differences in STEM entrepreneurship participation between native-born and foreign-born workers;¹⁰
The Committee for Equal Opportunity in Science and Engineering (CEOSE), an advisory committee to NSF and other government agencies, established under 42 U.S.C. §1885c, has been charged by the U.S. Congress with advising NSF in assuring that all individuals are empowered and enabled to participate fully in science, mathematics, engineering and technology. Every two years CEOSE prepares a congressionally mandated report that makes extensive use of NSCG and SDR data to highlight key areas of concerns relating to students, educators, and technical professionals;
The Council of Graduate Schools (CGS) used NSCG data to estimate the potential monetary cost and return on investment of pursuing advanced degrees,¹¹ which is a key element of CGS’s financial education website – www.gradsense.org.

Uses by NCSES

The NSCG data were used extensively in the latest version of the congressionally mandated biennial report Women, Minorities and Persons with Disabilities in Science and Engineering (WMPD), 2019,¹² and NSCG data will be used in the forthcoming WMPD 2021, set for release next year. Similarly, NSCG data were used in the congressionally mandated biennial report The State of U.S. Science and Engineering, 2020,¹³ and Science and Engineering Indicators Thematic Report: Science and Engineering Labor Force.¹⁴

NCSES used NSCG data in recent reports such as:

Characteristics of College Graduates, with a Focus on Veterans, October 2018
Prevalence of Certifications and Licenses among the College-Educated Population in the United States, January 2017
Immigrants’ Growing Presence in the U.S. Science and Engineering Workforce: Education and Employment Characteristics in 2013, September 2015
Characteristics of the College-Educated Population and the Science and Engineering Workforce in the United States, April 2015

All NCSES publications can be accessed on the NCSES website at https://www.nsf.gov/statistics/

Uses by Researchers and Analysts

NCSES makes the data from the NSCG available through published reports, an online data tool, downloadable public-use files, restricted-use licenses, and the Federal Statistical Research Data Centers. The online data tool, available at https://ncsesdata.nsf.gov/sestat/sestat.html, allows users to create customized data tabulations using NSCG data. The NSCG public-use files are available for download through the NCSES data downloads web page at https://ncsesdata.nsf.gov/datadownload/.

Since April 2015, NCSES has distributed nearly 600 copies of the 1993 NSCG public-use files, over 600 copies of the 2003 NSCG public-use files, over 700 copies of the 2010 NSCG public-use files, and over 1,900 copies of the 2013 NSCG public-use files to researchers in government, academia, and professional societies. The 2015 NSCG public-use files have been downloaded over 2,300 times since their release in January 2017, and the 2017 NSCG public-use files have been downloaded over 2,000 times since their release in October 2018. The 2019 NSCG public-use files will be available soon. The NSCG public-use files receive heavy use because they are the only data sets analysts can use to compare the S&E workforce to the general population of college degree holders in the United States.

In addition to users of the public‑use files, there are currently 33 restricted-use licensees with access to data from the NSCG, SDR, and/or National Survey of Recent College Graduates (NSRCG)¹⁵ microdata files under a licensing agreement with NCSES. Plus, the U.S. Census Bureau has approved four research projects to use NSCG data in the Federal Statistical Research Data Centers (FSRDC) and is currently reviewing two more proposals.

Some of the recent research based on the public-use NSCG data and the restricted-use data resulted in papers such as the following:

Witteveen, D., & Attewell, P. (2020). Reconsidering the ‘meritocratic power of a college degree. Research in Social Stratification and Mobility. doi:10.1016/j.rssm.2020.100479
Demirci, M. (2019). Transition of International Science, Technology, Engineering, and Mathematics Students to the U.S. Labor Market: The Role of Visa Policy. Economic Inquiry. doi:10.1111/ecin.12795
Dulani, S., Baney, H., Nguyen, H., & Yan, Y. (2019). Education and Job Match: Revisited. Undergraduate Economic Review, 15(1), Article 20.
Hur, H., & Hawley, J. (2019). Turnover behavior among US government employees. International Review of Administrative Sciences, 002085231882391. doi:10.1177/0020852318823913
Hur, H., Maurer, J. A., & Hawley, J. (2019). The role of education, occupational match on job satisfaction in the behavioral and social science workforce. Human Resource Development Quarterly. doi:10.1002/hrdq.21343
Jiang, X., & Kim, D. (2019). The Price of Being International: Career Outcomes of International Master’s Recipients in U.S. Labor Market. Journal of International Students, 9(3), 732-757. doi:10.32674/jis.v9i3.700
Okahana, H., & Hao, Y. (2019). Are They Worth it? Master's Degrees and Labor Market Outcomes in the STEM Workforce. Innovative Higher Education. doi:10.1007/s10755-019-9455-5
Pyne, J., & Grodsky, E. (2019). Inequality and Opportunity in a Perfect Storm of Graduate Student Debt. Sociology of Education. doi:10.1177/0038040719876245
Rangel, M. A., & Shi, Y. (2019). Early patterns of skill acquisition and immigrants' specialization in STEM careers. Proceedings of the National Academy of Sciences of the United States of America, 116(2), 484-489. doi:10.1073/pnas.1812041116
Wang, Q., Jung, J., Bozeman, B., & Gaughan, M. (2019). Collaboration cosmopolitanism: what are the effects on the “overlooked majority” of scientists and engineers? Higher Education. doi:10.1007/s10734-019-00385-5
Witteveen, D., & Attewell, P. (2019). The Vertical Transfer Penalty among Bachelor’s Degree Graduates. The Journal of Higher Education, 1-26. doi:10.1080/00221546.2019.1609323
Coon, M., & Chi, M. (2018). Visa Wait Times and Future Earnings: Evidence from the National Survey of College Graduates. Journal of Economics, Race, and Policy. doi:10.1007/s41996-018-0024-6
Gesing, P., & Glass, C. (2018). First Generation International Students and the 4Ds Shaping the Future of Global Student Mobility: A Comparative Report Analysis. Journal of Comparative and International Higher Education, 10, 24-27.
Pitt, R. N., & Zhu, L. (2018). The Relationship between College Major Prestige/Status and Post-baccalaureate Outcomes. Sociological Perspectives, 073112141880332. doi:10.1177/0731121418803325

3. USE OF AUTOMATED, ELECTRONIC, MECHANICAL, OR OTHER TECHNOLOGICAL TECHNIQUES

The data for the 2021 NSCG will be collected by the U.S. Census Bureau under an interagency agreement between NCSES and the Census Bureau. The 2021 NSCG data collection will use a multi-mode approach that begins with a mailed invitation to sample persons asking them to complete the survey on the Internet. Nonrespondents will be followed up using a paper questionnaire mailing and computer assisted telephone interviews (CATI). The data will be collected and managed by the Census Bureau using multiple complementary systems including Docuprint, Intelligent Mail Barcoding, Enterprise Internet Solutions, Adaptive Design and Intermittent Data Processing, and the Unified Tracking System. These systems are described below.

Docuprint and Intelligent Mail Barcoding

Invitation letters are produced through an in-house on-demand print process using a Docuprint system which allows personalization and the ability to tailor items to each specific respondent. The letters and questionnaire packets will be tracked using Intelligent Mail Barcoding (IMB). IMB requires separate outgoing and return barcodes to be placed on NSCG envelopes for tracking purposes. Using IMB has the potential to increase the overall efficiency of data collection enabling the collection of detailed tracking information including:

When an outgoing questionnaire or other mail piece reached a respondent's local post office;
When an outgoing mail piece left the post office with a postmaster for delivery;
If the outgoing mail piece was identified as undeliverable-as-addressed (UAA) and is being rerouted for return;
When a return questionnaire reaches a respondent’s local post office; and
When a return questionnaire reaches its destination.

This information will allow the NSCG to put cases on hold while the returned questionnaire is reviewed to determine whether it is a “good complete.” Placing cases on hold will reduce respondent burden by limiting unnecessary contacts. In addition, the IMB tracking will alert the NSCG staff to undeliverable mail pieces while they are still in circulation, allowing the Census Bureau to reduce the NSCG data collection costs by eliminating any future mailings to undeliverable addresses.

Enterprise Internet Solutions and Mobile Optimization

The Enterprise Internet Solutions (EIS) area of the Application Services Division (ASD) at the Census Bureau will host a web-based data collection instrument. Data will be transmitted and processed daily. The web instrument will be hosted on the fully certified and accredited Centurion system (infrastructure, security, and framework). The 2021 NSCG web instrument will be optimized for use on mobile devices, creating a better experience for mobile device users and, thereby, reducing survey breakoffs and the possibility of measurement errors.

Adaptive Design and Intermittent Data Processing

The 2021 NSCG will continue to expand the scope of adaptive design in an effort to attain high‑quality survey estimates in less time and at less cost than traditionally executed survey operations. In 2019, the focus of the adaptive design implementation was on protecting the quality of one key survey estimate while reducing data collection costs. Since then, the Census Bureau has improved the “flow processing” (i.e., intermittent editing, imputation, and weighting of incoming response data), which now allows the survey team to monitor quality measures in near real time throughout data collection.

The 2021 NSCG will build upon the lessons learned in prior rounds to monitor and protect the quality of several key survey estimates while reducing data collection costs. Additionally, the 2021 experiment has a goal of fully automating the adaptive interventions based on predefined survey goals. More detail about the 2021 NSCG adaptive design experiment is provided in Supporting Statement B, Section 4.

Unified Tracking System

In 2021, the NSCG will continue its use of the Census Bureau’s Unified Tracking System (UTS) to assist in various aspects of survey management. Since 2013 the UTS has provided a full contact history report for the NSCG, giving survey managers a single place to view all contacts integrated from all three survey modes along with the outcomes of those contacts. This contact history reporting system enables the examination of contact strategies in various ways. For example, if respondents call in to check on the status of their response, NSCG staff are able to access the respondents’ contact history quickly and easily. In addition, this report provides an easily interpretable audit trail of all contacts, allowing survey managers to immediately verify if NSCG interviewers are following proper contact protocols, particularly when questions or complaints from respondents arise. In 2017, this contact report was enhanced by the integration of the previously mentioned IMB data.

For the 2021 NSCG, the UTS will continue to provide daily updates for R-indicators analysis at the cohort-level, so that survey management can understand how data collection operations affect representativeness. Additionally, the UTS will provide two reports to monitor IMB data. These reports will focus on the difference between the dates provided by the Census Bureau’s National Processing Center (NPC) and IMB-provided dates for survey monitoring purposes. For outgoing mailings, the report will show the lag between the scheduled mail date and when NSCG packages actually enter the mail stream. For incoming mailings, the report will provide the dates when UAAs or return questionnaires enter the IMB system versus when they are checked in at NPC. Both reports will have these data broken down by mailing geographies. These reports help us understand the relationship between when sample persons receive their mail and when they respond to survey requests, thus helping us anticipate response relative to mailout operations.

Finally, a UTS report that documents the interactions of the NSCG sample with the web instrument will be continued for the 2021 NSCG. This report provides information like the number of sample persons that have logged in and with what type of device, statistics about the time spent responding, and whether they logged out or submitted the survey. This report allows such valuable web paradata to be monitored throughout the data collection period.

4. EFFORTS TO IDENTIFY DUPLICATION

Duplication, in the sense of a similar data collection, does not exist. No other data collection captures all components of scientists and engineers in the United States. Data from the Current Population Survey provides occupational estimates but does not collect information on degree field for postsecondary degrees. The American Community Survey (ACS) collects the field of bachelor’s degrees but does not collect detailed information on education history, work activities, and employment characteristics as the NSCG does.

The NSCG and ACS both collect demographic information including gender, race, ethnicity, marital status, citizenship status, and veteran status. This survey content duplication between the ACS and NSCG is necessary because of the confidentiality restrictions placed on the public release of ACS data. Due to these restrictions, it is not possible for NCSES to link the demographic information from the ACS with the detailed education and employment information collected on the NSCG. Because linkage between demographic, education, and employment information is needed for the analyses used in NCSES’s congressionally mandated reports, all this information is collected on the NSCG.

Overlap does exist in the target populations for the NSCG and some of the other NCSES surveys, including the SDR and the planned National Training, Education, and Workforce Survey (NTEWS). As a result, it is expected there will be approximately 400 individuals selected for sample in both the 2021 NSCG and the subsequent SDR. Given the ACS sample design, in combination with the NSCG and NTEWS plans to use different ACS years for sampling purposes, it is expected there will be very few individuals selected for sample in both the 2021 NSCG and the NTEWS.

In the 2013 NSCG survey cycle, the NSCG and SDR survey contractors identified the individuals selected for both surveys, removed the individuals from the NSCG data collection effort, and, at the completion of the SDR data collection effort, used the SDR responses for these individuals to complete the individual’s record on the NSCG data file. This NSCG/SDR deduplication process required the SDR survey contractor to create numerous files containing all SDR sample cases for use by the NSCG survey contractor. Moreover, given file format and processing differences between contractors, the NSCG survey contractor needed to reformat and manually manipulate many of the SDR files to use them in combination with the NSCG files. The NSCG/SDR deduplication process added over a week of staff time to both the NSCG and SDR processing during the 2013 survey cycle.

Information collected on the NSCG questionnaire but not on the SDR includes attainment of certifications and licenses, financial support for education, community college enrollment, and veteran status. Information collected on the NSCG questionnaire but not on the NTEWS includes educational history, financial support for education, and community college enrollment. Because of the content differences, the small number of expected duplicates, and the operational challenges of the deduplication process, NCSES will not deduplicate individuals selected for sample in the NSCG with either the SDR or NTEWS during the 2021 survey cycle.

5. IMPACT ON SMALL ENTITIES

Not applicable. The NSCG collects information from individuals only.

6. CONSEQUENCES OF LESS FREQUENT COLLECTION

The NSCG data are central to the analysis presented in a pair of congressionally mandated reports published by NCSES – Science and Engineering Indicators and Women, Minorities, and Persons with Disabilities in Science and Engineering. Since these reports are published on a biennial schedule, they rely on the availability of updated data on the S&E workforce every two years. Conducting the NSCG on a less frequent basis would prohibit NCSES from meeting its congressional mandate to produce a report that contains an accurate accounting and comparison, by sex, race, and ethnic group and by discipline, of the participation of women and men in scientific and engineering positions. The impact of not being able to meet this congressional mandate is that government, business, industry, and universities would have less recent data to use as a basis for formulating the nation's science and engineering policies.

A less frequent data collection would also impact the quality of the NSCG data. Follow-up surveys every two to three years on the same sampled persons are necessary to track changes in the S&E workforce as there are large movements of individuals into and out of S&E occupations over both business and life cycles. To ensure the availability of current national S&E workforce data, the NSCG has been conducted and coordinated with the SDR on a biennial basis since 1993. The degradation of any component jeopardizes the integrity and value of these surveys to provide comprehensive information on the S&E workforce.

Finally, because the NSCG is a panel survey, conducting the survey less frequently would make it more difficult and costly to locate the sampled persons in follow-up cycles because of the mobility of the U.S. population. The likely impacts would be a higher attrition rate, higher potential for nonresponse bias, and less reliable estimates.

7. SPECIAL CIRCUMSTANCES

Not applicable. This data collection does not require any one of the reporting requirements listed.

CONSULTATIONS OUTSIDE THE AGENCY

Federal Register Announcement

The Federal Register announcement for the 2021 NSCG appeared on 28 April 2020 (85 FR 23537, Document Number 2020-09000, pages 23537-23538, see Appendix C). NCSES received two public comments in response to the announcement. One comment inquired as to whether any substantive changes were planned for the 2021 NSCG compared with the 2019 NSCG. NCSES initially informed the commenter that no substantive changes were planned and that the 2021 NSCG survey instruments would be the same as the 2019 versions, aside from the updated survey reference date. After the decision was made to consider modifying some NSCG questionnaire items to accommodate COVID-19 impacts, NCSES followed up with the commenter to make him aware of the updated plans.

The second comment requested that NCSES include measures of sexual orientation and gender identity (SOGI) on the 2021 NSCG and on other future NCSES surveys. NCSES informed the commenters that it shares their interest in improving federal data collections and providing informative measures for underrepresented segments of the population. As such, research into the feasibility of collecting consistent and reliable SOGI data from individuals is part of NCSES’ research agenda. However, due to time and resource limitations, the planned SOGI research has had to be postponed. Hence, no SOGI measures would be included in the 2021 NSCG.

Consultation Outside the Agency

NCSES has sought the advice and guidance of survey methodologists, statisticians, demographers, researchers, data analysts, and policymakers to examine numerous issues related to the development of the NSCG.

Evaluation of the NCSES Effort to Measure the S&E Workforce Population

The National Academies of Science, Engineering, and Medicine’s Committee on National Statistics (CNSTAT), at the request of NCSES, convened an expert panel to review, assess, and provide guidance on NCSES’s effort to measure the S&E workforce population in the United States. Given the evolving data needs of NCSES stakeholders and the budget climate uncertainty under which NCSES operates, NCSES would like to develop a framework for measuring the S&E workforce that will enable the flexibility to examine emerging issues related to this unique population while at the same time allowing for stability in the estimation of trend data. This framework would provide direction for numerous issues related to measuring the S&E workforce population including content, data sources, survey design and methodology, data collection, data processing, data integration, data dissemination, and data promotion.

At the end of its review, the panel issued a report with findings, recommendations, and priorities for improving the relevance, accuracy, timeliness, and cost-effectiveness of S&E workforce data for the next decade and beyond.¹⁶ In the 2019 NSCG, NCSES pursued Recommendation 5.1 (research into optimal sample member contacts), Recommendation 5.5 (research aimed at expanding the use of adaptive design), and Recommendation 4.2 (evaluating modeling techniques for identifying foreign-trained doctorate holders). In the 2021 NSCG, NCSES will continue to pursue Recommendation 5.5, expanding the use of adaptive design to reduce nonresponse bias and control costs, by implementing an adaptive design experiment focused specifically on protecting the quality of key survey estimates (see Appendix H for details). Furthermore, through the inclusion of a non-production bridge panel (see Section B.4 for details), NCSES will pursue Recommendation 5.2 (evaluating the survey content to ensure the concepts are relevant and remain understandable to respondents, and ensuring that any survey content changes are implemented with careful consideration of the impact on trend data).

Evaluating Administrative Records as an NSCG Sampling Frame Source

The staff at the Census Bureau’s Center for Economic Studies (CES) is continuing research examining the potential use of administrative records as a supplementary NSCG sampling frame source. The current frame, drawn from the ACS, covers the target population well but may omit some college-educated individuals. Additionally, the current use of the ACS as a frame makes the future of the NSCG dependent on future changes to the ACS, and thus identifying potential alternate frame data sources is a prudent course of action. To date, CES has documented the strengths and weaknesses of possible frame sources and has conducted a detailed assessment of the quality and viability of two sources that show considerable promise: The National Student Clearinghouse (NSC) and the Census Bureau’s Master Address File Auxiliary Reference File (MAF-ARF). CES is currently in the process of examining an extract of the NSC data to analyze its coverage and potential usability compared to the ACS and will continue this investigation if new NSC data become available. Early results from this analysis suggest that while most college graduates in the NSC data are covered in the ACS, there is a sizable fraction of linked cases who are college graduates in the NSC but not in the ACS (and hence would be excluded from the NSCG frame). With respect to the MAF-ARF, CES examined whether MAF-ARF contact information could supplement the ACS information for NSCG respondent locating purposes. The MAF-ARF analysis motivated new research on the use of administrative records to reduce survey attrition through the development of residential history files. Ongoing research along this vein is incorporating new data sources from the Internal Revenue Service (IRS), including Form 1040 and Form 1099 information returns, to identify between-wave movers, how response rates vary for movers vs. non-movers, and whether these residential histories might be able to reduce attrition. Additionally, work is ongoing on identifying additional datasets that may be useful in augmenting the NSCG frame, such as IRS Form 1098-T data, and investigating how these data might be acquired and incorporated into NSCG research projects.

Evaluating Administrative Records to Inform Measurement Error Properties of NSCG Data

CES is also continuing research using administrative records and third-party data sources to compare with NSCG data to inform measurement error issues for NSCG survey estimates. The two projects underway are (1) an evaluation of earnings data and employment history data from the Census Bureau’s Longitudinal Employer-Household Dynamic (LEHD) program and (2) an evaluation of demographic data from a variety of data sources including IRS 1040s, IRS 1099s, NSC, and the Census Bureau’s Numident File. The results from this research will inform measurement error discussions and may provide guidance on the feasibility of using administrative records for NSCG survey replacement or supplementation. Additionally, longer term planning is investigating how administrative records and NSCG survey data might be combined to produce new blended data products, describing, for instance, how the income trajectories of college graduates in different STEM subfields evolve over time.

Paradata Analysis

The staff at the Census Bureau’s Demographic Statistical Methods Division (DSMD) began analyzing the NSCG web survey instrument paradata in 2013 and has continued in each cycle since then. This research has led to instrument changes to make the survey more intuitive and less burdensome for respondents. The changes have led to shortened completion times, reduced breakoffs, and fewer respondents moving backward in the instrument (i.e., clicking the Previous button). While not all suggested changes have been implemented, they have informed considerations for a larger instrument redesign. We will continue to evaluate the paradata from the 2021 NSCG to measure how device usage is changing over time and to ensure there are no new issues with the instrument.

Contact Strategies Research

The Census Bureau’s DSMD staff conducted an experiment in the 2019 NSCG cycle to examine the impact of deadlines on survey response. (As part of a 2017 NSCG experiment, a specific deadline was included in the first and second mailings instead of the standard “respond within two weeks” wording that had been used in previous NSCG cycles. However, there were many other changes associated with that test, so the impact of the deadline could not be measured independently.) The 2019 experiment avoided the limitations of the 2017 experiment and tested three treatments in a fully factorial design:

deadline on the envelope and in the letter in weeks 1 and 2,
deadline on the envelope and in the letter in week 12, and
deadline just in the letter in week 12.

These treatment groups were compared to the control group, which had a deadline inside the letter only in weeks 1 and 2. Of particular interest was whether a) an earlier deadline led to earlier response, which would result in less nonresponse follow up and reduced costs, or b) a later deadline would motivate sample cases that had yet to respond. Results showed that the earlier deadline did not result in faster response, but the later deadline did nominally increase the overall response rate by approximately 2.9 percentage points for the new cohort and 1.2 percentage points for the old cohort. Based on these results, the survey materials for the 2021 NSCG will include a deadline.

Another component of the 2017 NSCG contact strategy experiment was limiting the number of CATI non-response follow-up calls. That experiment evaluated an across the board limit of 10 CATI calls per sample case and resulted in lower response rates. Therefore, in the 2019 survey cycle, we focused on the effects of limiting calls that resulted in specific outcomes (e.g., answering machine, “ring no answer”). These limits were programmed into the CATI system and cases were flagged for analysis when they reached one of the limits. No changes were made in the actual calling operations in 2019. However, the resulting data are being evaluated and will continue to be monitored during the 2021 NSCG to determine whether limiting calls based on call outcome can result in cost savings without impacting response rates.

Monitoring Contact Option Developments

The NSCG currently contacts sample cases by mail, phone, and email (when available). However, with declining response rates, particularly among the younger cohort, the NSCG is seeking new ways to reach sample cases. Recent research shows text messaging can help increase response rates when combined with other contact modes, such as email,¹⁷ and younger and non-white individuals are more likely than others to consent to receiving text messages,¹⁸ potentially addressing a hard-to-reach demographic. Hence, the NSCG is pursuing the option of incorporating text messages into its contact strategy in an attempt to reach the younger cohort and to provide an additional contact mode for those who either do not have email addresses or do not check their email regularly.

Although the Census Bureau’s current policy does not allow for sample cases to be contacted via text messages, DSMD is monitoring developments surrounding this issue. Two surveys sponsored by the National Center for Education Statistics (NCES) for which the Census Bureau is the data collection contractor are experimenting with text messages. Additionally, the Census Bureau is currently allowing text messaging for the new Household Pulse Survey, an “emergency data collection” initiated to measure the impacts of COVID-19. While the Household Pulse Survey is not a standard data collection and does not result in the release of official statistics, it does provide an opportunity to measure sample cases’ reactions to text messages and may help inform future policy decisions.

For the 2021 NSCG cycle, we will not text message any sample cases but instead will use the occasion to evaluate how many cases are willing to consent to receiving text messages. We will add an opt-in checkbox (in all data collection modes) near the field where telephone number is collected, which is in accordance with the Census Bureau Policy Office’s standards regarding the NCES surveys. Along with the checkbox, there will be text to the following effect: “I consent to receiving text messages for follow-up purposes only.” The exact verbiage will be crafted to meet Census standards and respondent expectations. To achieve this, we will coordinate with the Policy Office, review the consent literature, and possibly conduct cognitive interviews with college graduates to understand their reactions to different statements. As a result, if the Census Bureau has changed its policy on texting by 2023, the NSCG will be positioned to send text messages in 2023 to the cases that provided their consent in 2021.

Adaptive Design

The 2013 NSCG Terms of Clearance stated that “OMB looks forward to NCSES collaborating actively with the National Center for Education Statistics [NCES] and the Census Bureau on ways to experiment with and apply "responsive design" methods to the NSCG in order to better measure and reduce bias and improve overall survey efficiency.” Since that time, NCSES staff have collaborated with the Census Bureau, NCES, and other agencies to take stock of the progress made in the field of adaptive design, to identify the obstacles that currently exist, and to explore the adaptive design possibilities for the future. Below are some of the latest examples of the outreach and collaboration efforts related to the NSCG’s adaptive design efforts.

In November 2019, the Census Bureau hosted the 6^th International Workshop on Adaptive and Responsive Survey Designs. The focus of the workshop was “From Theory to Practice,” and over two days, contributors and attendees were able to hear from federal statistical agencies, international statistical agencies, and other survey organizations about the current state of adaptive survey design methodologies. The workshop was well attended by federal agencies, including BLS, CDC, Census, NASS, NCES, NCHS, and NCSES. One of the keynote speakers, Dr. Michael R. Elliott, included a high-level discussion of the NSCG due to its importance for generating recent evidence of success in ASD experiments.
In March 2018, the Census Bureau and NCES participated in an invited panel at the Federal Committee on Statistical Methodology. The panel topic was “Nonresponse Bias Studies in the Federal Government.” The panel focused on the state of nonresponse bias studies, suggested necessary improvements, and the ability of various data collection methods, including adaptive design, to assist in reducing nonresponse bias. The NSCG was one of several surveys used for background information on the types of metrics used to evaluate survey quality and the risk of nonresponse bias.
NCSES, Census Bureau, and NCES staff attended meetings of the Adaptive Design Interagency Working Group. This working group, established by the OMB Office of Statistical and Science Policy in 2014, is a collaboration among federal statistical agencies.

Adaptive Design Publications and Presentations Using NSCG Data (2017 – Present)

Publications:

Coffey, S., Reist, B., Miller, P. (2019). Interventions on Call: Dynamic Adaptive Design in the National Survey of College Graduates. Journal of Survey Statistics and Methodology.

Presentations:

Miller, P., Coffey, S., Reist, B. (2018). Implementation of Adaptive and Responsive Design at the U.S. Census Bureau. Invited Seminar at the Responsive Survey Design Education Program. June 2018. Ann Arbor, MI.

Coffey, S., Zotti, A. (2017). Logistic Regression: Practical Examples in Experimental Design. Presentation at the National Center for Education Statistics Monthly Seminar Series. August 2017. Washington, DC.

Coffey, S. (2017). More Information is Better! Where Can We Get It and How Can We Use It? Presentation at the 2017 Joint Statistical Meetings. July 2017. Baltimore, MD.

Coffey, S. (2017). Adaptive Design in the NSCG: Insights from 2015 and Developments for 2017. Presentation at the 2017 AAPOR Conference. May 2017. New Orleans, LA.

Survey Design and Methodology

NCSES has sponsored and collaborated on multiple survey design and methodology research projects in an effort to ensure that the NCSES surveys, including the NSCG, are incorporating best practices for survey design and methodology. NCSES holds ongoing discussions with staff from NCES and the Census Bureau to discuss survey design and methodological issues of interest. In addition, NCSES funds research on survey design and methodological issues. The following provides a listing for some of the ongoing research funded by NCSES related to the NSCG:

To produce more reliable survey estimates, NCSES funded research to examine the most efficient manner to create weights for multiple panel estimation within the NSCG. Jean Opsomer (Westat) and Jay Breidt (Colorado State University) are the principal investigators for this research.
To address improvements to data quality and increases in nonresponse trends, NCSES funded research to examine these issues. Jolene Smyth and Kristen Olson (University of Nebraska – Lincoln) are the principal investigators for this research.

9. PAYMENT OR GIFTS TO RESPONDENTS

The 2010 NSCG and 2013 NSCG included incentive experiments to examine the impact of offering incentives on response, data quality, and cost. The results from the incentive experiments¹⁹^,²⁰ provided NCSES and the Census Bureau with guidance and direction for using incentives in the 2015 NSCG data collection effort. The incentive usage in the 2021 NSCG will follow the procedures used in the 2015, 2017, and 2019 survey cycles.

As was the case in the 2019 NSCG, we plan to offer a $30 prepaid debit card incentive to a subset of highly influential new sample cases at week 1 of the 2021 NSCG data collection effort. “Highly influential” refers to the cases with a large base weight and a low response/locating propensity. The highly influential cases will be identified by a model-based approach using a weighted response influence, which is the product of a sampled case’s base weight and predicted response propensity. We expect to offer $30 debit card incentives to approximately 14,800 of the 74,000 new sample cases included in the 2021 NSCG production sample, and approximately 1,000 of the 5,000 non-production bridge panel sample. The weighted response influence factor is calculated as follows:

, where .

The weighted response influence for a case, , is the product of the log of the base weight, , and the response influence, . The response influence is the inverse of the product of the locating propensity, , and the response propensity, .

In addition, using the findings from the 2013 NSCG incentive conditioning study and following our procedures from the 2015, 2017, and 2019 NSCG, we plan to offer a $30 prepaid debit card incentive to past incentive recipients at week 1 of the 2021 NSCG data collection effort. As a result, we expect to offer $30 debit card incentives to approximately 14,400 of the 90,000 returning sample members.

The $30 incentive amount proposed for use in the 2021 NSCG was chosen based on findings from the 2010 NSCG late-stage incentive experiment targeting hard to enumerate cases that had not responded to the survey after multiple contacts. As part of the 2010 experiment, the hard to enumerate cases were allocated to three treatment groups:

$30 debit card incentive
$20 debit card incentive
No incentive

Other than the use and amount of the debit card incentive, the three treatment groups in the 2010 NSCG late-stage incentive experiment received the same data collection contact strategy. At the conclusion of the experimental period (approximately six weeks), the response rate for the three treatment groups differed significantly. The $30 incentive treatment group had a response rate of 29.5%, the $20 incentive treatment group had a response rate of 24.1%, and the no incentive group had a response rate of 6.4%.

In addition to the increase in the response rate for the hard to enumerate cases that were targeted as part of this experiment, the use of the incentive also had a profound effect on the overall representation of the responding sample. The incentive was successful in obtaining responses from individuals who were demographically different from the set of respondents prior to the incentive stage. This ability to increase the demographic diversity of our responding sample helped decrease the potential for nonresponse bias in our estimates.

10. ASSURANCE OF CONFIDENTIALITY

NCSES and the Census Bureau are committed to protecting the confidentiality of all survey respondents. The NSCG data will be collected in conformance with the Privacy Act of 1974, the NSF Act of 1950, as amended, Title 13, Section 9 of the United States Code, and the Cybersecurity Enhancement Act of 2015. The Census Bureau is conducting the NSCG under the authority of Title 13, Section 8 of the United States Code.

The paper questionnaire cover will include the following confidentiality statement:

The information collected in this questionnaire is solicited under the authority of the National Science Foundation (NSF) Act of 1950, as amended. The U.S. Census Bureau is conducting this survey under the authority of Title 13, Section 8 of the United States Code. The Census Bureau is required by law to keep your information confidential and can use your responses for statistical purposes only. The Census Bureau is not permitted to publicly release your responses in a way that could identify you. Federal law protects your privacy and keeps your answers confidential (Title 13, United States Code, Section 9). Per the Federal Cybersecurity Enhancement Act of 2015, your data are protected from cybersecurity risks through screening of the systems that transmit your data. Your response is voluntary and failure to provide some or all of the requested information will not in any way adversely affect you. Actual time to complete the questionnaire may vary depending on your circumstances but on the average, it will take about 30 minutes. If you have any comments on the time required for this survey, please send them to the Reports Clearance Officer, Office of the General Counsel, National Science Foundation, 2415 Eisenhower Ave., Alexandria, VA 22314.

The cover letters will include additional statements in the Frequently Asked Questions section about the Census Bureau’s Title 13 as the data collection authority and assurances of confidentiality. The Census Bureau will include the same appropriate notices of confidentiality and the voluntary basis of the survey to respondents replying via the web instrument and CATI.

NCSES and the Census Bureau will operate within the guidelines established by the Privacy Act to protect respondents’ privacy and the confidentiality of the data collected. The Privacy Act states that “microdata files prepared for purposes of research and analysis are purged of personal identifiers and are subject to procedural safeguards to assure anonymity.”

The Census Bureau has demonstrated experience in handling sensitive data. Routine procedures will be in place to ensure data confidentiality, including the use of passwords and encrypted identifiers to prevent direct or indirect disclosures of information.

11. JUSTIFICATION FOR ASKING SENSITIVE QUESTIONS

No questions of a sensitive nature are asked in this data collection.

12. BURDEN ESTIMATE

NCSES estimates that it will contact approximately 169,000 sample persons by web, mail, or CATI as part of the 2021 NSCG data collection. Based on questionnaire completion times from last cycle, it is estimated that the 2021 NSCG questionnaire will take, on average, 25 minutes to complete. In fact, for returning sample members and the non-production bridge panel sample the average time will likely be closer to 20 minutes.

The amount of time to complete the questionnaire may vary depending on an individual’s educational history, employment status, past response to the NSCG, and selected survey mode. The questionnaire for new sample members is slightly longer than the two forms for returning sample members because it includes all the demographic questions and questions about one’s educational history. The time to complete the 2019 NSCG web survey ranged from 15.4 minutes for some returning sample members to 23.6 minutes for new sample members, and approximately 85% of respondents completed the web mode. Likewise, CATI interview times during the 2019 NSCG ranged from 29.6 minutes for some returning sample members to 34.3 minutes for new sample members, and about 5% of respondents completed via CATI. It was estimated that all forms of the 2019 NSCG paper questionnaire took about 30 minutes to complete, and roughly 10% of respondents completed the paper form.

NCSES expects the response rate to be 65 to 75 percent. Based on an estimate of approximately 126,750 completed cases (169,000 x .75), the total burden hours for the 2021 NSCG data collection are 52,813 (126,750 x 25 minutes). The total cost to respondents for the 52,813 burden hours is estimated to be $1,599,691. This estimate is based on an estimated median annual salary of $63,000 per NSCG employed respondent.²¹ Assuming a 40-hour workweek and a 52-week salary, this annual salary translates to an hourly salary of $30.29. Over the three-year OMB clearance period, the average annual cost to the public for the 2021 NSCG is estimated to be $533,230.

13. COSTS TO RESPONDENTS

Not applicable. The NSCG collection does not have a recordkeeping requirement that would impose additional costs (e.g., purchases equipment, software, or contract out services) to respondents.

14. COSTS TO THE FEDERAL GOVERNMENT

The total estimated cost to the Government for the 2021 NSCG is approximately $14.4 million, which includes survey cycle costs, and NCSES staff costs to provide oversight of the NSCG and coordination with the SDR. The estimate for survey cycle costs is approximately $13.8 million, which is based on sample size; length of questionnaire; administration; overhead; sample design; mailing; printing; sample person locating; web instrument development; telephone interviewing; incentive payments; data keying and editing; quality control; imputation for missing item responses; weighting and estimating sampling error; file preparation and delivery; and preparation of documentation and final reports. The NCSES staff costs are estimated at $562,500 (based on $150,000 annual salary of 1.5 FTE for 2.5 years). Over the three-year OMB clearance period, the average annual cost to the Government is estimated to be $4,787,500.

15. CHANGES FROM THE PRIOR CYCLE

Although the sample size is increasing from approximately 147,000 to 169,000 between the 2019 and 2021 survey cycles to offset the diminishing response rate among both new sample members and returning sample members, and to account for the non-production bridge panel, the overall burden impact is expected to decrease. The main explanations for the decrease in burden are the lower expected response rate (75% rather than 80%) and the shorter questionnaire completion time (25 minutes rather than 30 minutes) that was calculated using 2019 NSCG paradata.

PLANS FOR TABULATION OR PUBLICATION

NCSES does not plan to use any complex analytical techniques in publications using this data. Normally cross-tabulations of the data are presented in NCSES reports and other data releases.

The time schedule for 2021 data collection and publication is currently estimated as follows:

Data Collection	February 2021 – August 2021
Coding and Data Editing	May 2021 – December 2021
Final Edited/Weighted/Imputed Data File	February 2022
NSCG InfoBrief	Summer 2022
NSCG Public-Use Data File	Summer 2022

EXCEPTION TO DISPLAYING THE OMB EXPIRATION DATE

Not applicable. The OMB Control Number and expiration date will be displayed.

18. EXCEPTION TO THE CERTIFICATION STATEMENT

Not applicable. No exceptions to the certification statement are being sought.

1 Section 505, Pub. L. No. 111-358. See Appendix A.

2 The SDR is a repeated cross-sectional biennial survey that provides demographic and career history information about individuals with a research doctoral degree in a science, engineering, or health field from a U.S. academic institution. For more information, see http://www.nsf.gov/statistics/srvydoctoratework.

3 The S&E workforce includes individuals with degrees or occupations in computer and mathematical sciences, life sciences, physical sciences, social sciences, engineering, health sciences and related fields.

4 See Appendix B.

5 42 U.S. Code § 1863(j)(1)

6 42 U.S. Code § 1885(d)

7 42 U.S. Code § 1885(d)

8 National Research Council, Committee on National Statistics. 1989. Surveying the Nation’s Scientists and Engineers: A Data System for the 1990s. Washington: National Academy Press.

9 https://www.nsf.gov/nsb/publications/2020/nsb202015.pdf, https://www.nsf.gov/pubs/2018/nsb20187/nsb20187.pdf, and http://nsf.gov/nsb/publications/2015/nsb201510.pdf

10 https://www.sba.gov/sites/default/files/advocacy/rs432tot-Immigrant-STEM-Entrepreneurs.pdf

11 http://www.gradsense.org/gradsense/methodology

12 https://ncses.nsf.gov/pubs/nsf19304/

13 https://ncses.nsf.gov/indicators

14 https://ncses.nsf.gov/pubs/nsb20198/

15 Through 2010, the NSRCG complemented the NSCG and SDR data with the inflow of U.S.-degreed bachelor's and master's level scientists and engineers. Beginning in 2013, the NSCG began capturing the bachelor’s and master’s level inflow population and eliminated the need for the NSRCG. As a result, the NSRCG was discontinued after the 2010 survey without any impact on the coverage provided by the NSCG and SDR.

16 https://www.nap.edu/catalog/24968/measuring-the-21st-century-science-and-engineering-workforce-population-evolving

17 Kanticar, K., & Marlar, J. (2017). Novelty of Text Messages as Reminders for Web Surveys: Does it Last? Presented at the 2017 American Association for Public Opinion Research annual meeting. New Orleans, LA, May 18-21, 2017.

De Bruijne, M., & Wijnant, A. (2014). Improving response rates and questionnaire design for mobile web surveys. Public Opinion Quarterly, 78(4), 951-962.

Mavletova, A., & Couper, M. P. (2014). Mobile web survey design: scrolling versus paging, SMS versus e-mail invitations. Journal of Survey Statistics and Methodology, 2(4), 498-518.

18 McGeeney, K., Yan, H. Yanna (2016). Text Message Notifications for Web Surveys. Pew Research Center.

19 Zotti, Allison, “Report for the 2013 National Survey of College Graduates Methodological Research Incentive Timing Experiment,” Census Bureau Memorandum from Reist to Finamore and Rivers, April 15, 2014, draft.

20 Thornton, Thomas, “2013 National Survey of College Graduates (NSCG) Incentive Conditioning Study,” Census Bureau Memorandum from Reist to Finamore and Rivers, April 15, 2014, draft.

21Salary estimates were obtained using data from the 2017 NSCG, because data from the 2019 NSCG were not yet available.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Milan, Lynn M.
File Modified	0000-00-00
File Created	2021-01-13

2021 NSCG SupportingStatement_SectionA_21Sep20