REQUEST FOR RENEWAL OF EHR
PROGRAM MONITORING DATA COLLECTIONS
Forms Clearance Package
Submitted by:
National Science Foundation
4201 Wilson Boulevard
Arlington, VA 22230
The National Science Foundation (NSF) is an independent Federal agency that supports research at the frontiers of knowledge, across all fields of science and engineering (S&E) and S&E education (NSF, “Investing in Science, Engineering, and Education for the Nation’s Future,” NSF Strategic Plan for Fiscal Years (FY) 2014-2018). NSF awards grants and cooperative agreements to more than 2,000 colleges, universities, K-12 school systems, businesses, informal science organizations, and other research organizations throughout the U.S.1
NSF provides approximately 24 percent of federal funding for basic research to academic institutions.2 Within NSF, the Directorate for Education and Human Resources (EHR) has primary responsibility for promoting rigor and vitality within the Nation’s science, technology, engineering, and mathematics (STEM) education enterprise to further the development of the 21st century’s STEM workforce and public scientific literacy. In order to support the development of a diverse and well-prepared workforce of scientists, technicians, engineers, mathematicians, and educators and a well-informed citizenry that has access to the tools of science and engineering, EHR’s mission includes identifying means and methods to promote excellence in U.S. STEM education at all levels and in all settings (both formal and informal). To these ends, EHR provides support for research and implementation activities that may improve STEM learning and education from pre-school through postdoctoral studies, in traditional and non-traditional venues, among all United States citizens, permanent residents, and nationals. EHR also focuses on broadening participation in STEM learning and careers, particularly among those individuals traditionally underrepresented and underemployed in the STEM workforce, including but not limited to, women, persons with disabilities, and racial and ethnic minorities.
This request seeks renewal of OMB 3145-0226 for 11 data collections that have similar elements and purposes and will provide essential information for program monitoring purposes. The collections contain items in two categories of programs (e.g., scholarship/fellowship programs and implementation, development, and research programs).
Data collected by EHR program monitoring systems are used for program planning, management, evaluation, and audit purposes. Summaries of monitoring data are used to respond to queries from Congress, the public, NSF’s external merit reviewers who serve as advisors, including Committees of Visitors (COVs), and NSF’s Office of the Inspector General. These data are needed for effective administration, program and project monitoring, evaluation, and for measuring attainment of NSF’s program and strategic goals, as identified by the President’s Accountable Government Initiative, the Government Performance and Results Act (GPRA) Modernization Act of 2010, and NSF’s Strategic Plan.
The 11 program-specific collections included in this request (see attachments A1 through K4) are designed to assist in management of specific programs, divisions, or multi-agency initiatives and to serve as data resources for current and future program evaluations. Of the 11 collections contained in this request, 5 are for collection from remaining projects in legacy programs that are no longer making new awards. Because of changes in program focus and emphasis since this collection was last cleared, these legacy programs have either been replaced by new programs, combined with others, or have ended. In any case, the level of monitoring activity and total burden for the collection have decreased. The 5 legacy programs are identified in the table below. EHR believes it is important to complete the collection of data from currently active projects in those programs to assure that there is a complete data repository from all projects in those programs for use in a future evaluation or research project.
Program |
Type of Program |
Advancing Informal STEM Learning (AISL) Monitoring System |
Implementation, Development, & Research |
Centers of Research Excellence in Science and Technology (CREST) and Historically Black Colleges and Universities Research Infrastructure for Science and Engineering (HBCU-RISE) Monitoring System |
Implementation, Development, & Research |
Graduate STEM Fellows in K-12 Education (GK-12) Monitoring System (PROGRAM ENDED) |
Scholarships and Fellowships |
Integrative Graduate Education and Research Traineeship Program (IGERT) Monitoring System (PROGRAM ENDED) |
Scholarships and Fellowships |
Louis Stokes Alliances for Minority Participation (LSAMP) Monitoring System |
Implementation, Development, & Research; Scholarships and Fellowships |
Louis Stokes Alliances for Minority Participation Bridge to the Doctorate (LSAMP-BD) Monitoring System |
Scholarships and Fellowships |
Robert Noyce Teacher Scholarship Program (Noyce) Monitoring System |
Scholarships and Fellowships |
Research in Disabilities Education (RDE) Monitoring System (PROGRAM ENDED) |
Implementation, Development, & Research; Scholarships and Fellowships |
Scholarships in Science, Technology, Engineering, and Mathematic (S-STEM) Monitoring System |
Scholarships and Fellowships |
Science, Technology, Engineering, and Mathematics Talent Expansion Program (STEP) Monitoring System (PROGRAM ENDED) |
Implementation, Development, & Research |
Transforming Undergraduate Education in Science, Technology, Engineering, and Mathematics (TUES) Monitoring System (PROGRAM ENDED) |
Implementation, Development, & Research |
The programs that have been replaced were replaced with programs that have changed in nature such that monitoring systems for the new programs no longer make sense. In the case of TUES, which was replaced by IUSE (Improving Undergraduate STEM Education), the focus is on research on systemic transformation as well as on classroom learning (in the Engaged Student Learning track).
The NSF Directorate for Education and Human Resources (EHR) is responsible for analyzing and evaluating STEM education and human resource development activities and research in NSF’s Education and Training (E&T) portfolio.
EHR Monitoring Systems Clearance
Since the prior request for this collection, the EHR Evaluation and Monitoring Working Group (EMG) has made changes in the way monitoring information is viewed within EHR and how the information is used. The EMG is still working to determine the proper mix of mechanisms for evaluation and monitoring.
EHR has collaborated on NSF-wide initiatives intended to coordinate evaluation and monitoring efforts across NSF led by the NSF Office of Integrative Activities, including methods and processes for collection of standard data about projects currently limited to annual and final reports as well as new reporting requirements of the RPPR (Research Performance Progress Report).
EHR has led NSF in identifying and utilizing advanced analytic and text mining techniques and processes to improve the use of annual and final project reports for project and program summaries and outcomes.
EHR has developed a number of analytic tools for mining the text of annual and final reports and making institutional and monitoring data more readily available for staff.
EHR has also led an effort to partner with the National Institutes of Health to utilize an advanced thesaurus indexing and retrieval application to dynamically categorize proposal text by topic, which can be used to provide views of NSF proposals and awards in a range of areas defined by thesaurus terms generated by NSF staff.
A joint pilot program between EHR and the NSF Directorate for Engineering in which the NSF Division of Information Systems was to integrate data from monitoring applications from both organizations into the NSF data system as part of a data warehouse was concluded during the period of this collection. Monitoring data from the STEP program have been integrated with NSF corporate data and corporate data management and analysis applications. New data collected from the monitoring system are transferred to the NSF data warehouse annually. The data from that collection are available for generating reports by data warehouse users with access to those specific data sets. This pilot provided valuable lessons about the difficulty of integrating external monitoring data into the corporate data system with the level of validation required of corporate data systems, timely and regular transfer of external data to corporate systems, and the limitations in terms of time and costs associated with such an exercise. The pilot has concluded at this time without further consideration for how other external data sets might be migrated.
The EMG continues to develop processes and policies that address the ways in which monitoring data could be integrated into evaluation throughout NSF and in other federal programs.
Investigate the feasibility of creating a common core of indicators for EHR monitoring data collection systems.
Explore internal systems that allow for more effective access and use of performance data.
Examine the usefulness of emerging systems and strategies for collecting and reporting performance data.
Expand the range of questions in the monitoring collection systems to anticipate future evaluations and include items that might be used to identify effective practice or process outcomes.
The EMG and its members have addressed these issues and questions in a variety of ways including:
Presented workshops across NSF on evaluation and assessment and how to introduce the components into program planning and management, including the use of logic models.
Envisions the potential of informing future monitoring activities grounded in fully developed program logic models and how that would further enhance the design of program evaluations.
EMG members have attended the Innovation Forum sponsored by the Department of Education on best practices for monitoring and evaluation at other agencies.
Issues Addressed In the Initial Collection Request
The initial request that created OMB 3145-0226 addressed the extent to which monitoring data in the collection were used in two ways:
Do monitoring systems collect data needed to assess programs?
These monitoring systems provide data required to assess the progress of projects in each program. The monitoring data also contribute to the overall assessment of program performance.
In the case of programs that are primarily fellowship or scholarship programs, collection of information about participants in those programs is essential to any future tracking of their progress and determination of the impact of participation in the program. As an example, the S-STEM program recently was asked to identify the number of graduate students participating in the program. The source of this information was the monitoring data, without which the program would have been unable to respond in a timely fashion.
The importance of monitoring data is illustrated by the following description of the activities of the S-STEM monitoring data in the recent program management plan:
“The program monitoring system operated by ICF International requests and gathers responses from PIs to a common set of items on a semester/quarterly basis. The items are tailored to the information needs of the program and are strongly aligned with the goals of the program. Monitoring activities are administered through a web-based survey in which Principal Investigators (PIs) report student scholarship recipient demographics and status. For example, the system collects and stores demographic information (e.g., gender, race/ethnicity, scholarship amount, discipline, degree program). At the start of each semester/quarter PIs report the academic status of each unique recipient (e.g., still in school and active in project activities, graduated, left the program) and the types of activities in which the recipient participated. ICF International assists the leadership team in contacting PIs, administering the survey, and following-up with PIs to ensure the collection of the requisite information for each project.
Individual program directors (PDs) monitor compliance with the semester/quarterly reporting requirement for each award on which they serve as the cognizant PD. Each PD has access to the data for the projects they manage. PDs do not sign off on Annual or Final Reports until data compliance is complete and correct. The management team utilizes data from these reports to assess and report on the impact of S-STEM on student recruitment, retention, and graduation in STEM fields.
The monitoring system is the source for both (a) the collection of information to support the documentation of program performance metrics (e.g., number of STEM majors, number and type of support activities, and number of graduates) and (b) the primary and/or secondary data source for the program evaluation and a project’s Third Year Review.”
Programs whose goals are implementation or development require that detailed information about the initial efforts of individual projects be identified in order to track the potential impact of those efforts in successive locations.
The monitoring systems collect project-level information on the scale, scope, and state of each project along with information on types of activities implemented; results, such as publications and number of students and/or faculty involved in the project; and partners. This information is essential for documenting the development, implementation, adaptation, dissemination, and results of supported activities in institutions of higher education and across STEM disciplines. In addition to program management and reporting purposes, the program monitoring system data set has been designed as a primary source of information for a future evaluation of each program.
To what extent is monitoring data used to shape questions for a third-party evaluation?
As noted in the previous clearance request, and reiterated above, EHR relies on the program monitoring data to contribute to and inform third-party evaluations. Without these data, third-party evaluators would be required to collect data about program participants and program projects mainly after awards had been completed rather than during the period of performance of an award.
The EMG has established guidelines for creating logic models and including them in each program solicitation, incorporating appropriate measurements in the management plans of EHR programs which then should subsequently be used in evaluation designs and monitoring data collections.
Circumstances of Data Collection
To fulfill its planning and management responsibilities, and to answer queries from Congress, OMB, and NSF management, EHR needs current and standardized information about projects in NSF’s Education and Training System of Records portfolio. This information is specifically important to support studies and evaluations by EHR, and studies by other NSF organizational units for project monitoring and effective program administration. The information is retained in accordance with the Education and Training System of Records (63 Fed. Reg. 264, 272 January 5, 1998). The Education and Training System of Records has several purposes, including:
Providing a source of information on demographic and educational characteristics and employment plans of participants in NSF-funded educational projects, in compliance with Foundation responsibilities to monitor scientific and technical resources enabling NSF to monitor the effectiveness of NSF-sponsored projects and identify outputs of projects funded under NSF awards for management and for reporting to the Administration and Congress, especially under the GPRA Modernization Act of 2010, 5 U.S.C. 306 and 39 U.S.C. 2801-2805, and under the President’s Accountable Government Initiative, and Performance Improvement Guidance as represented by OMB’s guidance to agencies (M-10-24)
Creating public use files (which contain no personally identifiable information) for research purposes
The data collected under this request are focused on initiative-specific, division-specific, and program-specific quantitative and qualitative data collection activities. Data from these collections are focused on participant demographic detail (particularly for scholarship and fellowship programs) and activities and outputs (i.e., the accomplishments of program grantees (projects) in terms of specific objectives). These descriptive data collections provide essential information for documenting progress toward NSF’s major performance goals, as described in NSF’s Strategic Plan. (The Foundation’s FY 2014-2018 Strategic Plan describes three strategic goals: Transform the Frontiers of Science and Engineering, Stimulate Innovation and Address Societal Needs through Research and Education, and Excel as a Federal Science Agency. See NSF's Strategic Plan.)
The information collected under the this request is required for effective program administration, program and project monitoring, evaluation, and for measuring attainment of NSF’s program and strategic goals as laid out in NSF’s Strategic Plan. This section describes how the data to be collected under the clearance authority will be used for internal program management and administration; as a data source for NSF’s performance assessment activities, including Committees of Visitors and Directorate and Office Advisory Committees (ACs); for documenting the attainment of NSF’s program and strategic goals in accordance with the President’s Accountable Government Initiative and GPRA reporting; and as a foundation for the rigorous research required to evaluate the effectiveness of STEM education programs. For more general information about NSF’s performance assessment activities see NSF Performance Activities.
Program Management and Administration
One of the primary uses of data from the EHR Program Monitoring Clearance is for the general oversight of project and program activities by EHR staff. Because EHR has a limited number of staff members who must monitor hundreds of projects, large-scale data collection is the only way by which program officers can track project activities. The monitoring systems that fall under OMB 3145-0226 allow program officers and other NSF staff to integrate pre-existing data from the NSF administrative data system and newly generated data in a coherent and timely manner, giving them information needed to make adjustments to the program portfolio. This kind of monitoring can lead to corrections by respondents to their project activities, may facilitate changes in program guidelines and/or NSF funding levels to a particular project, and may result in improved benefits to participants in NSF projects.
Data for NSF’s Performance Assessments and Committees of Visitors
Data from the monitoring systems contribute to NSF’s performance assessment activities, and support the larger NSF evaluation model. NSF relies on the judgment of external experts to maintain high standards of program management and to provide advice for continuous improvement of NSF performance. Committees of Visitors (COVs) for divisions or programs meet once every three years. COV reviews provide NSF with external expert judgments in two areas: (1) assessments of the quality and integrity of program operations and program-level technical and managerial matters pertaining to proposal decisions; and (2) comments on how the results generated by awardees have contributed to the attainment of NSF’s mission and strategic outcome goals. Data collected in the monitoring systems are often used in these reviews. For example, the December 2014 GK-12 and IGERT and March 2015 Noyce and S-STEM program COV materials included summary data about program participants and their activities that had been collected via the respective monitoring systems. COV reports are available at http://www.nsf.gov/od/oia/activities/cov/covs.jsp.
One of the recommendations of two COVs completed in 2015 involving all programs in one EHR division, only one of which had a monitoring system (AISL), was that each program should develop a larger set of data from which a future COV might more effectively identify the progress of programs under review.
GPRA Reporting
Another central use of the EHR Program Monitoring Clearance data is to document attainment of NSF’s program and strategic goals and to report on the progress toward each of these goals. NSF’s performance assessment is guided by three elements: the GPRA Modernization Act of 2010, the President’s Accountable Government Initiative, and NSF’s Strategic Plan.
The Foundation’s FY 2014-2018 Strategic Plan describes three strategic goals: Transform the Frontiers of Science and Engineering, Stimulate Innovation and Address Societal Needs through Research and Education, and Excel as a Federal Science Agency. EHR’s portfolio of E&T programs is a critical part of the Foundation’s goals to Stimulate Innovation and Address Societal Needs through Research and Education and Transform the Frontiers of Science and Education. Under the Stimulate Innovation and Address Societal Needs through Research and Education goal specifically, EHR programs contribute to the strategic objectives of “Strengthen[ing] the links between fundamental research and societal needs through investments and partnerships” and “Build[ing] the capacity of the Nation to address societal challenges using a suite of formal, informal, and broadly available STEM education mechanisms” (p. 18). Under the Transform the Frontiers of Science and Engineering goal, EHR programs “Invest in fundamental research to ensure significant continuing advances across science, engineering, and education,” “Integrate education and research to support development of a diverse STEM workforce with cutting-edge capabilities,” and “Provide world-class research infrastructure to enable major scientific advances” (p. 18). Much of the information that enables EHR to report on these developments is derived from the data elements collected in the monitoring systems under OMB 3145-0226. Monitoring systems and the data they collect identified in this request enable the successful reporting and use of these performance assessments, which is essential in meeting GPRA requirements.
A Foundation for Future Evaluations
Finally, a key measure of NSF’s success at achieving its goals is the effectiveness of its STEM education programs. NSF is committed to implementing program evaluation in accordance with the President’s Accountable Government Initiative. While the monitoring systems used to collect data under this collection play a role in this work, it is understood that they are not evaluative studies. NSF does conduct program-level management reviews to ensure that programs are administered properly and in accordance with federal guidelines and agency missions. This is currently one use of data from the EHR monitoring systems.
In guidance from the Director of OMB, M-10-32, the need for rigorous evaluations and the objectives of program evaluations were clearly outlined, including the use of evaluation resources. Because the collection of data contained in these monitoring efforts contributes to the formal evaluation of programs and provides regular measures of program performance by accumulating operating information from each project in the programs included in this request, this guidance is particularly pertinent to this request. In this regard, the OMB guidance provides a rationale for the collections covered under this request and the activities implemented on behalf of the development of this request.
“Improving and coordinating the use of existing evaluation resources. In addition to the voluntary evaluation initiative, agencies should continue to carefully assess, report on, and allocate the base funds and resources that the agencies have for conducting evaluation. Agencies are encouraged to share information beyond what is requested in guidance and consult with OMB’s Resource Management Offices (RMOs) to coordinate and improve the design, implementation, and utilization of evaluations.”
This directive reinforces the need for EHR to engage in an integrative process of collecting information about its programs to improve program evaluation assessment processes.
EHR has encouraged the use of monitoring data in evaluation activities, creating a foundation for the kind of evaluation the President’s Accountable Government Initiative requires of federal agencies. While data collected under this collection were not used to evaluate program effectiveness, some of the data collected contributed to programmatic evaluations. For example, in order to conduct program-level or portfolio-level evaluations, both experimental and quasi-experimental evaluation research studies on STEM education interventions require researchers to identify individual-level and organizational-level or project-level control and treatment groups or comparison groups. NSF-funded contract or grantee researchers and evaluators have used the data to identify control, comparison, or treatment groups for NSF’s E&T portfolio using some of the descriptive data gathered through OMB 3145-0226 to conduct well-designed, rigorous research and portfolio evaluation studies.
A recent evaluation of the Louis Stokes Alliances for Minority Participation – Bridge to the Doctorate (LSAMP-BD) program of the EHR Human Resources Development division cited the use of data from the monitoring systems of both LSAMP and LSAMP-BD:
“This study relied on several data sources, including administrative data received from grantee institutions or submitted through surveys ...from the LSAMP monitoring data system.
The LSAMP monitoring data system is different from the LSAMP-BD data system used to collect data for BD. (For the evaluation) …data from the LSAMP data system was used to provide information needed for the analysis, such as whether students had participated in LSAMP as undergraduates.”3
The ongoing evaluation of the Science, Technology, Engineering, and Mathematics Talent Expansion Program (STEP) notes that:
“The STEP Monitoring System database contains outcome data related to 3 of the 11 evaluation questions…The STEP Monitoring System also contains data on STEP implementation characteristics (e.g. STEP strategies) that (the evaluation) will use as covariates to assess variation in STEP implementation in Substudy.”4
Other ways in which monitoring data might be used in evaluation include:
Creating a universe data set with which to compare and establish representativeness of sample data;
Data with which to verify/assess quality of evaluation data;
Data with which to establish population baseline and/or trend data
All of the collections included under this clearance request use Web-based data collection systems to minimize data duplication and respondent burden. EHR favors Web-based systems because they facilitate respondents’ data entry across computer platforms. One innovative feature of many of the individual Web systems is the thorough reviewing and editing of all submitted data for completeness, validity, and consistency. Editing and validation are performed as data are entered. Most invalid data cannot be entered into the system, and questionable or incomplete entries are called to respondents’ attention before they are submitted to NSF.
EHR Program Monitoring Web-based data collection systems employ user-friendly features such as automated tabulation, data entry with custom controls such as checkboxes, data verification with error messages for easy online correction, standard menus, and predefined charts and graphics. All of these features facilitate the reporting process, provide useful and rapid feedback to the data providers, and reduce burden.
All collections in the EHR Program Monitoring Clearance comply with Section 508, the 1998 amendment to the Federal Rehabilitation Act, which mandates that the electronic and information technology used by federal agencies be made accessible to all people with disabilities.
The EHR Program Monitoring Clearance does not duplicate efforts undertaken by the Foundation, other federal agencies, or other data collection agents. For example, NSF grants require the submission of annual and final project reports in accordance with OMB 3145-0058. Recipients of NSF grants, such as principal investigators (PIs), must create and submit annual and final project reports through Research.gov. Data collected under the EHR Program Monitoring Clearance are unique and not available in either the NSF annual or final reporting system. The introduction of the new annual and final reports based on the Research Performance and Progress Report (RPPR) format has improved the submission of project information, but does not change the need for additional information that monitoring systems provide on a program-specific basis.
Of the 11 collections in the EHR Program Monitoring Clearance, only TUES collects information from small businesses. TUES collects only a small amount of data from small business organizations, with the total small business response burden accounting for less than one percent of the total TUES response burden. Based on current data, fewer than five small businesses are affected by the TUES data collection. Together these businesses hold fewer than five awards in total, and each small business would spend no more than 4 hours responding per award.
Data collected for the EHR Program Monitoring Clearance are used to manage programs, monitor projects, inform project and program evaluations, coordinate with federal and non-federal education partners, provide Congress with information about government-supported activities, and report for GPRA and other requirements. In many cases, the data need to be collected annually to inform the NSF management and evaluation processes. Data collected under the EHR Program Monitoring Clearance can be used by NSF management to document and measure NSF’s success at achieving both Strategic Outcome Goals and internal Annual Performance Goals.
If the information were not collected, NSF would be unable to document the implementation of project activities and outcomes of its programs. It would be unable to meet its accountability requirements or assess the degree to which projects and programs are meeting their goals.
All data collections will comply with 5 CFR 1320.6. All collections under the EHR Program Monitoring Clearance ask respondents for data annually, with the exception of the S-STEM monitoring system (attachments I1 and I2), which asks respondents to submit data each semester/quarter. See attachment I1 for more information on the frequency of this collection.
The notice inviting comments on the EHR Program Monitoring Clearance (OMB 3145-0226) was published in the Federal Register November 10, 2015, Volume 80, Number 217, pages 69701-69702. No comments were received.
When developing collection instruments, EHR routinely consults with research and evaluation experts, PIs, and educators affected by EHR investments. The purpose of these consultations is to assess the relevance, availability, and clarity of items. As suggested by OMB guidelines, these consultations also enable EHR staff to obtain a reliable estimate of the respondent burden generated by new instruments. When a new collection is added or when an existing collection is modified to add new instruments, each instrument is pretested with nine or fewer individuals and revised following debriefings with participating respondents.
For data collections conducted earlier under the EHR Generic Clearance, consultations have included knowledgeable outsiders such as representatives of EHR contractors responsible for technical and evaluation tasks and fellows who work at the Foundation as guests under programs such as the Einstein Fellows Program or the American Association for the Advancement of Science Washington Fellows Program.
To date no payments or gifts have been provided to respondents. There are no plans to provide incentives to respondents because the value of program and project monitoring surveys is of value to the respondents as well as NSF. Program monitoring can be used by projects as a foundation for project-level evaluation.
Respondents are informed that any information on specific individuals is maintained in accordance with the Privacy Act of 1974. Every data collection instrument displays both OMB and Privacy Act notices.
Respondents are told that data collected for the EHR Program Monitoring Clearance are available to NSF officials and staff, evaluation contractors, and the contractors hired to manage the data and data collection software. Data are processed according to federal and state privacy statutes. Detailed procedures followed by EHR for making information available to various categories of users are specified in the Education and Training System of Records (63 Fed. Reg. 264, 272 January 5, 1998). This system limits access to personally identifiable information to authorized users. Data submitted are used in accordance with criteria established by NSF for monitoring research and education grants and in response to Public Law 99-383 and 42 USC 1885c.
The information requested through NSF monitoring systems may be disclosed to qualified researchers and contractors in order to coordinate programs and to a federal agency, court, or party in court or federal administrative proceedings, if the government is a party.
Seven of the proposed collections in the EHR Program Monitoring Clearance request information from respondents, including either name, address, Social Security Number (SSN), date of birth (DOB), and/or grade point average (GPA). These data are collected in order to monitor the award sites and evaluate the success of the award programs. Information of this nature is also used to track recipients of funding and training. For example, in the IGERT survey (attachments D1, D2, and D3), trainees’ SSNs are used as a tracking mechanism to permit follow-up studies that examine the long-term effect of the IGERT program on individuals’ success. However, in the IGERT collection and in all collections that request SSN, SSN is a voluntary field. Responses to all items of a sensitive nature are voluntary. Respondents may choose not to provide information that they deem as privileged, such as SSN, address, or DOB. Any individual-level data that are collected are provided only to program staff and consultants conducting studies using the data as authorized by NSF. Any public reporting of data is in aggregate form.
The table below shows which individual collections include questions of a sensitive nature.
Table 1. Questions of a Sensitive Nature
Attachments |
Collection Title |
Address |
DOB |
GPA |
Name |
SSN |
A1-A4 |
Advancing Informal STEM Learning (AISL) Monitoring System |
X*** |
|
|
X*** |
|
B1-B2 |
Centers of Research Excellence in Science and Technology (CREST) and Historically Black Colleges and Universities Research Infrastructure for Science and Engineering (HBCU-RISE) Monitoring System |
X |
|
|
X |
|
C1-C4 |
Graduate STEM Fellows in K-12 Education (GK-12) Monitoring System |
X |
|
|
X |
X |
D1-D3 |
Integrative Graduate Education and Research Traineeship Program (IGERT) Monitoring System |
X |
|
X* |
X |
X |
E1-E2 |
Louis Stokes Alliances for Minority Participation (LSAMP) Monitoring System |
|
|
X |
X |
X |
F1-F2 |
Louis Stokes Alliances for Minority Participation Bridge to the Doctorate (LSAMP-BD) Monitoring System |
X |
|
X |
X |
X |
G1-G3 |
Robert Noyce Teacher Scholarship Program (Noyce) Monitoring System |
|
X |
X |
X |
|
H1-H2 |
Research in Disabilities Education (RDE) Monitoring System |
X |
X** |
X |
X |
|
I1-I2 |
Scholarships in Science, Technology, Engineering, and Mathematics (S-STEM) Monitoring System |
X |
X |
X |
X |
|
J1-J3 |
Science, Technology, Engineering, and Mathematics Talent Expansion Program (STEP) Monitoring System |
X*** |
|
|
X*** |
|
K1-K4 |
Transforming Undergraduate Education in Science, Technology, Engineering, and Mathematics (TUES) Monitoring System |
|
|
|
X**** |
|
*IGERT does not collect GPAs, but does collect the Graduate Record Exam (GRE) scores of individual trainees.
**RDE collects just the birth year as opposed to the full date of birth.
***AISL and STEP collect names and addresses for PIs/respondents but not for individual students.
****TUES collects names for PIs/data collection personnel but not for individual students.
As shown in appendix A, and in table 2 below, the annual response burden for the 11 collections under OMB 3145-0226 is 57,249 hours (for 7,284 respondents and 7,784 responses). Given the diversity of respondent types, the methods used to arrive at individual collection burden estimates are described in detail in attachments A1 through K1.
Table 2. Respondents, Responses, and Annual Hour Burden
Attachment |
Collection Title |
No. of Respondents |
No. of Responses |
Annual Hour Burden |
A1 |
Advancing Informal STEM Learning (AISL) Monitoring System |
155 |
155 |
1,921 |
B1 |
Centers of Research Excellence in Science and Technology (CREST) and Historically Black Colleges and Universities Research Infrastructure for Science and Engineering (HBCU-RISE) Monitoring System |
40 |
40 |
1,810 |
C1 |
Graduate STEM Fellows in K-12 Education (GK-12) Monitoring System |
1,267 |
1,267 |
3,529 |
D1 |
Integrative Graduate Education and Research Traineeship Program (IGERT) Monitoring System |
3,307 |
3,307 |
12,282 |
E1 |
Louis Stokes Alliances for Minority Participation (LSAMP) Monitoring System |
563 |
563 |
12,949 |
F1 |
Louis Stokes Alliances for Minority Participation Bridge to the Doctorate (LSAMP-BD) Monitoring System |
55 |
55 |
2,090 |
G1 |
Robert Noyce Teacher Scholarship Program (Noyce) Monitoring System |
422 |
422 |
5,908 |
H1 |
Research in Disabilities Education (RDE) Monitoring System |
12 |
12 |
1,368 |
I1 |
Scholarships in Science, Technology, Engineering, and Mathematics (S-STEM) Monitoring System |
500 |
1,000 (500 respondents X 2 responses/yr.) |
6,000 |
J1 |
Science, Technology, Engineering, and Mathematics Talent Expansion Program (STEP) Monitoring System |
277 |
277 |
6,648 |
K1 |
Transforming Undergraduate Education in Science, Technology, Engineering, and Mathematics (TUES) Monitoring System |
686 |
686 |
2,744 |
|
Total |
7,284 |
7,784 |
57,249 |
EHR estimates that possibly one new collection will need to be cleared under the EHR Program Monitoring Clearance during the next three years, dependent on budgetary limitations and Congressional mandates. The overall response burden in any year should not exceed 90,000 hours.
Below is an example that shows how the hour burden was estimated for the CREST monitoring system (attachment B1).
The estimated average number of annual respondents is 40 (30 CREST center PIs/program coordinators and 10 HBCU-RISE award PIs/program coordinators), with an estimated annual response burden of 1,810 hours. The Web-based data collection is an annual activity of the CREST program. The respondents are either PIs or program coordinators. One PI or program coordinator per award completes the questionnaire. The estimated annual hour burden per respondent was determined using the burden information reported by respondents from the last two collection cycles.
The burden estimate is outlined below:
Respondent |
Estimated
Average Annual No. of |
Estimated Average Annual Burden Hours Per Respondent |
Estimated Annual Burden Hour Total |
CREST center PIs/program coordinators |
30 |
55 |
1,650 |
HBCU-RISE award PIs/program coordinators |
10 |
16 |
160 |
Total |
40 |
45.25 |
1,810 |
Details on the burdens of each form can be found in attachments A1 through K1. The table below is an example of how this burden was estimated for the CREST monitoring system (attachment B1):
Form Type |
Respondent Type |
No. of Respondents |
Burden Hours Per Respondent |
Total Burden Hours |
CREST data collection form |
PIs/program coordinators |
40 |
45.25 |
1,810 |
Total |
|
40 |
|
1,810 |
As shown in appendix A, the total annual cost to respondents generated by the 11 ongoing data collections is currently estimated to be $2,008,144. Below is an example of the method used to calculate cost burden for the CREST monitoring system (attachment B1):
The overall annualized cost to the respondents is estimated to be $76,020. The following table shows the annualized estimate of costs to PI/program coordinator respondents, who are generally university professors. This estimated hourly rate is based on a report from the American Association of University Professors, “Annual Report on the Economic Status of the Profession, 2014-15,” Academe, March–April 2015, Survey Report Table 4. According to this report, the average salary across all academic ranks and across all types of doctoral-granting institutions (public, private-independent, religiously affiliated) was $87,838. When divided by the number of standard annual work hours (2,080), this calculates to approximately $42 per hour.
Respondent Type |
No. of Respondents |
Burden Hours Per Respondent |
Average Hourly Rate |
Estimated Annual Cost |
PIs/Program Coordinators |
40 |
45.25 |
$42 |
$76,020 |
Total |
40 |
|
|
$76,020 |
The costs to respondents generated by each data collection are described in attachments A1 through K1.
There is no overall annual cost burden to respondents or record-keepers that results from the EHR Program Monitoring Clearance other than the time spent responding to online questionnaires that are described in specific detail in attachments A1 through K4. It is usual and customary for individuals involved in education and training activities in the United States to keep descriptive records. The information being requested is from records that are maintained as part of normal educational or training practice. Furthermore, the majority of respondents are active or former grantees or participants in programs or projects funded by NSF. In order to receive funding, institutions must follow the instructions in the NSF Grant Proposal Guide (GPG) that is cleared under OMB 3145-0058. The GPG requires that all applicants submit requests for NSF funding and that all active NSF awardees do administrative reporting via FastLane or Research.gov. Thus, PIs, K-12 administrators, faculty members, and college students, who are the primary respondents to the individual data collections within the EHR Program Monitoring Clearance, make use of standard office equipment (e.g., computers), Internet connectivity that is already required as a startup cost and maintenance cost under OMB 3145-0058, and free software (e.g., Netscape or Microsoft Explorer) to respond.
As shown in appendix A, the total annual cost to the Federal government of the 11 ongoing data collections is currently estimated to be $2,342,942. Details of the costs of each collection can be found in appendix A.
Below is an example of the costs to the Federal government from the CREST data collection (attachment B1):
Computing the annualized cost to NSF for the CREST data collection was done by taking the budget for the most recent year and calculating the costs for each of the following operational activities involved in producing, maintaining, and conducting the data collection:
Operational Activities |
Cost Over Three Years |
System Development (includes initial development of the database and Web-based application, and later changes requested by the program, e.g., increased reporting tools, additional validations) |
$159,202 |
System Maintenance, Updates, and Technical Support (system requires updates each year before opening the collection; maintenance is required to keep the system current with technology, e.g., database servers, operating systems) |
$79,601 |
Data Collection Opening and Support (e.g., online and telephone support to respondents and contacting respondents to encourage completion of the questions), Reporting (as defined by HRD), and Followup Activities (e.g., providing data to other consultants) |
$59,798 |
Three-Year Total for All Operational Activities |
$298,601 |
The annualized cost was computed as one-third of the total three-year costs; thus, the annualized cost to NSF for the CREST data collection is $99,534.
More details on the costs of existing collections can be found in attachments A1 through K1.
The current inventory numbers at OMB for the EHR Program Monitoring clearance covers 11 individual collection tasks. The OMB inventory records show a total number of responses of 9,845 and total hours of 62,649.
This renewal includes the same 11 tasks and requests 7,784 responses and 57,249 total hours; details can be found in appendix A. The change in burden is due to shifts in the number of respondents. The chart below shows the changes in burden in the individual tasks:
Table 3. Hour Changes in Task Burdens
Attachment |
Collection Title |
Previously Cleared Burden |
Currently Requested Burden |
Change in Burden |
A |
Advancing Informal STEM Learning (AISL) Monitoring System |
2,047 |
1,921 |
(126) |
B |
Centers of Research Excellence in Science and Technology (CREST) and Historically Black Colleges and Universities Research Infrastructure for Science and Engineering (HBCU-RISE) Monitoring System |
1,374 |
1,810 |
436 |
C |
NSF Graduate STEM Fellows in K-12 Education (GK-12) Monitoring System |
3,941 |
3,529 |
(412) |
D |
Integrative Graduate Education and Research Traineeship Program (IGERT) Monitoring System |
12,156 |
12,282 |
126 |
E |
Louis Stokes Alliances for Minority Participation (LSAMP) Monitoring System |
17,094 |
12,949 |
(4,145) |
F |
Louis Stokes Alliances for Minority Participation Bridge to the Doctorate (LSAMP-BD) Monitoring System |
3,600 |
2,090 |
(1,510) |
G |
Robert Noyce Teacher Scholarship Program (Noyce) Monitoring System |
4,108 |
5,908 |
1,800 |
H |
Research in Disabilities Education (RDE) Monitoring System |
1,439 |
1,368 |
(71) |
I |
Scholarships in Science, Technology, Engineering, and Mathematics (S-STEM) Monitoring System |
6,000 |
6,000 |
0 |
J |
Science, Technology, Engineering, and Mathematics Talent Expansion Program (STEP) Monitoring System |
6,050 |
6,648 |
598 |
K |
Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics (TUES) Monitoring System |
4,840 |
2,744 |
(2,096) |
|
NSF Burden Estimate Total |
62,649 |
57,249 |
(5,400) |
The total change of burden is a decrease of 5,400 hours.
Changes in the hour burden are accompanied by changes in the number of respondents. The table below shows the changes in total number of responses.
Table 4. Hour Changes in Number of Responses
Attachment |
Collection Title |
Previously Cleared No of Responses |
Currently Requested No. of Responses |
Change in No. of Responses |
A |
Advancing Informal STEM Learning (AISL) Monitoring System |
157 |
155 |
(2) |
B |
Centers of Research Excellence in Science and Technology (CREST) and Historically Black Colleges and Universities Research Infrastructure for Science and Engineering (HBCU-RISE) Monitoring System |
37 |
40 |
3 |
C |
NSF Graduate STEM Fellows in K-12 Education (GK-12) Monitoring System |
1,626 |
1,267 |
(359) |
D |
Integrative Graduate Education and Research Traineeship Program (IGERT) Monitoring System |
4,658 |
3,307 |
(1,351) |
E |
Louis Stokes Alliances for Minority Participation (LSAMP) Monitoring System |
518 |
563 |
45 |
F |
Louis Stokes Alliances for Minority Participation Bridge to the Doctorate (LSAMP-BD) Monitoring System |
50 |
55 |
5 |
G |
Robert Noyce Teacher Scholarship Program (Noyce) Monitoring System |
316 |
422 |
106 |
H |
Research in Disabilities Education (RDE) Monitoring System |
31 |
12 |
(19) |
I |
Scholarships in Science, Technology, Engineering, and Mathematics (S-STEM) Monitoring System |
1,000 |
1,000 |
0 |
J |
Science, Technology, Engineering, and Mathematics Talent Expansion Program (STEP) Monitoring System |
242 |
277 |
35 |
K |
Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics (TUES) Monitoring System |
1,210 |
686 |
(524) |
|
NSF Burden Estimate Total |
9,845 |
7,784 |
(2,061) |
The decrease in respondents is due largely to transitions in several programs that are no longer making new awards (i.e., GK-12, IGERT, RDE, STEP, and TUES).
Like many agencies, NSF no longer relies on formal (i.e., traditional) publication methods and publication formats. News media advisories, notices of funding opportunities for colleges and universities, and results from survey collections are all examples of the types of publications that NSF regularly publishes without putting ink to paper.
For content authored by NSF or by a third party at NSF’s request, the agency rarely uses paper to publish the information. NSF publishes most documents electronically only using the agency’s Web site, from requests for proposals to evaluation or statistical reports, using an archive called an On-Line Document System (ODS) or as in the case of reports such as the LSAMP-BD program evaluation cited above, are made available to the public directly from the EHR main Web page, part of the NSF main public Web page.
In addition, NSF runs a Custom News Service, an e-mail and Web-based alert service that sends documents newly published in the ODS (e.g., vacancy announcements, calls for proposals, statistical reports) to subscribers. Subscribers receive electronically those NSF documents of interest and not the agency’s entire publications line.
The other major venue for NSF publications is FastLane. The NSF FastLane system collects and publishes information from NSF’s clients (i.e., applicants for NSF funding) using the Web. When an applicant’s proposal has been funded, that applicant’s name and other key data are published on NSF’s Web site. Each week the FastLane Web site publishes a list of new awards using data gathered from the application process.
Like NSF itself, the scope of publication plans and practices by the OMB 3145-0226 EHR Program Monitoring Clearance has a dual nature. Some individual collections contribute to formal products (e.g., analytical reports) that can be published by NSF’s ODS. Some collections produce only the respondents’ replies that are posted verbatim on the EHR share of the NSF Web site for anyone to download.
Most of what the EHR Program Monitoring Clearance collects, however, is not published as a stand-alone product, because the data are an input to how NSF manages, documents, evaluates, and measures its performance as an agency. NSF’s GPRA Performance Report or an individual division’s annual report to the NSF Director may use information from the collection to report to Congress. This is an annual cycle.
The data collection efforts included under this request are administered by third-party contractors that deliver (1) analytical reports, (2) the raw data from the collections, or (3) both. Third parties are contractually forbidden from publishing results unless NSF has made a specific exception. In short, all products of the collections are the property of NSF. After the products are delivered, NSF determines whether the quality of the products deserves publication verbatim by NSF; i.e., NSF typically is the exclusive publisher of the information collected by the collections. Often it is only after seeing the quality of the information the collection delivers that NSF decides the format (raw or analytical) and manner (in the ODS or simply a page on the NSF Web site) in which to publish.
EHR recurring studies based on monitoring data are requested by program staff and are done to monitor, manage, and communicate with and about the clients funded by NSF’s investment in education and training. In most cases the primary purpose for each recurring study is program management. These studies generate data that enable both NSF and the funded education and training projects to improve management and performance. Typically, recurring studies generate information that NSF uses as inputs to other reports, and therefore EHR cites no specific publication plans other than internal or general use to meet reporting requirements.
EHR uses data from recurring studies to provide information that can be mined for program evaluation purposes, such as identifying best practices in the education of graduate and undergraduate students, or as a baseline for summative evaluation reports.
Not applicable
No exceptions apply.
1 National Science Foundation. (2015). How we work. Retrieved from http://www.nsf.gov/about/how.jsp
2 National Science Foundation. (2015). NSF at a glance. Retrieved from http://www.nsf.gov/about/glance.jsp
3 “National Evaluation of the Louis Stokes Alliances for Minority Participation – Bridge to the Doctorate (LSAMP-BD) Program,” Mathematica Policy Research, Washington, D.C., February 28, 2014, pg. 14.
4 “Science, Technology, Engineering, and Mathematics Talent Expansion Program (STEP) Evaluation,” Information Collection Request, January 2016, DRAFT
Page
| File Type | application/msword |
| File Title | Supporting Statement |
| Author | 21773 |
| Last Modified By | Plimpton, Suzanne H. |
| File Modified | 2016-02-23 |
| File Created | 2016-02-23 |