The National Science Foundation (NSF) requests that the Office of Management and Budget (OMB) approve, under the Paperwork Reduction Act of 1995, a one-time clearance to evaluate the Research Experiences and Mentoring (REM) supplement. The REM supplement is available to active NSF Emerging Frontiers in Research and Innovation (EFRI) research awards and active Engineering Research Center (ERC) awards.
The REM Program supports the active involvement of research participants (high school students, STEM teachers, undergraduate STEM students, and faculty) in hands-on research in order to bring participants into contact with suitable STEM mentors and expose them to this rich research experience. Research participants will be recruited as cohorts in order to facilitate mentoring and research activities, community building, and provide mutual student support.
The main goals of the REM Program are to provide research experiences and mentored opportunities to STEM students and/or educators that may ultimately enhance their career and academic trajectories while enhancing EFRI- and ERC-supported research. The REM Program may also enable the building of long-term collaborative partnerships among EFRI- and ERC-supported researchers, community colleges, local four-year colleges, and local school districts.
The REM supplement was first offered in 2011 for EFRI awardees and 2017 for ERC awardees. In January 2021, the NSF contracted the Science and Technology Policy Institute (STPI) to conduct a comprehensive evaluation of the REM supplement. The REM supplement evaluation is charged with addressing the following broad research questions:
What is the impact of the REM program on the research participants (RPs) during and subsequent to their participation in the program?
What is the impact of the REM program on mentors?
What are the impacts on EFRI and ERC research?
To what extent did the REM activities contribute to new partnerships with other institutions?
To address the research questions related to the impact of the REM program on the research participants (RPs), which is the primary question for this evaluation, NSF seeks to collect the primary data via a survey of REM RPs from all supplements funded between 2017 and 2021. Note that there are are multiple RPs per supplement. The survey data is augmented by the award information provided annually to NSF from the PIs. The annual reports offer mostly qualitative information on the progress of the research on parent award, but only offers few specifics about the REM supplement. Without the survey data, the program will not be able to measure the qualitative impact of the REM program on their continued interest and progress towards a STEM career.
Regarding the research impact, the annual reports will provide a list of publications, and the evaluation will attempt to consider the contribution of the RPs through looking at the bibliometric data.
To address the second and final research questions, the evaluation will examine administrative data that are collected through the annual reporting process. The evaluation will attempt to collect educational progress through obtaining Student Clearinghouse data, but it will only provide quantitative information. Appendix A displays each of the main research questions and sub-questions and the data sources that will be used to answer those questions.
Table 1 below summarizes the study’s broad data collection activities and when data are to be collected.
Table 1. Data Collection Instruments and Schedule
Data Sources |
Data Collection Schedule |
|---|---|
|
CY 2021-22 |
Extant Data Sources |
|
Bibliometric Data for 2017-2021 Awards |
X |
National Student Clearinghouse Data for 2017-2021 Awards |
X |
NSF Administrative Data for 2017-2021 Awards |
X |
Survey |
|
Research Participants (all) |
X |
Specifically, the evaluation study will:
Examine all 64 supplements awarded between 2017 and 2021, through an online survey of research participants; Each supplement has a range of 6-9 RPs per supplement, for a total of approximately 468 research participants.
Use survey data collection to describe the career decisions of research participants;
Use survey data collection to explore the relationships between program and research participant characteristics;
Use administrative data to understand program impacts on mentoring and other impacts as reported on RPs;
Use bibliometric data to understand the extent to which RPs contribute to research
Use Clearinghouse data to understand persistence of RPs in STEM
The primary purpose for collection is program evaluation. The data collected will allow NSF to describe the program components associated with the supplements, and research participant career direction and experience receiving mentorship. Describing short- and long-term impacts of the REM supplement on STEM retention will inform future REM funding decisions and contribute to the wider NSF discussion on STEM retention.
It is important to recognize that there are already annual program data being collected about the REM supplement. However, the basic level of information collected through annual data is not sufficiently comprehensive to describe the characteristics of REM supplements and the activities supported through the REM program, the types of research participants involved and their career directions and competencies, and the mentoring in which research participants receive. The proposed study will also collect information from recipients about their background information, the activities they participated in and their perception of the value of those activities. Consequently, the study will conduct a survey to answer the specified research questions that cannot be answered through existing sources alone.
To reduce burden, the survey will be administered online. Survey skip patterns will automatically move the respondent forward into the next appropriate section, creating less confusion and simplifying the survey-taking experience. This approach also allows for easy identification of nonrespondents and facilitates follow-up. All these features facilitate the reporting process, provide useful and rapid feedback to the data providers, and reduce burden. Email will be used to send respondents their invitations to complete the survey and follow-up with the non-respondents to ensure their participation.
This evaluation does not duplicate other NSF efforts. Program officers of the REM supplement have indicated there are no other studies that examine the impacts of the REM supplement.
Program data, as described in the previous section, will be available to the contractor and the NSF staff working on this research. A survey will be constructed to ask about elements of program participation that are not captured in the annual reports. Future data collection tasks likewise will also draw on whatever prior program data exists, thus preventing duplication.
No information for this research will be collected from small businesses.
If this information is not collected, NSF will be unable to document the initial and longer-term impacts of the REM supplement, nor will it be able to adequately assess program performance.
The data collections will comply with 5 CFR 1320.6.
Comments on this data collection effort were solicited in the Federal Register on April 30, 2021 in Volume 86, Number 82. During the first comment period prior to submission to OMB, no substantial comments were received.
Consultation on the study design and data sources and needs occurred during the design phase and will continue throughout the study. The purpose of such consultations is to ensure the technical soundness of the study and the relevance of its findings, and to verify the importance, relevance, and accessibility of the information sought in the study. Consultation was conducted by the Federally Funded Research and Development Center, The Science and Technology Policy Institute (STPI), contracted by NSF to evaluate the REM supplement. Senior technical staff from STPI who are conducting the study are listed below.
STPI |
Asha Balakrishnan |
202-419-5480 |
|
Brian Zuckerman |
202-419-5485 |
|
Lara Rubinyi |
202-419-5465 |
|
Logan Pratico |
202-419-3721 |
In addition, the proposed data collection instruments are being pilot tested with respondents drawn from the target populations. Respondents will be asked to comment on the clarity and content of the questions. The respondents will be asked overarching questions regarding:
Current studies or work
Exposure to research experiences
Experience with mentors
Communication with mentors and Principal Investigators since the experience ended
Skills gained during the summer
The duration of the data collections will be recorded to help with an accurate estimation of time burden.
No payment or gift will be provided to respondents.
Respondents will be advised that any information on specific individuals will be maintained to protect their confidentiality. Data collected will be available to the evaluation contractor and at the aggregate level to NSF officials and staff. The information requested may be disclosed to qualified researchers and contractors in order to coordinate programs and to a Federal agency, court or party in court, or Federal administrative proceeding, if the government is a party.
Individuals surveyed will be assured that the information they provide will not be released in any form that identifies them as individuals and their responses will be kept confidential. The contractor will be expected to maintain the confidentiality, security, and integrity of survey data. The web-based survey data will be maintained on a secure server with appropriate levels of password and other types of protection. The contractor will strip personally identifiable information (PII) from survey results and store the data on secure servers. In addition, the contractor will allow respondents to quit the survey if they do not want to finish it.
The proposed survey asks for demographic information (gender, race, ethnicity, and veterans’ status) from REM research participants on a voluntary basis, thus respondents may choose not to provide information that they believe is sensitive in nature. This information is being collected so that NSF can answer questions about how effective the REM program has been in the recruitment and preparation of a diverse population of students. All survey questions will be reviewed by the contractor’s Office of General Counsel (which handles their human subjects research compliance program) prior to fielding.
The target population for this study includes 468 individuals who participated in REM supplement funds between 2017 and 2021. Our study will include a total of 64 REM supplements. Research participants from 2017-2021 REM supplements (n= 64) will be surveyed in 2021.
The online survey will be administered to all REM research participants identified (about 468 participants). The survey instrument is included in Appendix B. The annual response burden for survey is estimated to be 115.83 hours per year for an average of 351 respondents.
Table 2 indicates the number of average respondents to be surveyed and the time demand the survey will place on each individual respondent.
Table 2. Estimated Annual Burden Hours
|
Average number of respondents |
Time per response (hours) |
Number of responses |
Total time burden (hours) |
|---|---|---|---|---|
Research Participant Survey |
351 |
.33 |
351 |
115.83 |
The total response burden for the survey is estimated to be 115.83 hours for 280 respondents. Table 3 indicates the total number of respondents to be surveyed and during the base study period.
Table 3. Estimated Total Burden Hours
|
Number of respondents |
Time per response (hours) |
Number of responses |
Total time burden (hours) |
|---|---|---|---|---|
Research Participant Survey |
351 |
.33 |
351 |
115.83 |
The overall annualized cost to the respondent is 4,082.40. Table 4 shows the estimated total annual costs to each group of respondents over one year for the survey.
Table 4. Estimated Cost to Respondents
|
Hourly Salary Estimate* |
Burden Time per Respondent (hours) |
Estimated Cost per Respondent |
Number of Respondents in Category |
Estimated Annual Cost Across All Respondents |
|---|---|---|---|---|---|
Research Participant Survey |
43.75 |
.33 |
14.58 |
351 |
5,118.00 |
*Based on Bureau of Labor Statistics, 2016 median hourly wage of an engineer across disciplines; Elka Torpey, "Engineers: Employment, pay, and outlook," Career Outlook, U.S. Bureau of Labor Statistics, February 2018.
There is no overall annual cost burden to respondents that results from this study other than the time spent responding to the survey attached in Appendix B to this request.
The estimated cost to the Federal Government for the data collection activities included in this request for approval is $80,000. This cost estimate includes instrument development and pretesting; data collection; and data processing and analysis over a one-year period. Indirect as well as direct costs are included in these estimates.
This is a new collection of information.
The purpose of this research is to examine the impact and outcomes of NSF–funded REM supplements on research participants who received the awards. This will help NSF respond to such questions as: What is the impact of the REM program on the research participants (RPs) during and subsequent to their participation in the program? What are the impacts on EFRI and ERC research?
An analytic evaluation report will be prepared based on study findings. The survey data will yield important details about how the REM supplement is being implemented across institutions as well as the paths REM research participants are taking throughout their career. The study team will also examine variation in program implementation and participant characteristics. Several survey blocks were adapted from the Undergraduate Research Student Self-Assessment tool (URSSA) developed by the University of Colorado, Boulder and validated for use in program evaluations of undergraduate research programs.1 This survey tool has been used for NSF’s BIO-REU program, and provides a precedent for the utility of the questions in NSF program evaluations of research experiences.
Analyses of survey data will include a detailed summary that utilizes appropriate descriptive statistics. For survey items using continuous scales, the study will calculate means. Frequency distributions and percentages will be used to summarize answers given on ordinal scales. The study will also compare differences in responses across different types of research participants and different demographics.
The analyses will examine how program characteristics, including mentoring provided; and research participant characteristics, including their demographics and type (i.e. high school teacher, undergraduate, etc.), influence program outcomes, including retention in STEM. The strength of the observed associations between program and recipient characteristics and outcomes will be estimated using Cochran’s Q test followed by a post-hoc McNamar’s test to determine (1) if the percentage of survey respondents who were in each student/employment status is significantly different from the other student/employment statuses and (2) if there are significant differences, which ones are significantly different from each other.
In addition, data from the national Clearinghouse as well as bibliometric data on research participant publications will be used to triangulate the data and complement self-reports of STEM retention. Before the conclusion of the study the NSF grants funded under this program may use preliminary data to improve management and performance.
Spring 2021 |
Study Design |
Summer/Fall 2021 |
Prepare online survey |
Fall 2021/Winter 2022 |
Administer online survey |
Winter/Spring 2022 |
Conduct any necessary follow-up with REM research participants |
Spring 2022 |
Analyze data |
Summer 2022 |
Prepare final report |
The data collection instruments will display the expiration date.
No exceptions are sought.
Weston, Timothy J., and Sandra L. Laursen. "The undergraduate research student self-assessment (URSSA): Validation for use in program evaluation." CBE—Life Sciences Education 14, no. 3 (2015): ar33.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Plimpton, Suzanne H. |
| File Modified | 0000-00-00 |
| File Created | 2024-11-21 |