3145-REM Supporting Statement A

Supporting Statement for OMB Clearance Request
for the National Science Foundation for the
Research Experiences and Mentoring Supplement
3145-NEW

A. Justification
Introduction
The National Science Foundation (NSF) requests that the Office of Management and
Budget (OMB) approve, under the Paperwork Reduction Act of 1995, a one-time clearance
to evaluate the Research Experiences and Mentoring (REM) supplement. The REM
supplement is available to active NSF Emerging Frontiers in Research and Innovation
(EFRI) research awards and active Engineering Research Center (ERC) awards.
The REM Program supports the active involvement of research participants (high
school students, STEM teachers, undergraduate STEM students, and faculty) in hands-on
research in order to bring participants into contact with suitable STEM mentors and expose
them to this rich research experience. Research participants will be recruited as cohorts in
order to facilitate mentoring and research activities, community building, and provide
mutual student support.
The main goals of the REM Program are to provide research experiences and
mentored opportunities to STEM students and/or educators that may ultimately enhance
their career and academic trajectories while enhancing EFRI- and ERC-supported research.
The REM Program may also enable the building of long-term collaborative partnerships
among EFRI- and ERC-supported researchers, community colleges, local four-year
colleges, and local school districts.
1.

Circumstances Requiring the Collection of Data

The REM supplement was first offered in 2011 for EFRI awardees and 2017 for ERC
awardees. In January 2021, the NSF contracted the Science and Technology Policy
Institute (STPI) to conduct a comprehensive evaluation of the REM supplement. The REM
supplement evaluation is charged with addressing the following broad research questions:

1

1. What is the impact of the REM program on the research participants (RPs) during
and subsequent to their participation in the program?
2. What is the impact of the REM program on mentors?
3. What are the impacts on EFRI and ERC research?
4. To what extent did the REM activities contribute to new partnerships with other
institutions?
To address the research questions related to the impact of the REM program on the
research participants (RPs), which is the primary question for this evaluation, NSF seeks
to collect the primary data via a survey of REM RPs from all supplements funded between
2012 and 2021. Note that there are are multiple RPs per supplement. The survey data is
augmented by the award information provided annually to NSF from the PIs. The annual
reports offer mostly qualitative information on the progress of the research on parent
award, but only offers few specifics about the REM supplement. Without the survey data,
the program will not be able to measure the qualitative impact of the REM program on
their continued interest and progress towards a STEM career.
Regarding the research impact, the annual reports will provide a list of publications,
and the evaluation will attempt to consider the contribution of the RPs through looking at
the bibliometric data.
To address the second and final research questions, the evaluation will examine
administrative data that are collected through the annual reporting process. The evaluation
will attempt to collect educational progress through obtaining Student Clearinghouse data,
but it will only provide quantitative information. Appendix A displays each of the main
research questions and sub-questions and the data sources that will be used to answer those
questions.
Table 1 below summarizes the study’s broad data collection activities and when data
are to be collected.
Table 1. Data Collection Instruments and Schedule
Data Sources

Data Collection Schedule
CY 2021-22

Extant Data Sources
Bibliometric Data for 2012-2021 Awards

X

2

National Student Clearinghouse Data for 2012-2021

X

Awards
NSF Administrative Data for 2012-2021 Awards

X

Survey
Research Participants (all)

X

Specifically, the evaluation study will:

2.

•

Examine all 118 supplements awarded between 2012 and 2021, through an online
survey of research participants; Each supplement has a range of 6-9 RPs per
supplement.

•

Use survey data collection to describe the career decisions of research participants;

•

Use survey data collection to explore the relationships between program and
research participant characteristics;

•

Use administrative data to understand program impacts on mentoring and other
impacts as reported on RPs;

•

Use bibliometric data to understand the extent to which RPs contribute to research

•

Use Clearinghouse data to understand persistence of RPs in STEM
Purposes and Uses of the Data

The primary purpose for collection is program evaluation. The data collected will
allow NSF to describe the program components associated with the supplements, and
research participant career direction and experience receiving mentorship. Describing
short- and long-term impacts of the REM supplement on STEM retention will inform future
REM funding decisions and contribute to the wider NSF discussion on STEM retention.
It is important to recognize that there are already annual program data being collected
about the REM supplement. However, the basic level of information collected through
annual data is not sufficiently comprehensive to describe the characteristics of REM
supplements and the activities supported through the REM program, the types of research
participants involved and their career directions and competencies, and the mentoring in
which research participants receive. The proposed study will also collect information from
recipients about their background information, the activities they participated in and their
perception of the value of those activities. Consequently, the study will conduct a survey
to answer the specified research questions that cannot be answered through existing sources
alone.

3

3.

Use of Information Technology to Reduce Burden

To reduce burden, the survey will be administered online. Survey skip patterns will
automatically move the respondent forward into the next appropriate section, creating less
confusion and simplifying the survey-taking experience. This approach also allows for easy
identification of nonrespondents and facilitates follow-up. All these features facilitate the
reporting process, provide useful and rapid feedback to the data providers, and reduce
burden. Email will be used to send respondents their invitations to complete the survey and
follow-up with the non-respondents to ensure their participation.
4.

Efforts to Identify Duplication

This evaluation does not duplicate other NSF efforts. Program officers of the REM
supplement have indicated there are no other studies that examine the impacts of the REM
supplement.
Program data, as described in the previous section, will be available to the contractor
and the NSF staff working on this research. A survey will be constructed to ask about
elements of program participation that are not captured in the annual reports. Future data
collection tasks likewise will also draw on whatever prior program data exists, thus
preventing duplication.
5.

Small Business
No information for this research will be collected from small businesses.

6.

Consequences of Not Collecting the Information

If this information is not collected, NSF will be unable to document the initial and
longer-term impacts of the REM supplement, nor will it be able to adequately assess
program performance.
7.

Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR
1320.6
The data collections will comply with 5 CFR 1320.6.

8.

Selection of Public Comments and Consultation with People Outside the
Agency
a. Federal Register Announcement

Comments on this data collection effort were solicited in the Federal Register on April
30, 2021 in Volume 86, Number 82. During the first comment period prior to submission
to OMB, no substantial comments were received.

4

b. Consultations Outside the Agency
Consultation on the study design and data sources and needs occurred during the
design phase and will continue throughout the study. The purpose of such consultations is
to ensure the technical soundness of the study and the relevance of its findings, and to
verify the importance, relevance, and accessibility of the information sought in the study.
Consultation was conducted by the Federally Funded Research and Development Center,
The Science and Technology Policy Institute (STPI), contracted by NSF to evaluate the
REM supplement. Senior technical staff from STPI who are conducting the study are listed
below.
STPI

Asha Balakrishnan

202-419-5480

Brian Zuckerman

202-419-5485

Lara Rubinyi

202-419-5465

Logan Pratico

202-419-5423

In addition, the proposed data collection instruments are being pilot tested with
respondents drawn from the target populations. Respondents will be asked to comment on
the clarity and content of the questions. The respondents will be asked overarching
questions regarding:
•

Current studies or work

•

Exposure to research experiences

•

Experience with mentors

•

Communication with mentors and Principal Investigators since the experience
ended

•

Skills gained during the summer

The duration of the data collections will be recorded to help with an accurate
estimation of time burden.
9.

Payments or Gifts to Respondents
No payment or gift will be provided to respondents.

10. Assurance of Confidentiality
Respondents will be advised that any information on specific individuals will be
maintained to protect their confidentiality. Data collected will be available to the evaluation
5

contractor and at the aggregate level to NSF officials and staff. The information requested
may be disclosed to qualified researchers and contractors in order to coordinate programs
and to a Federal agency, court or party in court, or Federal administrative proceeding, if
the government is a party.
Individuals surveyed will be assured that the information they provide will not be
released in any form that identifies them as individuals and their responses will be kept
confidential. The contractor will be expected to maintain the confidentiality, security, and
integrity of survey data. The web-based survey data will be maintained on a secure server
with appropriate levels of password and other types of protection. The contractor will strip
personally identifiable information (PII) from survey results and store the data on secure
servers. In addition, the contractor will allow respondents to quit the survey if they do not
want to finish it.
11. Questions of a Sensitive Nature
The proposed survey asks for demographic information (gender, race, ethnicity, and
veterans’ status) from REM research participants on a voluntary basis, thus respondents
may choose not to provide information that they believe is sensitive in nature. This
information is being collected so that NSF can answer questions about how effective the
REM program has been in the recruitment and preparation of a diverse population of
students. All survey questions will be reviewed by the contractor’s Office of General
Counsel (which handles their human subjects research compliance program) prior to
fielding.
12. Estimates of Response Burden
a. Number of Respondents, Frequency of Response, and Annual Hour Burden
The target population for this study includes individuals who participated in REM
supplement funds between 2012 and 2021. Our study will include a total of 118 REM
supplements. Research participants from 2012-2021 REM supplements (n= 118) will be
surveyed in 2021.
The online survey will be administered to all REM research participants identified.
The survey instrument is included in Appendix B. The annual response burden for survey
is estimated to be 178.20 hours per year for an average of 540 respondents.
Table 2 indicates the number of average respondents to be surveyed and the time
demand the survey will place on each individual respondent.

6

Table 2. Estimated Annual Burden Hours
Time per

Research

Average number

response

Number of

Total time

of respondents

(hours)

responses

burden (hours)

540

.33

540

178

Participant
Survey

b. Hours Burden Estimates and Aggregate Hours Burden
The total response burden for the survey is estimated to be 178.20 hours for 540
respondents. Table 3 indicates the total number of respondents to be surveyed and during
the base study period.
Table 3. Estimated Total Burden Hours
Number

Research

of

Time

per

Number

of

Total

time

respondents

response (hours)

responses

burden (hours)

540

.33

540

178

Participant
Survey

c. Estimates of Annualized Cost to Respondents for the Hour Burdens
The overall annualized cost to the respondent is 7,873.20. Table 4 shows the estimated
total annual costs to each group of respondents over one year for the survey.
Table 4. Estimated Cost to Respondents
Estimated

Burden Time

Research

per

Estimated

Number of

Annual Cost

Hourly Salary

Respondent

Cost to

Respondents

Across All

Estimate*

(hours)

Respondent

in Category

Respondents

43.75

.33

14.58

540

7,873.00

Participant
Survey
7,873.00

Total

7

*Based on Bureau of Labor Statistics, 2016 median hourly wage of an engineer across disciplines; Elka
Torpey, "Engineers: Employment, pay, and outlook," Career Outlook, U.S. Bureau of Labor Statistics,
February 2018.

13. Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs
to Respondents or Record Keepers
There is no overall annual cost burden to respondents that results from this study other
than the time spent responding to the survey attached as an appendix to this request.
14. Estimates of Costs to the Federal Government
The estimated cost to the Federal Government for the data collection activities
included in this request for approval is $80,000. This cost estimate includes instrument
development and pretesting; data collection; and data processing and analysis over a oneyear period. Indirect as well as direct costs are included in these estimates.
15. Changes in Burden
This is a new collection of information.
16. Plans for Publication, Analysis, and Schedule
The purpose of this research is to examine the impact and outcomes of NSF–funded
REM supplements on research participants who received the awards. This will help NSF
respond to such questions as: What is the impact of the REM program on the research
participants (RPs) during and subsequent to their participation in the program? What are
the impacts on EFRI and ERC research?
An analytic evaluation report will be prepared based on study findings. The survey
data will yield important details about how the REM supplement is being implemented
across institutions as well as the paths REM research participants are taking throughout
their career. The study team will also examine variation in program implementation and
participant characteristics. Several survey blocks were adapted from the Undergraduate
Research Student Self-Assessment tool (URSSA) developed by the University of
Colorado, Boulder and validated for use in program evaluations of undergraduate research
programs.1 This survey tool has been used for NSF’s BIO-REU program, and provides a
precedent for the utility of the questions in NSF program evaluations of research
experiences.
Analyses of survey data will include a detailed summary that utilizes appropriate
descriptive statistics. For survey items using continuous scales, the study will calculate
means. Frequency distributions and percentages will be used to summarize answers given
on ordinal scales. The study will also compare differences in responses across different
types of research participants and different demographics.

8

The analyses will examine how program characteristics, including mentoring
provided; and research participant characteristics, including their demographics and type
(i.e. high school teacher, undergraduate, etc.), influence program outcomes, including
retention in STEM. The strength of the observed associations between program and
recipient characteristics and outcomes will be estimated using Cochran’s Q test followed
by a post-hoc McNamar’s test to determine (1) if the percentage of survey respondents who
were in each student/employment status is significantly different from the other
student/employment statuses and (2) if there are significant differences, which ones are
significantly different from each other.
In addition, data from the national Clearinghouse as well as bibliometric data on
research participant publications will be used to triangulate the data and complement selfreports of STEM retention. Before the conclusion of the study the NSF grants funded under
this program may use preliminary data to improve management and performance.
a. Project Time Schedule
Spring 2021

Study Design

Summer/Fall 2021

Prepare online survey

Fall 2021/Winter 2022

Administer online survey

Winter/Spring 2022

Conduct any necessary follow-up with REM
research participants

Spring 2022

Analyze data

Summer 2022

Prepare final report

17. Approval to Not Display Expiration Date
The data collection instruments will display the expiration date.
18. Exceptions to Items 19 of OMB Form 83-I
No exceptions are sought.
19. References
1

Weston, Timothy J., and Sandra L. Laursen. "The undergraduate research student
self-assessment (URSSA): Validation for use in program evaluation." CBE—Life Sciences
Education 14, no. 3 (2015): ar33.

9