3145-REM Supporting Statement B

B. Collection of Information Employing Statistical Methods
1.

Respondent Universe and Sampling Methods

a. Survey Samples
The survey data collection process will include research participants from all 118
REM supplements funded between 2012 and 2021. Respondents include high school
students, STEM teachers, undergraduate STEM students, and faculty.
b. REM Research Participant Sample
The study will field one wave of survey data collection of REM research participants.
(See Appendix B for REM Research Participant survey). The wave will be conducted in
fall 2021 and will include all high school students, STEM teachers, undergraduate STEM
students, and faculty that received REM funding in years 2012 through 2021.
The study sample will include the census of research participants across each of the
four types of respondents described above. The rationale for this strategy is based on
several related factors. First, there are different numbers of recipients within each of the
possible recipient categories, and NSF is interested in learning about all types of recipients.
Second, some research participants who may have participated in the first cohort started
their REM activities up to seven years before the first round of data collection in 2021, the
study may not be able to locate and include all of these individuals in the study. In fact, it
is quite possible that it will be easier to locate and recruit REM research participants who
entered the program more recently, particularly, in the last two or three years. Third,
experience from the prior studies indicate that obtaining high response rates may be
challenging.
Taken together, these factors suggest that to obtain a sufficient number of survey
responses to compute estimates with reasonable precision (i.e. small standard errors,
modest confidence intervals), the most conservative approach would be to include the
census as our potential respondent sample.
2.

Procedures for the Collection of Information

The study team will field an online survey for research participants. The study team
will obtain the list of names and e-mail addresses of research participants from annual
reports and contact with the principle investigators. The study team first notified principal
investigators about the study and then provided them with a list of their awards’ research
participants extracted from the annual reports. This was done in order to ascertain whether
the complete list of research participants had been collected, and to collect updated contact
information for these recipients. This approach was used in the previous NSF evaluations,
and it yielded substantial, albeit not always fully current contact information.
10

Once relevant approvals are obtained, STPI will program the research participant
survey for online data collection in QualtricsGov. The study team will test the survey
system to ensure functionality and accuracy of data capture. STPI plans to launch survey
data collection in the early fall 2021. Given the academic calendar, this is an ideal time to
survey respondents. To ensure effective data collection, our approach will focus on three
areas:
Developing and Implementing an Accessible and Intuitive Survey. Achieving
strong response rates begins with a well-designed, user-friendly instrument, and continues
with providing a clear and convincing rationale for the survey and the importance of
respondents’ participation. One challenge of this study will be motivating participation and
following-up with initial nonrespondents to achieve our target response rate. The web
approach will allow us to easily identify non-respondents for follow-up emails to
encourage participation, thus substantially increasing response rates.
Supporting Respondents During Data Collection to Ensure a High Response
Rate. At the designated opening date, an e-mail message will be sent to respondents with
the link to the survey, detailed instructions, the closing date, and project staff contact
information. Throughout the data collection cycle, numbers and e-mail addresses will be
available for contractor and NSF staff to ensure that potential respondents can easily and
quickly obtain answers to questions or concerns. We will conduct up to three follow-ups
per respondent.
Protecting the Confidentiality of Data Collected. The contractor is a
conscientious guardian of the confidentiality of data. They have conducted numerous
contracts involving sensitive information; consequently, the organization and all project
staff employ both electronic and physical safeguards to protect data from unauthorized
access. Their facilities contain a PII Enclave where data can be stored in an air gaped
facility and not connected any network. Electronic project directories, files, and databases
are accessible to project staff only and are protected by discretionary access control lists.
The web-based survey data will be maintained on a secure server with appropriate levels
of password protection. The contractor will be using QualtricsGov, a FedRAMPModerate approved survey platform. According to the website, “FedRAMP, or Federal
Risk and Authorization Management Program, is a government-wide initiative to assess,
authorize, and monitor cloud software providers and protect the sensitive data housed in
federal agencies.” 1

1

https://www.qualtrics.com/platform/fedramp/

11

a. Statistical Methodology
As described above, the study team will be surveying the census of research
participants from 2012-2021 REM supplements. The primary purpose of surveying REM
research participants is to describe the career direction of REM research participants and
to see how various program components affect research participant outcomes from
different types of participants. Therefore, for this part of the study, a comparison group will
not be surveyed.
b. Impact Analyses Using Extant Data
To determine the impact of the REM supplement on research participant STEM
retention and research output, the study will be analyzing degree completion and
bibliometric output post receipt of the REM supplement. These analyses will be based on
extant degree completion data from the National Student Clearinghouse and published
research attributed to the participants. Preliminary inquiries have been made to the
clearinghouse to ensure the necessary data are available.
3.

Methods to Maximize Response Rates and Deal with Nonresponse

The study team will use the knowledge and lessons learned from the NSF evaluations
to design instruments that will be clear and transparent to the recipients.
One challenge of this study will be motivating participation and following-up with
initial nonrespondents to achieve our target response rate. The web approach will allow us
to easily identify non-respondents for follow-up emails to encourage participation, thus
substantially increasing response rates. In an effort to increase overall survey response rate,
email follow-up will be used for non-respondents.
At the designated opening date, the study team will send an e-mail message to
respondents with the survey link, detailed instructions, the closing date, project staff and
contact information. Detailed onscreen instructions will be included. Throughout the data
collection cycle, the study will use NSF and STPI staff phone numbers and e-mail
addresses to ensure that potential respondents can easily and quickly obtain answers to
questions or concerns.
4.

Test Procedures or Methods

The survey instruments developed for this data collection will be pilot-tested.

12

5.

Individuals Consulted on Statistical Aspects and Individuals Collecting and/or
Analyzing Data

The contractor for collection and analysis of data in this study is STPI. Staff from the
organization have knowledge of statistical methods, experience in evaluation of research
programs, and expertise in scientific research.
Key personnel involved in the statistical aspects and who will be involved in
collecting and analyzing data include:

STPI

Asha Balakrishnan

202-419-5480

Brian Zuckerman

202-419-5485

Lara Rubinyi

202-419-5465

Logan Pratico

202-419-5423

Appendix A: Research Questions by Data Sources
Appendix B: Research Participant Survey

13