The survey data collection process will include the 468 research participants from all 64 REM supplements funded between 2017 and 2021. Respondents include high school students, STEM teachers, undergraduate STEM students, and faculty.
The study will field one wave of survey data collection of REM research participants. (See Appendix B for REM Research Participant survey). The wave will be conducted in fall 2021 and will include all high school students, STEM teachers, undergraduate STEM students, and faculty that received REM funding in years 2017 through 2021. The sample excludes supplements awarded between 2012 and 2016 due to insufficient and less updated data on research participants.
The study sample will include the census of research participants across each of the four types of respondents described above. The rationale for this strategy is based on several related factors. First, there are different numbers of recipients within each of the possible recipient categories, and NSF is interested in learning about all types of recipients. Because it is quite possible that it will be easier to locate and recruit REM research participants who entered the program more recently, particularly, in the last two or three years the survey sample has been limited to recent participants in REM. Third, experience from the prior studies indicate that obtaining high response rates may be challenging.
Taken together, these factors suggest that to obtain a sufficient number of survey responses to compute estimates with reasonable precision (i.e. small standard errors, modest confidence intervals), the most conservative approach would be to include the census as our potential respondent sample.
The study team will field an online survey for research participants. The study team obtained the list of names and e-mail addresses of research participants from annual reports and contact with the principle investigators. The study team first notified principal investigators about the study and then provided them with a list of their awards’ research participants extracted from the annual and final reports. This was done in order to ascertain whether the complete list of research participants had been collected, and to collect updated contact information for these recipients. The principal investigators then verified or updated the contact information to provide the most up-to-date information on research participants. The study team, as well as NSF staff, followed up several times by email and phone with principal investigators to ensure accurate and complete research participant information. The resulting list of research participants includes verified contact information. This approach was used in the previous NSF evaluations, and it yielded substantial, albeit not always fully current, contact information.
Once relevant approvals are obtained, STPI will program the research participant survey for online data collection in QualtricsGov. The study team will test the survey system to ensure functionality and accuracy of data capture. STPI plans to launch survey data collection in the fall 2021. Given the academic calendar, this is an ideal time to survey respondents. To ensure effective data collection, our approach will focus on three areas:
Developing and Implementing an Accessible and Intuitive Survey. Achieving strong response rates begins with a well-designed, user-friendly instrument, and continues with providing a clear and convincing rationale for the survey and the importance of respondents’ participation. One challenge of this study will be motivating participation and following-up with initial nonrespondents to achieve our target response rate. The web approach will allow us to easily identify non-respondents for follow-up emails to encourage participation, thus substantially increasing response rates.
Supporting Respondents During Data Collection to Ensure a High Response Rate. To ensure a robust response rate, the contractor and NSF will contact respondents several times, using multiple approaches. See Table 5 for more details on when and with what means the respondents will be contacted. Appendix C contains draft email text or phone scripts for each contact or follow-up step listed in Table 5.
At the designated opening date, an e-mail message will be sent to respondents with the link to the survey, detailed instructions, the closing date, and project staff contact information. NSF will request PIs send emails to their research participants two days before the survey invitation is sent, letting them know to expect an email with the survey link and encouraging them to fill it out. In addition, NSF will send two emails directly to research participants, the first alerting them of the survey and encouraging participation the same day they receive the invitation and the second email two weeks later reminding the research participants to take the survey. NSF will also post about the survey and importance of the data collection on their LinkedIn group of REM research participants. Throughout the data collection cycle, numbers and e-mail addresses of the contractor and NSF staff will be available to ensure that potential respondents can easily and quickly obtain answers to questions or concerns. We will conduct up to three email follow-ups per respondent; one week, two weeks, and two and a half weeks after the initial invitation. If a phone number is available, project staff will provide a final phone call follow up with the research participant.
Table 5. Respondent Contact Schedule
Form of Contact |
When Contact Will Occur |
|---|---|
|
2 days before launch |
|
Day of launch |
|
Launch day |
|
1 week after launch |
|
2 weeks after launch |
|
2 weeks after launch |
|
2.5 weeks after launch |
|
3 weeks after launch |
Protecting the Confidentiality of Data Collected. The contractor is a conscientious guardian of the confidentiality of data. They have conducted numerous contracts involving sensitive information; consequently, the organization and all project staff employ both electronic and physical safeguards to protect data from unauthorized access. Their facilities contain a PII Enclave where data can be stored in an air gaped facility and not connected any network. Electronic project directories, files, and databases are accessible to project staff only and are protected by discretionary access control lists. The web-based survey data will be maintained on a secure server with appropriate levels of password protection. The contractor will be using QualtricsGov, a FedRAMP-Moderate approved survey platform. According to the website, “FedRAMP, or Federal Risk and Authorization Management Program, is a government-wide initiative to assess, authorize, and monitor cloud software providers and protect the sensitive data housed in federal agencies.”1
As described above, the study team will be surveying the census of research participants from 2017-2021 REM supplements. The primary purpose of surveying REM research participants is to describe the career direction of REM research participants and to see how various program components affect research participant outcomes from different types of participants. Therefore, for this part of the study, a comparison group will not be surveyed.
To determine the impact of the REM supplement on research participant STEM retention and research output, the study will be analyzing degree completion and bibliometric output post receipt of the REM supplement. These analyses will be based on extant degree completion data from the National Student Clearinghouse and published research attributed to the participants. Preliminary inquiries have been made to the clearinghouse to ensure the necessary data are available.
The study team will use the knowledge and lessons learned from previous NSF evaluations to design instruments that will be clear and transparent to the recipients.
One challenge of this study will be motivating participation and following-up with initial nonrespondents to achieve our target response rate. The web approach will allow us to easily identify non-respondents for follow-up emails to encourage participation, thus substantially increasing response rates. In an effort to increase overall survey response rate, email follow-up will be used for non-respondents.
At the designated opening date, the study team will send an e-mail message to respondents with the survey link, detailed instructions, the closing date, project staff and contact information. Detailed onscreen instructions will be included. Throughout the data collection cycle, the study will use NSF and STPI staff phone numbers and e-mail addresses to ensure that potential respondents can easily and quickly obtain answers to questions or concerns.
The survey instruments developed for this data collection will be pilot-tested.
The contractor for collection and analysis of data in this study is STPI. Staff from the organization have knowledge of statistical methods, experience in evaluation of research programs, and expertise in scientific research.
Key personnel involved in the statistical aspects and who will be involved in collecting and analyzing data include:
STPI |
Asha Balakrishnan |
202-419-5480 |
|
Brian Zuckerman |
202-419-5485 |
|
Lara Rubinyi |
202-419-5465 |
|
Logan Pratico |
202-419-3721 |
Appendix A: Research Questions by Data Sources
Appendix B: Research Participant Survey
Appendix C: Research Participant Survey Notifications, Invitations, and Reminders
1 https://www.qualtrics.com/platform/fedramp/
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Plimpton, Suzanne H. |
| File Modified | 0000-00-00 |
| File Created | 2024-11-21 |