S-STEM Supporting Statement Part A 5-22-13

Supporting Statement for Paperwork Reduction Submission
Program Evaluation of the Scholarships in Science, Technology, Engineering,
and Mathematics (S-STEM) Program

SECTION A: Justification
Introduction
The National Science Foundation (NSF) requests that the Office of Management and Budget
(OMB) approve, under the Paperwork Reduction Act of 1995, a three-year clearance for original
data collection to be used in the evaluation of the Scholarships in Science, Technology,
Engineering, and Mathematics (S-STEM) Program. The new data collections includes an SSTEM Recipient Survey (Appendix A); an S-STEM Principal Investigator Survey (Appendix C);
and Site Visit Interview and Focus Group Protocols (Appendix D).
The S-STEM program, which operates within NSF’s Division of Undergraduate Education,
awards grants to a geographically diverse set of two- and four-year institutions of higher
education (IHEs) that then provide scholarships for academically talented students, in science
and engineering disciplines, who have demonstrated financial need. Individuals may be granted
S-STEM scholarships for up to five years and receive up to $10,000 per year depending on
financial need. The institutions also provide resources and support services (e.g. academic
support, career counseling, recruitment, research opportunities) to students to support them in
becoming and/or remaining engaged in science and engineering through successful pursuit of
associate, baccalaureate, or graduate-level degrees. Institutions are not required to provide any
specific type of resources or support services (e.g., faculty advisors, peer tutoring, career
counseling) beyond student scholarships, so part of the evaluation will be to gather data on the
services provided (see the proposed Principal Investigator Survey, Appendix C, Module D for
specific survey items addressing the non-financial support offered to S-STEM scholarship
recipients at S-STEM grantee institutions).
S-STEM awards to institutions may last up to five years. The maximum S-STEM request is
normally not to exceed $600,000 in total direct costs; annual budgets are limited to $225,000
direct costs. As part of the direct costs, institutions may request funds up to 5 percent of the total
scholarship amount for expenses related to program administration, and up to 10 percent of the
total scholarship amount for student support services.
The goals of S-STEM are to: 1) improve educational opportunities for students; 2) increase
retention and degree attainment; 3) improve student support programs at institutions of higher
education; and 4) increase the number of well-educated and skilled employees in technical areas
of national need.1 Successful outcomes of the program include graduation with a STEM major,
1

National Science Foundation. NSF Scholarships in Science, Technology, Engineering, and Mathematics (S-STEM)
Program Solicitation (NSF 12-529). Retrieved 5/21/13 from
http://www.nsf.gov/pubs/2012/nsf12529/nsf12529.htm

1

transfer of STEM students from two-year to four-year colleges, pursuit of STEM graduate degree
studies, and employment in the STEM workforce. Funding for S-STEM comes from H-1B
VISAs, funding which was reauthorized in FY 2005 through Public Law 108-447. NSF receives
40 percent of the H-1B funding, and the agency uses 75 percent of its portion of these funds for
the S-STEM program (NSF, 2011). In 2006, S-STEM expanded to include technology and
science fields beyond the original computer science, engineering, and mathematics fields
included in its precursor program—the Computer Science, Engineering, and Mathematics
Scholarship (CSEMS) Program, targeting instead the development of the broader STEM
workforce.
The evaluation of S-STEM will explore the practices (academic and student support services)
and characteristics of the implementation of S-STEM projects; the educational and career
outcomes of the S-STEM scholarship recipients; the student or programmatic outcomes
associated with the receipt of an S-STEM award; the relationship between specific project
features or practices and student outcomes; and any promising practices or lessons learned about
implementation of S-STEM projects. The evaluation will draw on extant data as well as require
new data collection efforts. This package seeks OMB approval for the new data collection
efforts, which include a survey of S-STEM principal investigators, a survey of S-STEM
scholarship recipients, and site visit interview and focus group protocols. Although approval is
sought only for the new data that will be collected for the study, our description of the evaluation
includes both the extant and original data sources that will be considered in the study.
Because of the nature of the S-STEM program and the type of information being sought, a
mixed-methods evaluation design will be employed. The mixed methods approach to the
evaluation integrates both quantitative and qualitative data analyses and methods. Specifically,
the evaluation will consist of descriptive, relational, benchmarking, and quasi-experimental study
components:
 A descriptive implementation study that describes the ways in which S-STEM projects
(i.e., grantee institutions) recruit and retain students in STEM fields, allocate scholarship
funds, and provide educational and support programming for scholarship recipients;
 A relational study of associations between project characteristics and practices and
recipient outcomes;
 A benchmarking comparison of recipients’ educational and academic support experiences
to national trends; and
 A comparative, quasi-experimental study using Propensity Score Matching (PSM) at the
individual level to compare the educational and career outcomes of S-STEM scholarship
recipients to a matched sample of respondents from a national survey of postsecondary
students.
The data sources to be used in the evaluation include both new data collections and extant
sources (see Section A.1 for more information about the circumstances requiring these data
sources):
New Data Collections
 Survey data from S-STEM Principal Investigators to understand how the S-STEM
projects operate and how the S-STEM awards are implemented;

2



Survey data from S-STEM scholarship recipients to: (a) compare their educational and
career outcomes to a matched comparison group of respondents to Beginning
Postsecondary Students Longitudinal Study (BPS) surveys, an extant data source; and
(b) benchmark (contextualize) their undergraduate experiences and academic engagement
to a reference group of respondents to the National Survey of Student Engagement
(NSSE);2
 Site visits to a purposive sample of S-STEM projects at up to six institutions of higher
education per year to gather in-depth information through interviews and focus groups,
about project implementation in particular contexts.3
Extant Sources
 Extant program data from projects’ annual reports to NSF and the S-STEM Monitoring
System to examine the components and practices employed by S-STEM projects;
 Extant institutional data maintained by the Integrated Postsecondary Education Data
System (IPEDS; U.S. Department of Education), to which institutions of higher education
report annual data on a wide range of institutional characteristics including types of
degrees granted; public versus private control; non-profit or for-profit status;
demographic characteristics of student body (e.g., race, ethnicity, gender); undergraduate
and graduate student enrollment; retention and degree completion rates; percentage of
students majoring in academic fields; tuition and cost-of-attendance; percentage of
students receiving financial aid; expenditures on instruction, research and development,
operations; etc.;
 Extant data from a matched comparison group of respondents to the Beginning
Postsecondary Students Longitudinal Study (BPS) surveys, part of a longitudinal,
nationally-representative study conducted by the U.S. Department of Education to
examine rates of college enrollment, degree attainment, and how students pay for
college;4
 Extant data from a comparison group of respondents to the National Survey of Student
Engagement (NSSE), an annual survey of undergraduate students at participating fouryear institutions to examine students’ engagement in learning and co-curricular campus
activities, and students’ perception of the degree of faculty and staff support for their
educational and career goals. 5
While approval is sought only for the new data that will be collected for the study, we describe
both the extant and original data sources that will serve the evaluation.
S-STEM recipients will be matched on student level characteristics (see Exhibit A.1 below) to
BPS survey respondents. Details of the approach to matching are contained in Appendix E.

2
3

4

5

The NSSE is administered once annually at participating institutions of higher education (IHE) that award a four- or fiveyear Bachelor’s degree to a random sample of currently-enrolled students within each institution.
In selecting sites, the study will be purposeful in ensuring a diversity of institutional types and contexts, with visits to twoyear schools, minority serving institutions, public research universities, and private liberal arts colleges..
This longitudinal study of first-time beginning postsecondary students included three waves of data collection beginning
with the National Postsecondary Student Aid Study (NPSAS04), with two follow-up waves: the Beginning Postsecondary
Student surveys in 2006 and 2009 (BPS:04/06 and BPS:04/09).
To determine the feasibility of the matching, NSF’s evaluation contractor has obtained access to the extant datasets (IPEDS,
BPS, NSSE), processed the data, and identified the samples.

3

Exhibit A.1: Individual characteristics to be included in matching
Financial aid
Received Federal Stafford Loan
Received Pell Grant
Received school grant/scholarship
Received State grant/scholarship
Received any other financial aid for education
Academic information
SAT I math score
SAT I verbal score
ACT composite score
Cumulative GPA ( or an estimate of GPA) through the end of the first school
year
Demographic characteristics
Gender
Age
Race and Ethnicity
Citizenship
Disability
Other characteristics
Type of degree (Associates or Bachelor’s degree)
Major (Current field of degree)
Full-time enrollment status

A.1. Circumstances Requiring the Collection of Data
A descriptive study of the precursor program, Computer Science, Engineering and Mathematics
Scholarships (CSEMS), was conducted in 2003-2004 (Temple University, no date), but this
study is the first evaluation to be conducted of the S-STEM program, which was established in
2006. Funding for S-STEM comes from H-1B VISAs, funding which was reauthorized in FY
2005 through Public Law 108-447. NSF receives 40 percent of the H-1B funding, and the agency
uses 75 percent of its portion of these funds for the S-STEM program.
At all levels of higher education, cost is a major impediment to access and completion and there
is evidence that merit-based aid increases student retention (e.g. Cabrera, Stampen, and Hansen,
1990; Singe1l, 2004; Dynarksi, 2008). In an effort to increase the numbers of students retained in
STEM degree programs, in particular, NSF’s S-STEM program has supported thousands of
students since its inception in 2006. NSF’s S-STEM program provides funds to institutions who
in turn grant scholarships to academically talented, low-income students interested in or
currently pursuing STEM fields. Students must be U.S. citizens, permanent residents, nationals,
or refugees. Scholarships may be made for up to five years and may provide individual
scholarships of up to $10,000 per year, depending on individual student financial need.
NSF has contracted with Abt Associates to conduct an evaluation of S-STEM that will explore
the practices, and characteristics of implementation of exemplary S-STEM awardees; investigate
S-STEM-related outcomes among recipients; and investigate institutional-related outcomes of SSTEM grantees. (The description of the site visits below provides a description of the process for
selecting cases for in-depth study.) More specifically, the study will answer the following
overarching questions:

4

1. How do individual awardees implement their S-STEM projects?
2. What are the educational and career outcomes of the S-STEM scholarship recipients?
How do outcomes of S-STEM recipients compare to an appropriate comparison group or
national trends?
3. Are there student or programmatic outcomes associated with receipt of an S-STEM
award?
4. What is the relationship between specific project characteristics or practices and student
outcomes? Are there promising practices or lessons learned about implementation of SSTEM projects?
There were 513 S-STEM projects awarded from 2006 to 2010. The evaluation will include the
494 S-STEM projects that provided scholarships to undergraduate recipients, the PIs of these
projects, and the S-STEM scholarship recipients who were (or are currently) supported by these
projects. Projects supporting graduate students are not included in the evaluation due to the small
number of awards (n=19).
The evaluation will employ data from extant sources, web surveys and site visits to provide a
comprehensive picture of the activities supported through the S-STEM program, the faculty
involved, the student recipients, projects’ recruitment and selection processes, degree attainment,
the educational and career decisions made by recipients, and the career outcomes of recipients.
Original data will be collected via surveys from principal investigators, and from S-STEM
scholarship recipients. In addition, in-depth interviews and focus groups will be conducted with a
series of respondents during site visits to a subset of awards. Site visits are intended to provide
rich information about the implementation of the S-STEM program at a given institution as well
as outcomes related to participating individuals and the capacity of a given college/university to
educate and support STEM students. NSF requests clearance for the instruments associated with
the original data collection for the evaluation, including the:
 S-STEM Recipient Survey;
 S-STEM Principal Investigator Survey; and
 Site visit interview and focus group protocols.
While approval is sought only for the new data that will be collected for the study, below we
describe both the new data sources and the extant data that will serve the evaluation. Exhibit A.2
maps the research questions to the sources of data.
New data collections
Primary data collection activities, for which approval is being sought, will entail web surveys of
S-STEM scholarship recipients and S-STEM Principal Investigators (PIs) and in-depth
interviews and focus groups with a series of respondents during site visits to a subset of awardee
institutions. Supporting Statement B, Section B.1 and Exhibit B.3 provide details of the sample
sizes for the respondent groups.
Web survey of scholarship recipients. A survey will be administered to a sample of current and
former scholarship recipients to gather information on undergraduate enrollment history and
characteristics, academic and employment outcomes, S-STEM scholarship support received
5

(e.g., amount and number of semesters of scholarship funding), participation in research projects
or internships in industry, academic engagement, quality of faculty advising received, and
characteristics of any employment while enrolled (see Part B, Section B.1 and Exhibit B.3 for
more information on the sample sizes and sample selection). The survey includes items drawn
from national surveys (See Appendix A for the Recipient Survey and Appendix B for sources of
specific items and a mapping of items to the study’s research questions). Including items from
these surveys will allow both quasi-experimental analyses, comparing S-STEM recipients to a
matched comparison group, and for analyses that benchmark S-STEM recipients’ experiences
against existing norms and national trends. The proposed survey will also collect information
from recipients to allow propensity score matching to national samples of undergraduate
respondents in extant survey data; these items include information about recipients’
characteristics during their first year of enrollment at the S-STEM grantee institution, including:
first declared major, first-year grade point average, age, marital status, income, types of financial
aid (i.e., other than an S-STEM scholarship), college credit earned in high school, SAT or ACT
college entrance exam scores, gender, race, ethnicity, disability, and parents’ education.
Web survey of PIs. A survey of S-STEM PIs will provide information about the S-STEM
project activities and institutional support (See Appendix C for the PI Survey and Part B, Section
B.1 and Exhibit B.3 for information on the sample sizes and sample selection). Specific topics
include: recruitment and application, including the number of applicants for S-STEM relative to
the number of recipients; the average number of recipients per year and average scholarship
amount; formal faculty advising and other support services to assist recipients with academic
work; activities to foster recipients’ development of a professional identity in STEM; recipients’
participation in faculty research projects or internships in industry settings; data collected by the
PI on recipient academic or employment outcomes; and other NSF-funded education-oriented
projects (e.g. CCLI/TUES, STEP) at the institution that may contribute to the retention of
undergraduates in STEM fields.
Site Visits. In-depth qualitative data will be collected through interviews and focus groups
during site visits at S-STEM institutions, purposively selected to span a range of institutional
contexts in which S-STEM awards operate. Site visits are intended to provide rich information
about the implementation of the S-STEM program at a given institution and program outcomes
related to both the participating individuals and the institution, or institutional unit participating
in STEM. Site visit teams, comprised of three to five evaluation team members and content experts,
will conduct interviews or focus groups with key respondent types at each of up to six IHEs each
year. In consultation with NSF, the Abt study team will select a purposive sample of S-STEM
sites for which to conduct site visits, which will ultimately inform case study analysis. Case
studies can help provide a depth of understanding about the implementation of the S-STEM
projects as well as possible mechanisms between the program and associated outcomes (Yin,
2009; Stake 1995). Case selection in case study research requires a sample of cases that
represent dimensions of theoretical interest (Seawright & Gerring, 2008). Sites will thus be
selected to ensure they represent the variety S-STEM program types (e.g., institution type, size,
location, STEM fields supported) that will yield data on implementation and promising practices
at the program level. The format of the data collection, interview or focus group, will be
dependent on the particular organization and participation patterns of each institution. Interviews
or focus groups will be conducted with an average of 34 people per IHE, including PI/co-PIs,
6

STEM faculty, other campus officials, and current and former S-STEM recipients (see Appendix
D for interview and focus group protocols). In preparation for each site visit, the data from extant
sources (e.g. annual reports, data from the S-STEM monitoring system) will be analyzed for the
institution and award and compiled into a background document that site visitors will review
prior to the visit. This will avoid duplication of data collection and identify specific areas of
information that should be specifically probed.
Extant data
Extant data sources include both sources that provide information about the S-STEM program
directly as well as sources that will provide data for comparisons. See Exhibit A.2 below for a
map of the research questions to the sources of extant data and Appendix B for sources of
specific items and a mapping of items to the study’s research questions.
Extant NSF program data sources that will serve the evaluation include the annual reports
prepared by PIs as part of their grant requirements and the program monitoring system. PIs
submit Annual Reports to NSF that provide an overview of the current year’s activities and
scholarship recipients. The S-STEM Monitoring System, administered by ICF International
(approved under OMB generic clearance #3145-0226), contains information entered by PIs each
year about project activities, scholarship recipients, and recipient characteristics. Data from the
monitoring system and annual reports will be used in descriptive analyses to describe the
services offered by S-STEM as well as the characteristics of S-STEM participants.
The Integrated Postsecondary Education Data System (IPEDS) is a system of interrelated
surveys conducted annually by the U.S. Department of Education’s National Center for
Education Statistics (NCES) that gathers information from every college, university, and
technical and vocational institution that participates in federal student financial aid programs.
Institutional characteristics data from IPEDS (such as types of degrees granted; public versus
private control; non-profit or for-profit status) will be used to identify institutions from which a
matched comparison group of individuals can be selected, as necessary, for the comparative
analyses.
The Beginning Postsecondary Students (BPS) Longitudinal Study follows the academic
progress of a nationally-representative sample of first-time beginning post-secondary students,
contains information about academic progress and persistence in postsecondary education,
degree completion, financing of education and entry into the workforce. Outcomes (e.g. degree
attainment, time to degree) for the matched comparison will be drawn from the BPS and used in
the comparative analyses. Although the BPS includes a nationally-representative sample of
students, the comparison group used in our analyses will be restricted to students in the BPS that
are U.S. citizens, full-time students, undergraduates, and are enrolled in (or were previously
enrolled in) a STEM major.
The National Survey of Student Engagement (NSSE) is a survey of first year and senior
undergraduate students from participating institutions of higher education in the US and Canada.
The survey asks questions about how students allocate time and effort to coursework and other
activities, levels of academic challenge and collaboration, the nature of students’ interactions
with faculty, and other aspects of the campus climate. The NSSE survey is designed to be
7

representative at the institution level. The survey collects information on student major so we can
identify and restrict the data to STEM students. Data from NSSE will provide benchmark
comparisons for the experiences on campus of S-STEM recipients.

8

Exhibit A.2: Map of Research Questions to Data Sources and Analyses
Research Questions

Data Sources

1.
2.

How do individual awardees implement their S-STEM projects?
a. What are promising practices in key areas (e.g. recruitment, academic support, retention) of and/or
lessons learned from highly effective and successful S-STEM projects?
What are the educational and career outcomes of the S-STEM scholarship recipients? How do
outcomes of S-STEM recipients compare to an appropriate comparison group or national trends?
a.

X

4.

To what extent do S-STEM scholarship recipients transfer to a four-year program as compared to an
appropriate comparison or national trends?
c. To what extent do S-STEM scholarship recipients join the STEM workforce after graduation as compared
to an appropriate comparison or national trends?
d. To what extent do S-STEM scholarship recipients apply for and attend a STEM graduate program as
compared to an appropriate comparison group or national trends?
e. How effectively does the program meet the needs of academically talented financially needy STEM
students as compared to other need-based opportunities and/or mechanisms?
Are there student or programmatic outcomes associated with receipt of an S-STEM award?
a. What is the effect of the S-STEM program on student outcomes (e.g. recruitment, retention) in STEM at
institutions that have received an award compared to an appropriate comparison or national trends?
b. What is the added value of the S-STEM program on institutions that receive awards?
c. What is the added value of the S-STEM program on student support and educational opportunities for
recipients in institutions receiving S-STEM program funding?
d. What is the added value of the S-STEM program on STEM programs, STEM departments, and/or IHEs
that have received S-STEM funding?
What is the relationship between specific project features or practices and student outcomes? Are
there promising practices or lessons learned about implementation of S-STEM projects?
a. Are there any unintended positive and negative consequences/outcomes that can inform project and
program management and design?
b. To what extent can outcomes be attributed to components supported by the S-STEM program?
c. Are there other NSF-funded education-oriented projects at the institution?
d. What is the relationship among these NSF-funded education oriented efforts at the institution?

9

NSSE data

BPS data

Site visit
interview/focus
groups
Annual Reports/
Monitoring System

IPEDS

Descriptive,
Relational

X

X

X
X

X

Comparative

X

X

X

Comparative

X

X

X

Comparative

X

X

Comparative

X

X

Comparative

X

X
X

X

X

X

X

X

X

X

X
X
X

Descriptive,
Benchmarking

X

X
X

Analyses
Extant

X

X

What is the effect of the S-STEM program scholarships on recipients?

b.

3.

Recipient survey

PI survey

New

X

Descriptive

X
X

X

Descriptive
Descriptive,
Benchmarking

Descriptive,
Relational
Descriptive
Descriptive
Descriptive

A.2. Purposes and Uses of the Data
The primary purpose for the proposed information collection is program evaluation. The data
collected will allow NSF to describe the project components that are implemented with S-STEM
funds to support academically talented STEM students who have demonstrated financial need.
Describing educational and career outcomes of S-STEM scholarship recipients, contextualizing
them with comparisons to matched non-recipients, and investigating project features that are
associated with program outcomes will inform future S-STEM program decisions, as well as
future decisions about the necessity for introducing specific program requirements and will
inform future S-STEM funding decisions and contribute to the wider NSF discussion on the
preparation of STEM scholars as professionals (see Exhibit A.2 for a cross-walk of the research
questions and data sources that will be used to address each).
Data collected in the study would be used by program staff to help them manage the program. In
addition, outcomes of the program would be reported to the leadership of NSF’s Division of
Undergraduate Education, Directorate for Education and Human Resources, NSF, and the
general public. If the study yields findings of more general interest, the results may be submitted
for dissemination at conferences or publication in a peer-reviewed journal.
Although annual program monitoring data are collected about the S-STEM program via the SSTEM monitoring system, this system did not launch until 2009. Data for 2006 to 2008 were
entered retrospectively, if they were available. Even if the monitoring system had complete data
from S-STEM awardees from 2006 forward, the information collected is not sufficiently
comprehensive to describe the activities provided by the S-STEM projects, the contexts in which
they operate, the faculty involved, and their recruitment and selection process; most importantly,
it does not include sufficient data on the educational and employment outcomes of S-STEM
scholarship recipients, nor recipients’ experiences that may inform project- and program-level
decision making. Consequently, the study will conduct surveys and interviews to answer those
research questions that cannot be answered through existing sources alone.
A.3. Use of Information Technology to Reduce Burden
Online surveys will be used to collect information from participants. S-STEM scholarship
recipients and Principal Investigators (PIs) will be asked about program experiences,
implementation, and associated outcomes.6 This includes, but is not limited to, S-STEM
activities, student support services available to S-STEM students and to STEM students at large,
program experiences, and synergies between the S-STEM grant and other federally-funded
enrichment programs.
Web-based surveys have become a commonly used avenue for data collection in recent years
given their facilitation of respondents’ data entry across computer platforms. Web-based surveys
employ user-friendly features such as automated tabulation and data entry with custom controls
such as checkboxes, standard menus, and predefined charts and graphics. In addition, survey skip
patterns automatically move the respondent forward to the next appropriate section, simplifying
6

Potential respondents with a disability preventing them from completing the survey online will be accommodated with the
option of a phone administration of the survey.

10

the survey-taking experience. Most invalid data cannot enter the system, and questionable or
incomplete entries are called to respondents' attention before they are submitted. Also, given the
ease by which data can be automatically uploaded into standard analysis software, there is
increased efficiency for researchers seeking to analyze said data. All these features facilitate the
reporting process, provide useful and rapid feedback to the data providers, and thus reduce
burden.
Another avenue of data collection will be site visits to S-STEM campuses and on-site interviews
with S-STEM students and faculty. To reduce participant burden during this activity, prior to the
site visit, publically available data will be compiled and data from proposals, interim, annual and
evaluation reports, as well as any data reported by S-STEM PIs via the S-STEM Monitoring
System will be analyzed and findings documented in a site visit background notebook for site
visitors. This will eliminate the need to ask questions about project operations and outcomes on
which PIs have already reported.
Finally, the study team will analyze data from extant large-scale surveys (BPS, NSSE) for the
purposes of benchmarking recipient outcomes. Data from IPEDS will be investigated to situate
the institutions with S-STEM awards within the larger context of U.S. postsecondary institutions.
A.4. Efforts to Identify Duplication
This study does not duplicate other NSF efforts. There has been no evaluation of the S-STEM
program.
Data from the S-STEM Monitoring System will be available to the evaluation contractor and the
NSF program staff involved in this study. Surveys will be constructed to ask about elements of
the projects that are not captured in the Monitoring System, and site visit interviews will be
structured to gather information that cannot be easily summarized in close-ended survey
responses. Items in the Recipient Survey will be drawn primarily from existing, validated
surveys where possible, including the Beginning Postsecondary Students (BPS) surveys
(National Center for Education Statistics, U.S. Department of Education) and the National
Survey of Student Engagement (NSSE; Center for Postsecondary Research National Survey of
Student Engagement at Indiana University). The study will use extant institutional data from the
BPS and NSSE surveys as well as from the Integrated Post-secondary Educational Data System
(IPEDS) to create analysis files.
A.5 Small Business
No information for this research will be collected from small businesses.
A.6. Consequences of Not Collecting the Information
If this information is not collected, NSF will be unable to document the specific implementation
of S-STEM projects and the outcomes of S-STEM recipients. It will not be able to assess the
degree to which the S-STEM program is meeting its goals or comply with the mandate that NSF
evaluate the S-STEM program (see the NSF Authorization Act of 2002, P.L. 107-368).
A.7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6
All data collection will comply with 5 CFR 1320.6.
11

A.8. Selection of Public Comments and Consultation with People Outside the Agency
A.8.A. Federal Register Announcement.
Comments on this data collection effort were solicited in the Federal Register on December 14,
2012 (vol. 77, no. 241, p. 74516). No comments were received. A copy of the notice is included
in Appendix I.
A.8.B. Consultations Outside the Agency
Consultation on the evaluation needs, study design, data sources, and analyses occurred during
the design phase and will continue throughout the study. The purpose of such consultation is to
ensure the technical soundness of the study and the relevance of the findings, and to verify the
importance, relevance and accessibility of the information sought in the study.
Consultation was conducted by the research firm Abt Associates Inc., contracted by NSF to
design and conduct an evaluation of the S-STEM program, its data subsidiary, Abt-SRBI, and
subcontractor SageFox. Senior technical staff from these organizations who are conducting the
study are listed below:








Alina Martinez, Principal Associate, Abt Associates, Inc. (Principal Investigator)
Kelly Daley, Abt SRBI
Carter Epstein, Scientist, Abt Associates, Inc.
Lorelle L. Espinosa, Senior Analyst, Abt Associates, Inc.
Luba Katz, Senior Scientist, Abt Associates, Inc.
Amanda Parsad, Senior Scientist, Abt Associates, Inc.
Alan Peterfreund, Executive Director, SageFox Consulting Group

In addition, an external evaluation group of subject matter experts has provided input on the study
design and data collection plan. The study’s group of subject matter experts includes:






Dr. Deborah Allen, Director, Center for Educational Effectiveness and Associate
Professor of Biological Sciences, University of Delaware
Dr. Eun-Woo Chang, Instructional Dean of Science, Engineering and Mathematics,
Montgomery College
Dr. Bert E. Holmes, Philip G. Carlson Distinguished Chair, University of North Carolina,
Ashville
Dr. Patricia Mead, Professor of Engineering, Norfolk State University
Dr. Lizanne DeStefano, Professor of Quantitative and Evaluative Research
Methodologies, Educational Psychology, University of Illinois, Champaign-Urbana

In addition, the proposed data collection instruments were pilot tested with respondents drawn from
the target populations. Respondents were asked to comment on the clarity and content of the survey
questions, as well as the proposed duration of the data collection to help with accurate estimates of
time burden, which were based on response times reported in the corresponding BPS:04/09
Methodology Report (Wine, Janson & Wheeless, 2011) and the NPSAS04 Methodology Report
(Cominole, Seigel, Dudley, Roe & Gilligan, 2004). The interview protocols were each pilot tested
12

with 5 to 9 individuals from each respondent group. Interviews and focus groups lasted from 30 to 60
minutes, with an average of 40 minutes.

A.9. Payments or Gifts to Respondents
No payments or gifts will be provided to respondents.
A.10. Assurance of Confidentiality
Respondents will be advised that any information on specific individuals will be maintained in
accordance with the Privacy Act of 1974. Data collected will be available to the evaluation
contractors, contractors hired to manage data and data collection software, and at the aggregate
level to NSF staff. Data will be processed in accordance to Federal and State privacy statutes.
Detailed procedures for making information available to various categories of users are specified
in the Education and Training System of Records (63 Fed, Reg. 264, 272 January 5, 1998). The
system limits access to personally identifiable information to authorized users. Data submitted
will be used in accordance with criteria established by NSF for monitoring research and
education grants, and in response to Public Law 99-383 and 42 USC 1885c. The information
requested may be disclosed to qualified contractors in order to coordinate programs and to a
Federal agency, court or party in court, or Federal administrative proceeding, if the government
is a party.
All assurances of confidentiality will be reviewed by the contractor’s Institutional Review Board
prior to data collection.
A.11. Questions of a Sensitive Nature
The proposed survey asks for demographic information (gender, race, ethnicity) from S-STEM
recipients on a voluntary nature, thus respondents may choose not to provide information that
they believe is sensitive in nature. This information is being collected so that NSF can
understand whether the S-STEM program is supporting a diverse population of students. All
instruments will be reviewed by the contractor’s Institutional Review Board prior to fielding.
A.12. Estimates of Response Burden
The target population for the PI Survey are the PIs of two- and four-year institutions of higher
education that were awarded S-STEM grants from 2006-2010. The S-STEM Monitoring System
identifies 513 S-STEM grants awarded from 2006 through 2010; of these we exclude 19 because
the awards were to institutions that grant S-STEM scholarships only to master’s- or doctoratelevel students; this leaves a total of 494 awards that will be included in the study(see Exhibit
A.3). Because eleven PIs had multiple awards, there are only 483 unique PIs across these
projects. We propose to survey the census of 483 PIs of 2006-10 awards that provided S-STEM
scholarships to undergraduate (Associate’s or Bachelor’s degree) students.

13

Exhibit A.3: Study Samples for PI and S-STEM Recipients for Surveys
All S-STEM awards 2006 - 2010
N= 513
S-STEM awards in study
N= 494
(483 PIs, because there are 11 PIs with 2 awards)

EXCLUDED: Projects with
only graduate students
N=19

PI survey

S-STEM awards with data in monitoring system
N= 462
(86 at two-year IHEs, 376 at four-year IHEs)

EXCLUDED: Projects with
no student-level data
N=32

Unique IHEs
N=377
(Two-year IHEs: 75; Four-year IHEs: 302)

Two-year IHEs
N= 75
(Recipients N =5,477)

Four-year IHEs
N= 302
(Recipients N = 14,391)

Include All Two-Year
IHEs:
N=75

Sample Four-Year IHEs:
N=166

Sample S-STEM
Recipients at 2-year:
N = 3,074

Sample S-STEM
Recipients at 4-year:
N = 5,146

Recipient Survey

14

EXCLUDED: Recipients
pursuing Master’s
N=419, PhD N= 104,
unknown N=1 degree

Excluded from Sampling
Frame

The target population for the Recipient Survey includes S-STEM scholarship recipients enrolled
at S-STEM grantee institutions between 2006 and 2010. Because the S-STEM monitoring
system collects information about scholarship recipients from PIs, the monitoring system serves
as the source for identifying recipients to be included in the study. However, student-level
information is only available for 462 awards. The 32 awards with no student-level information
are excluded from the sampling frame for the recipient survey (Exhibit A.3). Because the
comparison group will be matched to recipients within IHEs the number of unique IHEs will
serve as the initial unit for sampling (details on sampling and matching are provided in Section
B.1). Among the 462 awards that have student data in the monitoring system, there are 377
unique IHEs (n=75 two-year IHEs and n=302 four-year IHEs). To be eligible for the recipient
survey, recipients must meet the following criteria:
 enrolled in an eligible 2006-2010 S-STEM awardee institution with student-level data in
the S-STEM Monitoring System (n = 377 IHEs); and
 enrolled in an associate’s or bachelor’s degree program during the first year they received
S-STEM support.
We will sample from two independent populations of scholarship recipients, classified by the
type of degree program in which they were enrolled during the first year they received S-STEM
support (see Exhibit A.3). These populations consist of those recipients who are currently or
were formerly enrolled:
 at eligible two-year awardee institutions (n=5,477 recipients at 75 unique institutions); or
 at eligible four-year awardee institutions (n=14,391 recipients at 302 unique institutions).
From the census of two-year awardee IHEs, we will sample 3,074 S-STEM scholarship
recipients to survey from within these 75 IHEs. From four-year awardee IHEs we will select a
sample of 166 IHEs. Within these IHEs, we will select a sample of 5,146 S-STEM scholarship
recipients to survey. Each group may include both former and current S-STEM scholarship
recipients. The total number of S-STEM recipients who will be invited to complete the recipient
survey is 8,220. Details of the power calculations supporting these sample sizes are provided in
Supporting Statement: Part B and Appendix E.
S-STEM PIs and scholarship recipients will each be asked to complete a web-based survey. The
Recipient Survey is expected to take an average of 19 to 22 minutes to complete (we have used
20 minutes to calculate the total annual burden in hours).7 The PI Survey is expected to take 20
minutes to complete.
In addition, up to six institutions will be visited each year. At each institution, we expect to
conduct interviews or focus groups with an average of 34 individuals per institution including
PI/co-PIs, STEM faculty, other campus officials, and current and former S-STEM recipients—
representing 204 participants over the series of site visits. Specifically, we expect to speak with
15 scholarship recipients per institution on average for a total of 90 scholarship recipients each
7

Mean response times for the Recipient Survey, which is comprised of items from the BPS:04/09 survey and the NPSAS:04
survey, are based on response times reported in the corresponding BPS:04/09 Methodology Report (Wine, Janson &
Wheeless, 2011) and the NPSAS04 Methodology Report (Cominole, Seigel, Dudley, Roe & Gilligan, 2004). The exact
number of items per respondent for the Recipient Survey depends on their current post-secondary enrollment status;
estimated burden takes this variation into account.

15

year; two PIs and/or co-PIs at each institution for a total of 12 interviews each year; four STEM
faculty at each institution for a total of 48 interviews each year; one program administrator per
institution for a total of 6 interviews each year; and up to 8 other campus representatives per
institution for a total of 48 interviews each year. These site visits will involve interviewees and
focus group participants for the purposes of qualitative data gathering; each will spend an
average of 45 minutes with the site visit team.
A.12.1. Number of Respondents, Frequency of Responses, and Annual Hour Burden
Web-based surveys will be administered to the 483 PIs and 8,220 recipients (survey instruments
are included in Appendices A and C). The total annual response burden for surveys is estimated
to be 2,872 hours across the total of 8,703 survey respondents. For the site visits, in-person
interviews and focus group sessions will be conducted with an average of 204 individuals per
year, representing an estimated total annual response burden of 143 hours. Exhibit A.4 illustrates
the number of average respondents per year to be surveyed and interviewed for each respondent
type and the time demands those surveys and/or focus groups/interviews will place on respective
respondents/participants.
A.12.2. Hour Burden Estimates by Each Form and Aggregate Hour Burdens
The total burden across all forms is estimated at 3,015 hours. The total response burden for
respondent surveys is estimated to be 2,872 hours per year across 8,703 respondents. The total
response burden for interviews/focus groups is expected to be 143 hours per year across 204
respondents. The hour burden by form and aggregate burden are included in Exhibit A.4. Data
will be collected from individuals using only one form, thus the hours by form is the same as the
hour estimates for respondent groups. The only exception is for those institutions selected for a
site visit, in which a given individual (such as a Principal Investigator or currently-enrolled
recipient) may respond to a survey and participate in an on-campus/site visit interview; this
occurrence will be limited given the small number of institutions selected for site visits relative
to the number of institutions in the survey sampling frame.
A.12.3. Estimates of Annualized Cost to Respondents for the Hour Burdens
The overall annualized cost is $59,673. Exhibit A.5 shows the estimated total annual costs to
each group of respondents.

16

Exhibit A.4: Estimated Total Annual Burden Hours
Respondent Type
Surveys
Current/former scholarship
1
recipients
Associate’s degree
Bachelor’s degree
PIs
Total
Interviews during Site Visits
(per year)
Scholarship recipients
PIs/co-PIs
STEM faculty
Program admins.
Other campus reps.
Total
Total
1

# of Respondents

3,074
5,146
483
8,703
Number of
Participants
90
12
48
6
48

Time Per Response
(Hours)

Total Time Burden
(Hours)

0.33
0.33
0.33

1,014
1,698
159.39
2,872
Total Time Burden
(Hours)
68
24
24
3
24
143

Time Per Response
(Hours)
0.75
2
0.5
0.5
0.5

204
8,907

3,015

Mean response times for the Recipient Survey, which is comprised of items from the BPS:04/09 survey and the
NPSAS:04 survey, are based on response times reported in the corresponding BPS:04/09 Methodology Report (Wine,
Janson & Wheeless, 2011) and the NPSAS04 Methodology Report (Cominole, Seigel, Dudley, Roe & Gilligan, 2004).
The exact number of items per respondent for the Recipient Survey depends on their current post-secondary enrollment
status; estimated burden takes this variation into account.

Exhibit A.5: Estimated Cost to Respondents

Respondent Type

Surveys

Current/former
2
scholarship recipients
Associate’s degree
Bachelor’s degree
PIs
Total
Groups/Interviews
(per year)
Scholarship recipients
PIs/co-PIs
Faculty
Program admins.
Other campus reps.
Total

Hourly Salary
1
Estimate

Burden Time
Per
Respondent

Estimated
Cost to
Respondent

Number of
Respondents
in Category

Estimated
Annual Cost
Across All
Respondents

$16.35
$18.75
$45.14

0.33
0.33
0.33

$5.40
$6.19
$14.90

3,074
5,146
483

$16,600.00
$31,854.00
$ 7,197.00
$55,651.00

$15.38
$45.15
$45.15
$22.12
$31.25

0.75
2
0.5
0.5
0.5

$11.54
$90.30
$22.58
$11.06
$15.63

90
12
48
6
48

$ 1,039
$ 1,084
$ 1,084
$
66
$ 750
$ 4,022
$ 59,674

Total
Notes:
1
Salary estimates are based on education and profession figures provided by the U.S. Department of Education, National
Center for Education Statistics. (2010). The Condition of Education 2010 (NCES 2010-028).
2
The salary for former scholarship recipients who have received an Associates or Bachelor’s degree will be higher than
those who are still seeking degrees, however, we won’t know the number of the former until analysis is underway. Thus,
this amount reflects an average of the two populations: those who have ‘some college’ and those who have received
their respective Associates or Bachelor’s degree.

17

A.13. Estimates of Total Capital and Startup Costs/Operation and Maintenance Costs to
Respondents or Record Keepers
There is no overall annual cost burden regarding capital, operation, or maintenance costs to
respondents as a result of this study. Respondent burden is confined to time spent responding to
the survey and/or participating in a focus group/interview.
A.14. Estimates of Cost to the Federal Government
The estimated cost to the Federal Government for the data collection activities included in this
request for approval is $1,281,000. This cost estimate includes instrument development and
pretesting; recruitment; data collection; and data processing.
A.15. Changes in Burden
This is a new collection of information.
A.16. Plans for Publication, Analysis, and Schedule
The purpose of this study is to examine the implementation of S-STEM at the local project level
and to assess the educational and career outcomes of S-STEM scholarship recipients. The study
will provide evidence that addresses the following questions: How do individual awardees
implement their S-STEM projects? What are the educational and career outcomes of S-STEM
scholarship recipients and how do their outcomes compare to an appropriate comparison group
and national trends? Are there student or programmatic outcomes associated with receipt of an SSTEM award? What are the relationships between specific project features or practices and
student outcomes? Are there promising practices or lessons learned about implementation of SSTEM projects?
Data from the S-STEM Monitoring System, PI Survey, and interviews/focus groups during site
visits will be examined through simple frequencies as well as descriptive summaries of emergent
themes in open-ended comments. Descriptive analyses of data from the Recipient Survey will be
conducted. For survey items using continuous scales, analyses will calculate means and standard
deviations to describe both central trend and variation. Frequency distributions and percentages
will be used to summarize answers given on categorical or ordinal items. Quasi-experimental
analyses will use both logistic and general linear regression analyses incorporating propensity
score matching to compare differences in dichotomous (e.g., did respondent complete a
Bachelor’s degree) and continuous (e.g., time to degree) outcomes between S-STEM scholarship
recipients and a matched comparison group of BPS respondents. Correlational analyses using
multi-level regression models (to account for clustering of recipients within institutions) will
examine associations between S-STEM program characteristics (using data from the PI survey)
and recipient experiences and outcomes (data from the recipient survey). Details of these
analyses are included in Supporting Statement, Section B and Appendix E.
An evaluation report will be prepared based on these analyses. This report will describe how SSTEM is being implemented at two- and four-year institutions of higher education, including, for
example, the types of educational opportunities and support services that were available to
scholarship recipients. In addition, the report will compare S-STEM recipient outcomes (e.g.,
degree attainment, time-to-degree, employment in STEM or non-STEM occupations) to a

18

matched comparison group. The full evaluation report will be preceded by an executive
summary of findings.
In addition to the full evaluation report and its executive summary, a separate dissemination
report will be prepared with the intent of making the key evaluation findings accessible to a
broad audience. The report will describe scholarship recipient experiences, academic and career
outcomes, and relationships between project characteristics and scholarship recipient outcomes.
It also will describe the results of a quasi-experimental comparison of recipient outcomes to a
comparison group of matched postsecondary students. The report will include a background
discussion of the context of the study, including program goals and relevant issues in STEM
education and training, explanations of the data collection and analytic methods, and a discussion
of the limitations of the study. Results will be discussed within the study and program context.
The report will include an executive summary and an appendix with technical information and
instrumentation.
NSF will make publicly available the executive summary, final report, and other documents
describing the findings of the evaluation of the S-STEM Program after review and clearance by
the NSF. The contractor conducting the program evaluation on behalf of NSF will submit the
report for review by the program, the COR, and other NSF officials for clearance. NSF will
review the quality of the reporting, data analyses, findings, and descriptions of the limitations of
the study; ensure that there are no errors in the description of the program and its awardees; and
assess compliance with privacy laws, regulations, and policies. To facilitate the timely
dissemination of the findings to the public and program stakeholders, NSF is reducing its
reliance on formal and traditional publication methods and publication formats. Once the review
is completed, the NSF will make public the executive summary, the final report, and other
documents describing the findings through a variety of means that are tailored to the information
needs of the public and the program stakeholders. In addition, on a case-by-case basis, NSF
could support dissemination of information through conferences and publication avenues by the
contractor. Exhibit A.6 outlines the study timeline.
Exhibit A.6: Project Timeline
Activity
Conduct site visits
Extant data extraction
Construct study sample
Update contact information of recipients
Field surveys
Analyze data
Prepare reports of findings

Timeframe
2 months after OMB approval
September 2012-June 2013
June-July 2013
August 2013-September 2013
4-12 months after OMB approval
13-19 months after OMB approval
19-27 months after OMB approval

A.17. Approval to Not Display Expiration Date
The data collection instruments will display the expiration date.
A.18. Exceptions to Items 19 of OMB Form 83-I
No exceptions are sought.

19