Memo for OMB; Responses to comments for AGEP

MEMORANDUM

To : Sharon Mar, Policy Analyst

Office of Management and Budget

From : Suzanne, Plimpton, Reports Clearance Officer

National Science Foundation

Date :October 21, 2009

Re :Request for OMB Clearance for Survey Instruments to be Used in the National Evaluation of the Alliances for Graduate Education and the Professoriate (AGEP) Program

The National Science Foundation has prepared the following responses to comments that the Office of Management and Budget has made with regard to the above referenced request for clearance. Our responses are numbered to parallel the numbering of OMB’s comments:

1. NSF has revised the language in the survey instruments and in the Clearance Request. The word “impact” is no longer included.

2. The Faculty Survey Sample: NSF will not administer the faculty survey.

The Student Survey Sample: The Principal Investigators of the AGEP grants have provided NSF with the names and e-mail contact information for all students taking part in AGEP sponsored activities. The names number about 5500. We believe it is important to survey the entire population for the reasons that follow. All have to do with the small size of the AGEP sample population.

The AGEP program focuses on providing support services to populations that are underrepresented in doctoral programs in science, technology, engineering, and math (STEM). African Americans, Hispanic Americans, and Native Americans/Pacific Islanders are the targeted populations. Because some AGEP sponsored programs target specific minority populations, and because others may be more efficacious with some populations than others, the analyses that will be conducted in the evaluation will look at differences in responses between the different ethnic/racial groups. We are not aggregating these populations into one group called “underrepresented minorities” or “non-whites.” Rather, we will need to make comparisons of responses between different ethnic/racial groups. Consequently, we will need to have large enough samples of respondents within each group in order to conduct statistically meaningful analyses. We do not know how many participants in AGEP there are from each minority group, so the best way to gather the data is to sample the entire population.

It is conceivable that differences in disciplinary cultures can affect student responses to AGEP programming. Consequently, our analyses of student responses will need to look at differences in responses by discipline. The number of minority students within any one STEM discipline is relatively small, and we need to sample the entire population in order to have a reasonable sample size. This is particularly true if we are looking at differences by ethnic or racial group within each discipline.

It is conceivable that we will see differences in responses to AGEP programs based on survey respondents socio-economic status, or in the case of Hispanic respondents whether the respondents are enrolled in institutions in Puerto Rico or on the U.S. mainland. We will need to have a large enough sample to allow for demographic analyses within each ethnic/racial group. Because we do not know in advance the demographic breakdown of AGEP participants within each ethnic/racial group or how people are distributed across the nation, surveying the entire population is the most efficacious way to gather this important information.

Native Americans are often called the “forgotten minority” in statistical analyses. Their numbers are small, particularly at the graduate school level. Our study needs to gather information on how Native American students respond to AGEP programs. Since there is no way to determine a priori how many Native American students are currently taking part in AGEP, the best way to gather information on Native American student participation in and responses to the program is to survey the entire AGEP student population.

3. a. The Faculty Survey will not be administered.

b. The questions on the student survey instrument were derived from work that the AIR has conducted on behalf of NSF over many years. AIR conducted a Pilot Study on AGEP in 2007. This study entailed extensive field work on campuses with AGEP sponsorship. AIR staff conducted interviews and focus groups with AGEP program administrators, faculty members, and students to determine specific AGEP program elements and program participant responses to these elements. As part of the current National Evaluation of AGEP, AIR also conducted site visits to a different set of AGEP campuses in the spring of 2009. Findings from both of these sets of interviews and focus groups are available in reports submitted to NSF.

The survey items were derived from these findings and from a previous survey instrument that AIR used in the Evaluation of the NSF Minority Institutions of Excellence (MIE) grant program. The MIE survey instrument was approved by OMB before the survey was administered. The survey instruments for the AGEP evaluation were field-tested by AIR during the 2008 Southern Regional Education Board (SREB)’s Institute on Teaching and Mentoring, at which 151 individuals currently taking part in AGEP filled out the survey. Survey responses were analyzed and individual survey items were modified according to questions raised by the respondents.

d. The language in the ethnicity and race questions has been modified in accordance with OMB’s suggestion. Specifically, the “Do not wish to provide” response has been deleted. Furthermore, the instructions accompanying the questions now say “Please select one” for the question concerning ethnicity, and “Please select one or more,” with regard to the particular race options.

4. The language in the survey documents and in the Request for Clearance concerning the Privacy Act has been modified. In both places language specifies that “the data will be protected under the Privacy Act,” pursuant to OMB’s suggestion.

5. The language in the original survey question 69 regarding “disability status” has been changed. The new questions 69-74 incorporate language from proposed census questions suggested by OMB and endorsed by The Washington Group on Disability Statistics. These questions are in track change format.

6. The language in the Support Document concerning the burden of time on respondents has been changed to reflect OMB’s concern about consistency. The changes in the Support Document are marked as track changes.