3145-0121 2006 Part B

3145-0121

Section B. Collections of Information Employing Statistical Methods

B1. Respondent Universe

Based on current NSF files of REU program awards, there were 413 PIs of ENG REU Supplement or Site awards in FY 2006. It is estimated that these 413 awards supported a total of 1,622 REU students in 2006 and utilized 1,349 PIs and faculty to mentor the students. All participants will be surveyed, so no sampling will be used. We expect a response rate of 90% from the PIs providing contact information for the faculty and students who participated in their awards and a response rate of 75% from the faculty and students.

Estimated Number of ENG REU Awards for 2006 and Estimated Number of Respondents
REU Division & Award Type	Number of Awards	Avg. No. Students/ Award	Est'd Total No. Students	Avg. No. Faculty/ Award	Est'd Total No. Faculty	Est’d Total Students & Faculty	Est’d Response Rate	Est’d No. of Respondents
BES Supplements	30	1.5	45	1.0	30	75	75	56
CMS Supplements	47	1.5	71	1.0	47	118	75	88
CTS Supplements	44	1.5	66	1.0	44	110	75	83
DMI Supplements	59	1.5	89	1.0	59	148	75	111
OII Supplements	52	1.5	78	1.0	52	130	75	98
ECS Supplements	55	1.5	83	1.0	55	138	75	103
EEC Supplements	22	1.5	33	1.0	22	55	75	41
ERC Supplements	20	7.5	150	10.0	200	350	75	263
EEC Sites	84	12.0	1,008	10.0	840	1,848	75	1,386
Total	413	--	1,622	--	1,349	2,971	75	2,228
PIs	413	--	--	1.0	413	413	90	372

B2. Procedures for the Collection of Information/Limitations of the Study

This study will be conducted as a Web survey. The respondent pools are the universe of participating undergraduate students, faculty mentors, and the PIs who received FY 2006 REU awards, so there will be no sampling. There is a short form for PIs to submit contact information for the faculty and students who participated in their award, two survey instruments each for faculty and students in the initial survey (one instrument is phrased for academic-year participants and another for summer participants), and one survey instrument for the follow-up survey of students. Each respondent will provide answers to the relevant survey instrument only once.

Names and contact information (e-mail and postal addresses and telephone numbers) for participants in FY 2006 REU awards made by ENG will be obtained from the PIs by SRI. All REU faculty and undergraduate students in the study will then be contacted by e-mail, provided with the URL of the survey questionnaire and asked to go to that Web site to complete the instrument. If requested, or if an e-mail address is not available for a participant, a hard copy of the questionnaire will be mailed to the individual’s home address, if available.

This study is a correlational design, and as such will not be represented to yield causal conclusions. Determinations of causality require objective pre- and post-measures as well as random assignment to condition (e.g, participation or non-participation in the program of interest). In the real world, these requirements are impractical, at best, and often impossible. Most program evaluation surveys, including the REU survey, rely on respondents' self-reports of program outcomes and impacts as the most practical alternative. Even though they do not provide strict evidence of causality, self-reports are widely considered to provide valuable information about program outcomes, impacts, areas of strengths and weaknesses, and so on.

B2.1. Statistical methodology for stratification and sample selection

Not applicable. All members of the survey universe are included in the survey.

B2.2. Estimation procedure

The profile of REU students’ survey responses by award type (Supplement vs. Site) and ENG division will be compared against the universe profile based on award type and ENG division. If there are statistically significant differences in the profiles of respondents, survey responses will be weighted so that the overall respondent profile parallels that of the universe.

B2.3. Degree of accuracy needed for the purpose described in the justification

Not applicable

B2.4. Unusual problems requiring specialized sampling procedures

Not applicable

B2.5. Use of periodic (less frequent than annual) data collection cycles

This is a one-time collection in 2007 about the FY 2006 REU awards with a follow-up collection in 2009 for the student participants.

B3. Methods to Maximize Response and Deal with Issues of Nonresponse

Several study design and timing situations or features are expected to bring about strong response rates:

(1) The REU survey is being conducted via the Internet to minimize the effort and time required of respondents to complete the questionnaire. All of the PIs and faculty and over 90% of the REU student population have access to the Internet, based on experience from the 2002 URO study;

(2) The PIs are regular NSF awardees who are accustomed to providing information about their project participants, activities, and results for GPRA purposes and individual evaluation studies;

(3) The surveying is timed to take place in late winter/early spring when universities and colleges are in session, maximizing the likelihood of reaching the PIs, faculty, and students well before the end of the academic year;

(4) All REU participants will have completed their REU experience no more than 12 months before being surveyed, so the contact information provided by the PIs is likely to be accurate and what the students did during REU along with the results of their experience should be easy to recall;

(5) A $20 gift certificate for a popular online retailer will be offered as incentive for students to complete the questionnaire. Use of a similar incentive for the 2002 URO study proved highly effective, resulting in 75% and 80% response rates for students.

(6) SRI will send follow-up e-mail reminders to non-respondents approximately once a week over at least a 6-week period following commencement of the surveys. Reminders will be sent on different days of the week and times of the day.

B4. Tests of Procedures or Methods

The questionnaire is similar to the instrument used in surveying the 2002 URO participants. A few questions were added to evaluate ENG encouragement of ethics and communication training and international experiences and activities specific to engineering research. Several questions were dropped so that the instrument wouldn’t increase in length and time needed to complete it. Pretesting of each instrument will be done with 6 to 8 people. The only appreciable change expected to the instrument as a result of the pretesting is the possible addition of a couple of response categories based on responses of pretest participants.

B5. Names and Telephone Numbers of Individuals Consulted

Survey Specialist:

Susan Russell, PhD

Director, Survey Research Program (Retired)

SRI International

333 Ravenswood Avenue

Menlo Park, CA 94025

(650) 859-4164

Agency Project Director:

Linda E. Parker, PhD

Engineering Program Evaluation Director

Division of Engineering Education and Centers

National Science Foundation

Arlington, VA 22230

(703) 292-5355

Agency ENG Program Managers:

Joanne Culbertson

Staff Associate for Planning and Evaluation

Office of the Assistant Director for Engineering

National Science Foundation

Arlington, VA 22230

(703) 292-4602

Mary Poats

ERC Program Manager

Division of Engineering Education and Centers

National Science Foundation

Arlington, VA 22230

(703) 292-5357

Esther Bolding

REU Program Manager

Division of Engineering Education and Centers

National Science Foundation

Arlington, VA 22230

(703) 292-5342