11/04/2010 Responses

Responses to OMB questions about NSF’s Evaluation of EAPSI/IRFP, Set 2

1. Please provide us with a cross-walk of questions in the questionnaire to the 7 research questions provided.

The evaluation is designed to answer the following seven research questions:

1. What are the characteristics of people who apply for and participate in the EAPSI and IRFP programs?

2. What motivates individuals to apply for and participate in the programs, and what are individuals’ experiences during the application process?

3. What are the program experiences of program participants and managers?

4. What are the perceived outcomes of program participation?

5. Do fellows’ post-award career activities and job characteristics differ from unfunded applicants?

6. Does the extent to which former Fellows engage in international collaborations differ from those of unfunded applicants?

7. Do the outcomes of program participation extent beyond the direct participants?

The Exhibit below summarizes, for each research question, specific topics of investigation (i.e., major constructs) and the survey items that address each topic.

Exhibit Q1. Crosswalk of Research Questions, Survey Topics and Individual Survey Items
Topic	NSF Extant data	IRFP Applicant Survey	IRFP Host Survey	EAPSI Applicant Survey	EAPSI Host Survey	EAPSI Advisor Survey	EAPSI interviews
RQ1: What are the characteristics of people who apply for and participate in IRFP?
Award status (fellow, unfunded)	x	A2.1-2.2		A2.1-2.2
Cohort year	x	A2.1-2.2		A2.1-2.2
Proposed host location	x	A3		A3
Demographic information	some	G1-G5a		G1-G5a
Academic background		A4a-b, A5, C1, C2a, C8		A4, A5, A5a-b, A6, A7, C1, C9
Prior international experience		C2b, C3a-b, C4		C2a-b, C3, C4, C5, C6		A2, A3
Publications		C7		C8
Publications with international collaboration		C7		C8		G1b
Prior collaboration with proposed host		C5a-b		C7a-b
Prior relationship between US and host institution		C6		C7c	A10	A2
RQ2. What motivates individuals to apply for and participate in the programs, and what are individuals’ experiences during the application process?
Motivation to apply/participate in general, specific location		B1, B2	B2	B1, B2	B2	C2	x
Experiences during application process, arrangements for fellowship		B3		B3, B6 B6a-b	C1, C1a		x
Support provided during application process		B4, B5		B4, B5		B3, C1, C2, C4
RQ3. What are the program experiences of program participants and managers?
Language preparation		E1	C2	E1	C4
Inter-cultural, professional opportunities		E2, E3, E3a		E2, E3, E3a
The research collaboration		E4	C1, C2	E4, E7	C3, C4, C6
Interaction between fellow/host scientist		E6, E7	C3	E6, E8	C2
Barriers encountered		E5		E5
Support provided to fellow		E7		E7, E8		D1, E1	x
Satisfaction with participation in fellowship		E7, E7a	C2, E1, E1a	E8, E8a	C4, C5, E1	H1, H2, H5

Exhibit Q1 (continued). Crosswalk of Research Questions, Survey Topics and Individual Survey Items
Topic	NSF Extant data	IRFP Applicant Survey	IRFP Host Survey	EAPSI Applicant Survey	EAPSI Host Survey	EAPSI Advisor Survey	EAPSI interviews
RQ4. What are the perceived outcomes of program participation?
Post-fellowship collaboration		F1, F1a-c	D1, D1a, D3	F1, F1a-c	D1, D1a, D3	C3	x
Effects of participation on career (educational) advancement/opportunities		F2, F2a-c, F6, F13		F2, F2a-c, F6, F13		C3, E4
Research or professional benefits of participation		F3, D6		F3, D6	E2	C3
Personal benefits of participation		F5		F5	E2	C3
Would recommend participation to others?		F9, F9a, F10, F11	E7	F9, F9a, F10, F11	E7	H3
Overall satisfaction/recommendations for change		F7, F8, F12	E4, E5, E8	F7, F8, F12	E3, E5, E8	H5	x
RQ5. Do fellows’ post-award career activities and job characteristics differ from unfunded applicants?
Employment characteristics		D1a, D1c, D2, D2a-d, D3, D9		D1, D1c, D2, D2a-d, D3, D9
Research: External funding awards/honors		D5, D5a-b		D5, D5a-b
Publications		D6		D6
RQ6. Does the extent to which former fellows engage in international collaborations differ from those of unfunded applicants?
Employment (includes postdoctoral) outside U.S.		D1b, D4b, D7, D7a		D1b, D4b, D7, D7a
Collaboration with colleagues outside U.S.		D4, D4a, D6		D4, D4a, D6
Fostering international engagement of others		D8, D10		D8, D10
RQ7. Do the outcomes of program participation extent beyond the direct participants?
Benefits of participation to US/foreign colleagues/ institutions		F4, D8, D9, D10	E2, E3, E6	E7, F4, D8, D9, D10	A8a, D2, D2a-b, E2, E4, E6	E2, E3, F1, F2, F3, G1a-b, G2, G4, H4	x

2. Related, what are the key outcomes on which you wish to compare the two groups (e.g., number of international collaborations since graduation)?

The two research questions that compare outcomes for the two groups are questions #5 (career activities and job) and #6 (international collaborations). The Exhibit below lists the specific outcomes under each of these research questions

Exhibit Q2. Specific Outcomes for Research Questions #5 and #6.
Outcomes for RQ #5: Do fellows’ post-award career activities and job characteristics differ from unfunded applicants?
Total number of postdoctoral fellowships (IRFP only)	D1a
Grant(s)/award(s)/honor(s) for research from international professional association or other institution outside U.S.	D5b
Current employment as research faculty at 4-year college/university, medical school, or university-affiliated research institute	D2a, D2b
Current faculty rank of Assistant, Associate or Full Professor	D2c
Currently has tenure (controlling for # of years since PhD)	D2d
Total number of “post-award” publications	D6
Outcomes for RQ#6: Does the extent to which former Fellows engage in international collaborations differ from those of unfunded applicants?
Number of international postdoctoral fellowships	D1b
In current job, works with individuals located in other countries	D4
Number, proportion of publications co-authored with a foreign collaborator (ratio of # publications with foreign co-author to total # of publications).	D6
Employment outside the U.S. since [year marking end of fellowship period]	D7
Has mentored others from the U.S. traveling to another country to conduct research	D8
Leadership in fostering international collaboration: established a program to foster international collaborations; Hosted researchers or colleagues from another country; Led a delegation of U.S. colleagues to another country; and/or Established or served as a leader in an international professional association	D10
Duration of employment outside the U.S.	D7a
Type of current work with individuals in other countries includes joint publications and/or jointly-developed products	D4a

3. What MDE for these outcomes does NSF consider realistic with this evaluation? What literature are these sizes based on?

We expect the program effects to be small in outcomes areas of interest. These small effects are substantively meaningful and important in the outcomes we will measure, because innovation and transformative scientific discovery can occur within a single lab or research collaboration.

Evaluations and research with comparison groups have not been conducted of international fellowship programs such as EAPSI and IRFP. Thus, we have extrapolated from research conducted of science research and fellowship programs to obtain expected effects sizes for this evaluation. Below provide some details from some evaluations of NSF’s graduate and early career programs.

In the recent evaluation of NSF’s IGERT program, ¹ a graduate training program, differences between IGERT trainees and comparison group members on outcomes related preparedness for research careers ranged from .04 to .16. The evaluation of NSF’s CAREER program, ² a fellowship program for early career faculty that compared outcomes for CAREER fellows and non-awardees effect sizes ranged from 0.01 to 0.38. Some examples of effect sizes for outcomes similar to the EAPSI and IRFP evaluation outcomes include tenure (.21), publications (.08), patents (.24), and research collaborations with target groups (.09). In a prior evaluation of NSF’s Graduate Research Fellows,³ a program that has some similarities to EAPSI, many of the differences between fellows and the comparison group were less than ten percent. For instance, among seven different accomplishments during graduate school, the largest difference found between the fellows and the comparison group was eight percentage points.

4. We understand that the method will likely be the same for NSF to locate evaluation respondents whether they are treatment or control. However, what are the expected response rates of unfunded applicants? We would imagine that this population is less motivated and inclined to respond. Additionally what are the expected response rates of hosts/advisors?

We agree that unfunded applicants may be less inclined to participate, and thus expect the response rate to be lower among unfunded applicants by 10 percentage points than among fellows, as displayed in the Exhibit below. We make this estimate based on the differential in recent studies of NSF funding programs of early career researchers and graduate students. In the evaluation of NSF’s CAREER program the response rate for awardees was 84% and for non-awardees 80%. In the recent IGERT programs, the response rate among IGERT trainees was 74% and among the comparison group 52%, however, the low response rate for the comparison group was due in part to the inability to find individuals – the cooperation rate among those located was 69%.

Advisors and hosts are research scientists who are familiar with the National Science Foundation. Thus, we expect the response rates to be just slightly lower than those of Fellows.

Exhibit Q4. Expected Response Rates
Length of Time Between Participation and Data Collection	Fellows	Unfunded Applicants	Hosts/Advisors
0-5 years	85%	75%	80%
6-10 years	65%	55%	60%

5. NSF’s justification for using a census approach of applicants seem reasonable. However, why would NSF conduct a census of hosts/advisors?

Data gathered from hosts and advisors will be used to meaningfully represent the program and to inform subsequent decisions about the program. Thus, the desired sample size is driven by our attention to the precision of the estimates that will results from this study.

Using the response rates that account for when the hosts and advisors were associated with the program that were provided in question 4 above, the Exhibit below illustrates the effect these projected response rates will have on the number of hosts for the IRFP and EAPSI programs and advisors of EAPSI applicants.

Exhibit Q5a. Projected Responses for Advisors and Hosts
Respondent Type	Target group Total	Target Group by Length of Time Between Participation and Data Collection		Projected Responses
		0-5 years	6+ Years	0-5 years	6+ Years	Total
EAPSI US Advisors	1,241	778	463	622	278	900
EAPSI Foreign Hosts	1,156	741	415	593	249	842
IRFP Foreign Hosts	559	192	367	154	220	374

Assuming a simple random sample and 95% confidence level the table below displays the sample size we would need to achieve various levels of precision; sample size is calculated as (p*(1-p))/ (Precision / 1.96)^2, where p was set equal to 0.50.

Exhibit Q5b. Number of Respondents and Precision
Precision	n
0.020	2401
0.030	1067
0.035	784
0.040	600
0.045	474
0.050	384

Ideally, we would design the study to have a precision of .3, which is a plus or minus 3 percentage point margin of error. Thus, the study is designed to go the full census in order to get as near this level of precision as possible.

6. Based on the propensity score model on the bottom of p. 3 of NSF responses to OMB comments, how many applicants fell into the tail ends of those included in the impact analyses? Why would those applicants in the tail end be included in the survey if they will be excluded from the analysis?

The propensity score model will include variables that are present in extant data, as well as variables that are collected for the first time via the study’s survey. Hence we do not know which applicant will be in the tail ends before data collection. A list of the variables to be included and their source is included in the Exhibit below.

Exhibit Q6. Pre-award data used to construct comparable groups of IRFP/EAPSI fellows and unfunded applicants for impact models
Pre-award characteristic	Reason for inclusion in propensity score model	Data Source(s)
Mean proposal score	Mean across reviews indicates quality of application	NSF Extant Data1
Cohort year	Control for cohort differences	NSF Extant Data1
Host location in application	Desire for geographic balance in portfolio; limited openings in some EAPSI sites	NSF Extant Data1
Geographic density2	Desire for geographic balance in portfolio; limited openings in some EAPSI sites	NSF Extant Data1
		Applicant Survey Item
		IRFP	EAPSI
Gender	Preference given for females in fields with under-representation	G1	G1
Under-represented minority status	White or Asian = 0, other race(s)/ethnicity = 1	G2, G3	G2, G3
Citizenship status	US citizenship (birth, naturalized) required	G4, G4a	G4, G4a
Disability status	Preference given to disabled applicant	G5, G5a	G5, G5a
STEM discipline	Desire for disciplinary balance in program portfolio	A5	A7
Undergraduate GPA	EAPSI application requests undergraduate transcripts	--	C1
Had tenure-track position	Unfavorable for IRFP applicant	C1	--
Highest degree held	Had PhD at time of application (=1) or expected by time of award (=0)	C2a	--
Graduate degree program	Master’s- or Doctoral-level program	--	A5
Degree from non-US institution	Unfavorable for IRFP applicant	C2b	--
Study-abroad as undergraduate	Prior international experience favorable for EAPSI, unfavorable for IRFP	C3a	C2a
Study-abroad as grad student	Prior international experience favorable for EAPSI, unfavorable for IRFP	C3b	C2b
Prior visit to host location	Prior exposure to host location favorable for applicant	--	C3
Prior relevant language(s)	Language aptitude or achievement favorable for applicant	--	C4
Participation in international club(s)	Favorable for EAPSI applicant	--	C5
Other prior international residential	Prior international experience favorable for EAPSI, unfavorable for IRFP	C4	C6
Prior international collaboration	Likely to be beneficial to applicant	C4	C6
Letter of support from host	Strongly favorable for EAPSI applicant	--	C7a
Prior collaboration with host	Likely to be beneficial to applicant	C5b	C7b
Already at host institution	Unfavorable for IRFP applicant	C5a	--
Prior international exposure	Prior exposure to foreign colleagues or former program fellow	C4	C6
Link between US, host institutions	Likely to be beneficial to applicant	C6	C7c
Total pre-award publications	Prior record of achievement favorable	C7	C8
% publications w/foreign collaborator	Likely to be beneficial to applicant	C7	C8
National post-collegiate fellowship	Prior record of achievement favorable	C9	C9

1 Carney, J., Martinez, A., Dreier, J., Neishi, K., & Parsad, A. (2009). Evaluation of the National Science Foundation’s Integrative Graduate Education and Research Traineeship Program (IGERT): Follow-up Study of IGERT Graduates. Prepared for the National Science Foundation. Cambridge, MA: Abt Associates.

2 Carney, J., Smith, W. C., Parsad, A., Johnston, K. & Millsap, M.A. (2008). Evaluation of the Faculty Early Career Development (CAREER) Program. Prepared for the National Science Foundation. Cambridge, MA: Abt Associates.

3 Goldsmith, S.S., Presley, J.B., and Cooley, E.A. (2002). National Science Foundation’s Graduate Research Fellowship Program: Final evaluation report. Prepared for the National Science Foundation. Los Alamitos, CA WestEd.

Response to second set of OMB comments for NSF Evaluation of EAPSI/ IRFP programs 26