June 24, 2009
Revised November 3, 2009
Name: G. Stephane Philogene, Ph.D.
Address: National Institutes of Health
31 Center Drive, Room B1C32
Bethesda, MD 20892-2027
Telephone: 301.402.3902
Fax: 301.402.1150
Email: PhilogeS@OD.NIH.GOV
B. Collection of Information Employing Statistical Methods 1
B.1 Respondent Universe and Sampling Methods 1
B. 2. Procedures for the Collection of Information 1
B.3. Methods to Maximize Response Rates 2
B.4. Test of Procedures or Methods to be Undertaken 3
B.5. Individuals Consulted on Statistical Aspects and Individual Collecting and/or Analyzing Data 3
List of Attachments:
Attachment 1: NDPA Program Description and Requirements
Attachment 2: Privacy Procedures of Contractor
Attachment 3a: NDPA Applicant Awardee Interview Protocol
Attachment 3b: NDPA Survey of Unfunded Applicants
Attachment 3c: NDPA Pioneer Lab Member Interview Protocol
Attachment 4: Respondent Instructions
Attachment 5: Introductory & Follow-up Instructions
This request seeks approval for OMB clearance for the NIH Director’s Pioneer Award (NDPA) Program Outcome Evaluation. The data collection consists of interviewing NDPA awardees (pioneers), members of each Pioneer’s labs, and a Survey of Unfunded Applicants. The information gathered from these interviews, visits to Pioneer labs, and focus group panels will document whether pioneers are conducting pioneering research and the impact of the NDPA program on NIH. The findings will be used as a guide to the program officers in their future strategic and management decisions.
A census will be conducted of all NDPA awardees for the FY 2004 and FY 2005 competitions. A census will be conducted for all NDPA unfunded applicants. The total number of NDPA awardees is 22, and the NDPA unfunded applicants is 440. A census is requested rather than a sample because of the program’s small number of awardees and unfunded applicants. Sampling would result in the number of respondents from important sub-groups being too small to permit comparative analyses, as well as preclude analyses of factors that influence various outcomes.
In addition, the study design includes site visits to a subset of 5 Pioneer institutions. Members of the Pioneer lab will be interviewed during each site visit. The subjects will be chosen to include scientists from different stages in their careers – graduate students, postdocs, research technicians. This sample is estimated to include 20-30 individuals, 4-6 per research lab.
At the outset of the survey process, an email from NIA officers will contact all participants to describe the purpose of the evaluation, the procedures being used, and requesting that they participate in the survey. They will also introduce the Science and Technology Policy Institute. To review all communications with the participants please consult Attachments 4 and 5 to this document.
This request for clearance includes four types of data collection: (1) interviews with Pioneers (2) a survey of unfunded applicants, and (3) Pioneer lab members interviews.
Interviews. Interviews will be conducted by telephone, with the exception of in-person interviews during site visits. Basic information about the interviewee will be collected in advance to the extent possible (e.g., title, affiliation, research topic) and verified with respondents during the interview. Interview protocols are included in Attachments 3a and 3c.
Survey of Unfunded Applicants: The Science and Technology Policy Institute will contact all participants with information on how they should locate the survey on the Internet, and procedures for completing the survey. Respondents will be allowed 3 weeks to complete the survey. In order to facilitate completion, individuals may either fill out the survey in one sitting, or they may pause at any time, and return later to finalize the survey. The survey will be organized by topics and participants can move from section to section via a central navigation interface.
Once the individual surveys are underway, the Science and Technology Policy Institute will monitor the status of the submissions (Never Logged In, Not Started, In Progress, and Complete). After two weeks, the first reminder will be sent to individuals who had never logged in. Individuals who have initiated but have not yet completed the survey will receive a note encouraging them to complete their survey; hard-copy of the survey is mailed to participants who have requested it in paper form. All persons who have completed the survey will be sent a thank you note. The second reminder email will be sent two weeks after the first (and about four weeks after the start of the survey process). All new respondents will be thanked at that time, and the persons who had started the survey will be encouraged to complete it. The survey will be closed at the end of week 8. The Survey of Unfunded Applicants is in Attachment 3b.
Staff at the Science and Technology Policy Institute will perform all analyses.
This study is being conducted with the aim of describing whether NDPA Awardees are conducting pioneering research with NDPA funds. Definitions of “pioneeringness” aimed at research conduct (and outcomes) can be measured, but require in-depth understanding of each researcher’s activities. The program’s experimental nature and complex/hard-to-measure definition of “pioneering” suggests that an ethnographic, case study approach be used. The “spillover effects” of the NDPA program on awardees, their students, and their institutions, as well as on NIH and on the scientific community will also be assessed.
Once the analyses are complete, the Science and Technology Policy Institute will prepare a thorough documentation of the findings and conclusions, and suggest specific recommendations regarding program activities, accomplishments, and outcomes.
The Outcome Evaluation of the NDPA program will make an important contribution to the program evaluation field, as it will explore the suitability and reliability of the data collection and analysis strategies, and these approaches may then be adapted to evaluate other programs at the NIH. To disseminate the evaluation findings to the evaluation and the scientific communities, efforts also will be made to publish the results of the Outcome Evaluation in a professional journal and to present results at conferences.
Several methods will be used to maximize response rates, such as:
Providing sufficient timeframe for data collection. For example, interviews will be carried out over the course of several months, to make sure that busy schedules of respondents can be accommodated;
Reminder emails will be sent to unfunded applicants at regular intervals, and
Reminding Pioneers that they are required to participate in evaluation activities by the terms of the grant.
Phone and email will be sent to the awardees (pioneers) and their lab members to encourage participation in the study.
Interview protocols will be tested on awardees and pioneer lab members. The Survey of Unfunded Applicants will be tested during a pilot study. Focus group questions will be reviewed by the Science and Technology Policy Institute staff, NIH staff, and NDPA advisors.
Several individuals from Science and Technology Policy Institute participated in the development of the study plan. The team included persons with knowledge of statistical methods, experience in evaluation of science programs, and expertise in scientific research. Bhavya Lal is Project Director of the NDPA process evaluation and the point of contact for this project; she has extensive evaluation experience with NIH and other Federal agencies. She can be contacted at:
Bhavya Lal
Science & Technology Policy Institute
1899 Pennsylvania Avenue NW, Suite 520
Washington DC 20006
Telephone: (202) 419-3720
Facsimile: (202) 223-1695
E-mail: blal@ida.org
Follow-up emails and telephone calls will be conducted by qualified researchers at the Science and Technology Policy Institute. The project team from the Science and Technology Policy Institute can be reached at (202) 419-3720
| File Type | application/msword |
| File Title | NDPA Proc Eval 30 day Fed Reg Supporting Statement |
| Author | STPI |
| Last Modified By | sshipp |
| File Modified | 2009-11-04 |
| File Created | 2009-11-04 |