June 24, 2009
Revised November 3, 2009
Name: G. Stephane Philogene, Ph.D.
Address: National Institutes of Health
31 Center Drive, Room B1C32
Bethesda, MD 20892-2027
Telephone: 301.402.3902
Fax: 301.402.1150
Email: PhilogeS@OD.NIH.GOV
A.1 Circumstances Making the Collection of Information Necessary 1
A.2 Purpose and Use of the Information 2
A.3 Use of Information Technology and Burden Reduction 3
A.4 Efforts to Identify Duplication and Use of Similar Information 3
A.5 Impact on Small Businesses or Other Small Entities 3
A.6 Consequences of Collecting the Information Less Frequently 4
A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 4
A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency 4
A.9 Explanation of Any Payment or Gift to Respondents 5
A.10 Assurance of Confidentiality Provided to Respondents 5
A.11 Justification for Sensitive Questions 5
A.12 Estimates of Hour Burden Including Annualized Hourly Costs 6
A.13 Estimate of Other Total Annual Cost Burden to Respondents or Record-keepers 7
A.14 Annualized Cost to the Federal Government 7
A.15 Explanation for Program Changes or Adjustments 8
A.16 Plans for Tabulation and Publication and Project Time Schedule 8
A.17 Reasons Why Display of OMB Expiration Date is Inappropriate 8
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 8
List of Attachments:
Attachment 1: NDPA Program Description and Requirements
Attachment 2: Privacy Procedures of Contractor
Attachment 3a: NDPA Awardees Interview Protocol
Attachment 3b: NDPA Survey of Unfunded Applicants
Attachment 3c: NDPA Pioneer Lab Member Interview Protocol
Attachment 4: Respondent Instructions
Attachment 5: Introductory & Follow-up Instructions
This request seeks approval for OMB clearance to conduct an Outcome Evaluation of the NIH Director’s Pioneer Award Program. The data collection consists of interviewing awardees (pioneers), surveying unfunded applicants, and pioneer lab members. The information gathered from these surveys will document the NDPA outcomes and will be used as a guide to the program officers in their future strategic and management decisions.
The NIH Director’s Pioneer Award (NDPA) program was established in FY 2004 as part of the NIH Roadmap for Medical Research. The goals of the NDPA Program are two-fold: (1) to support individual scientists of exceptional creativity with pioneering ideas and approaches to major contemporary challenges in biomedical research, and (2) to support new research directions that are not supported by other NIH funding mechanisms. The program awards are $2.5 million each ($500,000 per year) in direct costs for five years. The program was established to address a view prevalent in the academic community that the NIH peer review system is excessively conservative and discourages many investigators from submitting innovative applications to the agency. The NDPA is a 5-year pilot program, which uses unique application and selection processes designed to select scientists who conduct high-risk, innovative research. The award process has been highly competitive, with only a dozen grantees selected each year from hundreds of applicants through an intensive multi-stage selection process. Between 2004 and 2008, five rounds of awards have been made, for a total of 63 individuals selected. For a description of the program and its requirements, please see Attachment 1.
NIH wishes to receive OMB clearance to perform an outcome evaluation for the first two years (FY2004 and FY2005) of the NDPA program. The primary objectives of the outcome evaluation are to determine whether (1) NDPA Awardees are conducting pioneering research with NDPA funds and (2) to assess the 'spillover' outcomes of the NDPA program.
NIH has tasked the Science and Technology Policy Institute to conduct the NDPA Outcome Evaluation. The Science and Technology Policy Institute (STPI), one of three federally funded research and development centers run by the Institute for Defense Analyses, assists the Executive Branch of the US government by providing objective, high-quality analytic support to inform program managers and others who run S&T programs.
The unique mode and public nature of the program make its monitoring and evaluation a high priority for the NIH. Additionally, because it is a pilot program, an evaluation is necessary to determine the NDPA program’s long-term value to the NIH and biomedical science more generally. Because of its importance to the NIH, two complementary evaluation studies of the NDPA program have been conducted: a process evaluation that is monitoring the grantee selection and program implementation process and a Feasibility Study that yielded the design for the Outcome Evaluation, which is a subject of this application.
The program is authorized under Sections 301 and 405 of the Public Health Service Act as amended (42 USC 241 and 284) and under Federal Regulations 42 CFR Part 52 and 45 CFR Parts 74 and 92. The proposed data collection supports the management and evaluation of this program.
Information collected during the NDPA Outcome Evaluation will be used in multiple ways. First, the results of the evaluation will inform the NIH Director and senior staff on the accomplishments of the NDPA program and will guide their decisions on whether to continue and/or to modify the program. Second, several of the NIH Institutes and Centers are considering establishing similar programs and the lessons learned from the NDPA program will be valuable for the design and implementation of these new initiatives. Third, due to its high visibility, the research community has shown substantial interest in the NDPA program, including the outcomes of this unique funding mechanism, and the results of the OE will be distributed to these wider audiences. Finally, creativity in science is a notoriously difficult subject, because of its complexity and the lack of rigorous empirical data, measures, or methods. The Outcome Evaluation of the NDPA program will make an important contribution to the program evaluation field, as it will explore the suitability and reliability of the data collection and analysis strategies, and these approaches may then be adapted to evaluate other programs at the NIH. To disseminate the evaluation findings to the evaluation and the scientific communities, efforts also will be made to publish the results of the Outcome Evaluation in a professional journal and to present the findings at conferences.
The goal of this phase of the Outcome Evaluation is to assess short-term program accomplishments. Intermediate and long-term accomplishments may be evaluated in subsequent studies, but these are not the subject of this request.
Products of the Outcome Evaluation will include case studies, a final report and briefings to program stakeholders, including senior NIH staff, on the short-term outcomes of the NDPA Program, as well as a research paper and an NIH press release to the public.
Components of the NDPA Outcome Evaluation included in this clearance request are as follows:
Interviews with the 22 NDPA awardees (pioneers) from FY 2004 and FY 2005 (using an interview protocol – see attachment 3A).
Survey of Unfunded Applicants (using a survey conducted on the internet – see attachment 3C) with follow-up interviews for sub-sample of unfunded applicants (for FY 2004-2006). (Abt conducted the first Survey in 2007 for the 2004-2005 unfunded applicants.)
Interviews with Lab Members of the 22 awardees (using an interview protocol – see attachment 3C)
The interviews with each NDPA awardee (pioneers) will provide the foundation for individual case studies, which will then be reviewed by a study section to assess if pioneering research is being undertaken.
The OMB Clearance number will be appended to each interview protocol and survey. (Standard Form 424, OMB SF-424 - OMB #0925-0001, exp. date 11/30/10.
Interviews with awardees. Interview protocols will be used to collect information from awardees. To reduce respondent burden, information that is available from other sources (such as from their proposals and progress reports, as well as internet searches) will be collected prior to interviews with NDPA Awardees so that respondents will need only to verify it.
Survey of Unfunded Applicants: The proposed NDPA Unfunded Applicant Survey uses Internet technology in order to minimize burden on respondents. NDPA unfunded applicants will be sent a link to a Web-based survey that can be completed and submitted on-line. Non-respondents will be followed-up by email and/or telephone. In the process of the survey design significant effort was made to ensure easy access on-line. In addition, the majority of the NDPA Unfunded Applicant Survey questions are closed-ended (e.g. ‘yes/no’) to reduce burden on respondents. A limited number of unfunded applicants will be interviewed.
Interviews with members of Pioneer laboratories. An interview protocol will be used to collect information from members of each Pioneer’s lab, to ask them about their experiences working on high-risk research.
This data collection will be associated with an IT system to collect, use, store, maintain, disclose and possibly transmit data if necessary. Prior to starting data collection, a Privacy Impact Assessment (PIA) will be undertaken with the Privacy Act Coordinator and Information Systems Security Officer to assess privacy and security risks of the IT system.
The proposed interview protocols and discussion guides do not gather data that duplicates other efforts.
The NDPA staff has carefully examined the data that are available on individual NDPA awardees and unfunded applicants in order to assure that no information is replicated under the present evaluation. Factual information from their proposals, progress reports, and their websites will be preloaded into the interview protocol, as discussed in section A.3.
Only direct contact with program participants can yield information such as:
What has the NDPA grant allowed you to do that you could not do without it? If you had not received the NDPA, do you believe that you would have pursued this work? If so, how?
Is there a difference in the way you approach your NDPA funded research versus your other research? What is this difference? Did that change over the time of the award?
How did the goals/expectations for your research change over the time of the NDPA award?
To what extent did you need to modify your research concept or approach since the initial award?
Have you experienced any obstacles to implementing your proposed work?
Do you think that your reputation, recognition, or statute has changed as a result of receiving the NDPA?
Did the unfunded applicants choose to pursue the idea proposed in their NDPA application?
How did the applicants’ careers develop in the years following their NDPA application?
What other factors influenced the applicants’ careers?
Small businesses are not involved in this study. All survey respondents are biomedical researchers who have participated in the NDPA program.
Respondents will be contacted only once to collect information for this study. The NDPA is a pilot program, and if the data for the Outcome Evaluation are not collected, the NIH will be unable to document whether the NDPA program resulted in pioneering research that ultimately led to new methodologies or breakthroughs in biological or medical sciences.
This project fully complies with all guidelines of 5 CFR 1320.5.
The 30-day Federal Register Notice was published (Vol. 74, No. 71 / April 15, 2009, pp. 17497-17490). No comments were received in response to this Federal Register Notice.
The 60-day Federal Register Notice was published (Volume 74, No. 135/July 16, 2009, pp. 34581-34582
The Science and Technology Policy Institute had several in-person meetings and conference calls with NIH staff, which provided important background information and made suggestions on the evaluation design. The NIH staff is:
Juliana Blome, Ph.D., Chief, Office of Program Analysis and Evaluation, National Institute of General Medical Sciences
Judith H. Greenberg, Ph.D., Director, Division of Genetics and Developmental Biology, National Institute of General Medical Sciences
Teresa Levitin, Ph.D., Director, Office of Extramural Program Review, National Institute on Drug Abuse
James Onken, Ph.D., Chief Analyst, Office of Research Information Systems, Office of Extramural Research
Elizabeth L. Wilder, Ph.D., Deputy Director, Office of Strategic Coordination, Division of Program Coordination, Planning, and Strategic Initiatives (DPCPSI), Office of the NIH Director
The Science and Technology Policy Institute had several in-person meetings and conference calls with external experts, who provided important background information and made suggestions on the evaluation design. These experts are:
Teresa Amabile, the Edsel Bryant Ford Professor of Business Administration, Entrepreneurial Management Unit, Harvard Business School.
Arthur Kleinman, the Esther and Sidney Rabb Professor of Anthropology, Department of Anthropology, Faculty of Arts and Sciences, Harvard University.
Robert S. Langer, one of 14 Institute Professors (the highest honor awarded to a faculty member) at the Massachusetts Institute of Technology (MIT).
Julia Lane, Program Director Science of Science and Innovation Policy (SciSIP), National Science Foundation
Robert Sternberg, Dean, School of Arts and Sciences, Tufts University.
Awardees and interviewees will not be paid for participating in this study and will not receive any gifts in return for participation. Participation is completely voluntary.
The subjects will be informed that their responses to the Interview Protocols are to be reviewed by the Science and Technology Policy Institute for purposes of analysis and reporting. Given the nature of the study, assurance of confidentiality is provided to respondents. Some materials may be disseminated in aggregate to the public by NDPA staff in order to inform the research community of the results of the study. NIH’s System of Records and privacy procedures under the Privacy Act are included in the Privacy Act System of Records Notice (Federal Register Vol. 67 No. 187, pages 60741-60794, September 26, 2002).
The NDPA Awardee and Interviewee Interview Protocol will contain the following general assurance of confidentiality:
“…Your responses will be kept strictly confidential: If you choose to participate, respondent confidentiality will be protected to the extent provided by law, and STPI will report only aggregate information concerning overall impressions of the process to the NIH.”
In order to ensure data security, all employees of the Science and Technology Policy Institute, are required to adhere to strict standards and sign a non-disclosure agreement as a condition of employment (for details, please consult Attachment 2). The Science and Technology Policy Institute has extensive experience collecting information and maintaining its confidentiality and security. All data files on multi-user systems will be under the control of the database manager, with access limited to project staff on a “need to know” basis only.
Data collection interview protocols will be prefilled with the name of the pioneer (awardees) (attachment 3a) or interviewee (attachment 3b). The name and title of the pioneer lab members will be collected during the interview (attachment 3c). These data collection interview protocols do not include any sensitive questions. The only personally identifiable information (PII) on the data collection form is their name. Our data system also includes their employment status, address and email address, which is obtained from the NIH IMPAC II system.
There are no questions of a sensitive nature neither in the survey nor in the interview protocol. In order to understand the background characteristics of applicants, the questionnaire does ask where a given applicant has received previous funding. This information will be used analytically to determine if the NDPA awardees are conducting pioneering research. Respondents may not know or may choose not to provide the information that they feel is privileged, such as previous funding sources. As discussed in the previous section, the Science and Technology Policy Institute will hold individual data strictly confidential, and any public reporting of the data will be in aggregate form that will not allow for the identification of individuals.
Every effort will be made to minimize the burden on the respondents. The interview protocols have been reduced to the minimum possible length (e.g., through the use of largely closed-ended question formats). The interview protocol for NDPA awardees should take, on average, 60 minutes to complete. The Interview Protocols will be conducted over a 6-month timeframe. The interview protocol will be conducted only once, not repeatedly. The Interview Protocol for the Pioneer Lab Members will take, on average, 30 minutes. Respondents do not need to retain or consult records for purposes of these interviews.
The Survey of Unfunded Applicants should take, on average, 15 minutes to complete. The Survey will be conducted over an 8-week period. This Survey will be conducted at approximately 2-year intervals. (Abt conducted the first Survey in 2007 for the 2004-2005 unfunded applicants.) STPI will conduct the second survey for 2004-2005 unfunded applicants and the first survey for 2006 unfunded applicants in 2009. STPI will conduct the first survey for 2007 unfunded applicants in 2010. Respondents do not need to retain or consult records for purposes of these interviews.
Further reductions in the interview protocol would jeopardize the power of the interview protocol to accurately assess the program. Because program participants come from many institutions and backgrounds, reducing the respondent population would have a negative impact of our ability to identify true differences between important subpopulations: for example, researchers from different scientific disciplines or males versus females.
Based on estimated response rates and hour burden from the NIH Director’s Pioneer Award (NDPA) Survey, we anticipate the following total burden:
Table 1: Annualized Estimate of Hour Burden |
||||
Type of Respondents |
Number of Respondents |
Frequency of Response |
Average Time for Response (hr) |
Total Hour Burden* |
Awardees (Pioneers) |
22 |
1 |
1.0 |
22.00 |
Unfunded Applicants (2004-2007) |
440 |
1 |
0.25 |
110.0 |
Pioneer Lab Members |
25 |
1 |
0.5 |
12.50 |
|
|
|
|
|
Total |
487 |
1 |
0.30 |
144.5 |
Total Burden = N Respondents*Response Frequency*(minutes to complete/60)
Thus, the expected burden level for this study is 144.5 hours.
An hourly earning rate for participants was estimated using the U.S. Department of Labor, Bureau of Labor Statistics National Compensation Survey (http://www.bls.gov/ncs/ncswage2007.htm).
NDPA program officers provided a list of research areas of applicants. Based on this list, relevant professions listed in the NCS (e.g. natural scientists, physicians, and college and university professors or relevant disciplines) were averaged. The average hourly earnings of this group is $46.23. This is the rate assigned to the Pioneer Lab Members. Given that nominators and nominees are likely leading biomedical scientists, we assumed an additional 40% in hourly earnings, for an estimated hourly wage of $64.72. With an average of 60 minutes of time required for awardees to complete the interview protocol, 15 minutes for unfunded applicants, and 30 minutes for pioneer lab members, the annual cost for the awardees and unfunded applicants for a NDPA Outcome Evaluation would equal approximately $9120.92
Table 2: Annualized Cost to Respondents |
||||
Type of Respondents |
Number of Respondents |
Response Frequency |
Approx. Hourly Wage Rate |
Total Respondent Cost** |
Awardees |
22 |
1 |
$64.72 |
$1,423.84 |
Unfunded Applicants |
440 |
1 |
$64.72 |
$ 7,119.20 |
Pioneer Lab Members |
25 |
1 |
$46.23 |
$ 577.88 |
|
|
|
|
|
Total |
487 |
1 |
$63.77 |
$9120.92 |
**Total Respondent Cost = Total Hour Burden * Hourly Wage Rate
There are no capital, maintenance or operating costs to respondents.
A.14.1 Annualized Cost to Contractor
The cost to maintain and implement the survey, including contractor’s fixed fee will be $20,000. This does not include analyses of collected data or preparation of reports.
A.14.1 Estimate of Other Total Annual Cost Burden to Contractor |
|
Phone interviews & expert panel coordination |
$10,000 |
Record keeping and follow up |
$10,000 |
Total |
$20,000 |
A.14.2 Annualized Cost to the Federal Government
Annualized Cost to the Federal Government is composed, in part, of an aggregate estimate from Items A.12 and the information above, as this is a one-time survey that will require less than one year to complete. In addition, there are costs of the NDPA Project Officer, NIH OMB Clearance Officer, other NDPA professional staff, and support staff time. Based upon a discussion with the Project Officer, we have estimated that approximately a quarter of a year’s time is required in association Based on the discussion with the project officer, we estimate that approximately a quarter of a year’s time in total would be required for the study. With an average salary of $80,000, this adds $20,000 in NIH staff costs. Thus the total costs of the Information Collection is $49,120.92 (Table A.14.2).
A.14.2 Total Cost Burden of Information Collection |
|
Annualized Cost to Respondents |
$9120.92 |
Other Annual Cost to Contractor (from A.14.1) |
$20,000.00 |
NIH/NDPA Staff Time |
$20,000.00 |
Total |
$49,120.92
|
The NDPA Outcome Evaluation is a new request. We expect that all awardees will respond and 80 percent of the pioneer lab members to respond. We expect that 34 percent of the Unfunded Applicants will respond.
The NDPA Outcome Evaluation began in September 2008, and will end in August 2010. The NDPA Project Officer has reviewed and approved the interview protocols and set up for the Focus Group Panelists. The interview protocols will be conducted during the summer of 2009 for years 1 and 2 of the NDPA program. The evaluation contractors are required to deliver a draft final report on the evaluation by Fall 2010.
A.16 Estimated Annual Project Time Schedule |
|
Activity |
Time Schedule |
Set up and conduct phone interviews of awardees |
1-2 months after OMB approval |
Set up and conduct Survey of Unfunded Applicants |
3-4 months after OMB approval |
Set up followup interviews with limited number of unfunded applicants |
5-6 months after OMB approval |
Conduct interviews with awardee lab members |
5-6 months after OMB |
|
|
FY08 Data analyses |
7-10 months after OMB approval |
FY08 Report writing, dissemination |
11-12 months after OMB approval |
No exceptions are sought; the OMB Expiration Date will be displayed on the interview protocols and other information sent to the respondents.
No exceptions are sought from the Paperwork Reduction Act.
| File Type | application/msword |
| File Title | Supporting Statement for the |
| Author | sshipp |
| Last Modified By | sshipp |
| File Modified | 2009-11-04 |
| File Created | 2009-11-04 |