Supporting Statement A for
National Institutes of Health
National Cross-site Evaluation of the Broadening Experiences in Scientific Training (BEST) Program for the Office of Strategic Coordination, an office of the Division of Program Coordination, Planning, and Strategic Initiatives, within the
Office of the Director
March 19, 2015
Patricia Labosky, Ph.D.
Office of Strategic Coordination
Division of Program Coordination, Planning, and Strategic Initiatives
Office of the Director, NIH
1 Center Drive, MSC 0189
Building 1, Room 214A
Bethesda, MD 20892-0189
Telephone: 301-594-4863
Fax: 301-435-7268
Email: Workforce_Award@mail.nih.gov
A.1 Circumstances Making the Collection of Information Necessary 4
A.2 Purpose and Use of the Information 6
A.3 Use of Information Technology and Burden Reduction 10
A.4 Efforts to Identify Duplication and Use of Similar Information 11
A.5 Impact on Small Businesses or Other Small Entities 11
A.6 Consequences of Collecting the Information Less Frequently 11
A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 12
A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency 12
A.8.1. Comments in Response to the 60-day Federal Register 12
A.8.2. Comments in Response to Consultations Outside Agency 12
A.9 Explanation of Any Payment or Gift to Respondents 13
A.10 Assurance of Confidentiality Provided to Respondents 13
A.11 Justification for Sensitive Questions 14
A.12 Estimates of Hour Burden Including Annualized Hourly Costs 15
A.12.1 Estimates of Hour Burden 15
A.12.2 Annualized Cost to Respondents 16
A.13 Estimate of Other Total Annual Cost Burden to Respondents or Record-keepers 17
A.14 Annualized Cost to the Federal Government 17
A.15 Explanation for Program Changes or Adjustments 19
A.16 Plans for Tabulation and Publication and Project Time Schedule 19
A.17 Reasons Why Display of OMB Expiration Date is Inappropriate 20
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 20
List of Attachments
Attachment A.1.1: RFA-RM-12-022
Attachment A.1.2: RFA-RM-13-019
Attachment A.2.1: Data Sharing Agreement (DSA) – (Data Collection by Awardee)
Attachment A.2.2: Data Sharing Agreement (DSA) – (Data Collection by the NIH)
Attachment A.2.3: Screenshots of the Entrance Survey for Graduate Students
Attachment A.2.4: Screenshots of the Interim Survey for Graduate Students
Attachment A.2.5: Screenshots of the Exit Survey for Graduate Students
Attachment A.2.6: Screenshots of the Entrance Survey for Postdoctoral Scientists
Attachment A.2.7: Screenshots of the Exit Survey for Postdoctoral Scientists
Attachment A.2.8: Screenshots of the 2-year Post-Exit Survey for Graduate Students
Attachment A.2.9: Screenshots of the 2-year Post-Exit Survey for Postdoctoral Scientists
Attachment A.2.10: Program Staff Phone Interview Questions
Attachment A.2.11: Data Form, Section 1—Report Annually
Attachment A.2.12: Data Form, Section 2—Report Annually
Attachment A.2.13: Data Form, Section 3—Report Once
Attachment A.2.14: Data Form, Section 4—Report Once
Attachment A.8.1: 60-Day FRN
Attachment A.8.2: Public Comments to the 60-Day Federal Register Notice
Attachment A.8.3: Members of NIH Strengthening the Biomedical Research Workforce Working Group and the BEST Evaluation Subcommittee
Attachment A.10.1: Letter from the NIH Office of Human Subjects Research Protections
Attachment A.10.2: Letter from the NIH Privacy Officer
Supporting Statement for the
Paperwork Reduction Act Submission
A: Justification
National Institutes of Health
National Cross-site Evaluation of the Broadening Experiences in Scientific Training (BEST) Program
This request seeks approval for OMB clearance to conduct a national cross-site evaluation of the Broadening Experiences in Scientific Training (BEST) Program. This request for clearance includes data collection efforts for three populations from the institutions that received the BEST award: graduate students, postdoctoral scientists, and program staff (Principal Investigator (PI), Co-Principal Investigators (co-PIs), Program Director, and local evaluator). The purpose is to identify best practices in the field of biomedical research training. This will be accomplished by assessing two desired outcomes for graduate students and postdoctoral scientists, and one desired outcome for the awardee institutions. The three desired outcomes are: (1) changes in understanding of career opportunities, confidence to make career decisions, and attitudes towards career opportunities; (2) reduced time to desired, non-training, non-terminal career opportunities, and reduced time in postdoctoral positions; and (3) creation/further development of institutional infrastructure to continue BEST-like activities. The third outcome includes actions which will lead to sustainability of BEST programs and the extension of BEST activities within and across multiple graduate programs. Surveys will be used to gather data from graduate students and postdoctoral scientists to assess the first and second outcomes. A Data Form will be used to gather data for all three outcomes. Phone interviews with program staff will be used to gather data for the third outcome. The information gathered from graduate students, postdoctoral scientists, and program staff will document the BEST program operations and activities, and assess its effectiveness.
The surveys will help fulfill the requirements of:
• Executive Order 12862, “Setting Customer Service Standards,” which directs Agencies to continually reform their management practices and operations to provide service to the public that matches or exceeds the best service available in the private sector; and
• The March 3, 1998 White House Memorandum, “Conducting Conversations with America to Further Improve Customer Service,” which directs Agencies to determine the kind and quality of service its customers want as well as their level of satisfaction with existing services.
The NIH Director’s Workforce Innovation Award to Enhance Biomedical Research Training (also generally referred to as the BEST program) was established in FY 2012 in response to recommendations provided by the Advisory Committee to the Director (ACD). The NIH funded 10 institutions for the amount of $3,726,979 in FY2013 and seven institutions for the amount of $2,635,580 in FY2014. The award is for five years and provides up to approximately $250,000 in direct costs per year. The BEST program is one of the NIH Common Fund programs and is managed by the Office of Strategic Coordination (OSC), located in the Division of Program Coordination, Planning and Strategic Initiatives (DPCPSI) at the Office of the Director (OD). The ACD report confirms that although the vast majority of people holding biomedical Ph.Ds. are productively employed, the proportion of Ph.Ds. that move into tenure-track or tenured faculty positions represents a minority of the postdoctoral scientist outcomes. Despite the broad range of career options available to U.S.-trained Ph.D. biomedical scientists, graduate programs and postdoctoral training focus almost exclusively on preparing individuals for careers as academic researchers. Exposing graduate students and postdoctoral scientists to a multitude of career paths that utilize their Ph.D. training will better prepare them to enter the dynamic biomedical workforce landscape and ultimately strengthen the biomedical research enterprise.
The NIH BEST award provides support for institutions to develop innovative approaches to complement traditional research training in biomedical sciences. The goal is to complement and broaden both doctoral and postdoctoral traditional training experiences so that the graduate student and postdoctoral scientist participating in the BEST program are better prepared for careers in a variety of venues, including science policy, technology transfer, science writing, research management, industry, government, academia, or entrepreneurial enterprises. The BEST program at each institution may include coursework, workshops, internships, and hands-on training experiences. The support given to the institutions is expected to directly impact graduate students and postdoctoral scientists in biomedical research training programs. Also, the program at each institution is expected to promote culture change in the field of biomedical research training. Institutions are encouraged to partner with industry or other entities to provide a wealth of diverse training opportunities for their graduate students and postdoctoral scientists, and PIs are expected to form networks to share experiences and assist the NIH in determining best practices. The transformative potential of the NIH BEST program will be realized as successful models of training are disseminated and adopted nationwide.
The NIH proposes to conduct a national cross-site evaluation for a twenty-four year period (2015-2038) and requests OMB clearance for the first three years (2015-2018). The NIH will seek an extension of OMB clearance after the first three years. The RFAs are listed in Attachment A.1.1 and A.1.2. The attachments in this Statement A are numbered based on the section in which they are referenced.
The evaluation will include 17 institutions. Of the 17, 10 institutions received the BEST award in FY2013 and their award will end in FY2018. Seven received the BEST award in FY2014 and their award will end in FY2019. Throughout this document, the term “awardee institution” refers to the 17 institutions that received the NIH BEST award.
The three desired outcomes of the study are: (1) changes in understanding of career opportunities, confidence to make career decisions, and attitudes towards career opportunities; (2) reduced time to desired, non-training, non-terminal career opportunities, and reduced time in postdoctoral positions; and (3) creation/further development of institutional infrastructure to continue BEST-like activities. This third outcome includes actions which will help the sustainability of the BEST programs after the funding ends, and the extension of BEST activities within and across multiple graduate programs. The high profile and public nature of the BEST program makes its evaluation a high priority for the NIH. An evaluation is necessary to determine best practices for biomedical research training programs.
The data collection includes online surveys for graduate students and postdoctoral scientists, phone interviews with program staff, and a Data Form for PIs to provide information listed in the RFAs for the BEST program. The purpose of each data collection is described below:
Online Surveys for Graduate Students and Postdoctoral Scientists
The purpose of the online surveys for graduate students and postdoctoral scientists is to assess the first two desired outcomes: (1) changes in understanding of career opportunities, confidence to make career decisions, and attitudes towards career opportunities; and (2) reduced time to desired, non-training, non-terminal career opportunities, and reduced time in postdoctoral positions. The surveys will be administered to participants and nonparticipants of the BEST program. Data will be collected from the graduate students and postdoctoral scientists while they are at the institution and after they exit the institution.
Two approaches will be used to administer the surveys based on awardees’ preference and capacity. Approach 1: Some surveys will be administered by the awardee institutions and some surveys will be administered by the NIH. Approach 2: All the surveys are administered by the NIH. Data will be gathered using a common set of survey questions, which the NIH has developed with input from BEST awardee institutions. The questions in the surveys may require minor revisions in the future since this is a long-term study. Any revisions will be submitted to OMB for approval.
The awardee institutions and the NIH will enter into a Data Sharing Agreement (DSA) to share data from all surveys conducted for the national cross-site evaluation. The NIH Office of the General Council has reviewed and approved the DSA, and the Awardee Institutions have agreed to participate in the national cross-site evaluation. An evaluation ID will be assigned to each graduate student and postdoctoral scientist and used to de-identify survey data. In instances where awardee institutions administer the surveys, they will provide the NIH the evaluation IDs and the survey responses. The DSA for this instance is in Attachment A.2.1. In instances where the NIH administers the surveys, the awardee institutions will provide the evaluation IDs and the email addresses of the graduate students and postdoctoral scientists, and the NIH will provide the survey responses with the evaluation IDs to the awardee institutions. The DSA for this instance is in Attachment A.2.2.
Awardee institutions use different names to refer to the BEST program at their institutions. Some institutions use “BEST” in the title of their program and some institutions do not. For example, Cornell University calls their program the “Cornell BEST Program” but Vanderbilt University calls their program “Augmenting Scholar Preparation and Integration with Research-Related Endeavors (ASPIRE).” When NIH administers the surveys, the name of each program will be listed in the section called “Impact of Career Development Activities” and graduate students and postdoctoral scientists will choose the program from their particular institution. When the awardee administers the surveys, the name of their specific program will be included in the question wording in the “Impact of Career Development Activities” section along with examples of specific activities that are part of their BEST program. The purpose of identifying the specific name of the awardee institution BEST program in the surveys is to improve data quality overall.
The screenshots of the surveys located in the attachments include the approach for when NIH administers the surveys, as well when the Awardee institutions administer the surveys.
Online Surveys While Graduate Students and Postdoctoral Scientists are at the Institution
Graduate Student Entrance Survey – This survey will assess the first desired outcome and serve as a baseline. This survey will also ask questions about graduate degree program, department, what year they are in graduate school, perceptions of faculty and graduate program support, and research productivity. In the first year of the evaluation, the Entrance survey will be administered to all graduate students from the graduate programs participating in the BEST program. In subsequent years, the Entrance survey will be administered only to new graduate students. The Graduate Student Entrance Survey is in Attachment A.2.3, as screenshots of the survey.
Graduate Student Interim Survey – This survey will assess the first desired outcome, and perceptions of faculty and graduate program support. The Interim survey will be administered in 2017 and 2019. The pool of respondents is all graduate students who received the Entrance Survey previously, whether or not they completed it. The Graduate Student Interim Survey is in Attachment A.2.4, as screenshots of the survey.
Graduate Student Exit Survey – This survey will assess the first and second desired outcomes and perceptions of faculty and graduate program support. The Exit survey will be administered each year to graduate students when they submit their thesis, or before graduation. The Graduate Student Exit Survey is in Attachment A.2.5, as screenshots of the survey.
Postdoctoral Scientist Entrance Survey - This survey will assess the first desired outcome. It will also ask about past postdoctoral positions, research productivity, and perceptions of PI and department support. In the first year of the evaluation, the Entrance survey will be administered to all postdoctoral scientists from the departments participating in the BEST program. In subsequent years, the Entrance survey will be administered only to new postdoctoral scientists. The Postdoctoral Scientist Entrance Survey is in Attachment A.2.6, as screenshots of the survey.
Postdoctoral Scientist Exit Survey – This survey will assess the first and second desired outcomes. An annual status update survey consisting of four questions will be used to determine if the postdoctoral scientist meets the criteria to receive the Exit survey. The administration of the Exit survey is based on the respondent’s postdoctoral appointment status. If the respondent no longer holds a postdoctoral position at the awardee institution, or is no longer employed by the awardee institution, the respondent will be directed to the Exit survey. The Postdoctoral Scientist Exit Survey is in Attachment A.2.7, as screenshots of the survey.
Post-Exit Surveys after Graduate Students and Postdoctoral Scientists
Graduate Student 2-year Post-Exit Survey – This survey will be administered by the NIH to graduate students after they exit their institution to assess the second desired outcome. This survey will be administered at two years after the Exit survey. The 2-year Post-Exit survey for graduate students is in Attachment A.2.8, as screenshots of the survey. Participants will be followed up every 6 months via email after they graduate from the awardee institutions to ensure that the email address they provided when they completed one or more of the previous surveys is correct. In the follow-up email, participants will be asked to click on a link and confirm that their email address is correct or provide up to two additional email addresses.
Postdoctoral Scientist 2-year Post-Exit Survey – This survey will be administered by the NIH to postdoctoral scientists after they exit their institution to assess the second desired outcome. This survey will be administered at two years after the Exit survey. The 2-year Post-Exit survey for postdoctoral scientists is in Attachment A.2.9, as screenshots of the survey. Participants will be followed up every 6 months via email after they completed the Exit survey to ensure that the email address they provided is correct. In the follow-up email, participants will be asked to click on a link and confirm that their email address is correct or provide up to two additional email addresses.
The questions from the 2-year Post-Exit survey will be used to develop Post-Exit surveys at six, ten, and fifteen years after the Exit survey. NIH will request clearance for the administration of these surveys.
Graduate Student Exit Survey Data Collected After the Grant Ends
The NIH will administer Exit surveys up to four years after the grant ends to graduate students who provide their email address(es) in the surveys, but graduate after the grant ends. The awardee institutions will share with the NIH a list of the evaluation IDs of students who will graduate from the institution in each academic semester or quarter so that the NIH can administer the Exit surveys four years after the grant. If the institution has an institutional exit survey, the NIH will administer the Exit survey to coincide with the schedule of the institution. If the institution does not have an institutional exit survey, the NIH will administer the Exit survey prior to graduation.
Testing of Data Collection Instruments
All surveys for graduate students and postdoctoral scientists were pilot tested in November and December of 2014 under OMB # 0925-0046-07. The purpose of the pilot test was to gather input from graduate students and postdoctoral scientists from the awardee institutions. Participants were asked to provide feedback on the wording of the questions, flow of the survey questions, appropriateness of the skip patterns, and the length of time to complete the online survey. They were also asked to provide their overall impression of the survey assigned.
A total of 100 individuals (50 graduate students and 50 postdoctoral scientists) were invited to participate in the pilot. The overall response rate was 71 percent. In addition to the online surveys, the pilot test included 10 interviews (five graduate students and five postdoctoral scientists) to gather feedback on the 2-year Post-Exit survey. The feedback collected in the pilot was incorporated into the final version of each online survey. Additional information about the pilot is included in Section B.4. of Statement B entitled, “Test of Procedures of Methods to be Undertaken.”
Interviews with Program Staff
Program Staff Phone Interviews. The purpose of the phone interviews with program staff from each institution is to assess the third desired outcome, that is, creation/further development of institutional infrastructure to continue BEST-like activities after the grant ends. The phone interviews will be conducted within a three-month period at the end of each calendar year. The following program staff from each institution will be interviewed: PI, co-PIs, Program Director, and local evaluator. The questions for the phone interview are in Attachment A.2.10.
Data Form
Data Form. The purpose of the Data Form is for PIs to report on the activities of their programs, the participation of graduate students and postdoctoral scientists in those activities, and provide institutional data requested in the RFAs. This information will be used to assess all three outcomes. The Data Form will be completed by PIs and submitted to the NIH in a user-friendly Excel format. The Data Form is organized into four sections: (1) Program Description and Participation in BEST Program Activities (reported annually), (2) Aggregate Data from Participating Departments/Graduate Programs (reported annually), (3) Baseline Information for the Five Years Prior to Receiving the BEST Award (reported once), and (4) Information to Report in Year Four of the Award (reported once).
Four excel files have been created, one for each section of the Data Form. A description of the four sections of the Data Form is as follows. For section one, the awardee will provide information on the BEST program, such as, the criteria for participation, requirements for certificates, and additional requirements to participate in specific activities. The awardee will provide information on activities included in the BEST program, along with data on the individual graduate students and postdoctoral scientists who participated in these activities. Examples of BEST activities include workshops, seminars, courses, internships, and mentorships. Individual graduate student and postdoctoral scientist level data will be reported by the evaluation ID number and will include information such as the name of activity, number of participation hours for the activity, length of time for the activity, topics covered, and any other pertinent information about each BEST activity. For example, for an internship, the awardee will report the following: type of participant (graduate student or postdoctoral scientist), source of payment, internship partner name, full-time/part-time, duties, number of hours per week, span of time in weeks, course credit, and mandatory/voluntary. Section 1 of the Data Form is in Attachment A.2.11.
For section two, the awardee will provide aggregate data on graduate students, postdoctoral scientists, and faculty participation in the BEST program. The data are generally reported for all BEST participants, and includes: (1) the number of graduate students and postdoctoral scientists in participating departments/graduate programs, (2) elapsed time to doctorate, (3) the postdoctoral placement of postdoctoral scientists, (4) length of time in postdoctoral training, (5) summary data on graduate students who did not complete the Ph.D. program and left to pursue non-biomedical science fields, (6) faculty participation in BEST program, (7) BEST activities offered to faculty, (8) faculty attitudes towards career development activities, (9) the role of external partners, and (10) non-NIH funding sources. Section two also includes “The Trainee Diversity Report” of gender, ethnicity, race, disability, and disadvantaged backgrounds. The “Trainee Diversity Report” has been approved under OMB #0925-0002 (REV. 08/12). This report is required by the RFA. Section 2 of the Data Form is in Attachment A.2.12.
For section three, for the five years prior to the BEST award, the Awardee will provide aggregate data for the departments and/or graduate programs participating in BEST. This data will include: (1) the elapsed median time to doctorate, (2) the career paths chosen by Ph.D. recipients, and (3) the length of time in postdoctoral training. Section 3 of the Data Form is in Attachment A.2.13.
For section four, in year four of the BEST award, the Awardee will provide aggregate data for the departments and/or graduate programs participating in BEST for the prior four years. This data will include: (1) the elapsed median time to doctorate, (2) the career paths chosen by Ph.D. recipients, and (3) the length of time in postdoctoral training. Section 4 of the Data Form is in Attachment A.2.14.
Use of the Information Collected
Information collected during the BEST national cross-site evaluation will be used to assess the effectiveness of the program and disseminate best practices. First, the information will be used by the NIH program staff to assess progress towards the three desired outcomes of the program. Second, the results of the evaluation will inform the NIH Director, DPCPSI Director, and the Office of Strategic Coordinator staff, about best practices. Third, the results of the national cross-site evaluation and best practices will be disseminated to the wider research community and biomedical training programs. Efforts will be made to publish the results of the national cross-site evaluation in professional journals and to present the findings at conferences.
The surveys for graduate students and postdoctoral scientists are online to reduce the burden on respondents. For this clearance, the surveys for graduate students include: Entrance, Interim, Exit, and 2-year Post-Exit. For this clearance, the surveys for the postdoctoral scientists include: Entrance, Exit, and 2-year Post-Exit.
As appropriate, the online surveys will use skip-patterns so that each respondent is only presented with questions relevant to his or her specific situation. In addition, several questions on the surveys are multiple choice or closed-ended to reduce the burden on respondents.
Awardee institutions will receive a codebook for each survey to standardize the data collection. They will adhere to the security protocols specified by their institutions and the Institutional Review Board. The NIH Center for Information Technology (CIT) will develop the online graduate student and postdoctoral scientist surveys. A Privacy Impact Assessment (PIA) will be conducted for the online surveys. The NIH Privacy Act Officer and the NIH Information Systems Security Officer will assess privacy and security risks of the NIH CIT IT system.
The questions for the phone interview with the program staff at each institution have been designed to reduce the burden on the respondent. The phone interviews will be limited to one hour.
The Data Form contains pre-formatted data collection tables in a user-friendly Excel format and is designed to reduce respondent burden. Each section of the Data Form is saved in its own Excel file and each file contains instructions on how to complete each table, along with pre-formatted tables.
The data to be collected for the NIH cross-site evaluation from graduate students, postdoctoral scientists and awardee program staff do not duplicate other data collection efforts. BEST is a new program, established to transform the culture of research training in the biomedical sciences for graduate students, postdoctoral scientists, and mentors. The BEST national cross-site evaluation will assess which activities produce the desired outcomes of: (1) changes in understanding of career opportunities, confidence to make career decisions, and attitudes towards career opportunities, (2) reduced time to desired, non-training, non-terminal career opportunities, and reduced time in postdoctoral positions, and (3) creation/further development of institutional infrastructure to continue BEST-like activities. The program activities can be disseminated throughout the biomedical research training community. The BEST national cross-site evaluation will compare program activities and outcomes across the awardee institutions. Additionally, collecting information from the program staff will allow assessment of the institution’s creation/further development of institutional infrastructure to continue BEST-like activities past the BEST awards. This evaluation will enable the best program activities to be institutionalized across biomedical training programs.
There is no anticipated impact on small businesses or other small entities.
In all the data collection, if the information is collected less frequently than proposed, then the feedback on program performance and best practices will be stretched out and will not provide as much value to the NIH to make program adjustments and disseminate best practices. If the graduate student and postdoctoral scientist surveys are not conducted on the proposed frequency, the NIH will be less able to determine if the BEST program produced the desired outcomes. Also, if the surveys are not conducted on the proposed frequency, the program activities which produce the desired outcomes could not be identified and disseminated to the biomedical research training community. In order to assess the first and second outcomes, it is necessary to administer the surveys over the period of 24 years to see long-term changes. It is also necessary to maintain contact with participants after they exit their institution to track their career outcomes. The timeline of surveys allows for adequate tracking of changes in the desired outcomes.
If information is not collected from program staff in the phone interviews annually, the NIH will be less able to assess how the programs are being implemented and progress made in sustainability and institutionalizing the program. The NIH will not be able to take timely steps to improve implementation, sustainability, and institutionalization throughout the BEST program with all awardee institutions.
If information is not collected from PIs in the Data Form on the proposed frequency, the NIH will not be able to assess the degree of participation in BEST program activities. This information will allow identification of the program activities which produce the desired outcomes. Also, it will allow the NIH to assess baseline rates and changes over time in career outcomes and length of time in graduate school and postdoctoral positions.
This project fully complies with all guidelines of 5 CFR 1320.5.
The 60 day notice required in 5 CFR 1320.8(d) was published in the Federal Register on August 26, 2014 (Volume 79, Number 165, Pages 50921-50922). This notice is in Attachment A.8.1 and is called the 60-Day FRN. The public comments and the NIH response to the public comments are shown in Attachment A.8.2.
During the 60-day comment period, two requests for information were received. Both individuals requested the data collection plans and the data collection instruments (surveys) and one also asked for background information. In response to these two requests, electronic copies of the following surveys were emailed: Entrance, Interim, Exit, and 2-year Post-Exit. Additionally, both individuals were provided with a brief overview of the program and a link to the NIH website that contains detailed information about the BEST program and the awardees. No additional comments were received from the two individuals who requested this information.
The proposed design is grounded in extensive background research and discussions with the NIH program staff and evaluators of similar programs. The plan for the national cross-site evaluation was developed in consultation with Windrose Vision, an independent contractor, who has experience evaluating NIH programs, the BEST Evaluation Subcommittee, and some members of the NIH Strengthening the Biomedical Research Workforce Working Group for the BEST program (See Attachment A.8.3 for a list of members of each group). These groups provided feedback on the outcomes to measure, the study design, surveys, and phone interview questions. Also, the BEST awardees were consulted in designing the NIH national cross-site evaluation and data collection instruments.
Participation is completely voluntary. Program staff from the Awardee institutions will not be paid or receive any gifts for participating in the phone interviews or for submitting the Data Form. Graduate students and postdoctoral scientists will not be paid for participating in the BEST program. As a token of appreciation for participation in the surveys, respondents will be entered into a raffle to win various prizes.
We are planning to provide $1,200 to each Awardee institution so that they can raffle various prizes. The money will not be given directly to the survey participants, rather it will be used to purchase prizes for the raffle. The total amount for the 17 Awardee institutions is $20,400 (17 * $1,200). The estimated number of survey participants is 15,310, meaning that there will be approximately $1.33 allotted per participant ($20,400/15,310).
At the time of this submission, it is estimated that of the 17 Awardee institutions, ten will be administering the surveys on behalf of NIH, and NIH will administer the surveys for four of the institutions.
The NIH Office of Human Subjects Research (OHSR) has reviewed the national cross-site evaluation for the BEST program and determined that the study is not human subjects research. The OHSR determination of Not Human Subjects Research is based on the interpretation of 45 CFR and Guidance of Engagement of Institutions in Human Subjects Research. (See letter from the NIH Office of Human Subjects Research Protections in Attachment A.10.1.) The NIH Privacy Act Officer also reviewed this OMB package and has determined that the Privacy Act Systems of Record Notice 09-25-0156 applies to the data collection. Attachment A.10.2 includes the letter from the NIH Privacy Officer.
The graduate students, postdoctoral scientists, and program staff will be informed that their responses to the data collection efforts will only be disclosed to authorize users for analysis and reporting. The awardee institutions, the NIH, and the NIH contractor will use the NIH Secure Email/File Transfer Service (SEFT) to share survey data and the completed Data Form. The NIH SEFT allows users to send and receive emails securely on a secure socket layer (SSL)/encrypted connection. Using the SEFT ensures the protection of personally identifiable information (PII) and thoroughly secures all data and information being sent via email.
Given the nature of the study, assurance of privacy will be provided to respondents. Results of the evaluation will be disseminated to the research community in aggregate. The NIH CIT will be responsible for ensuring the security of the survey data. The NIH will conduct a Privacy Impact Assessment. Awardee institutions will be responsible for ensuring the security and privacy of the survey data they collect based on the protocols and procedures reviewed and approved by their Institutional Review Board (IRB).
The following safeguarding procedures will be followed:
The NIH CIT and the NIH contractor will follow the NIH Privacy Act Systems of Record Notice 09-25-0156 and all computer-based systems will comply with the Privacy Act. The authorized users are the NIH CIT programmers, the NIH BEST program staff and the evaluation team of the NIH contractor.
The data collected from the surveys, phone interviews, and Data Form will be stored in the secure server of the NIH Office of Strategic Coordination. A password-protected directory will be created and only authorized users will have access to the data.
All paper files (such as handwritten interview notes) shall be stored in locked cabinets.
The safeguarding protections offered to all participants will be included in the invitations and reminder emails for the survey instruments. Respondents will be informed that their participation is voluntary and that no consequences will be associated with either responding or not responding. For example, the invitation for online surveys for graduate students and postdoctoral scientists will have the following statements: “Your participation is completely voluntary and you can withdraw at any time without penalty” and “All information collected will be kept private and protected on a secure server to prevent disclosure. Only designated staff from your institution, the NIH, and the NIH Contractor, Windrose Vision, will have access to the data for analysis purposes. All personally identifiable information will be removed from your survey responses. When the results of this study are published or presented, the data will be reported in aggregate and no information will be included that would reveal any respondent’s identity.”
For purposes of analysis, email addresses from online surveys shall be removed.
Publications shall only report the data in aggregate and will not contain any identifying information.
The NIH is committed to providing high-quality service to the biomedical research training community. Given the diversity of its constituents, it is important for the NIH to collect survey data from a wide range of respondents. Hence, the graduate student and postdoctoral scientist Entrance surveys contain questions regarding respondents’ race, ethnicity, age, and sex. The Exit and 2-year Post-Exit surveys for graduate students and postdoctoral scientists will contain questions regarding work-related information (type of employer organization and career field). This information will allow the NIH to analyze the survey data by subgroups and support the NIH’s long standing efforts to strengthen diversity of those it serves. The majority of questions asked will not be of a sensitive nature.
Graduate students and postdoctoral scientists may skip any or all of the Personally Identifiable Information (PII) questions that they do not wish to answer. To avoid fear of disclosure of sensitive information, participants will be told that their responses will be kept private, and will be reported in the aggregate. Those who choose to provide the demographic data will do so on a strictly voluntary basis.
The Program Staff Phone Interviews will not contain any sensitive questions.
The Data Form asks PIs to report on the gender, ethnicity, race, disability, and disadvantaged backgrounds of their graduate students and postdoctoral scientists in the “Trainee Diversity Report” [OMB 0925-0002 (REV. 08/12)]. The RFA requires awardees to submit this report. This information is only reported in aggregate and will be maintained in a secure location.
As requested by the NIH Privacy Act Officer, the Privacy Act Systems of Record Notice 09-25-0156, “Records of Participants in Programs and Respondents in Surveys Used to Evaluate Programs of the Public Health Service, HHS/PHS/NIH/OD” will be distributed to staff responsible for handling any PII. The contractor that will conduct the evaluation and the NIH CIT, that will develop the surveys, will comply with the Privacy Act.
Every effort will be made to minimize the burden on the respondents. All surveys will be given online and interviews will be over the phone. Estimates of hour burden are provided only for the surveys that will be administered during the three years of clearance.
Graduate Students – The Entrance survey for the graduate students will take an average of 20 minutes to complete. The Interim survey, Exit survey, and 2-year Post-Exit survey will take an average of 15 minutes.
Postdoctoral Scientists – The Entrance survey for the postdoctoral scientists will take an average of 20 minutes to complete and the Exit survey and 2-year Post-Exit survey will take an average of 15 minutes.
Program Staff – The phone interview will take an average of 60 minutes to complete. Program staff includes PIs, co-PIs, Program Director, and a local evaluator from each awardee institution.
PIs - The Data Form which PIs will complete consists of four sections. Section 1 will be submitted annually and should take an average of 3 hours to complete. Section 2 will be submitted annually and should take an average of 90 minutes to complete. Section 3 will be submitted only in 2015 and should take 30 minutes to complete. Section 4 will be submitted only in year four of the award and should take 30 minutes to complete.
Further reductions in the completion times for online surveys, phone interviews, and Data Form would jeopardize accurate assessment of the program. Table A.12.1 displays the annualized estimate of hour burden. The expected burden level for this study is 8,106 hours.
A.12.1: Annualized Estimate of Hour Burden |
||||
Type of Respondents |
Number of Respondents* |
Frequency of Response |
Average Time per Response (in hours) |
Total Annual Hour Burden |
Graduate Student – Entrance Survey (online survey) |
4,519 |
1 |
20/60 |
1,506 |
Graduate Student – Interim Survey (online survey) |
11,296 |
1 |
15/60 |
2,824 |
Graduate Student – Exit Survey (online survey) |
3,012 |
1 |
15/60 |
753 |
Graduate Student – 2-year Post-Exit Survey (online survey) |
3,012 |
1 |
15/60 |
753 |
Postdoctoral Scientist – Entrance Survey (online survey) |
3,137 |
1 |
20/60 |
1,046 |
Postdoctoral Scientist – Exit Survey (online survey) |
2,091 |
1 |
15/60 |
523 |
Postdoctoral Scientist – 2-year Post-Exit Survey (online survey) |
2,091 |
1 |
15/60 |
523 |
Program Staff – Annual Phone Interview |
83 |
1 |
1 |
83 |
PIs – Data Form Section 1 (reported annually) |
17 |
1 |
180/60 |
51 |
PIs – Data Form Section 2 (reported annually) |
17 |
1 |
90/60 |
26 |
PIs – Data Form Section 3 (reported once) |
17 |
1 |
30/60 |
9 |
PIs – Data Form Section 4 (reported once) |
17 |
1 |
30/60 |
9 |
Total |
|
|
8,106 |
|
*These are estimates.
An hourly earning rate for graduate students was estimated by calculating an average rate for the NIH NRSA Stipends for FY2014 (http://grants.nih.gov/grants/guide/notice-files/NOT-OD-14-046.html) and NSF stipends (http://www.nsfgrfp.org/). The average hourly earnings for graduate students is $26.19.
An hourly earning rate for postdoctoral scientists was estimated by calculating an average of the rates of stipends for postdoctoral scientists with between 0 years and 7 or more years of experience in the NIH NRSA Stipends for FY2014 (http://grants.nih.gov/grants/guide/notice-files/NOT-OD-14-046.html). The average hourly earnings for postdoctoral scientists is $46.51.
An hourly earning rate for the program staff was estimated using the American Association of University Professors’ Annual Report on the Economic Status of the Profession (http://www.aaup.org/reports-publications/2013-14salarysurvey).The amount of average hourly earnings rate was based on the salary of full professors. The average hourly earnings for the PIs, evaluators, and BEST Program Directors is $64.78.
The annual cost for graduate students, postdoctoral scientists, and program staff (PI, co-PIs, Program Director, and local evaluator) to participate in the BEST national cross-site evaluation would equal approximately $261,675.
A.12.2: Annualized Cost to Respondents |
|||
Type of Respondents |
Annual Hour Burden* |
Approx. Hourly Wage Rate |
Total Cost |
Graduate Student – Entrance Survey (online survey) |
1,506 |
$26.19 |
$39,442 |
Graduate Student – Interim Survey (online survey) |
2,824 |
$26.19 |
$73,961 |
Graduate Student – Exit Survey (online survey) |
753 |
$26.19 |
$19,721 |
Graduate Student – 2-year Post-Exit Survey (online survey) |
753 |
$26.19 |
$19,721 |
Postdoctoral Scientist – Entrance Survey (online survey) |
1,046 |
$46.51 |
$48,649 |
Postdoctoral Scientist – Exit Survey (online survey) |
523 |
$46.51 |
$24,325 |
Postdoctoral Scientist – 2-year Post-Exit Survey (online survey) |
523 |
$46.51 |
$24,325 |
Program Staff – Annual Phone Interview |
83 |
$64.78 |
$5,377 |
PIs – Data Form Section 1 (reported annually) |
51 |
$64.78 |
$3,304 |
PIs – Data Form Section 2 (reported annually) |
26 |
$64.78 |
$1,684 |
PIs – Data Form Section 3 (reported once) |
9 |
$64.78 |
$583 |
PIs – Data Form Section 4 (reported once) |
9 |
$64.78 |
$583 |
Total |
8,106 |
|
$261,675 |
*This is an estimate.
There are no capital, maintenance or operating costs to respondents.
The annual cost to the Federal Government includes both the costs for developing the surveys and the cost of federal personnel time. The annual total cost to the government is estimated to be $198,893.55 ($175,000 for the surveys, $22,410 for the phone interviews, and $1,484 for the NIH staff time cost). Table A.14 shows the total annualized cost to the Federal Government.
Assume the following costs for a total of 7 surveys: 4 for graduate students (Entrance, Interim, Exit, and 2-year Post-Exit), and 3 for postdoctoral scientists (Entrance, Exit, and 2-year Post-Exit). The cost for each of seven online surveys is estimated to vary between $20,000–$25,000, depending on the complexity of the survey. The total is estimated to be $175,000 (7 X $25,000). The cost of the phone interview is estimated to be $22,410 (Approximately 83 interviews, 1 hour each, with a senior contractor at $175, and 1 hour for a junior contractor at $95). Using these estimates, the total approximate cost is $197,410. The estimates provided do not include the preparation of the survey questions, interview questions, and Data Form, scheduling of phone interviews, analyses of collected data, or preparation of reports.
Total federal government personnel costs will be $1,483.55 in the first year, based on the hourly rates (on the Office of Personnel Management [OPM] website) for the three NIH staff. It is anticipated that three NIH staff from the Office of Strategic Coordination will be involved at the GS-15, GS-14, and GS-13 levels. The costs for the first year assumes one GS-15 with an annual salary of $158,700 working at 10 hours; one GS-14 with an annual salary of $114,480 working at 5 hours; and one GS-13 with a salary of $102,632 working at 9 hours. The costs for the next two years are estimated to be $180.82, when each of the three NIH staff devote one hour each to the project. Salaries are based on the January 2015 General Schedule for the Washington, DC Metropolitan area (http://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2015/DCB.pdf).
A.14: Annualized Costs to the Federal Government |
||||
Estimated Costs for Federal Government Personnel |
||||
Item |
Annual Salary ($) |
Hourly Rate (Salary/2,080)* ($) |
Number of Hours |
Annualized Cost ($) |
Personnel from the NIH Office of Strategic Communication (GS-13) |
102,932 |
49.49 |
9 |
445 |
Personnel from the NIH Office of Strategic Communication (GS-14) |
114,480 |
55.04 |
5 |
275 |
Personnel from the NIH Office of Strategic Communication (GS-15) |
158,700 |
76.30 |
10 |
763 |
Total Cost for Government Personnel: |
1,484 |
|||
Estimated Costs for Survey Administration |
||||
Item |
Number of Surveys |
Cost per Survey |
Annualized Cost ($) |
|
Cost for four Graduate Student Surveys: Entrance, Interim, Exit, 2-year Post-Exit |
4 |
25,000 |
100,000 |
|
Cost for three Postdoctoral Scientist Surveys: Entrance, Exit, 2-year Post-Exit |
3 |
25,000 |
75,000 |
|
Total Cost for Surveys: |
175,000 |
|||
Estimated Costs for Contractor to conduct Phone Interviews with Program Staff |
||||
Item |
Number of Phone Interviews |
Hours per interview |
Hourly Rate ($) |
Annualized Cost ($) |
Phone Interviews with Program Staff (Senior contractor personnel) |
83 |
1 |
175 |
14,525 |
Phone Interviews with Program Staff (Junior contractor personnel) |
83 |
1 |
95 |
7,885 |
Total Cost for Contractor: |
22,410 |
|||
Total Annualized Cost to the Federal Government: |
198,894 |
|||
*2,080=Number of working hours per year |
||||
This is a new collection of information.
The Entrance, Interim, and Exit surveys for graduate students and the Entrance survey for postdoctoral scientists will be administered by BEST awardees and the NIH. The data analysis will be conducted by the company that the NIH has contracted to conduct the evaluation. The contractor will provide reports that describe the study and findings.
A.16 Estimated Annual Project Time Schedule |
|
Activity |
Time Schedule (Assume clearance in July 2015) |
Awardee institutions and the NIH administer online Entrance Survey to all graduate students and postdoctoral scientists from the departments participating in BEST |
1 month after OMB approval
|
Data Form Section 1, 2, and 3 completed by PIs from each awardee institution |
1 month after OMB approval |
Conduct phone interviews with Program Staff from each awardee institution |
3 months after OMB approval
|
Awardee institutions and the NIH administer online Graduate student Exit Survey to any graduating students from the departments participating in BEST |
5 months after OMB approval
|
The NIH administers online Postdoctoral Scientist Exit Survey to any exiting postdoctoral scientists from the departments participating in BEST |
10 months after OMB approval
|
Data Form, including section 1 and 2, completed by PIs from each awardee Institution |
12 months after OMB approval |
Analyze data and report results |
12 months after OMB approval |
Awardee institutions and the NIH administer online Entrance Survey to any new graduate students and postdoctoral scientists who have entered the departments participating in BEST |
13 months after OMB approval
|
Conduct phone interviews with Program Staff from each awardee institution |
15 months after OMB approval
|
Awardee institutions and the NIH administer online Graduate student Exit Survey to any graduating graduate students from the departments participating in BEST |
17 months after OMB approval
|
Awardees institutions and the NIH administer online Interim Survey to all graduate students from the departments participating in BEST |
22 months after OMB approval
|
The NIH administers online Postdoctoral Scientist Exit Survey to any exiting postdoctoral scientists from the departments participating in BEST |
22 months after OMB approval
|
Data Form, including section 1, 2, and 4 submitted by PIs from each awardee institution |
24 months after OMB approval |
Analyze data and report results |
24 months after OMB approval |
Awardee institutions and the NIH administer online Entrance Survey to any new graduate students and postdoctoral scientists who have entered the departments participating in BEST |
25 months after OMB approval
|
Awardee institutions and the NIH administer online Graduate student Exit Survey to any graduate students graduating from the departments participating in BEST |
29 months after OMB approval
|
Data Form, including section 1 and 2, submitted by PIs from each awardee institution |
36 months after OMB approval |
The NIH administers online 2-Year Post-Exit Survey to any prior graduate students who graduated two years prior from the departments participating in BEST |
To be administered after the 3-year OMB approval period |
The NIH administers online 2-Year Post-Exit Survey to any prior postdoctoral scientists who exited two years prior from the departments participating in BEST |
To be administered after the 3-year OMB approval period |
Analyze data and report results |
36 months after OMB approval |
No exceptions are sought; data collection instruments will display the OMB Expiration Date.
No exceptions are sought from the Paperwork Reduction Act.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Melissa Newberry |
| File Modified | 0000-00-00 |
| File Created | 2021-01-25 |