Mini Supporting Statement A For
“A Generic Submission for Formative Research, Pretesting, and Customer Satisfaction of NCI’s Communication and Education Resources”
OMB No. 0925-0046-07, Expiration Date 5/31/2016
Title of Sub-Study: Pretest of Surveys for the Broadening
Experiences in Scientific Training (BEST) Program
October 230, 2014
Patricia Labosky, Ph.D.,
Office of Strategic Coordination,
Division of Program Coordination, Planning, and Strategic Initiatives,
Office of the Director,
National Institutes of Health (NIH)
List of Attachments
Attachment 1: Invitation Email for Online Survey Pretest
Attachment 2: Reminder Email for Online Survey Pretest
Attachment 3: Screenshots of Graduate Student Entrance Survey
Attachment 4: Screenshots of Graduate Student Interim Survey
Attachment 5: Screenshots of Graduate Student Exit Survey
Attachment 6: Screenshots of Graduate Student Post-Exit Survey
Attachment 7: Screenshots of Postdoctoral Scientist Entrance Survey
Attachment 8: Screenshots of Postdoctoral Scientist Exit Survey
Attachment 9: Screenshots of Postdoctoral Scientist Post-Exit Survey
Attachment 10: Phone Interview Guide
Attachment 11: Invitation Email for Phone Interview
Attachment 12: Reminder Email for Phone Interview
Attachment 13: List of Working Group Members
Section A
A.1 Circumstances Making the Collection of Information Necessary
Section 410 of the Public Health Service Act (42 USC § 285) authorizes the collection of the information. Executive Order 12862, “Setting Customer Service Standards,” authorizes agencies to continually reform their management practices and operations to provide service to the public that matches or exceeds the best service available in the private sector.
The mission of the NIH is to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability. The NIH implements this mission by supporting innovative life science research and development largely through grant awards authorized by Sections 301 and 405 of the Public Health Service Act as amended (42 USC 241 and 284) and under Federal Regulations 42 CFR Part 52 and 45 CFR Parts 74 and 92.
The NIH institutes and centers support research training and research career development opportunities. The NIH Director’s Workforce Innovation Award to Enhance Biomedical Research Training (also generally referred to as the BEST program) is a trans-NIH program that was established in FY 2012 in response to recommendations provided by the Advisory Committee to the Director (ACD). The BEST program is managed by the Office of Strategic Coordination, located within the Division of Program Coordination, Planning, and Strategic Initiatives, in the NIH Office of the Director. As a trans-NIH program, BEST has a Working Group that benefits from cross-institute collaboration and participation by NIH staff from many institutes. The National Cancer Institute (NCI) is an active participant in the Working Group and provides expertise with training programs. The BEST program provides support for institutions to develop innovative approaches to complement traditional research training in biomedical sciences. The goal of the program is to complement and broaden both doctoral and postdoctoral traditional training experiences so that the graduate students and postdoctoral scientists participating in BEST activities are better prepared for careers in a variety of venues, including science policy, technology transfer, science writing, research management, industry, government, academia, or entrepreneurial enterprises.
This collection of information fits within the scope of the full generic as stated in the original Supporting Statement A:
“The formative research process is used to determine whether or not a draft message or message concept is effective in reaching and communicating with its audience. Pretesting involves presentation of draft messages designed to convey specific information to a sample of the audience for whom the materials are intended. These respondents are asked to give their reaction to the messages…. Information collected to determine the level of customer satisfaction with products helps NCI identify strategies for improving the accessibility of materials/programs, their user-friendliness, and their relevance… Respondents' input and reactions to each of these areas provide insight into how the audiences for these messages may react, how the messages should be formulated or revised to communicate most effectively, and the usefulness, relevance and appropriateness of these products to its customers.” (SSA written 4/8/2013, p. 4-5)
A.2 Purpose and Use of the Information Collection
This request is to pretest the draft online surveys for graduate students and postdoctoral scientists. There are four surveys that will be pretested for graduate students and three surveys for postdoctoral scientists. The focus of the pretesting activity is to assess the clarity of the instructions and survey questions, appropriate use of skip patterns, and the length of time needed to complete the survey. The feedback will be used to modify the surveys. The pretesting will improve the accessibility and user-friendliness of the final surveys.
This collection of information fits within the scope of the full generic as stated in the original Supporting Statement A:
“The formative research process is used to determine whether or not a draft message or message concept is effective in reaching and communicating with its audience. Pretesting involves presentation of draft messages designed to convey specific information to a sample of the audience for whom the materials are intended. These respondents are asked to give their reaction to the messages…. Information collected to determine the level of customer satisfaction with products helps NCI identify strategies for improving the accessibility of materials/programs, their user-friendliness, and their relevance… Respondents' input and reactions to each of these areas provide insight into how the audiences for these messages may react, how the messages should be formulated or revised to communicate most effectively, and the usefulness, relevance and appropriateness of these products to its customers.” (SSA written 4/8/2013, p. 4-5)
Principal Investigators (PIs) from the trainees’ institutions will provide contact information for a potential pool of pretest participants. The total number of participants 110: 55 graduate students and 55 postdoctoral scientists. Of the 55 Graduate students and 55 postdoctoral scientists, 50 graduate students and 50 postdoctoral scientists will complete the draft surveys online and provide their reactions and feedback in a comments section at the end of the draft survey. Feedback will also be gathered on specific questions by including a comment box after the question. The invitation email to participate in the online survey pretest is in Attachment 1 and the reminder email is in Attachment 2.
The surveys to pretest are four graduate student surveys (Entrance, Interim, Exit, and Post-exit 2-year) and three postdoctoral scientist surveys (Entrance, Exit, and Post-exit 2-year). The screenshots of the graduate student surveys are listed in Attachments 3-6, and the screenshots of the postdoctoral scientist surveys are listed in Attachments 7-9.
Of the 55 graduate students and 55 postdoctoral scientists, five graduate students and five postdoctoral scientists will be invited to participate in a 30 minute telephone interview regarding the Post-exit 2 year survey. The interview guide is included in Attachment 10. Volunteers for the phone interviews will provide their phone number via email. The invitation email to participate in the phone interviews is in Attachment 11 and the reminder email is in Attachment 12.
The revised surveys will be included in the OMB package that NIH plans to submit for the evaluation of the BEST program.
A.3 Use of Information Technology to Reduce Burden
The surveys will be presented in an online format to reduce burden. The reactions and feedback will inform the final surveys. The pretest will use SurveyMonkey. A Privacy Impact Assessment (PIA) was done by the NIH Office of the Senior official for privacy on the use of SurveyMonkey for all Institutues and OD Offices within NIH.
A.4 Efforts to Identify Duplication
This data collection is unique and does not duplicate any existing data collection.
A.5 Impact on Small Businesses or Other Small Entities
There are no small businesses or other small entites participating.
A.6 Consequences of Collecting the Information Less Frequently
The pretest will be a one-time-only data collection.
A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
This survey will be implemented in a manner that fully complies with 5 C.F.R. 1320.5.
A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency
N/A
A.9 Explanation of Any Payment of Gift to Respondents
Respondents will receive no remuneration for their participation in the pretest.
A.10 Assurance of Confidentiality Provided to Respondents
The participation of graduate students and postdoctoral scientists in the pretesting of the surveys will be strictly voluntary. The review of feedback on the draft surveys will be conducted with names and email addresses removed.
A.11 Justification for Sensitive Questions
The NIH is committed to providing high-quality service to its customers. Given the diversity of its constituents, it is important for NIH to collect survey data from a wide range of customers. Hence, the graduate student and postdoctoral scientist entrance surveys contain questions regarding respondents’ race, ethnicity, age, and sex. It is noted in the surveys that graduate students and postdoctoral scientists may skip any or all of the Personally Identifiable Information (PII) questions or the questions they do not wish to answer.
A.12-1 Estimates of Hour Burden Including Annualized Hourly Costs
The respondents for the surveys are graduate students and postdoctoral scientists. There will be a total of 110 respondents (55 graduate students and 55 postdoctoral scientists). There will be 50 graduate students respondents completing the online surveys and 5 completing the phone interview. In addition, 50 postdoctoral scientists will complete the online surveys and 5 will be postdoctoral scientists completing the phone interview. The total burden hours are 89 hours.
Estimates of Annual Hours Burden |
||||
Type of Respondents |
Number of Respondents |
Estimated Frequency of Response |
Average Respondent Time (in hours) |
Annual Hour Burden |
Graduate Student - Entrance Survey |
14 |
1 |
1 |
14 |
Graduate Student – Interim Survey |
11 |
1 |
45/60 |
8 |
Graduate Student - Exit Survey |
13 |
1 |
45/60 |
10 |
Graduate Student - Post-exit 2-year Survey |
12 |
1 |
45/60 |
9 |
Graduate Student – Phone interview for Post-exit 2-year Survey |
5 |
1 |
30/60 |
3 |
Postdoctoral Scientist - Entrance Survey |
18 |
1 |
1 |
18 |
Postdoctoral Scientist - Exit Survey |
16 |
1 |
45/60 |
12 |
Postdoctoral Scientist - Post-exit 2-year Survey |
16 |
1 |
45/60 |
12 |
Postdoctoral Scientist – Phone interview for Post-exit 2-year Survey |
5 |
1 |
30/60 |
3 |
Total |
|
89 |
||
A.12-2 Cost to Respondents
An hourly earning rate for graduate students was estimated by calculating an average rate for the NIH NRSA Stipends for FY2014 of $22,476 (http://grants.nih.gov/grants/guide/notice-files/NOT-OD-14-046.html) and NSF stipends of $32,000 (http://www.nsfgrfp.org/general_resources/about). The average annual salary we estimated is $27,238. The average hourly earnings for graduate students is $26.19 based on a 20 hour work week.
An hourly earning rate for postdoctoral scientists was estimated by calculating an average of the rates of stipends for postdoctoral scientists with between 0 years and 7 or more years of experience in NIH NRSA Stipends for FY2014 of $48,375 (http://grants.nih.gov/grants/guide/notice-files/NOT-OD-14-046.html). The average hourly earnings for postdoctoral scientists is $23.26 based on a 40 work week.
The total cost for graduate students and postdoctoral scientists to participate in the pretest would equal approximately $3,247.
Annualized Cost to Respondents |
|||
Types of Respondents |
Total Burden Hours
|
Estimated Hourly Wage Rate |
Total Respondent Costs |
Graduate Student - Entrance Survey |
14 |
$26.19 |
$367 |
Graduate Student – Interim Survey |
8 |
$26.19 |
$210 |
Graduate Student - Exit Survey |
10 |
$26.19 |
$262 |
Graduate Student - Post-exit 2-year Survey |
9 |
$26.19 |
$236 |
Graduate Student – Phone Interview for Post-exit 2-year Survey |
3 |
$26.19 |
$79 |
Postdoctoral Scientist - Entrance Survey |
18 |
$46.51 |
$837 |
Postdoctoral Scientist - Exit Survey for Post-exit 2-year Survey |
12 |
$46.51 |
$558 |
Postdoctoral Scientist - Post-exit 2-year Survey |
12 |
$46.51 |
$558 |
Postdoctoral Scientist – Phone Interview |
3 |
$46.51 |
$140 |
TOTAL |
|
$3,247 |
|
A.13 Estimate of Other Total Annual Cost Burden to Respondents or Record Keepers
There are no costs to the respondents other than their time.
A.14 Annualized Cost to the Federal Government
The total cost to the federal government to perform this project is $4,728. It is anticipated that two NIH staff from the Office of Strategic Coordination will be involved at the GS-15/9 and GS-14/3 levels. The costs for the pretest assumes one GS-15 with an annual salary of $157,000 working for 6 hours and one GS-14 with an annual salary of $113,356 working for 10 hours. The total cost for federal government personnel will be $998. Due to the small number of hours that we are projecting for this activity, we are presenting this information on an hourly basis instead of on a FTE basis.
Salaries are based on the January 2014 General Schedule for the Washington, DC Metropolitan area
(http://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2014/DCB.pdf).
The cost for the pretest assumes one senior contractor working for 20 hours at $154 and one junior contractor working for 10 hours at $65. The total cost for contractor time is $3,730.
A.14-1 Cost to the Federal Government
Task |
Time (in hours)
|
Total Costs |
Federal Staff – GS 15/9 ($157,000 or $75.48/ hr) |
6 |
$453 |
Federal Staff – GS 14/3 ($113,356 or $54.50/hr) |
10 |
$545 |
Contractor Staff (Senior contractor) @ $154/hr. |
20 |
$3,080 |
Contractor Staff (Junior contractor) @ $65/hr. |
10 |
$650 |
Total |
|
$4,728 |
A.15 Explanation for Program Changes or Adjustments
This is a generic sub-study under 0925-0046.
A.16 Plans for Tabulation and Publication and Project Time Schedule
Feedback on the surveys will be compiled and used to revise the surveys. The results from this pretesting are not generalizable or used to make broad, expansive conclusions from this sample size.
The study time schedule is outlined in Table A.16-1.
Table A16-1. Study Time Schedule
Activity |
Months after OMB Approval |
Graduate students and postdoctoral scientists complete pretest online surveys and phone interviews |
2 weeks after OMB approval |
Revise draft surveys with feedback from participants |
4 weeks after OMB approval |
A.17 Reason(s) Display of OMB Expiration Date is Inappropriate
No exceptions are sought; data collection instruments will display the OMB Expiration Date.
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions
No exceptions are sought from the Paperwork Reduction Act.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Sarah Ward |
| File Modified | 0000-00-00 |
| File Created | 2021-01-29 |