Mini SSA NIH External Constituency Surveys

Mini Supporting Statement A

NIH External Constituency Surveys

Section A

A.1 Circumstances Requiring the Collection of Data

This is a request to conduct a voluntary customer satisfaction survey of the National Institutes of Health’s (NIH’s) Enhancing Peer Review Initiative. These surveys will help fulfill the requirements of:

• Executive Order 12862, “Setting Customer Service Standards,” which directs Agencies to continually reform their management practices and operations to provide service to the public that matches or exceeds the best service available in the private sector; and

• The March 3, 1998 White House Memorandum, “Conducting Conversations with America to Further Improve Customer Service,” which directs Agencies to determine the kind and quality of service their customers want as well as their level of satisfaction with existing services.

A.2 Purposes and Uses of the Data

The planned survey will be directed to scientists who conduct research relevant to the mission of the National Institutes of Health (NIH) and have applied for research grant funding in the past five years. The NIH mission is to “seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability.” The primary objective of the survey is to define appropriate expectations for peer reviewers’ service commitments. The information provided by the survey will also be used to better understand the factors that influence scientists’ decisions to accept an invitation to serve as peer reviewers for NIH, and thus optimize our efforts to identify highly qualified scientists who are likely to serve as reviewers. The survey will focus on scientists’ preferences in terms of review venues, review formats and the grant activities assigned to them. It will also examine the competing demands for scientists’ time and help NIH to gauge reasonable expectations in terms of how much of scientists’ professional effort should be devoted to peer review.

A.3 Use of Information Technology to Reduce Burden

The mode of data collection for the survey was carefully considered with respondent burden in mind. It was determined that automated information technology will be used to collect and process the information. The survey will be conducted online. Invitations to participate will be sent to the selected sample members via email and later by mail, if needed.

A.4 Efforts to Identify Duplication

Collected information will be limited to that which is needed to assess customer satisfaction. Some of the data we are seeking is available through NIH data systems where administrative information relating to research grants and contracts is stored. For example, this includes administrative data on individual grant applications (e.g., date of submission, type of application, and application status). However, these data may not be linked to the customer satisfaction survey responses to achieve the goals of this effort. The proposed survey instrument minimizes the duplication to the maximum extent possible. Only essential demographic data are requested.

A.5. Small Business

Not Applicable

A.6 Consequences of Not Collecting the Information

Scientists will be asked to complete the survey only once in FY2014. NIH has faced consistently increasing application submission rates in the past ten years, and the number of grantees who are deemed eligible and agree to serve as reviewers has remained relatively constant. NIH must consider changes to its criteria for selecting qualified reviewers or the agency may face a time when it cannot recruit a sufficient number of reviewers to meet the demand. Input is needed from the scientific community to ensure the potential changes considered reflect NIH stakeholder opinions. Absent this survey, changes to NIH’s peer review recruitment processes might not be adapted to meet the scientific community’s needs based on customer satisfaction, because formal data on satisfaction with the system would not be ascertained.

A7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 C.F.R. 1320.5

This data collection fully complies with 5 C.F.R. 1320.5.

A.8. Consultation Outside the Agency

Not Applicable

A.9. Payments or Gifts to Respondents

No payment or gift will be offered to survey participants.

A.10. Assurance of Confidentiality

The NIH Privacy Act Officer has reviewed this OMB request and determined that the Privacy Act is applicable (Attachment 4).

Concern for privacy and protection of respondents’ rights will play a central part in the implementation of the survey. Strict procedures will be followed for protecting the anonymity of information gathered from the participants. Participation will be fully voluntary, and the choice not to participate will have no impact on eligibility for or receipt of future funding.

Safeguarding procedures that we will implement include:

• The safeguarding protections offered to survey participants are described in the informed consent language in the introduction to the survey instruments. Respondents will be informed their participation is voluntary and that no consequences will be associated with not responding or with responding. Individuals contacted in the course of these surveys will be assured of their private under the Privacy Act (42 USC 1306, 20 CFR 401 and 422, 5 USC 552 (Freedom of Information Act), 5 USC 552a (Privacy Act of 1974), Privacy Act System of Records Notice: 09-25-036, and OMB Circular No.A-130.

• All data will be analyzed and reported in an aggregate form that does not personally identify any applicants or reviewers.

• An independent contractor, RTI International (RTI), will collect and collate the surveys electronically. RTI will also be responsible for initial analysis and reporting of the data. The data sets that will be transferred back to NIH staff will be fully de-identified. RTI has the required security clearances in order to assure privacy and protection of the data.

• RTI’s Institutional Review Board (IRB) has determined that these surveys are exempt from IRB review (IRB ID Number 13560) based upon information provided by the RTI project manager (Attachment 5). In addition, all study staff members will receive Human Subjects Protection Awareness training. This training will promote awareness of the human subjects’ protection offered by the survey design, ethical issues and concerns, and regulations and assurances by which the survey is governed.

• Access to data will be restricted to project staff members on an as-needed basis.

RTI will observe high standards of information technology (IT) security to protect the privacy, integrity and availability of all computer-based systems and the data they contain. RTI IT security policies and procedures are designed to protect information systems and data from a wide range of risks and will educate their staff to be aware of their responsibilities for ensuring information security and to comply with these policies. RTI also participates with agencies to ensure that their policies conform to agency information security requirements and applicable laws and regulations as required by contract. RTI has System Security Plans for its infrastructures in which it documents how they secure their systems using administrative, technical, and physical controls.

All computer-based systems employed by RTI will comply with the Privacy Act of 1974. The system security features will include:

• User ID and Password authentication required to access all computer systems

• The Website will operate on a certified and accredited Internet-accessible Standard Security Infrastructure which has received an Authority to Operate in accordance with NIST special publication 800-37 (Guide for the Security Certification and Accreditation of Federal Information Systems).

• Web content delivery will be on FIPS 140-2 compliant hardware.

• Access from the Internet is available to authorized staff only and is controlled by RTI’s Internet firewalls. Remote access to RTI’s data networks is provided through the use of client-computer-installed VPN software, a clientless SSL/VPN portal, and direct dial-in connections. The use of RSA SecurID two-factor authentication for remote access is supported.

A.11 Questions of a Sensitive Nature

The NIH is committed to providing high-quality service to its customers.  Given the diversity of its constituents, it is important for NIH to collect survey data from a wide range of customers. The survey contains questions regarding respondents’ race, ethnicity, gender, and age. The survey also contains questions about work-related information (type of employer organization, job title, education, prior research support from and review service to the NIH, the National Science Foundation (NSF) and the Canadian Institutes for Health Research (CIHR)).  This information will allow NIH to analyze the survey data by key analysis subgroups and support NIH’s long-standing efforts to strengthen the diversity of the membership of its applicants and reviewers.  

Respondents may skip any or all of the questions concerning race, ethnicity, gender, age and work-related information in the surveys.  Those who choose to provide these demographic data will do so voluntarily.  The surveys will not collect any personally identifiable information.  Thus, any demographic information gathered by the surveys will not be linked to individual respondents.     

A.12 Estimates of Response of Burden

The total number of participants who will be sampled is 4,779 and the response rate will be approximately 50% based upon previous surveys of this pool of scientists. These participants are university faculty and other members of the NIH research community. It is estimated that the survey will take an average of 15 minutes to complete. The annual hour burden is, therefore, estimated to be 597 hours for approximately 2390 respondents (Table A.12-1).

Table A.12 – 1 Estimates of Annual Hours Burden (Based on Expected 50% Response)

Types of Respondents	Number of Respondents	Frequency of Response	Average Response Time	Annual Hour Burden
Adult Science Professionals	2390	1	15/60	598
Total

Estimated costs to the respondents consist entirely of their time. Costs for time were estimated using a rate of $47.00 per hour for adult science professionals. The estimated annual cost burden for respondents for the first year for which the generic clearance is requested is $28,059 (Table A.12-2).

Table A.12 – 2 Annualized Cost to Respondents (Based on Expected 50% Response)

Types of Respondents	Number of Respondents	Frequency of Response	Average Time per Respondent	Hourly Wage Rate	Respondent Cost
Adult Science Professionals	2390	1	15/60	$47	$28,059
Total

A.13. Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to Respondents or Record Keepers

We do not require any additional record keeping.

A.14. Estimates of Costs to the Federal Government

The approximate annualized cost to the government for this data collection effort is approximately $262,229 (Table A.14-1). Total government personnel costs will be $22,617, taking into account benefits. This figure assumes a upper level GS-15 annual salary of $157,100 for NIH professionals who manage the projects, an upper level GS-14 salary of $138,136 for additional NIH staff members to provide expert reviews and analysis. Salaries are based on the January 2014 General Schedule for the Washington, DC metropolitan area (http://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2014/DCB.pdf).

Contractor support will be required to carry out the data collection efforts. It is estimated that contractor support costs are approximately $239,612. The NIH anticipates completing this project in 4-6 months.

Table A.14-1. Annualized Costs

Activity	Cost
Administration of the Clearance and Contractor Oversight
NIH staff (2 GS-15) – 10% FTE @ $157,100/year in October 2013 – September 2014	$15,710
NIH staff (1 GS-14) – 5% FTE beginning @ $138,136/year in October 2013 – September 2014	6,907
Contract Support for Data Collection
1 survey; 12-month project period	239,612
Total	$262,229

A.15. Changes in Burden

Not Applicable

A.16. Plans for Publication, Analysis and Schedule

The analysis plan is designed to examine the degree to which survey responses differ across key analysis subgroups or combinations of those groups. Key analysis groups are defined by combining the following information to form groups of interest, such as grant funding agencies, grant funding status, employment information, race and ethnicity.

Comparisons across key subgroups will focus on topics such as prior experience with the peer review process, satisfaction ratings about the peer review process, as well as the format of grant applications. Analyses will focus mainly on descriptive information including two-way tables to compare groups of interest.

Two non-NIH comparison groups are planned: A group of scientists who have not had research support from NIH in the past five years, but who have had research support from NSF, and a group who have not had research from NIH or NSF but have had research support from CIHR. These two groups are of interest because scientists supported by NSF and CIHR are often recruited by NIH staff to serve as reviewers, and thus they represent areas of scientific overlap. It is of interest to examine whether attitudes about peer review service differ as a function of the agency providing research support, since differences in research disciplines may influence grantee burden or other factors that affect competing demands on reviewers’ time and subsequent willingness to commit to review service.

Data collected for this study will be aggregated. No results will be reported that identify respondents by name or another identifier that allows respondent’s identity to be disclosed. Specific procedures for analyzing the data are described in the following paragraphs.

Descriptive Information

Analysis will begin with a description of the respondents to the peer review survey. The survey is provided as Attachment 1. One analysis table will be created with the demographic variables collected at the end of the questionnaire.

Data will be presented in tabular format with frequencies and percents for categorical variables; means, minimum and maximum values will be displayed for continuous variables. Table A.16-1 is an illustration of the table that will be compiled during analysis for the descriptive and demographic related questions shown above. The overall numbers of respondents reported in each column will be given in the column headers.

Table A.16-1: Demographic Information – Sample Table Shell

Demographic Question	N =
Ethnicity
Hispanic	n (%)
Non-Hispanic	n (%)
Type of Employer Organization
Institution of Higher Education	n (%)
Hospital/Medical Center	n (%)

Assessing Unit and Item Non-response

Three sampling strata are planned: scientist status (NIH applicant, or NIH applicant/reviewer), race, and ethnicity.

After an overall descriptive summary of the sample respondents, a Unit and Item non-response analysis will be carried out. While sampling weights will be adjusted for unit non-response within sampling strata, if the response rate within sampling strata is low (less than 75%), then the sample respondents may not be representative of the relevant target population. In order to assess whether or not unit response rates are low, response rates will be tabulated for each race and ethnicity group within the two selected samples (Applicant only, Reviewer only, and individuals who are both Applicant and Reviewer).

Even when unit response rates are high, item nonresponse amongst respondents may reduce the degree to which inferences about such an item is trusted. Since there are a variety of analyses that may be carried out using the peer review surveys’ responses, one could calculate item nonresponse for a variety of analytical subgroups. We will tabulate item response rates, separately for the Applicants and Reviewers overall and within some key analytical subgroups, where applicable (e.g., race and ethnicity).

Analysis of Survey Responses

Survey responses to various questions will be analyzed by comparing survey responses between the key groups described in the first section. Categorical responses will be analyzed by cross-tabulating weighted responses across given groups (such as race or ethnicity). Statistical differences will be assessed by performing sample survey appropriate Chi-square tests of proportions to test for independence of survey responses across the groups. Continuous responses will be analyzed by reporting weighted means across given domains. Statistical differences will be assessed by performing sample survey appropriate t-tests to test for differences in mean response across the domains. Two-way tables will be created for all satisfaction/opinion questions in order to compare the groups of interest. All categorical variables will contain the frequency counts of the responses as well as their respective percentage of non-missing data. All continuous variables will be displayed with means along with the number of non-missing responses, minimum and maximum values.

Tables A.16-2 and A.16-3 are examples of tables to display the results of the analysis.

Table A.16-2. Experience of Applicants – Sample Table

	N =
Research Support in the past five years
National Institutes of Health	n (%)
National Science Foundation	n (%)
Canadian Institutes of Health Research	n (%)
Review Service in the past twelve months
National Institutes of Health	n (%)
National Science Foundation	n (%)
Canadian Institutes of Health Research	n (%)

Table A.16-3. Experience of Past NIH Peer Reviewers – Sample Bivariate Table

Question		Future Service Preference
		Regular	Ad Hoc	Mail
		N =	N =	N =
Most Recent Prior Capacity as a NIH reviewer	N =	n (%)	n (%)	n (%)
Regular (appointed)	n (%)	n (%)	n (%)	n (%)
Ad hoc (temporary)	n (%)	n (%)	n (%)	n (%)
Mail	n (%)	n (%)	n (%)	n (%)
Reasons for Accepting an Invitation to Review	N =	Regular	Ad Hoc	Mail
Opportunity to travel	n (%)	n (%)	n (%)	n (%)
Networking with other scientists	n (%)	n (%)	n (%)	n (%)
Honorarium	n (%)	n (%)	n (%)	n (%)
Grantsmanship experience	n (%)	n (%)	n (%)	n (%)
Opportunity to contribute intellectually to my research field	n (%)	n (%)	n (%)	n (%)
Learning the latest news about the funding agency	n (%)	n (%)	n (%)	n (%)
Increased opportunity for tenure/promotion	n (%)	n (%)	n (%)	n (%)
Sense of responsibility to serve as a reviewer	n (%)	n (%)	n (%)	n (%)

Plans for Publication

A written report with accompanying charts will be provided to NIH management for internal use. There are no plans to publish the results of these surveys.

Project Time Schedule

The project time schedule is provided in Table A.16-4. OMB clearance is being requested for one year.

Table A.16-4. Project Time Schedule

Activity	Time Schedule
Launch survey website and email invitations	August 8, 2014
Conduct data collection	August 8 – September 23, 2014
Create analysis file and analyze data	September 24 – November 30, 2014
Document findings	October – December, 2014

A.17. Approval to Not Display Expiration Date

We are not requesting an exemption to the display of the OMB Expiration date.

A.18 Exceptions to Item 19 of OMB form 83-I

These surveys will comply with the requirements in 5 CFR 1320.9.

8