CSR SupportingStatementB

Supporting Statement B for

GENERIC CLEARANCE FOR SURVEYS

OF CUSTOMERS AND PARTNERS OF

THE CENTER FOR SCIENTIFIC REVIEW (CSR)

NATIONAL INSTITUTES OF HEALTH

Date: August 22, 2011

Name: George Chacko, Ph.D.

Address: 6701 Rockledge Drive, Room 3030, Bethesda, MD, 20892-7776

Telephone: 301-435-1133

Fax: 301-480-3965

Email: chackoge@csr.nih.gov

Table of Contents

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS 1

B.1 Respondent Universe and Sampling Methods 3

TABLE B1-1 3

B.2 Procedures for the Collection of Information 4

B.3 Methods to Maximize Response Rates and Deal with Non-response 4

B.4 Test of Procedures or Methods to be Undertaken 5

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting

and/or Analyzing Data………………………………………………………………….5 `

Supporting Statement B

B.1 RESPONDENT UNIVERSE AND SAMPLING METHODS

The respondent universe will be separately identified for each initiative whose customers and or partners are to be surveyed. Developmental activities, such as focus groups, will be designed to assure inclusion of an appropriate range of stakeholders; quantitative activities will survey 100 percent of the universe or use sampling procedures developed to be properly representative of the universe. Surveys will be designed to minimize burden on respondents while obtaining essential information. The expectation is that information collection instruments will require no more than 15 minutes response time, on average. Focus groups will generally last for two and a half hours (plus travel time). In virtually all instances, there will be existing lists of "customers" readily available for sampling (e.g., charter members of review study sections, meeting rosters, program official listings, etc). Appropriate probability sampling techniques will be used to select samples. Respondents are expected to be a mix of adult scientists connected to the research community. It is estimated that e-mail/mail surveys will take 15 minutes to complete. The annual hour burden is, therefore, estimated to be 1,250 hours for approximately 5,000 respondents in each of the 3 fiscal years. Focus groups: It is projected that in each year of this approval approximately 5 focus groups will be convened, primarily for the purpose of customer input into the design of satisfaction surveys. Each focus group is expected to require 2.5 hours and will include approximately 15 members. The focus groups are expected to take 2.5 hours to complete. The annual burden for the 5 annual focus groups is estimated to be 187.5 hours for each of 3 fiscal years. The expected response rate for these surveys is 80 percent.

B1-1 EXPECTED ANNUALIZED NUMBERS AND

CHARACTERISTICS OF SAMPLE PERSONS

Type of Respondents	Number of Respondents	Frequency of Response
Adult science professionals - Focus groups	75	1
Adult science professionals -mail/telephone/ e-mail surveys	5,000	1

B.2 PROCEDURES FOR THE COLLECTION OF INFORMATION

A roster of reviewers and their e-mail addresses will be developed for each review meeting related to the topic being evaluated. For example, studying the use of a 2 stage review process (Editorial Board) for grant reviews is expected to be most beneficial for very complex grants. Therefore, the data collection universe will only include review meeting for grants that meet criteria for being complex. After the list is generated and checked for duplications, all e-mail addresses will be checked and verified. When the rosters are finalized, CSR will email each intended respondent a letter explaining the purpose and importance of the survey and will ask each one to participate. The letter will also inform each intended respondent of the Internet address where they can find the survey. The same day, at a later time, an email will be sent from the Scientific Review Officer (SRO) for each of the surveyed scientific review groups. It will reiterate what the invitation letter said and will contain the unique access code and password needed to gain entry to the survey. No access to any other survey instrument except for the one assigned to the respondent will be allowed. The survey will be submitted electronically by clicking a submission button. With submission, the password will be automatically deactivated. No re-access to the site will be allowed for security reasons.

All data collection will be conducted in a manner that is consistent with the following principles:

o Appropriate sample sizes will be determined for each activity to assure that burden is minimized while reliable estimates are produced.

o Participation will be fully voluntary, and non-participation will have no impact on eligibility for or receipt of future services. If necessary, steps will be taken to ensure unbiased completion of questionnaires by use of third-party distribution and receipt by a party not directly involved in provision of the service being assessed.

o Information to be collected will be limited to that needed to assess customer satisfaction. Repeated implementation of quantitative surveys will be at an interval appropriate to measure the impact of changes and to monitor ongoing levels of satisfaction.

o Efforts will be made to obtain the highest possible response rates, given the voluntary nature of the data collection efforts. To the extent feasible, efforts will be made to assess non-response bias.

• Previous customer satisfaction surveys have achieved response rates from 50 to 60 percent. As discussed in B3, efforts for increasing response rates will be enhanced and emphasized.

B.3 Methods to Maximize Response Rates and Deal with Non-response

Consistent with sound survey methodology, the design of each quantitative survey will include approaches to maximize response rates, while retaining the voluntary nature of the effort. For e-mail surveys, for example, this is expected to include a second e-mailing of the questionnaire, a reminder e-mail, and possibly some telephone follow-up, if phone numbers are available.

B.4 Test of Procedures or Methods to be Undertaken

It is anticipated that some surveys will begin with focus groups or similar efforts to identify the views and concerns of customers. More formal pre-testing will be carried out at a level and in a manner consistent with the specific survey. All mail and telephone surveys are expected to include pre-testing with a small number of potential respondents with telephone debriefing of pretest respondents as needed to clarify responses.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Each program will obtain input from statisticians in the development, design, conduct and analysis of customer satisfaction surveys. This statistical expertise will be available from agency evaluators or from contractors.

5