Form 1 Applicants Focus Group Guide

OMB Control Number: 0925-0474 Expiration Date 2/28/2018

Public reporting burden for this collection of information is estimated to average 90 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-0474). Do not return the completed form to this address.

Applicants' Experiences with the NIH Peer Review Process

Focus Group Opening Script

Hello and welcome. Thank you for meeting with us today. My name is Dr. Ami Lynch and I am from Social Solutions International, Inc., a research firm that works on a wide range of issues to improve the health and well-being of people and communities. Thank you for being here to share your thoughts and ideas. My role today is simply to guide our discussion and to make sure that everyone has an opportunity to share their experiences and opinions. 

Opening Comments: 

Social Solutions has been contracted by the NIH Center for Scientific Review to help gain a better understanding of the experiences of Applicants with the NIH Peer Review Process. The purpose of this focus group is to better understand Applicant’s experience and satisfaction with the peer review process. CSR’s mission is to ensure that every applicant receives a fair, independent, expert, and timely review. They want to know your experience and honest feedback about the NIH peer review process. Ultimately, results of this focus group will be combined with the views of other key stakeholders to enable NIH decision-makers to identify what all stakeholders like best about the process and what could be done better. Emerging critical issues and sentiments will be further explored via future quantitative research. CSR hopes this research may lead to the identification of possible new directions for the peer review process. Your input today will be critical to this process.

Focus Group Overview:

This focus group discussion should last approximately 90 minutes. We would like to tape record the discussion so we can keep track of what you tell us. We will not refer to you by name during the audio recording, and ask that other participants refrain from doing so as well. This will ensure that no comments you make can be connected to your name to protect your privacy to the extent permitted by law. When you wish to join in on the discussion and respond to a question, please raise your hand. What you say is very important and we do not want to miss anything. We are only recording the discussion to make sure we don’t miss any of the information you share.

Your names will be tracked to account for the distribution of incentives, but this information will not be shared with anyone at NIH or CSR. The information you share with us today will not be linked to your name in anyway, will not affect your ability to apply for or secure funding from NIH in the future, and will never be associated with your NIH applications.

We would like to ask you not to discuss any of the information you hear from this discussion group with anyone outside of this room. We want you to feel comfortable sharing your personal opinions and experiences with the group so that we can learn from you.

Your participation in this group is entirely voluntary, and you are free to leave at any point if you find it necessary. I will be available after the session this evening to answer any questions you may have about the discussion or the project in general. If you need to get up and stretch, get food or drink, and/or use the rest room during the focus group, please feel free to do so. We just ask that you do so quietly because of the recording.

Following the Focus Group, Social Solutions may invite select participants to appear in a video clip to be presented to CSR with the study findings. Participants will be asked to repeat their particularly insightful comments they made during the focus group. If approached, you are under no obligation to appear in a video clip. If you do chose to do so, Social Solutions cannot be responsible for any of the statements you choose to make on video.

If you DO NOT wish to be contacted with this further opportunity, please let us know at the conclusion of the discussion.

Discussion Group Rules

First, since each and every person's comments are important, we ask that only one person speak at a time. This will help us to hear each other. There is no such thing as a right or wrong answer in this group. We have no expectations about what is going to be said. Tell us what you honestly think, and feel free to share whatever is on your mind. If you disagree with something that you hear, let us know. If you agree, don't just say, "I agree", but try to add your own perspective or opinions. We want to hear as many different opinions, ideas, and stories as possible.   

Consent

As you came in, you were asked to sign an informed consent statement. In order to participate today, we must have your written consent. Is there anyone who has not signed the consent statement?

[Social Solutions researchers will ensure that the number of participants matches the number of signed consent forms prior to proceeding. If the numbers do not match, Social Solutions will confirm whether they have completed theirs privately with each participant, to avoid identifying participants to each other.]

Does anyone have any questions at this time?

I am going to turn on the tape recorder now so we can get started…

Applicants' Experiences with the NIH Peer Review Process

Focus Group Probes/Questions

Introduction: Over 70% of peer reviews are conducted by CSR, and the remainder by individual Institutes and Centers. During this focus group discussion, if you are referring to your experience with an I/C review, please let us know.

1. Let’s first think about the success of the NIH/CSR review process, in terms of the overall outcomes and what is or is not being achieved. Please describe your experience with the peer review process. Would you characterize the current review process as successful? Why or why not?

• Probes: How many applications have you submitted? Revised and resubmitted?

2. What conditions or circumstances influence your satisfaction with peer review other than a fundable score?

• Probe: Quality or utility of Summary Statements? NIH policies regarding peer review? Communication with NIH and/or I/C staff?

3. We would like to focus on CSR reviews for this next question. CSR’s mission is to ensure “NIH grant applications receive fair, independent, expert, and timely reviews—free from inappropriate influences—so NIH can fund the most promising research.” Is this mission reflected in the day-to-day reality of NIH peer review? How can NIH strengthen the quality of the peer review process to align more with CSR’s mission?

• Probes: How well does the peer review process embody NIH’s mission regarding fairness? Independence? Expert? Timeliness?

4. How knowledgeable do you feel about the NIH peer review process?

• Probes: Where do you get most of your information about the NIH peer review process? Who do you feel has been most helpful in explaining or providing information about the NIH peer review process?

5. Sometimes there is a divergence of opinion among reviewers. Have you ever had an application that was given very positive feedback from two of the reviewers, and negative feedback from the third reviewer? What did you think/do in this situation?

• Probes: Did you speak to a Program Officer? Did you revise/resubmit? Were you discouraged from doing so? What was the outcome? Did your score improve?

6. Are your applications being reviewed in Study Sections with the best scientific fit?

• Probes: Did you request a specific Study Section or Study Sections for the review of your application? Was your request honored? Were there individuals with appropriate levels of expertise on the Study Section that reviewed your applications? If not, did you request a different Study Section when you resubmitted?

7. How many of you have been in contact with CSR or an NIH I/C prior to or after submitting an application? Please describe this contact. In what ways was the contact helpful/unhelpful?

• Probes: Through what mechanism did you communicate? With whom? What was the outcome of the interaction? If you have communicated with both CSR and I/C POs, is there any difference in the interaction?

8. Regarding the usefulness of summary statements to you, how useful is the Overall Impact paragraph to summarize the drivers of the overall final score? Are criterion scores useful? Do they provide useful feedback to you about the impact or scientific merit of your application? Why or why not?

• Probes: Which feedback is most helpful: Summary Statements or feedback from the Program Officer? Describe the ideal feedback you wish to receive on a non-funded application. How similar is this to the feedback that you receive? Is there anything not typically included in a Summary Statement that you think is important to know? Has the new bulleted format improved the quality/utility of the Summary Statement, when compared with the old paragraph format?

9. What do you think of the new policy on resubmissions?

• Probes: What do you plan to do differently in response to the new policy?

10. Is there anything else that would be helpful for NIH to know about (your experiences with/impressions of) the current peer review process? Anything about the topic that we didn’t ask? Is there anything else you want to share?

Thank You