CSR Attachment 2- Reviewer Survey Instrument

Asynchronous Electronic Discussion

Reviewer Experience

Dear NIH reviewer,

The National Institutes of Health (NIH) thanks you for your service as a reviewer of grant applications. Your expertise and independent insight are invaluable to the peer review process, which helps ensure that NIH continues to fund the most promising research.

We would like to solicit your feedback on your most recent experience as a reviewer using the Asynchronous Electronic Discussion review format. NIH will use the results as part of assessments of the effects and benefits of different review formats on the peer review process.

Any information you provide will be treated as confidential, and reported only in a form that does not personally identify you. Your response is voluntary; however, we hope that you will take a few minutes and provide us with your perspective. If you have any questions about this information request, please contact Dr. Andrea Kopstein, Director of Planning, Analysis, and Evaluation (Tel: 301-435-1133; email kopsteina@csr.nih.gov).

Thank you!

USE OF AED

1. Have you previously participated in a review using the AED format?

• No

• Yes

2. If you previously participated in an AED meeting, was this experience better, the same, or worse than your prior experience(s)?

• Better

• The same

• Worse

3. Were you satisfied with the following aspects of AED?

	Y	N
Access to AED
Navigation of AED
Instruction on how to use AED
Technical support during AED session
Speed of AED
Appearance of AED (e.g. font, margins)
Timing out of the session
Access to other reviewers’ comments
Entering your comments
Following a discussion
Access to an SRO
Scoring of applications
Overall experience

4. Approximately how many hours per day have you spent participating in the AED meeting? Please enter a number in the box below.

_____ hrs

5. What is the maximum number of applications that should be reviewed during one AED meeting? Please enter a number in a box below.

_______

DISCUSSION PROCESS

6. Were you the chair of this AED meeting?

• Yes

• No

7. Was the duration of the discussion phase sufficient to allow meaningful consideration of the applications?

• Too short

• About right

• Too long

• No opinion

8. Would you characterize the AED discussion as sufficiently rigorous?

• Yes

• Somewhat

• No

• No opinion

9. If not, do you have any suggestions on how to make the AED discussion more rigorous?

10. Did the AED review format enhance or limit your ability to assess scientific merit?

• Enhanced

• Neither enhanced nor limited

• Limited

• No opinion

11. How would you compare your level of participation in the AED discussion to a face-to-face review?

• I was more active during the AED meeting

• About the same

• I was less active during the AED meeting

• No opinion

12. How would you compare the study section chair’s participation in the AED discussion to face-to-face review?

• Chair appeared to be more involved than during face-to-face meeting

• About the same

• Chair appeared to be less involved than during face-to-face meeting

• No opinion

13. Were you sufficiently well-informed at the end of the AED process to make a fair evaluation of the applications assigned to you?

• Yes

• Somewhat

• No

14. Did the AED review process result in outcomes that were fair to the applicants?

• Yes

• Somewhat

• No

15. For applications not assigned to you, did you use any of these primary resources to develop scores for applications? Please select all that apply.

• Applications

• Critiques provided in IAR

• Discussion

• Final scores provided by assigned reviewers

• None of the above

• Other. Pease specify ___________________

16. Given the structure and format of the AED review process, do you have any suggestion for encouraging reviewer discussion of non-assigned applications?

GENERAL VIEWS

17. How would you compare the overall burden of participation in the AED meeting to a face-to-face meeting?

• AED was less burdensome

• The burden was about the same

• AED was more burdensome

• No opinion

18. Do you consider not traveling to a review meeting a benefit or a disadvantage of using the AED format?

• Benefit

• Neither benefit nor disadvantage

• Disadvantage

• No opinion

19. If you had a choice for your next review, would you prefer to participate as a reviewer in an AED meeting, at a face-to-face meeting, or have no preference?

• Prefer AED

• No preference

• Prefer face-to-face meeting 

20. Do you agree or disagree with the following statements?

	Agree	Disagree
I am more likely to agree to participate in a review if I do not have to travel
AED is a more efficient use of my time than in-person review
Getting immediate feedback to my points during discussions helps me think
AED format helps me focus more on content and less on presentation
AED discussion threads help me scan for important points
AED supports iterative and reflective evaluation of applications
Being able to participate at my own pace helps me focus
AED allows for more views to be considered than in-person review
AED review is more objective than in-person review
It is easier to reach consensus using AED than in in-person review
AED technology supports effective peer review

21. What do you like best about AED (max. 500 characters)?

22. What do you like least about AED (max. 500 characters)?

23. Please provide any additional comments that you may have regarding AED.

Thank you for your feedback!