Proposed Content Revisions for the Survey of Science and Engineering Research Facilities

Appendix C: Telephone Discussion Protocol

Protocol for Facilities Survey Testing of Questionnaire Revisions

Thank you for taking the time to speak with us today. My name is <NAME> and I am an employee of Westat, the National Science Foundation’s contractor for the Survey of Science and Engineering Research Facilities, also referred to as the Facilities Survey. Before each cycle, we examine the survey content to see if updates are needed due to changes in the way research is conducted, as well as to address evolving data needs.

Before we begin our next survey cycle in fall 2017, we are talking with a number of universities to get their feedback on proposed revisions to the research definition, the definition of project start date for new construction, and a new question on shared research space. We are not evaluating you or your institution, and are not collecting data for publication. Rather, we are looking to improve the survey questionnaire, so there are no right or wrong answers.

This discussion is voluntary, so you may end it at any time and you are not obligated to answer any questions. We will prepare a summary of all our discussions. It would be very helpful if we could audio record this discussion to ensure we have accurate notes. The names and content of this discussion, including the audio recording, will only be used for internal analysis and questionnaire development by the NSF survey staff and the NSF survey contractor. Is it ok to record our talk?

Do you have any questions before we begin?

(START RECORDER UNLESS THE RESPONDENT DECLINES TO BE RECORED: Thank you for agreeing to have this discussion recorded. We will now begin with our first question.)

1. Did you get a chance to review the documents about the survey revisions that we are going to discuss today? [IF YES, CONTINUE WITH THE NEXT QUESTION. IF NO, E-MAIL THE DOCUMENTS WHILE ON THE PHONE or RESCHEDULE THE CALL]

2. On page 2, you will see revised wording for the definition of research. Wording changes are highlighted in yellow for your convenience. [READ THE DEFINITION ALOUD IF NECESSARY]

   1. How well or how poorly does the survey definition match your institution’s definition of research?

   2. Do you think the new wording would or would not change the way you report research space or costs on the survey compared to the way you reported for FY 2015? If so, would you please describe how your responses would change?

   3. Do you find the revisions helpful or not helpful? In what ways?

   4. Do you think there is some aspect of the definition that could cause problems or questions for other institutions?

   5. Do you have any recommendations or comments on this proposed revision?

3. On page 3, you will see a new question about shared space (question 6). The goal of this question is to obtain more data about space that is shared among fields or used for other purposes in addition to research.

[IF THEY DID NOT RECEIVE Q6 BEFORE THE CALL, COLLECT Q6 ANSWERS BY PHONE]

[REVIEW THEIR ANSWERS TO Q6 AND PROBE IF THE FIELDS WITH RESEARCH SPACE ARE NOT CONSISTENT IN Q2 AND Q6]

1. In your FY 2015 response to Q2, did you pro-rate any of the space because it was shared among fields? Did you pro-rate because space was used for other purposes in addition to research?

2. In your FY 2015 response, was there any shared space that you did not pro-rate in Q2? If so, what was the reason you did not pro-rate this shared space? Was the space shared across fields or by purpose?

Now let’s talk about the new question 6.

3. What steps did you take to answer this question?

4. Did you need to talk to multiple departments to obtain the information to answer this question or does your institution have a system or single source that maintains this type of information?

   1. IF MULTIPLE SOURCES: Could you describe the type and number of departments that you worked with?

5. How easy or difficult was it for you to answer this new question? (very difficult, somewhat difficult, somewhat easy, very easy)

6. About how long did it take to respond to this question?

7. Do you have any recommendations for improvements or other comments about this question?

4. On pages 4, 5, and 9, you can see the proposed change to the definition of project start date for new construction, which is highlighted.

1. How well or how poorly does the survey definition match your institution’s definition of project start date?

2. Would the new definition change your FY 2015 response to any of the questions 9, 9B or 13, or would the new definition not change your response? If so, would you please describe the reason the response(s) would change?

3. Do you find the revisions helpful or not helpful? In what ways?

4. Do you think there is some aspect of the definition that could cause problems or questions for other institutions?

5. Do you have any recommendations for improvements or other comments about this proposed revision?

5. As you see on pages 4, 6 and 9, we’ve added a sentence to the instructions for reporting completion costs.

1. [BASED ON FY 2015 RESPONSES TO Q9D AND Q13, ASK ABOUT THE APPROPRIATE ITEMS]: Would the additional instructions change your response to question [9D or 13]? If so, would you please describe the reason the response(s) would change?

2. Do you find the additional instruction helpful?

3. Is there some aspect of the additional instruction that you think may cause response issues or questions for institutions?

4. Do you have any recommendations for improvements or other comments about this proposed revision?

6. Let’s look now at the Individual Project Form. Does your institution typically assign the Individual Project Form to someone other than the coordinator to complete? If so, are copies of the form provided from the PDF version of the questionnaire or in some other format, such as screenshots from the web survey or providing access to the web survey?

7. We’ve added a note to the bottom of each page of the Individual Project Form (pages 5–8) to inform or remind respondents that a survey help desk is available. We added this because coordinators may assign these pages to someone who may not realize that help is available. However, we did not include the help desk contact information in order to allow the survey coordinator at the institution to serve as the gateway for questions, if desired.

1. Do you find it helpful to include this note, or is it not helpful?

2. Do you think it’s useful to leave off the help desk contact information from the note, which requires contacts to go through the institution coordinator, or should we provide the help desk contact information in the note?

8. Do you have any general comments about the survey that you’d like to share? Do you have any questions for us?

Those are all of the questions that I have. Your feedback is very helpful as we finalize the FY 2017 survey. Survey invitations for the next cycle will go out this fall, typically in mid-to-late October. Thank you again for your continued participation in the Facilities Survey and for your help today.