NCSES - Cognitive Interview Research for the 2020 Higher Education Research and Development Survey

HERD Cognitive Interview Protocol Before Completing Questions

HERD Cognitive Interview Protocol Before Completing Questions

[Interviewer introduces herself.] Thank you very much for talking with me today. As you know from our initial contact, we are considering adding some questions about head counts and full time equivalents (FTEs) in the upcoming FY 2020 NSF-NCSES Higher Education R&D Survey.

Before we begin our interview, I want to review some information with you.

• Your participation in this interview is voluntary.

• The interview should last no more than 30 minutes.

• There is another person on the telephone with me who is taking notes.

• If appropriate: <Name>, from NSF-NCSES, is also listening to our discussion today. Is that okay? If no, NCSES staff member will hang up the phone.

• I would like to audio record this interview so that I get all of your comments. Is that okay with you?

• Before we continue with the interview, I need your consent to be interviewed. I’m going to read you the consent form and then I’ll ask for your verbal consent.

The National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation designs, conducts, and sponsors surveys on science, engineering, and technology. In order to produce the best information possible, NCSES routinely evaluates its surveys.

You have volunteered to take part in a study to evaluate one of the NCSES’s surveys. In order to have a complete record of your comments, your interview session will be audio recorded. We plan to use the recording to improve the survey. NCSES and ICF staff directly involved in this research project will have access to the recording.

This study is authorized by law (42 U.S.C. 1862 Section 3.a.6). The OMB control number for this study is 3145-0174.

I have volunteered to participate in this study, and I give permission for my recording to be used for the purposes stated above.

If you consent to participate in this interview, please state your name and that you consent.

Do you have any questions before we start?

We just sent you a copy of your completed FY 2019 survey. The proposed new questions are included in that attachment. Did you receive the copy of the survey we sent?

[If they did not receive a copy of the revised survey, be prepared to e-mail it to them immediately.]

[Give them a few minutes to review before beginning interview]

(Head counts) Let’s start by going to page 29, and the proposed new question 1. I’ll give you a moment to read through it. Please let me know when you’re finished.

1. What information is this question requesting?

2. What are your initial reactions to this question?

   1. Can you tell me more about that?

3. This question refers to “functions” of personnel. What does “functions” mean to you, in the context of this question?

4. What office(s) or department(s) at your institution would have this information?

5. Does your institution classify its own employees by function in the way the question asks?

   1. What sort of employees would your institution consider to be “researchers”?

   2. What about “R&D technicians”?

   3. What about “R&D support staff?”

   4. How would you classify student R&D personnel?

6. Are the instructions asking you to do something that you would normally do? Or are they asking you to do something that seems unusual?

(FTEs) Now let’s look at the proposed new question 2, on page 30. I’ll give you a moment to read through it. Please let me know when you’re finished.

7. What information is this question requesting?

8. What are your initial reactions to this question?

   1. Could you tell me more about that?

9. This question refers to “full-time equivalents (FTEs).” Does your institution typically have this information?

10. What office(s) or department(s) at your institution would have this information?

11. Are the instructions asking you to do something that you would normally do? Or are they asking you to do something that seems unusual?

12. What are your reactions to the examples provided?

    1. Are they too long, too short, or about right?

    2. Are they helpful or not helpful?

Overall Thoughts for Proposed Questions

13. Do you think you would be able to answer these questions yourself, or would you have to consult with others?

    1. If others, who will you consult with? Is it different for each question?

    2. If others, how will you go about getting information on this topic from that person/those people? (Phone call? E-mail? Share the questionnaire or not?)

14. Approximately how long do you think it will take you, or others pulling this data together for you, to answer these two questions, including the time it takes to gather information and talk to others?

15. Do you have any questions or comments about anything that we have not covered?

Those are all of our questions for this interview today. The next step is for you to complete the two proposed new questions we’ve just talked about. Just a reminder, the responses provided will not be released publicly. They will only be used for research and evaluation purposes.

Once you’re finished with the new questions, we’d like to talk again. Will you be able to finish these questions within 3 weeks? Would you prefer to schedule that meeting now, or schedule it later?

Thank you so much for your time today.

2