NCSES - Cognitive Interview Research for the 2020 Higher Education Research and Development Survey

HERD Respondent Debriefing Protocol After Completing Questions

HERD Respondent Debriefing Protocol After Completing Questions

[Interviewer introduces herself.] Thank you very much for talking with me today and for taking the time to complete the proposed questions for the upcoming FY 2020 NSF-NCSES Higher Education R&D Survey.

Before we begin our interview, I want to remind you of some things.

• Your participation in this interview is voluntary.

• The interview should last no more than 30 minutes.

• There is another person on the telephone with me who is taking notes.

• If appropriate: <Name>, from NSF-NCSES, is also listening to our discussion today. Is that okay? If no, NCSES staff member will hang up the phone.

• I would like to audio record this interview so that I get all of your comments. Is that okay with you?

• Before we continue with the interview, I need your consent to be interviewed. I’m going to read you the consent form and then I’ll ask for your verbal consent.

The National Center for Science and Engineering Statistics within then National Science Foundation designs, conducts, and sponsors surveys on science, engineering, and technology. In order to produce the best information possible, NCSES routinely evaluates its surveys.

You have volunteered to take part in a study to improve one of NCSES’s surveys. In order to have a complete record of your comments, your interview session will be audio recorded. We plan to use the recording to improve the survey. NCSES and ICF staff directly involved in this research project will have access to the recording.

This study is authorized by law (42 U.S.C. 1862 Section 3.a.6). The OMB control number for this study is 3145-0174.

I have volunteered to participate in this study, and I give permission for my recording to be used for the purposes stated above.

If you consent to participate in this interview, please state your name and that you consent.

Do you have any questions before we start?

If you don’t have it in front of you already, please pull up your responses to the proposed new questions.

(Head counts) Let’s start with your overall reactions to proposed new question 1.

1. Can you tell me how you went about gathering this information?

   1. Were you able to do it yourself, or did you get help from other people?

      1. If you got help from others, were those people in your own office or department, or in other offices/departments? What other offices or departments were they in? Did they have any feedback on the question?

2. Do the numbers you reported represent estimates, or figures from records?

   1. How did you estimate?

   2. What records did you use? Where are those records kept? (In what offices or departments are they kept? In what system(s) are they kept?)

3. Did you have to make any assumptions about what the question was asking? If so, what were they?

4. Did you or anyone that you consulted with have questions while working on this question?

   1. If yes, what were your (or their) questions? Were you able to answer them?

5. How easy or difficult was it to gather this information?

   1. If difficult, tell me more about that.

   2. Were any of these things easier or harder to gather than others?

6. [If some of the survey question was not answered] I noticed that [line/column X] was blank. Is there any particular reason for that?

   1. Do you believe you will be able to provide those data points for the FY 2020 survey, or would they also be blank?

      1. If no, why not and do you think the information could be provided for a survey beyond FY 2020?

7. Did you refer to the instructions as you were completing the question, or not?

   1. If referred, what parts of the instructions were helpful? Which parts were not helpful? Were there any parts of the instructions that you thought were confusing, or could be confusing to another person?

8. Did you or any of your colleagues have concerns about reporting this data?

(FTEs) Now let’s look at proposed new question 2.

9. Can you tell me how you went about gathering this information?

   1. Were you able to do it yourself, or did you get help from other people?

      1. If you got help from others, were those people in your own office or department, or in other offices/departments? What other offices or departments were they in? Did they have any feedback on the question?

10. Do the numbers you reported represent estimates, or figures from records?

    1. How did you estimate?

    2. What records did you use? Where are those records kept? (In what offices or departments are they kept? In what system(s) are they kept?)

11. Did you have to make any assumptions about what the question was asking? If so, what were they?

12. How easy or difficult was it to gather this information?

1. If difficult, tell me more about that?

2. Were any of these things easier or harder to gather than others?

13. Did you or anyone that you consulted with have questions while working on this question?

a. If yes, what were your (or their) questions? Were you able to answer them?

14. [If some of the survey question was not answered] I noticed that [line/column X] was blank. Is there any particular reason for that?

a. Do you believe you will be able to provide those data points for the FY 2020 survey, or would they also be blank?

1. If no, why not and do you think the information could be provided for a survey beyond FY 2020?

15. Did you happen to refer to the instructions as you were completing the question, or not?

1. If referred, what parts of the instructions were helpful? Which parts were not helpful? Were there any parts of the instructions that you thought were confusing, or could be confusing to another person?

2. What did you think of the instruction that “an individual cannot be more than 1.0 FTE?”

16. Did you happen to refer to the examples for calculating the FTEs as you completed the question?

a. Were they helpful or not helpful?

b. What did you think of the level of detail in the examples: was it too much, too little, or about right?

Overall Thoughts for Proposed Questions

17. Approximately how many hours did it take you, and others who contributed, to answer these two questions, including the time it takes to gather information, talk to anybody you’d need to talk to, etc.?

18. Do you have any questions or comments about anything that we have not covered?

Those are all our questions. Thank you so much for your time today. If we decide to put the question on the survey, we will send an email announcing the change as soon as we can.

4