NCSES - Respondent Debriefing Research for the FY 2020 Survey of State Government Research and Development

State Government R&D Survey: Respondent Debriefing Protocol

Draft 10/16/20

Revised 10/27/20

Purpose

• To gain feedback from respondents on the data sources used and reporting routines for the following questions:

  • Q4: internal R&D

  • Q5: external R&D

  • Q9: internal R&D by type of R&D

  • Q10: R&D employees

  • Q11: R&D FTEs

• To determine the best placement of Q12

Interview mode

Zoom

Materials Needed

• Consent form

• Interview protocol

• Copy of submitted data (PDF)

Introductions

• Thank you for completing the State Gov R&D Survey, and for agreeing to participate in this follow-up interview.

• Purpose: to learn how you went about answering questions – records/reports you looked at, people you talked to, anything that was particularly easy or hard about answering the questions.

• Start with general questions and experiences, then move to more specific questions.

• Who is in the meeting, titles, roles

• Consent form

• Start recording

General Questions

• Overall, how easy or difficult was it to complete this survey?

  • Very difficult

  • Somewhat difficult

  • Somewhat easy

  • Very easy

• Do any questions stand out in your mind as being particularly difficult?

• Do any questions stand out in your mind as being particularly time-consuming?

• Did you answer the questions in order, or did you skip around?

  • If skip around: do you happen to remember which questions you answered first? Which ones you answered last?

• Ok, let’s take a closer look at some of the questions on the survey.

Question 4 (Expenditures for R&D Performed Internally)

Let’s start with #4.

• In your own words, what is this question asking you?

• Can you tell me how you went about answering this question?

  • What records did you consult to answer this question?

  • Did you get help from others in your agency to answer this question?

• Can you give me some examples of these types of expenditures?

• What did the term “internal R&D performance” mean to you? Do you use this term yourself? If not, what term or terms do you use to describe this type of activity?

• Do your records allow you to easily distinguish among these different sources of funds?

• What does “state funds used for internal R&D” mean to you? Can you give me an example of this?

Question 5 (Expenditures for R&D Performed Externally)

Let’s go on to question 5.

• In your own words, what is this question asking you?

• Can you tell me how you went about answering this question?

  • What records did you consult to answer this question?

  • Were the records you reviewed for this question the same as the ones for Q4, or different?

  • Did you get help from others in your agency to answer this question?

• What did the term “external R&D performance” mean to you? Do you use this term yourself? If not, what term or terms do you use to describe this type of activity?

• Do your records allow you to easily distinguish among these different sources of funds?

Question 9 (Expenditures for R&D Performed Internally by Type of R&D)

We’re going to skip over several questions now. Let’s move to Question 9.

• In your own words, what is this question asking you?

• How did you go about answering this question?

  • Did you use records to come up with an answer, or did you estimate?

    • If estimate, can you tell me how you went about estimating?

    • If records, what records did you consult to answer this question?

  • Did you get help from others in your agency to answer this question?

• How easy or difficult was it to provide a response for this question?

• Do you remember whether or not you referred to the examples of basic research, applied research, and experimental development (at the top of the page)?

• Does your agency classify its R&D work in these three buckets (basic research, applied research, experimental development) or not?

  • If not, does your agency classify its R&D projects some other way? Can you tell us about it?

• Were there any projects or expenditures that you were unsure how to classify? What made it hard? How did you decide what to do with it/them?

Question 10 (R&D Employees)

• In your own words, what was this question asking?

• How did you go about answering this question?

  • Did you use records, or did you estimate?

  • Did you talk to other people, or were able to answer it on your own?

• Does your agency classify its own employees in the way this question asks? In other words, were you able to identify researchers vs. R&D technicians and equivalent staff vs. R&D support staff, or not?

  • How did you go about determining which employees fit in each category?

  • Do your agency’s records have this information? How easy or difficult was it to get this information from the records?

• What sort of employees did you consider to be “researchers”?

  • Can you give me some examples of job titles?

• How about “R&D technicians and equivalent staff”?

  • Examples of job titles?

• How about “R&D support staff”?

  • Examples of job titles?

• How easy or difficult was it to obtain the numbers of employees in these three categories (rather than a headcount for the whole agency) – very difficult, somewhat difficult, neither easy nor difficult, somewhat easy, or very easy?

• Does your agency have any full-time employees who spend only part of their time on R&D?

  • If yes: Did you include or exclude those employees in your answer?

• Does your agency have any part-time employees that would fit into any of these three categories (researchers, R&D technicians/equivalent staff, R&D support staff)?

  • Did you include or exclude those part-time employees in your answer?

• What time period does your data cover?

  • Did you happen to notice the reference date (pay period including March 12, 2020) when you were completing the questionnaire?

  • Did you use that date, or some other date?

Question 11 (FTE R&D Personnel)

• In your own words, what was this question asking? How was it the same or different, compared to Question 10?

• How did you go about answering this question?

• Did you use records, or did you estimate?

  • If records:

    • What records did you use?

    • Did you access those records yourself, or did you work with someone else to get them?

  • If estimate:

    • How did you estimate?

    • What process did you use?

    • Did you estimate in your head, use some sort of tool?

    • Did you talk to other people, or were able to answer it on your own?

• Does your agency already calculate the number of full-time equivalent employees it has? Was the survey asking you to do something you already do, or not?

• Did you happen to refer to the examples in the bulleted list?

  • What did you think of the examples?

  • Were they too long, too short, or about right?

  • About how many times did you look at the examples?

  • Were they helpful or not helpful?

  • Did you think they were easy or hard to understand?

(If agency has part-time employees working on R&D) Were your agency’s part-time employees who work on R&D included or excluded in your answer to this question?

• What time period does your data cover?

  • Did you happen to notice the reference date (pay period including March 12, 2020) when you were completing the questionnaire?

  • Did you use that date, or some other date?

Question 12 (Expenditures for R&D Performed Externally by Type of Entity)

Let’s take a look at Question 12, Expenditures for R&D Performed Externally by Type of Entity.

• How did you go about answering this question?

• Did you happen to look back at Question 5 as you were answering this question?

• What did you think of where Question 12 is located in the survey? Did it seem odd for it to come after Question 11, or was it ok?

Wrap Up

Those were all the questions we had for you today. Did you have any questions for us?

Thank you for your time.

State Government R&D Survey: Respondent Debriefings

Page 6 of 6