Request for Approval for Cognitive Testing of Innovation Activities as Input to Annual Business Survey

Attachment 2 – Innovation Activities Cognitive Testing Interview Protocol
Introduction
(To be given by NSF representative if present, or Westat interviewer if NSF is not available)
Points to cover in the introduction; these will be fleshed out into a script
Hello, we are [introduce attendees from NSF, Westat].
The National Science Foundation (NSF) is in the process of developing a survey to collect information
from small and medium sized U.S. businesses. The focus of this interview will be on innovation. There
are no right or wrong answers.
We would like to record this interview so that we have all your comments. Would that be okay?
[Have respondents read and sign consent forms.]
Background
Please tell me, in 1-2 sentences, what your company does. [Use the response to this question to glean
whether the respondent uses products vs. goods vs. services. Then go into the topic of innovation.]
Innovation Activities
Now let’s talk about the new or improved goods or services that your company has developed over
the last three years. Please think of all the developmental, financial and commercial activities your
company undertook to develop the actual products, goods, or services. We are also interested in
understanding the costs associated with these activities.
Which of these activities has your company performed in the last three years? [Use ATTACHMENT 2A
and have the respondent mark yes or no for each.]









Research and development activities
Engineering, design, and other creative work
New approaches to marketing and brand equity activities
Purchasing, selling, or protecting intellectual property
Training employees to carry out or market the new or improved products, goods, or
services
Software development and database activities to develop new or improved products,
goods, or services
Investing in machinery, equipment, and other tangible assets to support your new or
improved products, goods, or services
Management activities to support new or improved products, goods, or services

What other activities, if any, has your company performed to support the new or improved products,
goods, or services? [Record any new activities.]

7

Let’s talk about those activities your firm has performed in the areas you marked. For now, let’s focus
on the [good/service – use the word(s) that respondent has used] that required the greatest effort or
cost to investigate or make. It doesn’t need to have been successful, or even to have been
completed. What would that new or improved product, good, or service be? [Walk through the
following for each of the areas marked “yes” above. For example, “First, let’s talk about your
research and development activities, and how you used them to support your new or improved
product, good, or service.”]


Speaking very generally, describe what your company did with regard to … [e.g., research
and development] What activities did your company engage in at this stage?



What kind of resources did that require? Personnel time? Equipment? Supplies?
Outside contractors?



Were these resources dedicated just for this new or improved product, or were the
resources used for other purposes? If they were used for other purposes, what were
they?



Would your company be able to provide a reasonably accurate cost estimate of what was
required with regard to developing or supporting your new or improved product?
o [IF NOT] What would hinder your company from being able to provide an accurate
estimate?
o [IF SO] How would your company determine this cost?



Has your company worked with any other company or organization on this activity? [IF
YES]: Would you be able to separate the costs for this new or improved product from
other work that organization has done for you? [IF NO]: Why not?

Activities and Costs
[After the list is completed] Suppose that instead of asking you about cost for the individual activities
(that we’ve discussed on the list), we asked you instead for the TOTAL cost of all such activities
supporting the new or improved product, good or service. Would you be able to provide a reasonably
accurate estimate at this higher level? How would you determine such an estimate? Is it easier or
harder to estimate or provide the total cost rather than the cost of the individual activities?
We’ve talked about discrete activities such as research and development, or employee training. Let’s
talk more generally about your company’s budget process and budget categories
[IF NEEDED]:
•
•

At what level are expenses tracked - at the department level, at the project level?
Some other level? Does tracking change if the expense is over a particular threshold?
Are expenses associated with developing new or improved products, goods, and
services intermingled with other expenses, or are they kept separate?
8

•
•

Does one person have all of the data needed, or would multiple people need to be
consulted?
Who at your company is responsible for making decisions regarding your company’s
budget? How are the decisions made?

So far we have just been talking about your company’s new or improved products, goods, or services.
Your company may also have made important improvements to its business processes, such as how
your company manufactures or produces goods or services, markets them, and provides services
after sale, and whether your company has made improvements to your information and
communication systems, and to your administrative or management procedures.
Has your company made or worked on any of these types of process improvements?
[IF YES]:
•

Would you be able to provide the costs for such work, including all of the steps from
developing the improvements to carrying them out? Roughly speaking, how do these costs
compare to the costs of new or improved products or services?

Partnerships
Let’s talk briefly about the degree to which your company worked with other types of organizations,
such as other businesses, universities, or the government, when making changes to goods, services,
or business processes. For example, this might include contracting another organization to do some
of the work for you, acquiring or selling a license to share proprietary information, or partnering
together on a particular project. Have you engaged in any such cooperative relationships?
[IF SO]:
• What kind of organizations did your company work with?
• Did that organization contribute its own funds for the work, or did your company provide all
of the funding?
• What information would your company be able to provide about the other organization’s
financial role? For example, if your company provided the funding, what information do you
have about how the other organization spent those funds? What categories of expenses (if
any) are shown in your own company’s records? Are you able to discern how much money the
other organization spent on activities associated with new or improved products, goods, or
services, or improved processes, as compared with other activities? If the other organization
provided some of its own funding, do your records show how much they provided? Do your
records show any details about the funding they provided? What sort of details are available?
Reasons affecting innovation
Now I would like to ask you about some factors that can affect a company’s ability to innovate.
Please review the following question [USE ATTACHMENT 2B].
During the three years 2015 to 2017, how important were the following factors in interfering with
this business from innovating?
9

A.
B.
C.
D.
E.
F.
G.
H.

Lack of internal finance
Lack of credit or private equity
Innovation costs too high
Lack of skilled employees within the business
Lack of collaboration partners
Difficulties in obtaining government grants or subsidies
Uncertain market demand for your ideas
Too much competition in your market

•

In your own words, what do you think this question is asking you?

•

What time period were you thinking of when you answered this question? How easy or
difficult was it to think of factors in this time period only?

•

How did you decide which boxes to check?

•

I noticed you noticed you spent a lot of time thinking before you answered this question/line,
can you tell me about that?

•

To you, what does “interfering with this business from innovating” mean?

•

Which parts of this question were the most difficult to answer? Why?

•

Which parts of this question were easiest to answer? Why?

•

Do these items get at the real reasons why companies are not able to innovate? What other
reasons should be added to the list?

Thank you

10

Attachment 2A
Which of these activities has your company performed in the last three years?
Research and development activities
Engineering, design, and other creative work
New approaches to marketing and brand equity activities
Purchasing, selling, or protecting intellectual property
Training employees to carry out or market the new or improved products,
goods, or services
Software development and database activities to develop new or
improved products, goods, or services
Investing in machinery, equipment, and other tangible assets to support
your new or improved products, goods, or services
Management activities to support new or improved products, goods, or
services

11

ATTACHMENT 2B
During the three years 2015 to 2017, how important were the following factors in interfering with
this business from innovating?
Very
Somewhat
Important Important

Not at all
Important

A. Lack of internal finance
B. Lack of credit or private equity
C. Innovation costs too high
D. Lack of skilled employees within the business
E. Lack of collaboration partners
F. Difficulties in obtaining government grants or subsidies
G. Uncertain market demand for your ideas
H. Too much competition in your market

12