Attachment 3 - Innovation Module Cognitive Testing
Interview Protocol
Introduction
(To be given by NSF representative if present, or Westat interviewer if NSF is not available)
Points to cover in the introduction; these will be fleshed out into a script
Hello, we are [introduce attendees from NSF, Westat].
The National Science Foundation (NSF) is in the process of developing a survey to collect information from small and medium sized U.S. businesses. The focus of this interview will be on innovation.
The survey is currently in the design phase. Today, we would like to go over the draft questions with you and get your opinions about the questions -- what you think the questions are asking for, whether these questions are relevant for your company, and how easy or difficult it is to answer them and why.
We would like to record this interview so that we have all your comments. Would that be okay? [Have respondents read and sign consent forms.]
Characterization of Company (Icebreaker and initial exploration of nature of business)
Before we get into the main part of the interview, I’d like to start off by learning a little more about your company.
1. Very briefly, how would you describe the type of business your company conducts?
[INTERVIEWER: LISTEN FOR WHAT MIGHT BE INNOVATIVE (FOR CATEGORIES PRODUCTS, PROCESSES, MARKETING, ORGANIZATIONAL).]
What is your role in this company?
INTERVIEWER: Probes to Apply to Any Questions, as Needed (These probes are for use as appropriate to get participants to elaborate, in addition to any specific probes that will be listed for individual questions)
Comprehension
Relevance
Feasibility
Response process
For subset of questions that provide a definition or explanation
For subset of questions that do not provide a definition but might need one
|
(Section 1)
[BEGIN PROCESS OF HAVING PARTICIPANT SILENTLY READ AND ANSWER EACH QUESTION, OR STATE WHY IT IS NOT RELEVANT. FOR ANY QUESTION THAT CANNOT BE ANWERED IMMEDIATELY, ASK PARTICIPANT TO BRIEFLY STATE HOW THE QUESTION WOULD BE ANSWERED OR WHY IT IS NOT RELEVANT.]
1. During the three years 2015 to 2017, how important were each of the following strategies to your company?
|
Degree of importance |
|||
|
High |
Medium |
Low |
Not important |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
What do you think this question is asking for?
What strategies, if any, are important to your company are missing from this list?
How did you decide on your answers?
[IF NEEDED, for “High” responses]:
Can you give us an example of why you marked “High” for that strategy?
Are
these established strategies for your company or strategies that
are
in the works?”
Which parts of this question were the most difficult to answer? Why?
To you, is there a difference between asking about reducing costs and asking about low price?
2. During the three years 2015 to 2017, did your company introduce new or significantly improved:
|
Yes |
No |
A. Goods (exclude the simple resale of new goods and changes of a solely aesthetic nature). A good is usually a tangible object such as a smartphone, furniture, or packaged software, but downloadable software, music and film are also goods. |
|
|
B. Services. A service is usually intangible, such as retailing, insurance, educational courses, air travel, consulting, etc. |
|
|
C. Methods of manufacturing for producing goods or services |
|
|
D. Logistics, delivery or distribution methods for your inputs, goods or services |
|
|
E. Supporting activities for your processes, such as maintenance systems or operations for purchasing, accounting, or computing |
|
|
How do you interpret or define the phrase “new or significantly improved”?
Is there anything your company did that was new or significantly improved but that doesn’t fit well into any of these categories?
[IF YES]: What was it? Would you still try to include it in one of these categories? Which one?
Let’s look at each part of the question. How did you decide on your answer for…
Goods (part a.)?
[IF YES]: Can you give us an example of goods your company produces?
How did you interpret the phrase in parentheses?
What is your reaction to the explanation given here for Goods?
Services (part b.)?
[IF YES]: Can you give us an example of services your company offers?
What is your reaction to the explanation given here for Services?
Methods of manufacturing (part c.)?
[IF YES]: Can you give us an example of the method(s)?
Logistics, delivery or distribution methods (part d.)?
[IF YES]: Can you give us an example of the method(s)?
Supporting activities (part e.)?
[IF YES]: Can you give us an example of the activities?
What is your reaction to the examples given for activities (the “such as…” text)?
IF RESPONDENT ANSWERED ‘YES’ TO Q2A OR Q2B, GO TO QUESTION 3.
IF RESPONDENT ANSWERED ‘NO’ TO Q2A AND Q2B AND ‘YES’ TO Q2C, Q2D, OR Q2E, GO TO QUESTION 5
IF RESPONDENT ANSWERED ‘NO’ TO Q2A, Q2B, Q2C, Q2D, AND Q2E, GO TO QUESTION 10.
If you answered yes to any of the items above, then we consider that to be an innovation.
3. If you answered “Yes” to either question 2a or 2b, were any of your product innovations during the three years 2015 to 2017:
|
Yes |
No |
|
New to your market? |
Your company introduced a new or significantly improved product (good or service) onto your market before your competitors (it may have already been available in other markets) |
|
|
Only new to your company? |
Your company introduced a new or significantly improved product (good or service) that was already available from your competitors in your market |
|
|
This question uses the phrase “product innovations.” What does this term mean to you? Is there another term you use to describe the sorts of things described in 2a and 2b?
When you answered “yes” to question 2a or 2b, were you thinking of those activities as innovation?
In your own words, what is this question asking?
[IF NEEDED]:
What does product innovation mean to you?
What do you think about “product innovation” being introduced here?
How did you determine if your good or service was new to the market or new only to your company?
[IF NEEDED]:
What is an example of what your company introduced…
…That was new to your market?
…That was only new to your company?
[IF NEEDED]:
Did you consider both innovations that were new to your market and those that were only new to your company when you answered question 2?
4. Using the definitions above, please give the percent of your total sales in 2017 from:
New or significantly improved products (goods or services) introduced during the three years 2015 to 2017 that were new to your market |
|
|
|
|
|
|
|
|
% |
|
New or significantly improved products (goods or services) introduced during the three years 2015 to 2017 that were only new to your company |
|
|
|
|
|
|
|
|
% |
|
Products (goods or services) that were unchanged or only marginally modified during the three years 2015 to 2017 (include the resale of new products purchased from other companies) |
|
|
|
|
|
|
|
% |
|
|
|
|
|
|
Total sales in 2017 |
1 |
0 |
0 |
% |
How did you determine your answers for this question? (Probe on any records and/or estimating strategies used.)
Are the answers you reported here exact figures, or estimates?
How confident are you that the answers you reported here are correct?
Do you need to consult your company’s financial records to answer this question?
How easy or difficult {is it/would it be} to report for the period of time the question asks about?
[IF NEEDED]: Why would it be {easy/difficult}?
[FOR INTERVIEWER]: Did respondent mention the different time frames in the question? If not, ask, Did you notice the different time frames in the question?
5. During the three years 2015 to 2017, did your company engage in the following innovation activities? If yes, please also enter the total investment in each, rounding to the nearest thousand.
|
|
Yes |
No |
Total Investment |
A. In-house R&D
|
Research and development activities undertaken by your company to create new knowledge or solve scientific or technical problems (include software development that meets this requirement) |
|
|
$_______,000 |
|
If yes, did your company perform R&D during the three years 2015 to 2017: |
|
|
|
|
Continuously (your company had permanent R&D staff in-house) |
|
|
$_______,000 |
|
Occasionally (as needed only) |
|
|
$_______,000 |
B. External R&D |
Your company contracted-out R&D to other companies (include affiliated companies) or to public or private research organizations |
|
|
$_______,000 |
C. Acquisition of machinery, equipment, software & buildings |
New machinery, equipment software and building that were acquired for the purpose of developing the goods, services, manufacturing or logistics mentioned previously in Question 2 |
|
|
$_______,000 |
D. Acquisition of existing knowledge from other companies or organizations |
Acquisition of existing know-how, copyrighted works, patented and non-patented inventions, etc. from other companies or organizations for the development of new or significantly improved products and processes |
|
|
$_______,000 |
E. Training for innovative activities |
In-house or contracted out training for your personnel specifically for the development and/or introduction of new or significantly improved products and processes |
|
|
$_______,000 |
F. Market introduction of innovations |
In-house or contracted out activities for the market introduction of your new or significantly improved goods or services, including market research and launch advertising |
|
|
$_______,000 |
G. Brand Building |
In-house or contracted out activities to build your company’s brand identity |
|
|
$_______,000 |
H. Design |
In-house or contracted out activities to alter the shape, appearance or usability of goods or services |
|
|
$_______,000 |
I. Other |
Other in-house or contracted out activities to develop or implement new or significantly improved products or processes such as feasibility studies, testing, tooling up, industrial engineering, etc. |
|
|
$_______,000 |
[INTERVIEWER: Note if respondent reads and/or answers follow-up items in Line A.]
In your own words, what do you think this question (“… did your company engage in the following innovation activities?” is asking for?
When you saw the phrase “innovation activities” in the stem of this question, what came to mind -- for your company?
When you answered this, did you read the information in the middle column?
[IF NEEDED]:
Did these seem like definitions, or not? Why?
Which, if any, of these were difficult to understand? Which were easiest to understand?
Did these in any way...
change your understanding of the item?
change how you responded?
Were there any terms or definitions that you had problems or concerns with?
For part C:
In your own words, what does the phrase “machinery, equipment, software and buildings” mean?
For part F:
In your own words, what does the phrase “the market introduction” mean?
Would you consider training salespeople to sell new products to be training for innovative activities, or not? If so, where would you report it?
For part G:
In your own words, what does it mean to “alter the shape, appearance, or usability of goods or services?”
For part H:
[IF YES]: What activity were you thinking of here?
Are there any additional “innovation activities” that your company does, that you don’t see here?
[IF NEEDED]: When you consider the activities necessary for [CITE EXAMPLE RESPONDENT MENTIONED EARLIER] is this list exhaustive or is something missing?
For total investment questions: How did you come up with the amount for the total investment? What time period does it cover? How confident are you in the figures you reported? Would you use records to come up with these numbers, or would you estimate? (Probe on records used.) Is this something you could answer yourself, or would you need to get information from someone else? (Probe on who.)
6. During the three years 2015 to 2017, did your company have any innovation activities that did not result in a product or process innovation because the activities were:
|
Yes |
No |
Abandoned or suspended before completion |
|
|
Still ongoing at the end of the 2017 |
|
|
NOTE FOR INTERVIEWER: DO WE HAVE TO INFORM RESPONDENT THAT QUESTION2 DEFINED WHAT WE MEAN BY PRODUCT AND PROCESS INNOVATION?
When you read this question, what “product or process innovation” were you thinking of?
[IF YES TO EITHER PART]: What is an example of this for your company?
[IF NEEDED]: Why was it abandoned/suspended?
7. During the three years 2015 to 2017, did your company receive any public financial support for innovation activities from the following levels of government? Include financial support via tax credits or deductions, grants, subsidised loans, and loan guarantees. Exclude R&D and other innovation activities conducted entirely for the public sector under contract.
|
Yes |
No |
||||
|
Local or State Government |
|
|
|
||
|
Federal Government |
|
|
|
||
How did you decide on your answers to the two parts of this question?
[IF YES TO EITHER PART]: What is an example of this for your company?
How easy or difficult is it to answer this question? Why is it {easy/difficult}?
8. During the three years 2015 to 2017, did your company co-operate on any of your innovation activities with other companies or organizations? Innovation co-operation is active participation with other companies or organizations on innovation activities. Both partners do not need to commercially benefit. Exclude pure contracting out of work with no active co-operation.
Yes (GO TO QUESTION 9)
No (GO TO QUESTION 10)
What is your reaction to the explanation of innovation cooperation?
How did you interpret the sentence “Exclude pure contracting out of work with no active co-operation.?”
What does “Both partners do not need to commercially benefit” mean?
What is your definition of “partner?” Would you consider a supplier a partner if you worked with that supplier? What about a client?
How did you decide on your answer?
How easy or difficult is it to answer this question? Why is it {easy/difficult}?
9. Please indicate the type of innovation co-operation partner by location.
(Mark all that apply)
Type of co-operation partner |
United States |
All other countries |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In your own words, what is this question asking you?
How easy or difficult is it to answer this question? Why?
[FOR EACH MARKED ANSWER]: Can you give us an example of your company's innovation cooperation with [FILL WITH TYPE OF PARTNER]?
When deciding whether to mark the US box or the other countries box, were you considering where the innovation was taking place, where the partner was physically located, or something else?
Would you have considered all of these as potential partners when you answered question 8?
Are there other potential partners that are not included?
10. During the three years 2015-2017, did your company introduce new:
|
Yes |
No |
a. Business practices for organizing procedures (i.e. first time use of supply chain management, business re-engineering, knowledge management, lean production, quality management, etc.) |
|
|
b. Methods of organizing work responsibilities and decision making (i.e. first time use of a new system of employee responsibilities, team work, decentralisation, integration or de-integration of departments, education/training systems, etc.) |
|
|
c. Methods of organizing external relations with other companies or public organizations (i.e. first time use of alliances, partnerships, outsourcing or sub-contracting, etc.) |
|
|
d. Aesthetic design or packaging of a good or service (exclude changes that alter the product’s functional or user characteristics – these are product innovations) |
|
|
e. Media or techniques for product promotion (i.e. first time use of a new advertising media, a new brand image, introduction of loyalty cards, etc.) |
|
|
f. Methods for product placement or sales channels (i.e. first time use of franchising or distribution licenses, direct selling, exclusive retailing, new concepts for product presentation, etc.) |
|
|
g. Methods of pricing goods or services (i.e. first time use of variable pricing by demand, discount systems, etc.)
|
|
|
IF RESPONDENT ANSWERED ‘YES’ TO ANY PART OF Q10A – Q10G, GO TO QUESTION 12.
IF RESPONDENT ANSWERED ‘NO’ TO EVERY PART OF Q2A – Q2E AND EVERY PART OF Q10A – Q10G, GO TO QUESTION 11.
How do you interpret the word “new” (here in the stem of the question)? [IF NEEDED]: For your company, would a variation or upgrade count as “new?”
[FOR EACH YES, ASK FOR AN EXAMPLE]: You said Yes for [part __]. Can you give an example for your company?
Let’s talk about some of the phrases in the parts of this question. How do you interpret…
…“Business practices for organizing procedures” (part a.)?
…”Methods of organizing work responsibilities and decision making” (part b.)?
…”Methods of organizing external relations” (part c.)?
In your own words, what is part d. asking?
Would you normally read the full definition, or just the part in bold when completing the questionnaire?
Did the definitions affect the way you thought about each item, or not?
NOTE TO INTERVIEWERS, THERE IS A QUESTION 11, BUT IT WILL NOT BE ASKED DURING THE COGNITIVE INTERVIEW. DO NOT RENUMBER THE DOCUMENT.
12. During the three years from 2015 to 2017, how important were the following factors in hampering or preventing your company from innovating?
|
|
Degree of importance |
|||
|
|
High |
Medium |
Low |
Not important |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
To you, what does “hampering or preventing your company from innovating” mean?
Which parts of this question were the most difficult to answer? Why?
Which parts of this question were easiest to answer? Why?
Which parts of this question, if any, were not relevant for your company? Why?
Are there other reasons, that are not included, that affected your company’s ability to innovate?
Do these items get at the real reason why companies do not innovate?
13. Please assess the effect of the following types of legislation or regulations on your company’s innovation activities during the three years 2015 to 2017.
|
|
|
|
|
|
|
|
Legislation or regulation |
Stimulated innovation |
Created no major problems |
Created uncertainty |
Generated an excessive burden |
|
Not relevant |
|
Product safety / consumer protection |
|
|
|
|
|
|
|
Operational and worker safety |
|
|
|
|
|
|
|
Environmental |
|
|
|
|
|
|
|
Intellectual property |
|
|
|
|
|
|
|
Tax |
|
|
|
|
|
|
|
Employment or social affairs |
|
|
|
|
|
|
|
Other, Specify______ |
|
|
|
|
|
|
|
Is it more appropriate to offer one response per row or would it be better to mark all that apply?
Which parts of this question were difficult to answer? Why?
Which parts of this question were easiest to answer? Why?
For any parts of this question, did you want to mark more than one answer in the row? Why? [FOR EACH ROW THAT IS RELEVANT]:
Does this look like an appropriate set of choices?
14. During the three years 2015 to 2017, did your company do any of the following?
|
Yes |
No |
a. Apply for a utility patent (patents for inventions) |
|
|
b. Apply for a design patent (patents for appearance) |
|
|
c. Register a trademark |
|
|
d. Claim copyrights |
|
|
e. Use trade secrets |
|
|
f. Use nondisclosure agreements |
|
|
[IF NEEDED]:
Which parts of this question were difficult to answer? Why?
Which parts of this question were easiest to answer? Why?
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Audrey Kindlon |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |