Download:
pdf |
pdfAttachment E
Business Innovation Expert Panel for Task Order 19
NCSES V Task Order 19
Workshop Summary Report
Reference: NCSES Contract No. NSFDACS16C1234
FINAL REPORT
January 7, 2019
Prepared by:
SRI International
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Table of Contents
Table of Contents ................................................................................................................................................. 2
Executive Summary ............................................................................................................................................. 4
Workshop Summary ............................................................................................................................................ 7
Continuing Examination of Innovation in the U.S. Business Sector by NCSES .......................................................... 7
Discussion - Continuing Examination of Innovation in the U.S. Business Sector by NCSES ................................. 9
Session 1: Business Innovation Activities................................................................................................................ 10
Discussion - Business Innovation Activities........................................................................................................ 11
Session 2: Business Capabilities for Innovation ...................................................................................................... 13
Discussion - Business Capabilities for Innovation .............................................................................................. 14
Session 3: Business Innovation and Knowledge Flows ........................................................................................... 15
Discussion - Business Innovation and Knowledge Flows ................................................................................... 16
Session 4: External Factors in Influencing Innovation in Firms ............................................................................... 17
Discussion - External Factors in Influencing Innovation in Firms ....................................................................... 18
Session 5: Objectives and Outcomes of Business Innovation ................................................................................. 19
Discussion - Objectives and Outcomes of Business Innovation ......................................................................... 20
Detailed Notes ................................................................................................................................................... 24
Welcome, Introductions, and Discussion of Meeting Goals ................................................................................... 24
Continuing Examination of Innovation in the U.S. Business Sector by NCSES ........................................................ 25
Discussion - Continuing Examination of Innovation in the U.S. Business Sector by NCSES ............................... 28
Session 1: Business Innovation Activities................................................................................................................ 29
Session 2: Business Capabilities for Innovation ...................................................................................................... 33
Discussion - Business Capabilities for Innovation .............................................................................................. 35
Session 3: Business Innovation and Knowledge Flows ........................................................................................... 37
Discussion - Business Innovation and Knowledge Flows ................................................................................... 38
Session 4: External Factors in Influencing Innovation in Firms ............................................................................... 40
Discussion - External Factors in Influencing Innovation in Firms ....................................................................... 41
Session 5: Objectives and Outcomes of Business Innovation ................................................................................. 44
Discussion - Objectives and Outcomes of Business Innovation ......................................................................... 45
Discussion Summary, Recommendations, and Conclusion ..................................................................................... 47
Appendix: Presentation Slides............................................................................................................................ 50
SRI INTERNATIONAL
2
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Note
Please note that this document was prepared by SRI International from the workshop on business innovation,
convened by SRI. Any opinions, findings, and conclusions expressed in this material are those of the respective
authors and do not necessarily reflect the views of the National Science Foundation or its staff. Because this report
directly quotes individuals, it is not intended to be shared publicly.
SRI INTERNATIONAL
3
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Executive Summary
On behalf of the National Center for Science and Engineering Statistics (NCSES) within the National
Science Foundation (NSF), SRI International (SRI) organized an Expert Panel on Business Innovation with
stakeholders from academia, government, and industry. The NCSES Business Innovation Workshop was
held on October 16, 2018, at SRI’s offices in Arlington, Virginia.
The workshop focused on the following objectives:
•
•
•
Prioritize the recommendations of the forthcoming new Oslo Manual edition regarding revised
and/or new metrics and data collections on business innovation.
Discuss how the recommendations from the forthcoming Oslo Manual should be incorporated
into a module on innovation to be included in the Annual Business Survey (ABS).
Identify the high-priority areas for future collections considering the needs for business
innovation data and the interests of U.S. policy makers and academic researchers.
ABS is conducted in partnership by NSF/NCSES and the Census Bureau and includes a module on
innovation. The module is currently based on the Community Innovation Survey (CIS). The Oslo Manual
sets forth guidelines for collecting and interpreting innovation data. A revised version was published in
October 2018. In the revised manual, numerous topics are presented for inclusion in future innovation
surveys. However, there are too many topics to include in the ABS innovation module. During this
workshop, participants discussed and recommended priorities on the topics for the future ABS. The
outcome of the workshop will be used to assist in the development of the innovation module for the
2018 ABS.
In addition to the Oslo Manual, workshop panelists were invited to review and discuss several
questionnaires that currently address innovation: ABS 2017 (innovation questions are in Section C,
pages 16–22); the Microbusiness Research & Development (R&D) and Innovation Survey (BRDI-M) 2016
(innovation questions are in Section D, pages 3–4); the Business Research and Development and
Innovation Survey (BRDIS) 2016 (innovation questions are in Section 1, pages 8–9); and CIS 2018
(innovation only).
The panelists included the following leading experts, who presented on various topics:
•
•
Jan Youtie, Georgia Tech University, Workshop Chair, presented Discussion 1: Business
Innovation Activities. Business Innovation Activities are activities that firms may undertake in
pursuit of innovation, including R&D; engineering, design, and other creative work; marketing
and brand equity activities; intellectual property (IP); employee training; software development
and databases; investment in buildings, machinery, equipment and other tangible asset, and
innovation management activities.
Wes Cohen, Duke University, presented Discussion 2: Business Capabilities for Innovation.
Business capabilities for innovation include the knowledge, competencies, and resources that a
firm accumulates over time and draws upon in the pursuit of its innovation objectives. Collecting
data on business capabilities is important to understand the effect of innovation on firm
performance and why some firms innovate, and others do not.
SRI INTERNATIONAL
4
Attachment E
TO 19 | INNOVATION EXPERT PANEL
•
•
•
WORKSHOP SUMMARY
Steve Landefeld, Bureau of Economic Analysis (Retired), presented Discussion 3: Business
Innovation and Knowledge Flows. Knowledge flows include how knowledge is accessed, how it is
deployed, and how it is exchanged between firms and other actors engaged in the innovation
system.
Stephen Ezell, Information Technology and Innovation Foundation, presented Discussion 4:
External Factors in Influencing Innovation in Firms. External factors can influence a firm’s
incentive to innovate, the types of innovation activities that it undertakes, and its innovation
capabilities and outcomes. External factors may also be the object of a business strategy, public
policy, or concerted social action by public interest groups.
E.J. Reedy, University of Chicago, presented Discussion 5: Objective and Outcomes of Business
Innovation. The planning and development stage for an innovation includes the identification of
a set of one or more objectives that the innovation is expected to achieve. The objectives can
concern the characteristics of the innovation itself, such as its specifications, and its market and
economic objectives. The outcomes of an innovation are similar to the objectives but consist of
the innovation’s realized effects. These can include unexpected effects that were not identified
among the firm’s initial objectives.
Other participants present were:
•
•
•
•
•
•
•
•
Mohan Sawhney, Director, Center for Research in Technology & Innovation, Kellogg School of
Management, Northwestern University
Nico Thomas, Performance Analyst, National Institute of Standards and Technology
Steve Campbell, Economist, Manufacturing Research and Program Evaluation, National Institute
of Standards and Technology
Adam Jaffe, Professor, Economics, Brandeis University
Mark Crowell, Executive Entrepreneur in Residence, Eshelman Institute for Innovation, UNC
Eshelman School of Pharmacy
Stuart Graham, Associate Professor, Scheller College of Business, Georgia Institute of
Technology
Ann Xu, Senior Technical Advisor for Impact and Assessment, ARPA-E
Kenneth Poole, Executive Director of C2ER, The Council for Community and Economic Research
The meeting was also attended by Emilda Rivers, Division Director of NCSES; Samson Adeshiyan, Acting
Deputy Division Director of NCSES; John Jankowski, R&D Statistics Program Director, NCSES; Mark
Boroush, Senior Analyst, NCSES; and Audrey Kindlon, Survey Statistician, NCSES. Additionally, several
other NCSES staff and Patrice Norman from the U.S. Census Bureau were present as observers.
The group recommended including most of the Oslo Manual topics reviewed during the discussion in the
business survey module of the NCSES survey. The only generally negative responses were towards the
questions about the drivers of and barriers to innovation.
SRI INTERNATIONAL
5
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Several experts stressed that co-development and co-creation by firms working together need to be
tracked, and the questions reviewed during the workshop may not capture adequately either legal or
less formal relationships between firms.
Other points emphasized during the meeting included the following:
•
•
•
Some information that the Oslo Manual suggests asking for may be available as administrative
data; however, although linkages to other data sets are possible, they are still difficult and not
an immediate option.
Asking questions about regulations in the United States can be tricky as “regulation” has
become a politically loaded term, and any questions of this type would have to be tested.
Questions about objectives and outcomes would also need extensive testing.
The group did not agree on whether data collection should be entirely subject-based (as recommended
by the Oslo Manual), or include an object-based component. The subject-based approach covers the full
range of innovation and innovation activities of the firm, whereas the object-based approach collects
data on a single most important innovation. Other unresolved disagreements included the ideal size of
business to survey, and how to align the Oslo Manual’s three-year windows with product life cycles.
SRI INTERNATIONAL
6
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Workshop Summary
The NCSES Business Innovation Expert Panel convened various stakeholders and end users from
academia, government, and industry in a workshop to identify and prioritize Oslo Manual
recommendations to be incorporated into the Annual Business Survey (ABS). Through extensive
conversation, SRI and NCSES developed a list of final panelists in academia, government, and industry.
Although not all could attend, SRI secured the participation of outside experts and scheduled the panel
meeting to take place at SRI International’s office in Arlington, Virginia, on October 16, 2018. Prior to the
meeting, several panelists were invited to present on the upcoming Oslo Manual chapters and review
current NCSES survey questionnaires (ABS, BRDI-M, BRDIS, and CIS). The meeting was attended by
thirteen panelists representing academia, industry, and U.S. government agencies; Emilda Rivers,
Division Director of NCSES; Samson Adeshiyan, Acting Deputy Division Director of NCSES; John
Jankowski, R&D Statistics Program Director, NCSES; Mark Boroush, Senior Analyst, NCSES; and Audrey
Kindlon, Survey Statistician, NCSES; and eight observers from the U.S. Census Bureau and NCSES.
Jan Youtie, Georgia Tech University, served as the workshop chair. After welcomes by NCSES leadership,
she presented an overview of the background and goals of the meeting. The meeting had been
organized as part of the effort by NCSES to improve and revise the upcoming ABS. ABS is conducted in
partnership by NCSES and the Census Bureau and includes a module on innovation. The workshop was
organized to leverage the rich experience of participants’ study of invention and innovation activities.
The goal for the workshop was to discuss and recommend priorities on the topics for the innovation
module of the future ABS. Question development was not a focus of discussion during this workshop.
Continuing Examination of Innovation in the U.S. Business Sector by NCSES
After introducing the Workshop Chair, Jan Youtie, John Jankowski, Program Director of NSF’s Research
and Development Statistics Program, began his presentation, NCSES’s Continuing Examination of
Innovation in the U.S. Business Sector. He provided an overview of the organizational background, NCSES
innovation surveys, the Oslo Manual, and the purpose of the workshop.
NCSES is a division within the Directorate for Social, Behavioral, & Economic Sciences in the National
Science Foundation, created by an act of Congress in 1950 to collect information on the science and
engineering enterprise, including competitiveness. It is one of the U.S. Federal Government’s 13
principal statistical agencies.
NCSES conducts three surveys of businesses in partnership with the Census Bureau: BRDIS, the
Microbusiness R&D and Innovation Survey (Micro-BRDIS), and ABS. As of 2017, BRDIS no longer collects
innovation data and has been renamed the Business R&D Survey (BRDS). Many of the innovation
questions are derived from the Oslo Manual. Historically, the manual defined innovation in terms of
various types of innovation achieved by the enterprise, specifically, Product Innovation, new or
improved goods or services; Process Innovation, new or improved production processes, distribution
methods, or support activities; Organizational Innovation, new business practices, new methods of
organizing work and decision making, or new methods of organizing external relations; and Marketing
Innovation, changes to the design or presentation packaging of goods or services, or new methods of
pricing goods or services. However, the objective of the workshop was not to look at what has been
SRI INTERNATIONAL
7
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
done, but to consider what data to collect in the future, based on guidance from the revised Oslo
Manual.
Jankowski continued with an overview of BRDIS, for which data have been collected annually from 2009
to 2016. This survey had about an 80% survey response rate in the last cycle. Questions ask about
product and process innovation as well as novelty (new to market or new to firm), but the survey does
not define innovation.
In 2016, NCSES distributed the Micro-BRDIS to 200,000 businesses that had up to nine employees. Some
Oslo Manual elements were incorporated into the questions. Cognitive testing indicated that different
wording worked better for small business respondents and, therefore, the original BRDIS questions were
revised for the Micro-BRDIS. Micro-BRDIS asked small businesses about product innovation, process
innovation, and organizational and marketing innovation. NCSES partnered with the Census Bureau to
include innovation modules on ABS. ABS combines the Survey of Business Owners, the Annual Survey of
Entrepreneurs, Micro-BRDIS, and a standalone business innovation module with approximately 15
innovation questions based on the 2005 Oslo Manual and the CIS survey. The 2017 sample size was
850,000.
The ABS 2018 will feature fewer innovation questions when compared to ABS 2017. However, it is
anticipated that the ABS 2019 will be closer to the ABS 2017 innovation module in terms of the number
of questions asked.
Three years ago, the Organisation for Economic Co-operation and Development (OECD) and Eurostat
began updating the Oslo Manual. The revisions are completed, after extensive review, and the new
edition was released on October 22, 2018. The revision was guided by a steering group from many
countries, including Japan, Germany, the United Kingdom, representatives from Eurostat, and the
United States. Part of the effort was to align R&D terminology with the System of National Accounts
(SNA) and other statistical manuals, such as the Frascati Manual. The review committee also looked at
the role of digitization and globalization. The new Oslo Manual encourages a subject-based approach,
but also recognizes the object-based approach. The subject-based approach covers the full range of
innovation and innovation activities of the firm, whereas the object-based approach collects data on a
single most important innovation. Each chapter ends with key indicator questions as well as
supplementary indicators.
Jankowski noted that his presentation covers material from the Oslo Manual’s Chapter 3, Concepts and
definitions for measuring business innovation, and gave an overview of the chapters and topics that the
remaining presenters would discuss:
•
•
•
•
•
Chapter 4. Measuring business innovation activities (Jan Youtie, Workshop Chair)
Chapter 5. Measuring business capabilities for innovation (Wes Cohen)
Chapter 6. Business innovation and knowledge flows (Steven Landefeld)
Chapter 7. Measuring external factors influencing firm innovation (Stephen Ezell)
Chapter 8. Objectives and outcomes of business innovation (E.J. Reedy)
SRI INTERNATIONAL
8
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Jankowski explained that the workshop would review the topic areas, discuss how to interpret them in
terms of U.S. business innovation, and then consider how to prioritize them in an innovation module for
the 2019 ABS. The purpose of this workshop is to identify the most important topics on the next U.S.
innovation survey, but not to write the questions themselves, which will be another step in the process.
Next, Jankowski presented the Manual’s definition of innovation:
•
An innovation is a new or improved product or process (or combination thereof) that differs
significantly from the unit’s previous products or processes and that has been made available to
potential users (product) or brought into use by the unit (process).
Jankowski noted that the product or process must be brought into use to count as innovation (it cannot
be abandoned), though it may succeed or fail. Also, in the digital world, an innovation may contain
intermixed products and processes.
A business innovation is a new or improved product or business process (or combination of the two)
that the firm has brought into use and that differs significantly from the firm’s previous products or
business processes and that has been introduced on the market or brought into use by the firm. A
business innovation may be either a product innovation or a business process innovation.
A product innovation is a new or improved good (a tangible object) or service (intangible activity).
Digital knowledge-capturing products, used for data storage, security, and communication, may behave
as either a goods or services.
A business process innovation is a new or improved business process and may concern any of six types
of functions: (1) production of goods and services, (2) distribution and logistics, (3) marketing and sales,
(4) information and communication systems, (5) administration and management activities, and (6)
product and business process development.
Jankowski added that for countries that wish to maintain their time series from earlier surveys, these
categories can be mapped to the previous types of innovation (i.e., product, process, organizational, and
marketing innovation).
Discussion - Continuing Examination of Innovation in the U.S. Business Sector by NCSES
Youtie began the discussion by encouraging participants to offer their commentary and feedback on the
presentations and methods to measure innovation. The goal of NCSES is to balance successfully
collecting up-to-date innovation data with a reasonable respondent burden.
The participants raised the following points:
•
Must an “innovation” be new to the market to count as innovation? Some studies focus on
innovation imitators. But the important issue is whether a new idea is really important or
impactful. For example, when it appeared, the first 3-D printer was more important than that
season’s new toothpaste formulas. There should be questions that speak to the importance of a
new product. This suggests that NCSES should ask these types of object-oriented questions in
addition to the firm-level questions.
SRI INTERNATIONAL
9
Attachment E
TO 19 | INNOVATION EXPERT PANEL
•
•
•
•
WORKSHOP SUMMARY
NCSES should add questions that measure impact (e.g., to address social welfare). These types
of variables could be a foundation for subsequent innovation research and address several of
the dimensions that change our understanding of social welfare and its impact. Jankowski
commented that those variables will be addressed later in the Chapter 7 presentation.
Could NCSES increase data quality by fielding targeted sampling modules? Jankowski responded
that the ABS 2019 sample size will be around 300,000 for all businesses of one or more
employees and cover all industries, as compared to this year’s ABS with a sample size of around
850,000. Some observe that the nature of value-creating innovations has taken on significantly
new characteristics. Now platforms are important; they create the ecosystem constructs upon
which innovations are built. How will surveys designed to capture product, process, and
business model innovation measure this? Jankowski responded that these variables are covered
in the new Oslo Manual, which addresses product, process, the combination thereof, and new
digital components. One takeaway could be to ensure that the survey captures something on
value creation, captures something about platform-based innovation and how it creates value.
Google, Uber, and other massively successful innovative companies are extremely rare. Can a
broad-based survey capture these unusual cases of business model innovation? The high-impact
factors are very rare, and that in itself is a policy issue.
Those with a macroeconomics perspective would like to see more data on prices and
transaction rather than yes/no responses. Valuations can be used to look at the relative
importance of different innovations. What economic growth flows from the innovations?
Researchers are scrambling for answers about U.S. and world economic growth and how
innovation and technology are affecting it. What about the location of growth? What about
offshore production? What do we expect to see from this new tax act? Two-way flow data are
necessary for this research.
Session 1: Business Innovation Activities
Youtie defined business innovation activities as developmental, financial, and commercial activities that
are intended to result in an innovation. They may be performed internally or externally. They may
succeed or fail in their original purpose or may succeed in the sense that they do not meet their original
goals but produce some other knowledge.
Youtie then discussed the eight types of business innovation activities:
•
•
•
Research and experimental development activities – The five criteria of R&D specified by the
Frascati Manual are novel, creative, uncertain outcome, systematic, and
transferable/reproducible. Not all innovations meet all five of the R&D criteria.
Engineering, design, and other creative work activities – Engineering and design activities that
do not meet the five R&D criteria include testing, set-up, tooling, production, and quality
testing. However, engineering may not always be innovative. Service firms may use design for
appearance or ease of use. Reverse engineering is another example of non-innovative activity.
Marketing and brand equity activities – Innovative marketing and branding activities include
pricing, product placement, and advertising (often linked to a new or improved product).
SRI INTERNATIONAL
10
Attachment E
TO 19 | INNOVATION EXPERT PANEL
•
•
•
•
•
WORKSHOP SUMMARY
IP-related activities – IP activities include patents, utility patents, industrial designs, trademarks,
copyright, integrated circuit designs, plants, and trade secrets for new products and processes.
Employee training activities – Employee training would not be innovative if it is for existing
products or general skills.
Software development and database activities – Innovative software development and database
innovation activities do not include maintaining the IT system, supporting existing systems, or
conducting routine functions.
Activities related to the acquisition or lease of tangible assets – These include buildings,
instruments, equipment, machinery, and computer hardware
Innovation management activities – These include funding, organization, external
collaborations, and performance (for firms that innovate only).
The chapter’s key questions for data collection are:
•
•
•
•
Were the eight activities conducted, regardless of the reason, with innovation as the aim?
Was each activity conducted in house or procured from external organizations?
Of the total expenditures for each of the eight activities, was innovation management
recommended?
What were the funding sources for innovation?
Supplementary questions for data collection requested additional information about specific activities,
such as whether R&D is continuous or occasional, innovation expenditures by funding source, follow-up
activities, and planned innovation activities and expenditures. Questions about the number of
innovation projects by activity may include innovation undertaken, completed, or ceased before
completion during the period as well as projects that are still in progress at the end of the period.
Follow-up activities are those that occur after implementation but within the period; they include
marketing, training of employees or customers in the use of the product or business process
innovations; and after-sales services, such as installation, update and repair services, guarantee and
return support, and general information. Questions about planned activities may ask whether the firm
plans to conduct any innovation activities in the one or two years after the reference year and whether
total innovation expenditures compared to the reference year are expected to increase, decrease, or
stay the same.
Discussion - Business Innovation Activities
The discussion centered around the following points:
•
•
•
What was the thought process behind the prioritization of the list? Jankowski responded that it
had been a collaborative consensus effort.
Some participants did not like the use of the terms procure or acquire. Jankowski responded
that the term procure was intentional on the part of the committee, but NCSES would take the
comment into consideration.
What is the unit of analysis? Jankowski explained that the Oslo Manual says to collect data from
the lowest level legal entity with decision-making authority, that is, the business unit rather the
firm as a whole. Some participants still favored using the innovation project as the unit of
SRI INTERNATIONAL
11
Attachment E
TO 19 | INNOVATION EXPERT PANEL
•
•
•
•
•
•
WORKSHOP SUMMARY
analysis. Jankowski thanked his colleagues for bringing the object-based approach to the
community’s attention; he added that had it not been for the earlier research of some of the
present participants, the object-based approach probably would not have been included in the
manual at all. He stated that, nevertheless, the focus should be on the enterprise. Because this
is the unit of analysis for all international statistics, NCSES must concentrate on the subjectbased approach. One participant noted that NCSES could have both.
Location is an important issue that may be overlooked. If a firm conducts training or deploys
new technology, it could be in a different location from where the design is taking place. Data
collection at the national level will not capture how multi-national organizations function.
Some participants felt that co-development and co-creation were being left out. The world
today is built upon partnerships, and a firm’s boundaries are no longer rigid. Jankowski assured
the group that the Oslo Manual addresses co-creation, and NCSES will consider this point when
developing the questions. One participant suggested a three-part question: Did you make it? Did
you buy it? Did you partner with another firm?
Many firms are engaged in work that meets the definition of R&D (e.g., it is novel and creative),
but they would not label or identify their work as R&D. Perhaps more generic phrases, such as
new activities would capture novel activities more successfully. Youtie responded that the Oslo
Manual suggests more detailed questions beyond these, including should intramural and
extramural activities be separated to try to get to the economics flows? However, she
acknowledged that some firms do not use an accounting approach to track R&D. In these cases,
the Oslo Manual suggests a full-time equivalent (FTE) approach.
The group did not come to a consensus about the ideal size(s) of business that NCSES should
cover in a national innovation survey. Youtie’s experience in Georgia has shown that companies
with fewer than 10 employees generally do not do manufacturing primarily; they focus on
distribution or sales. Companies with 10 or more employees should be the threshold. On the
other hand, Xu noted that many performers with whom she has worked have only two or three
employees. Jankowski commented that NCSES might need to develop different standards for
industries in different sectors.
Several participants emphasized that small firms are involved in innovation, but they may be
difficult to identify. How do we find small firms that conduct R&D? On the other hand, it cannot
be assumed that R&D itself is a descriptor of innovation. People who work for especially small
firms may have multiple roles, taking care of both traditional business activities and innovation
activities. Another issue with small firms is that those who have not yet introduced an
innovation into the market may not have any revenue, so tracking activity by accounts activity
may not work.
Some participants wished that the NCSES survey would collect data at the state and local levels,
at least for a few industries. Jankowski commented that NCSES will be sensitive to those
concerns, though the primary focus of the workshop was national level data.
SRI INTERNATIONAL
12
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Session 2: Business Capabilities for Innovation
Wes Cohen began the next discussion with a presentation on business capabilities for innovation, noting
that he suspected, based on earlier clarifications by Jankowski, that some of the points he had planned
to raise might be addressed by other chapters.
Cohen then discussed the types of business capabilities for innovation given by Chapter 5. As it pursues
its objectives over time a firm accumulates knowledge, competencies, and resources. Other resources
include the physical plant, human capital and experience, sources of finance, and access to outside
resources. These would include cooperative venture, licensing deals, and other relationships, which
NCSES should try to capture. NCSES should consider asking about cooperative ventures, licensing deals,
and other relationships. The business strategy should include cost competitiveness or focus on quality
features (e.g., functionality, durability, flexibility of use). Firms should also be surveyed on their
organizational, managerial, and innovation management capabilities. NCSES should also consider
workforce qualifications, occupational structure, and competences. Regarding human resource
management, an increasingly important source of information for innovation is the ability to use
technological tools and digital resources. Lastly, NCSES should consider the innovative capabilities
related to digital technologies and data analytics (i.e. technical expertise and design capabilities).
Cohen recommended breaking out aggregate statistics from this category by industry. Then, he
reviewed the key questions for data collection: What are the number of full-time employed persons by
business unit (not the firm as a whole)? What are the total sales, business unit sales—the key correlate
of R&D? What is the firm age by the year the firm began business activities (e.g., date of entry by
market)? What is the firm ownership status (e.g., stand alone, part of a national group, part of a
multinational group)? What is the geographical distribution of sales (e.g., local, national, international
markets)? What are the export shares of sales by business unit? What is the importance of cost versus
quality for the firm’s competitive strategy (the suggested measures are: percentage of process versus
product R&D, and, for the latter, percentage that is dedicated to developing new products versus
product improvement)? What is the share of employed persons with a tertiary education? What is the
business level of design capability and cash flow?
Having reviewed these, Cohen asked why the Oslo Manual needed supplementary questions and how
had questions been sorted into the key and supplementary groups: by feasibility, or by vagueness?
Jankowski responded that questions in the key groups were generally agreed on by the whole
committee, or at least not strongly opposed by anyone. Supplementary questions were generally those
that one or a few members thought were important (but none opposed).
Pursuing this line, Cohen asked, what approach could be used to determine whether these data are
important or not? Can a method be developed to determine this, perhaps through working with others
at NCSES or organizations other than NSF? Jankowski responded that NCSES would consider that in
refinements of future innovation surveys. However, this workshop’s focus is on guiding next year’s
survey, and NCSES was relying on the stakeholders in the room because they had much more experience
with these kinds of topics and issues than NCSES’ colleagues elsewhere.
SRI INTERNATIONAL
13
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Cohen gave an overview of supplementary questions for data collection: Is the firm family-owned? How
many product lines does the firm manage? Who within the firm is responsible for innovation? How is
innovation managed and how does the firm handle the internal exchange of knowledge? What emerging
technologies does the firm have expertise in? What are the firm’s digital competences?
Cohen wondered, who is the audience for these data, and would these topics address the needs of
policymakers, business management, researchers, and others? He wondered if the Oslo Manual makes a
clear statement about this. Also, he recommended that a vehicle be included within these surveys that
will allow analysts to tease out cause and effect; this would allow researchers to answer the question,
why should we care about these data? To do that, he suggested distinguishing between firms that
innovate, firms that imitate, and firms that do neither. Rather than using Likert scale questions to assess
that, he prefers to ask about behaviors; for instance, did a university suggest new products to a firm? In
response, Jankowski noted that the scales help to provide a range of importance.
Discussion - Business Capabilities for Innovation
The discussion brought up the following points:
•
•
•
•
•
•
Some recent work by Cohen’s colleagues shows that an important driver for innovation
performance at the systemic level is the relationships across nodes, not just the nodes
themselves (e.g., individual firms, universities, and others). When asked from where the most
important new products came, 49% of the respondents said that they were acquired from an
external source, and many said they were acquired through co-development with other entities.
It is reasonable to focus on the firm, but it is necessary to think about relationships and
capabilities as well as how to find managers of co-operative ventures, licensing deals, etc.
Perhaps a proxy measure for capabilities would be a measure of resources in the ecosystem.
Relationship capital will not appear on the balance sheet. Where did you get the idea? Who built
the prototype? What are the knowledge flows?
At what level of the firm should these questions be asked? At what level do they reside? Who
can respond knowledgably? Upper management might be unaware of the innovative activity,
and the department that undertakes it might have a different perspective from management. It
may also vary across firms. NCSES needs to think about how to get this information.
One participant’s research shows that different departments and levels within firms give
different responses about innovation capabilities. Management and marketing are more
optimistic, whereas sales and engineering are less optimistic. People in one function do not
understand other functions, and few understand the picture as a whole. Capabilities may need
to be distinguished across managerial, technical, and other divisions. Marketing, sales, and
manufacturing are essential for data collection.
Cohen stated that when measuring performance, he prefers to use financial performance data.
Regarding products, he has in the past asked about incremental innovation versus new products
and he has found that data on behaviors—even though self-reported—can be very useful.
Questions about tertiary education could yield important data; however, it is worth
remembering that some firms hire those with a tertiary education for the front office but not for
the back office. Also, some firms make the decision to hire people without advanced skills, so it
SRI INTERNATIONAL
14
Attachment E
TO 19 | INNOVATION EXPERT PANEL
•
•
•
•
•
•
•
WORKSHOP SUMMARY
becomes a matter of business strategy. In that case, it is necessary to know who in the firm
understands the strategy.
If tertiary education data are collected, it would be useful to separate out those with tertiary
education in a technical field (as opposed to, say, individuals with master’s degrees in business
administration or social scientists).
Digital capabilities and data analytics are becoming increasingly important, but researchers must
be careful not to categorize design activities, such as user functionality and aesthetics, as R&D.
However, the Oslo Manual recognizes that digitization is taking place across all sectors and aims
to capture these types of innovation activity.
Some of the key questions address things that researchers can already find out about firms.
Perhaps many of these data can be imported from administrative record sources?
A clear distinction needs to be made between business capabilities and individual employee
capabilities. If the focus is on employees, the business (or firm) may not be the right unit of
measurement.
Looking at new products and financial performance gives a certain measure of success, but
technology failure can also teach a great deal. Also, what about the lives affected? Did the firm
learn anything from the experience? What about the firm’s relationships with its customers and
other firms?
What are the incentives and what are the performance evaluation criteria for innovative project
management? These trigger incentives and may vary across type of innovation, even within the
same firm. These might yield a clue as to how firms prioritize projects. Could they be market
performance or market share?
Questions that capture institutional value at advancing stages of investment (as is often
apparent in the pharmaceutical field) might help to quantify values around innovation.
Jankowski commented that knowing how businesses are organized at the functional level is critical for
any survey that reports quantitative data on R&D. But what is the cut-off size? Businesses with nine or
fewer people typically do not have super-specialized roles, so many staff members may have a good
idea of what is going on, whereas large companies have people with specialized knowledge. Innovation
statistics from BRDIS show that much business innovation activity did not have R&D.
Youtie asked Cohen whether he thought the business innovation capability questions should be included
in the survey. Cohen responded that some should be included, but not the whole list from the Manual.
He cautioned that it would probably take a year or two of research before the community could agree
on two to four reasonable indicators.
Session 3: Business Innovation and Knowledge Flows
Steve Landefeld presented material from Chapter 6 of the Oslo Manual on business innovation and
knowledge flows. He commented that IP is key to addressing these issues. Market transactions
deliberately transfer from one knowledge area to another. Customers may suggest improvements that
are incorporated into products or processes, but we will never see these in transaction records.
Offshoring of resources and jobs may be technology transfer or tax avoidance. Trade regulations and
tariffs also have an important effect on the flow of knowledge. Also, some flows are deliberate, whereas
SRI INTERNATIONAL
15
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
others are accidental. Landefeld also encouraged NCSES to strongly consider some of the more detailed
questions presented in the chapter, such as geography.
He discussed the key questions for data collection.
•
•
•
•
What inbound knowledge flows contribute to innovation? The Oslo Manual provides mainly a
qualitative yes/no response; however, NCSES has great experience in economic flows and can
use BRDS and other surveys, as well as the Oslo Manual, as guides for this question.
Where are the firm’s collaborative partners in innovation located? If firms provide access to
economic data, this would be very interesting in the Global Value Chain (GVC) context.
What are the sources of ideas and knowledge for innovation? (Landefeld referred the group to
Table 6.2, which includes internal and external sources of innovation.)
What are the barriers to knowledge interactions? (Landefeld noted that Table 6.9 provides an
interesting classification of barriers to transactions, many of which are amenable to increased or
decreased policy action.)
Supplementary questions for data collection include: What are the locations of the sources of inbound
knowledge flows? What are the firms’ outbound knowledge flows? What channels for knowledge
sharing exist between the firm and Higher Education Institutions and Public Research Institutions?
Landefeld noted that the locations of inbound knowledge, though very important for GVCs, is very
difficult to collect on surveys, and may only be possible with world input-output tables. Collecting data
on outbound knowledge flows is also difficult, but both inbound and outbound flows are needed;
perhaps NCSES can approach these questions by build on existing economic data and surveys. Use of
intellectual property rights for knowledge flows is also very difficult to collect due to tax and transfer
pricing for IP, but it is extremely important.
Jankowski noted that NCSES wanted to distinguish internal sources versus external sources but that the
key recommendations are not always the easiest to collect nor even necessarily the sources of the most
important information.
Discussion - Business Innovation and Knowledge Flows
The discussion touched on the following issues:
•
•
•
One participant felt that the term accidental made the knowledge flow sound like a leak and
preferred the term unintentional. Landefeld acknowledged the point but noted both that
sometimes there are real leaks and that the language comes from the Oslo Manual.
NCSES could add trade secrets as an object that captures knowledge. Value chains are critically
important today, but it is difficult to collect information on them; the only way researchers have
been able to get at those data is through the world input-output model.
Questions on barriers to innovation might be difficult for respondents to answer and increase
item nonresponse. Youtie added that her work asking about barriers had rarely led to identifying
new barriers; the same answers, usually time and money, appeared frequently. Jankowski
mentioned the concerns listed in the manual, such as anti-trust policy enforcement, high
coordination costs, cost of dispute settlements, and financial resources. He agreed that although
time and resources are common barriers, these answers do not provide much insight.
SRI INTERNATIONAL
16
Attachment E
TO 19 | INNOVATION EXPERT PANEL
•
•
•
•
WORKSHOP SUMMARY
NCSES can use federal labs and clusters as test cases to see if they show innovation signals.
These observations might help people develop better questions. This information would be
useful for the National Institute of Standards and Technology, which focuses on industry
questions about challenges; however, the issue with questions like these is that it is difficult to
find one approach that is broad enough to address every industry.
A significant challenge is tracking follow-up funding. It is difficult to know the full chain upstream
and downstream but getting that information would dramatically improve the results.
Outbound knowledge flows consider both sides (i.e., flows and asset transfers).
Co-creation is more than a flow and more than transactional; it is persistent creation, not just an
accidental leak; co-development should also be addressed.
IP questions are very important; IP is observable in some areas of a firm, as it underpins
royalties and tangible flows. Although IP provides some detailed information, it does not cover
everything.
Landefeld cautioned that transaction data can be misleading (e.g., Apple has a much lower tax rate
through maintaining a headquarters in Ireland).
Session 4: External Factors in Influencing Innovation in Firms
The fourth session leader, Stephen Ezell, held a discussion on the factors in the external environment
that affect innovation. External factors define the firm’s operating environment. A firm operating in the
technology innovation sector may be in a different position than a firm elsewhere in the economy. An
innovative firm must protect IP and manage R&D, design, innovation, and cloud computing services, all
of which are affected by external factors. Likewise, the markets for the firms’ products; society and the
natural environment, including climate change; levels of societal trust; and innovation in society, are all
important external factors.
Ezell reviewed the chapter’s key questions: What is the firm’s industry and main market? What is the
firm’s competition and what is their product market characteristics? What support does government
provide for innovation? What are the drivers or barriers to innovation?
Ezell stated that government support for innovation should be a key component of the data collection
and gave an overview of the main types of government policies supporting innovation: grants, equity
finance, debt finance, guarantees for debt financing, payments for goods and services, tax incentives,
and public infrastructure and services. Working capital and access to suppliers are also drivers to
innovation.
The chapter’s supplementary questions include: What are the locations of business activities and value
chains? What is the firm’s main customer’s industry and share of sales? What is the effect of regulations
on innovation? Ezell asked the group if it is worth investigating the firm’s location and space
domestically and globally in the survey. Ezell also encouraged the group to consider the value of
questions about competition.
Ezell noted that the chapter also contains questions about consumer responsiveness to innovation and
trust. However, he recommended not asking these questions, as the World Values Survey already
SRI INTERNATIONAL
17
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
collects similar data. For the chapter overall, he concluded that he liked the questions but not the way
they were approached.
Discussion - External Factors in Influencing Innovation in Firms
Points raised by the group during the discussion included the following:
•
•
•
•
•
•
•
•
•
•
There are traditional competitors and disruptive competitors; the latter play a different game.
Sometimes, the actions or identity of your next competitor cannot be predicted (e.g., Amazon
getting into insurance).
Defining external can be tricky. It requires a lot of attention to measurement. Aside from
intensity of competition, there is demand, technological opportunity (i.e., technical performance
value for dollar in R&D), and appropriability (i.e., the degree to which firms can hope to profit
from their innovations).
There are also demand-side externalities, including both direct and indirect externalities. Today,
there are marketplaces mediated by platforms. The phenomenon of indirect externalities is
new. Poole noted that the U.S. Bureau of Labor Statistics fields the Contingent Worker Survey
that gets at some of the questions about how dependent firms are on external or platform
technologies from the worker’s perspective. Sawhney suggested asking, “Are you dependent on
outside technology for how you do your work and how you are paid?”
There are many all-digital platforms, such as small businesses that write software for Uber, for
example. It is important to differentiate the digital platform from other types. Research should
move to where the profit is going to be, because value is de-materializing. Data and
digitalization will drive value in the future.
Many groups have endorsed the concept of linkages to other data sets. Landefeld noted that
there had been some very good work between the U.S. Economic Development Agency and the
U.S. Census Bureau, and it may be helpful to link to that data.
There are ways of linking non-Census data at a firm level with data from funding surveys. It may
be possible to use linkages as substitutes for some questions. However, respondent burden
must also be considered. Linking data is difficult but can be done.
NCSES should think about linkages ahead of time, as this will help researchers in their work. Ann
Xu links self-reported data to two other data sets in her own research, but it is difficult.
The traditional analysis of innovation has looked at the firm as the unit of analysis and the
product as the output; but now the product is moving towards platform and the firm is moving
towards ecosystem. The problem is how to measure a network in which the product is the
output. Sawhney suggested researchers take an external view and consider it as an ecosystem.
NCSES should not only consider the federal level but collect state data as well. NCSES should
push the envelope on data linkage but not be overly optimistic about what the outcomes will
be. This will rely in large part on data quality improvements.
NCSES could also consider cyclical survey components to add value to the data variables that are
already available for linkages across time, which has been done with the U.S. Census Bureau’s
Annual Survey of Entrepreneurs. However, someone needs to go into the field to negotiate for
linkages and data improvements to support linkage.
SRI INTERNATIONAL
18
Attachment E
TO 19 | INNOVATION EXPERT PANEL
•
•
•
•
•
•
•
•
WORKSHOP SUMMARY
In terms of external data, competition is not always a great measure. Past research (cited by
participants) has shown that firms believed that there were fewer competitors than the actual
number of businesses suggest. It is important to distinguish what researchers want to collect
from what they want to infer.
Several years ago, the U.S. Bureau of Labor Statistics tried to look at the issue of outsourcing.
Either it was not successful, or the results were not reported. The outcomes of this work may be
helpful for NCSES if someone can find their results. One participant heard that the difficulties
were more bureaucratic than technical. Budget resources in the statistical agencies are limiting
this type of research.
The underlying science and engineering base is a big external driver, but it does not appear in
the Oslo Manual list. Is there a profitable way to ask something meaningful about regulatory
effects on product innovation? NCSES might have to ask particular questions for particular
industries.
The role of regulation is extremely important, but how does one obtain solid, objective data
about its effect on innovation? And how deep should the questions go? One purpose of
regulation is to keep people safe, but the social effects are far down the line, beyond NCSES’
remit.
NCSES should not use the term regulation; perhaps the survey could ask about specific policies
for different industries instead.
Rather than asking yes or no questions for many variables, it may be helpful for NCSES to collect
quantitative data to measure the external factors influencing innovation.
Over 75% of federally funded R&D is conducted by multinational companies. A good chunk of
multi-national data would be more valuable than that 300,000 U.S. respondents. NCSES should
ask questions about multi-national operations.
Participants generally recommended including variables on topics such as the firm’s industry,
competition, and government support for innovation. They were not enthusiastic about
questions regarding barriers to innovation. One participant suggested NCSES collect data on
barriers without asking explicitly if something is a barrier.
Session 5: Objectives and Outcomes of Business Innovation
For the final session, E.J. Reedy reviewed innovation outcomes, areas of influence, and strategies. He
shared a casual conversation he had had with someone from the chemical manufacturing industry who
makes products for agriculture. This industry representative discussed the difficulties of getting his
products into the marketplace. His average customer is around 65 years old, and though consumers are
becoming savvier, they do not always believe statistics. Even when his product offers a 10% increase in
yield, he does not feel like he has a good value proposition to offer. In bad years, he must convince
customers that their 50% lower yield would have been 60% lower yield without his product. His firm has
developed other products with 1-2% yield increase rates that they have not brought to market. They do
not believe they can sell them to their customer base because of the low yield rate. Reedy noted that
this suggests that there may be patents for inventions that are not taken to market though they may still
have tangential added value at the firm; however, that will not show up in the measurements. This
SRI INTERNATIONAL
19
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
means that surveys do not capture the projects that fail. Those data are revealing about barriers and
constraints. Some questions at the project level could address questions on projects that fail.
Reedy provided an overview of the definitions. Most of the chapter used similar terminology for
objectives and outcomes, but Reedy was not convinced that the same structure could be used for both;
in some cases, it might not be possible or feasible. Researchers often start with outcomes for innovative
firms, for which the manual suggests a three-year period; then, these transition into the objectives.
Reedy felt that the three-year period—around which the Oslo Manual is built— provides sufficient time
to measure impact.
Next, he gave an overview of the questions for data collection.
•
•
•
What are the firm’s innovation objectives and outcomes by area of influence (e.g., markets,
production and delivery, business organization, economy and society)?
What are the innovation objectives and outcomes for business strategies?
What percentage of innovation sales share in total business sales?
Reedy then presented supplementary questions for data collection from the chapter: What is the count
of product innovations? What are the major impacts of innovations for markets?
Discussion - Objectives and Outcomes of Business Innovation
The discussion continued with a debate on the three-year period. One year may not be enough time to
measure impact. NCSES can use impact as an outcome measure to determine how successful firms are.
It could also describe market-level intensity around innovation.
The Manual can help NCSES interpret outcomes. Although some effects would only be observable to
customers, impact on society as a whole might be more visible.
In terms of objectives, it was not clear to what extent the focus should be on the last three years.
Jankowski felt that for the revision steering group, objectives are probably linked to the three-year
period, but it is probably not a tight timeline relationship.
The R&D lifecycle is likely longer than three years, and the time horizon for innovation is a function of
objectives and outcomes. But this would require distinguishing different types of innovation. Some firms
do not care about the next year. This is not just a large-firm issue, as even small companies may take 15
years to get something to market. NCSES could ask about the distribution of projects across horizons or
create a metric of product planning for product strategy that would measure the product life cycle.
NCSES can add a question that asks about lifecycle and another that asks about the types of innovation.
Reedy drew a diagram to illustrate the two types of taxonomies that apply to objectives and outcomes.
The first is based on strategy and includes variables that assess positioning, internal capabilities, and
positioning within the market. The second is based on area of influence and include variables that assess
the markets for products; productivity and delivery; business organization; and economy, society, and
environment (i.e., quality of life and well-being).
SRI INTERNATIONAL
20
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
NCSES can also consider social impact by asking firms about their prioritization of those issues. Although
it may be difficult to collect data about environmental and societal outcomes, many businesses cite
them as part of their goals (i.e., objectives). Firms try to balance between being good corporate citizens
(by having a societal impact) and making money.
Participants cautioned that adding questions on social impact to the ABS may lead to firms responding
with what they think NCSES wants to hear. These types of questions will be difficult to write and will
require extensive testing for bias. It might be more useful to conduct qualitative research on finances.
Another issue is that often respondents do not know the market or the demand and need to learn about
them. This may require activities that do not bear directly on the objectives. Although the ultimate
objective is to make money, these more intermediate objectives may be important. Additionally, how do
researchers get to objectives other than strictly commercial ones (e.g., the malaria vaccine)? Many
young entrepreneurs have a hybrid business vision, with a high level of intentionality about societal
impact. NCSES is not inclined to include these on the first survey round. These sorts of goals are part of
the European Union’s Horizon 2020 plan, but they are not yet in the language of U.S. policy.
Xu suggested that the potential way for NCSES to confirm answers to social impact questions is to ask
how firms measure their outcomes. Investments are measurable. Graham suggested NCSES find out
what managers’ bonuses are linked to.
Reedy concluded by saying that he was concerned about the granularity of the questions and would not
recommend implementing all of them; he also suggested asking at the firm level rather than object
level.
Summary and Recommendations
In the final session, Youtie reviewed the group’s conclusions and recommendations for the topics by
chapter. She noted that, overall, the group recommended including most of the questions, except those
about drivers and barriers to innovation.
Chapter 5. Measuring Business Capabilities for Innovation
Youtie noted that the group appeared to have become more intense and probing as the day progressed
and wondered if the discussion on capabilities had happened too soon. The general conclusion was that
the questions in this section should probably be asked, especially if dollar figures can be collected.
However, this chapter also had the least consensus among the group, and many concerns were voiced.
There was disagreement about the preferred unit of analysis. NCSES plans to collect data at the firm
level, per the manual recommendations, but some participants felt that that without project-level data,
the survey would miss business relationships and many functions of multi-national firms.
Sawhney felt that co-development and co-creation were not adequately addressed. He suggested a
three-part question: Did you make it? Did you buy it? Did you partner with another firm?
SRI INTERNATIONAL
21
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Some firms are engaged in work that meets the definition of R&D but does not track it as such in the
accounting systems. Perhaps NCSES should consider more generic terminology (e.g., new activities).
Youtie noted that the manual suggests an FTE approach in these cases.
The group did not come to a consensus about the ideal size(s) of businesses that NCSES should cover in a
national innovation survey. Some favored 10 or more employees, but others noted that small businesses
also conducted innovative activity. One possibility is that NCSES consider different standards for
industries in different sectors.
Small businesses present particular challenges. They may have people who take care of both traditional
business activities and innovation activities. Those that have not yet introduced an innovation into the
market may not yet have any revenue.
Youtie doubted that the planned activities questions would yield useful data. (Many promise, but few
follow up.)
Cohen liked supplementary questions that ask about the number of innovation projects by activity.
Poole suggested that it might be easier to get at capabilities by asking about occupations and human
capital (e.g., how many people do this type of work?). Landefeld added that labor data can provide
information on the mix of occupations.
Chapter 6. Business Innovation and Knowledge Flows
Youtie noted that the group had shown great interest in this chapter. Landefeld would prioritize
anything that can be monetized. Sawhney felt that the topics got at some part of, but not all of, cocreation. Cohen suggested that researchers consider trying to capture spillovers, co-development
activities, and upstream knowledge, in addition to the variables presented in the manual. Another
suggestion was to track performance evaluation criteria for innovative project management, as these
might show how firms prioritize projects.
One concern was identifying the correct persons in the firm of whom to ask the questions—who knows
the operation? Who knows the company strategies? Do people in one unit understand what is
happening in other units or in the firm overall?
Some wondered if administrative records could provide some of the information data this chapter
discusses. Another point raised was that whereas new products and financial performance give a certain
measure of success, failure is also a teacher. What did the firm learn? Also, how did an innovation affect
people’s lives? Did it affect the firm’s relationships with its customers and other firms?
Chapter 7. Measuring External Factors Influencing Firm Innovation
The group liked most of the questions in this section. However, the questions about drivers of and
barriers to innovation caused concern. Why ask about working capital and access to suppliers when
secondary data can be used to control for how competitive a market is? Many firms have no idea how
many competitors they have, and would not be able to answer questions about their domestic and
global space. Also, data about consumer responsiveness to innovation and trust are already collected by
the World Values Survey.
SRI INTERNATIONAL
22
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
The discussion covered the possibility of linking to other data sources, while acknowledging the difficulty
of doing this. Participants also acknowledged that regulation could be both a driver of and a barrier to
innovation, but it is problematic to ask the question because the term “regulation” has become
politically loaded in the United States. Some noted that external drivers to innovation were not on the
list and should be added.
Chapter 8. Objectives and Outcomes of Business Innovation
Reedy was concerned about the granularity of these questions and would not recommend
implementing all of them. Concerns raised by the group included the practical application of the threeyear periods suggested by the manual. Do these apply to objectives as well as outcomes? More
importantly, do they align with product life cycles? In many cases, the answer is no. Another concern
was how to ask questions about outcomes and objectives in such a way that respondents would not
simply choose an attractive sounding answer because there is no penalty for doing so. This line of
questioning will likely need extensive testing, but many new businesses have a hybrid vision that
includes social impact, and the group agreed that it is important to capture it. As a check on the
outcome-related answers, Xu suggested asking how respondents measure their outcomes, and Stuart
Graham suggested asking respondents what triggers bonuses.
Participants made several suggestions in closing. Poole stated that any increase in granularity in
geography would increase the usage of the data dramatically. National data does not tell the story that
researchers and policy makers need at the state and sub-state levels. Landefeld suggested that NCSES
interview government analysts directly for their feedback. Crowell noted that universities are becoming
more like commercial firms in the way they manage their innovation assets and urged NCSES to look
more closely at this.
Cohen that the discussion missed the adoption and diffusion of platforms, and non-high-tech
innovations that have a huge economic impact (e.g., the supermarket, containerization). Jankowski
responded that the Oslo Manual explicitly states that the innovation does not have to be high-tech,
while admitting that the last version was clearer about this.
Youtie thanked the stakeholders for their participation and wonderfully diverse comments. Jankowski
told the participants that NCSES would keep them updated on future developments and asked that they
feel free to contact NCSES with any questions or updates on any interesting developments in their work
on innovation.
Next Steps
NCSES will proceed with updates to the ABS and potentially schedule individual interviews with
government agencies and other end users of the data for additional feedback. During these processes,
NCSES will consider the recommendations presented by each of the stakeholders. After this review,
NCSES will contact businesses about the topics it plans for the survey. NCSES will be in touch with the
workshop participants again to share its conclusions.
NCSES plans to launch the survey module in about a year. NCSES will also watch closely over the next
two years as Eurostat, Japan, and Statistics Canada launch surveys based on the 2018 Oslo Manual.
SRI INTERNATIONAL
23
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Detailed Notes
Welcome, Introductions, and Discussion of Meeting Goals
John Benskin, Project Leader, Center for Innovation Strategy and Policy (CISP), SRI International
John Benskin opened the workshop by welcoming the assembly and provided an overview of the
workshop logistics. Benskin introduced fellow workshop organizers, CISP Executive Director Christina
Freyman and CISP Research Analyst Tina Davis. Benskin then introduced NCSES Division Director Emilda
Rivers.
Emilda Rivers, Division Director, NCSES, NSF
Emilda Rivers welcomed everyone and thanked those in the room for participating in the workshop.
Rivers summarized the objective of the Innovation Expert Panel and the goals for NCSES national and
global innovation indicators. She stated that innovation is a very important topic that NCSES has heavily
invested in to be able to have useful information not only on the United States, but in a global context.
This workshop will engage stakeholders early on to determine the necessary measures and discuss the
challenges and opportunities as NCSES moves forward to include more information in the business
innovation survey. This is also an important topic for NCSES Directorate for Social, Behavioral, &
Economic Sciences, as one of their grand challenges addresses innovation. She added that often when
innovation is considered from a research perspective, measurement is not a part of the discussion. She
would like to develop national and global ways of talking about innovation. Rivers closed by thanking
everyone again for being a part of the workshop discussion.
Benskin suggested that each of the participants introduce themselves and provide their organizational
background.
The participants who attended the workshop are as follows, in order of introduction:
•
•
•
•
•
•
•
•
•
•
•
Samson Adeshiyan, Acting Deputy Division Director, NCSES, NSF
Audrey Kindlon, Survey Statistician, NCSES, NSF
Stephen Ezell, Vice President, Information Technology and Innovation Foundation
Mohan Sawhney, Director, Center for Research in Technology & Innovation, Kellogg School of
Management, Northwestern University
Nico Thomas, Performance Analyst, National Institute of Standards and Technology
Steve Campbell, Economist, Manufacturing Research and Program Evaluation, National Institute
of Standards and Technology
Mark Boroush, Senior Analyst, NCSES, NSF
Jan Youtie, Workshop Chair, Principal Research Associate, Enterprise Innovation Institute,
Georgia Tech
John Jankowski, R&D Statistics Program Director, NCSES, NSF
Adam Jaffe, Professor, Economics, Brandeis University
Mark Crowell, Executive Entrepreneur in Residence, Eshelman Institute for Innovation, UNC
Eshelman School of Pharmacy
SRI INTERNATIONAL
24
Attachment E
TO 19 | INNOVATION EXPERT PANEL
•
•
•
•
•
•
WORKSHOP SUMMARY
Stuart Graham, Associate Professor, Scheller College of Business, Georgia Institute of
Technology
Ann Xu, Senior Technical Advisor for Impact and Assessment, ARPA-E
E.J. Reedy, Director, Polsky Center for Entrepreneurship and Innovation, The University of
Chicago
Wes Cohen, Professor, Fuqua School of Business, Duke University
Kenneth Poole, Executive Director of C2ER, The Council for Community and Economic Research
Steve Landefeld, Retired Director, U.S. Bureau of Economic Analysis
Next, the observers in the room introduced themselves and provided an overview of their organizational
background.
•
•
•
•
•
•
Patrice Norman, U.S. Census Bureau
Ray Wolfe, NCSES, NSF
Gary Anderson, NCSES, NSF
Dan Foley, NCSES, NSF
Francisco Moris, NCSES, NSF
Carol Robbins, NCSES, NSF
Continuing Examination of Innovation in the U.S. Business Sector1 by NCSES
John Jankowski, R&D Statistics Program Director, NCSES, NSF
Jankowski began the discussion with an introduction of the Workshop Chair, Jan Youtie. Youtie will be
guiding the conversation. Jankowski then began his presentation with an introduction to the work down
by NCSES in examining innovation in the U.S. business sector. He provided a presentation of the NCSES
organizational background, NCSES innovation surveys, and the Oslo Manual and purpose of this
workshop.
Jankowski continued his presentation with a brief overview of NCSES’s organizational background. He
gave a brief history of the innovation surveys and then discussed the revisions to the Oslo Manual.
NCSES is a division within the Directorate for Social, Behavioral, & Economic Sciences. NCSES is one of
the 13 Federal Government’s principal statistical agencies. NCSES Legislative Mandate of 1950 outlines
that NCSES is an independent federal agency “to promote the progress of science; to advance the
national health, prosperity, and welfare; to secure the national defense” and one role is to collect
information on the science and engineering enterprise.
All the BRDIS and Micro-BRDIS innovation questions have derived from the Oslo Manual. Jankowski
asked if anyone in the room did not know of the Oslo Manual. As all participants were aware of the
manual, Jankowski proceeded. The Oslo Manual itself defines innovation historically in terms of what
the enterprise did. It discusses various types of innovation: product innovation, process innovation,
organizational innovation, and marketing innovation.
1
See Appendix for presentation slides.
SRI INTERNATIONAL
25
Attachment E
TO 19 | INNOVATION EXPERT PANEL
•
•
•
•
WORKSHOP SUMMARY
Product Innovation – New or significantly improved goods or services with respect to their
capabilities, user friendliness, components, or sub-systems.
Process Innovation – New or significantly improved production processes, distribution methods,
or support activities for goods or services.
Organizational Innovation – New business practices; new methods of organizing work
responsibilities and decision making; or new methods of organizing external relations with other
firms or public institutions.
Marketing Innovation – Significant changes to the design or packaging of a good or service; new
media or techniques for product promotion; new methods for product placement; or new
methods of pricing goods or services.
These factors govern what type of data are collected. Although historically the Oslo Manual itself defines
innovation based on what enterprises do, the objective of the workshop today is not to look back but to
continue looking forward.
Jankowski gave an overview of the BRDIS questions. Innovation data on BRDIS have been collected
annually from 2008 to 2016. BRDIS had about an 83% response rate last year. NCSES introduced
questions about product and process innovation as well as novelty, new-to-market sales, and more to
the survey instrument. Internationally, this is the focus for innovation. In 2016, NCSES developed the
micro-BRDIS survey that went out to 200,000 businesses with up to 9 employees. The survey asked
these micro-businesses about product, process, organizational, and marketing innovation, and those
data have been delivered to NCSES from the U.S. Census Bureau. NCSES has now partnered with Census
to include innovation modules on the new ABS.
With the development of the ABS, where businesses of all sizes are asked questions about innovation,
BRDIS then became the Business Research and Development Survey (BRDS). The 2017 BRDS contains no
questions on innovation. The ABS combines the Survey of Business Owners, Annual Survey of
Entrepreneurs, Micro-BRDIS, and a stand-alone business innovation module. This will collect business
innovation statistics with a sample of 850,000. NCSES has considered various business strategies and the
results of those activities. The module will ask approximately 15 questions looking at innovation.
In June 2019, NCSES will scale back the innovation module on the ABS to ask approximately eight
questions. This transition will mirror the transition from the old Oslo Manual to the new Oslo Manual
and ask new innovation questions (e.g., types of innovation activities, total cost). Three years ago, the
OECD and Eurostat began an effort to review and revise the Oslo Manual. The manual is completed,
having gone through extensive review. It will be released on October 22, 2018. The revision was guided
by a steering group from multiple countries, including Japan, Germany, Norway, representatives from
Eurostat, and the United States. There was an extensive effort to review each chapter in terms of
measuring innovation so that it is consistent with other manuals. NCSES also looked at the role of
digitization and globalization in terms of the economy and the role of innovation. The manual
encourages companies to focus on the subject-based approach as the primary collection source for
innovation statistics, although it also recognizes the object-based approach. Each Oslo Manual chapter
ends with key indicator questions as well as supplementary indicators.
SRI INTERNATIONAL
26
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Jankowski commented that five presenters will introduce the discussions in each of the topic areas of
interest. Jankowski provided an overview of Chapter 3, Concepts and definitions of measuring business
innovation, and the basic definitions of the Oslo Manual; the remaining presenters focus on Oslo Manual
Chapters 4 through 8, as follows:
•
•
•
•
•
Chapter 4. Measuring business innovation activities (Jan Youtie, Chair)
Chapter 5. Measuring business capabilities for innovation (Wes Cohen)
Chapter 6. Business innovation and knowledge flows (Steve Landefeld)
Chapter 7. Measuring external factors influencing firm innovation (Stephen Ezell)
Chapter 8. Objectives and outcomes of business innovation (E.J. Reedy)
Jankowski explained that the workshop would review the basic topic areas and then hold a discussion
regarding how to understand them in terms of U.S. business innovation and how to prioritize them to
create an innovation module for the 2019 ABS. He added that the purpose of this workshop will identify
the most important topics on the next U.S. innovation survey based on the guidance in the Oslo
Manual—not to write the questions themselves, which will be another step in the process.
Jankowski introduced the survey definition of innovation:
•
An innovation is a new or improved product or process (or combination thereof) that differs
significantly from the unit’s previous products or processes and that has been made available to
potential users (product) or brought into use by the unit (process).
o This definition is to be the basis for measuring innovation in all sectors of the economy.
Jankowski added that the product or process must be used to count as innovation, though it may
succeed or fail. Also, in this digital world, products and processes can intermix.
•
•
A business innovation is a new or improved product or business process (or combination
thereof) that differs significantly from the firm's previous products or business processes and
that has been introduced on the market or brought into use by the firm.
o A product innovation is a new or improved good or service that differs significantly from
the firm’s previous goods or services and that has been introduced on the market.
o A business process innovation is a new or improved business process for one or more
business functions that differs significantly from the firm’s previous business processes
and that has been brought into use by the firm.
There are two major types of innovation by object: innovations that change the firm’s products
(product innovations), and innovations that change the firm’s business processes (business
process innovations):
o Product innovations can involve two generic types of products: goods (tangible objects)
and services (intangible activities). Knowledge-capturing products, as identified in SNA,
can have the characteristics of either a good or service and concern the provision,
storage, safekeeping, communication, and dissemination of digital information that
users can access repeatedly.
SRI INTERNATIONAL
27
Attachment E
TO 19 | INNOVATION EXPERT PANEL
o
WORKSHOP SUMMARY
There are six types of business process functions: (1) methods for producing goods or
providing services; (2) distribution and logistics; (3) marketing and sales; (4) information
and communication systems; (5) administration and management activities; and (6)
product and business process development, including activities to identify, develop, or
adapt products or a firm’s processes.
Jankowski added that for countries that wish to maintain their time series, these functions will crosswalk
to the previous other types of innovation (i.e., organizational, marketing, and process innovation). He
added that this is where things stand after much work that has gone into the Oslo Manual. He suggested
that the same amount of work will have to be put into a good survey on business innovation in the
United States. Jankowski concluded his presentation and thanked the participants.
Discussion - Continuing Examination of Innovation in the U.S. Business Sector by NCSES
Jan Youtie, Chair, Principal Research Associate, Enterprise Innovation Institute, Georgia Tech
Youtie introduced herself as the Workshop Chair and noted that she would welcome feedback for
methods to measure innovation. Youtie added that she does not want to overburden firms by asking
everything in the Oslo Manual, but NCSES would like to collect up-to-date information to measure
innovation. She encouraged commentary and updates from the stakeholders in the room. Youtie then
opened the floor for discussion.
Wes Cohen, Professor, Fuqua School of Business, Duke University
Cohen discussed his ideas on the importance of copycat innovation and asked if an innovation must be
new to the market; he doubts this factor is important to the discussion today. His last survey identified
those who were innovation imitators. Cohen asked participants how to decide when to pay attention to
new innovative products—is it a new toothpaste or the first 3-D printer? Which is more significant?
There should be questions that summarize the importance of a new product. This will also speak to the
importance of object-oriented questions. NCSES should ask these questions in addition to the firm-level
questions. He added that if NCSES wants respondents to answer meaningfully, specific questions and
measures should be kept in mind (e.g., to address social welfare). Cohen recently co-authored a paper
that discussed those variables. This could be a foundation for subsequent innovation research and
address several of the dimensions that change the understanding of social welfare and its impact.
Jankowski commented that those variables will be addressed in the presentation on Chapter 7. He
added that each of the five presenters has seen the individual chapter that pertains to his or her
presentation; however, no one else in the room has seen it or the whole manual yet. Many the issues
that Cohen has identified will be covered in later discussions. (The 2018 Oslo Manual will be launched
next week.)
Stuart Graham, Associate Professor, Scheller College of Business, Georgia Institute of Technology
Graham asked how much space would be available on the survey for questions and how many questions
could realistically be included for businesses? Graham suggested that it might be helpful to make
comments on those questions today to get an informal, internal ranking on the first-order, high-priority
issues. Jankowski responded that the goal, at this point, is to include 15 to 20 questions.
SRI INTERNATIONAL
28
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Reedy asked about the sample size and industry focus. Jankowski responded that the 2019 sample size
will be approximately 300,000 for all businesses of one or more employees and cover all industries. One
question is whether it is possible to have targeted sampling modules. ABS, which is in the field now, has
a sample size of approximately 800,000, and the only oversampling is on the Micro-BRDIS portion.
Sampling modules will not be realistic for the business innovation survey.
Mohan Sawhney, Director, Center for Research in Technology & Innovation, Kellogg School of
Management, Northwestern University
Sawhney noted that the end users provide use cases for what happens in the data. He asked, what do
we do with all these data? Jankowski responded that the workshop participants are the end users or
represent a particular interest in these data. One of the problems with BRDIS is that the questions have
not changed much and are not granular enough; they have not kept up with end users’ interests.
Sawhney feels that the nature of value-creating innovations has transformed. He asked, how do surveys
capture that in product, process, and business model innovation? Sawhney added that everyone may be
using industrial viewpoints to measure ecosystem constructs. Now, platforms have become very
important; iOS and Android create the foundation.
Jankowski responded that all these variables that Sawhney mentioned (addressing product, process, and
the combination thereof) are covered in the new Oslo Manual. The new survey includes the digital
components that could overlap. One of the takeaways could be to ensure that the survey captures
something on value creation or platform-based innovation; NCSES should be aware of those variables
and include questions that will help to capture those topics.
Youtie invited one final question. Cohen added that Google and Uber are trillion-dollar companies, and
he is worried about the ability of a broad-based survey to capture these rare cases of massively
successful business model innovation. The real high-impact factors are very rare, and that in itself is a
policy issue.
Steve Landefeld, Retired Director, U.S. Bureau of Economic Analysis
Landefeld stated that he comes from a macroeconomics perspective. The relative importance for
macroeconomics is huge; he would like to see more data on prices and dollars instead of yes/no
responses to observe valuations and put a relative importance on it. What is the residual and the impact
on economic growth? In terms of U.S. and world growth, everyone is scrambling for answers on what’s
happening in innovation and technology and how that is impacting growth. What about the location of
growth and offshore production? What is expected from this new tax act? He would like to see two-way
flows. Data on both sides of the flow is critical.
Session 1: Business Innovation Activities2
Youtie closed the discussion points and began her presentation. This discussion captures Business
Innovation Activities. Youtie introduced the definition for this chapter.
2
See Appendix for presentation slides.
SRI INTERNATIONAL
29
Attachment E
TO 19 | INNOVATION EXPERT PANEL
•
WORKSHOP SUMMARY
All developmental, financial, and commercial activities that are intended to result in an
innovation.
She gave an overview of the eight activities and reviewed the manual definitions and criteria, as follows:
Eight Types of Business Innovation Activities:
•
•
•
•
•
•
•
•
Research and experimental development activities
Engineering, design, and other creative work activities
Marketing and brand equity activities
IP-related activities
Employee training activities
Software development and database activities
Activities related to the acquisition or lease of tangible assets (e.g., buildings, equipment,
machinery, and other tangible assets)
Innovation management activities
Youtie briefly discussed how each activity must meet five criteria of R&D: novel, creative, uncertain
outcome, systematic, and transferable/reproducible. Software and new-economy companies are where
these R&D often activities take place. Not all innovations meet all five of the R&D criteria.
•
•
•
•
•
•
•
Engineering and design activities which do not meet the five R&D criteria include testing, set-up,
tooling, production, and quality testing; service firms use design for look and use. (Engineering
need not always be for innovation and can even be reverse engineering.)
Marketing and brand equity activities include pricing, placement, and promotion advertising
(usually if linked to a new, improved product).
IP activities include patents, utility patents, industrial designs, trademarks, copyright, IC designs,
plants, and trade secrets for new products and processes.
Employee training is not for existing products or general skills.
Software development and database innovation activities would not include maintaining the IT
system, supporting existing systems, or routine functions.
Tangible assets are buildings, instruments, equipment, and computer hardware.
Innovation management includes funding, organization, external collaborations, and
performance (applicable only to firms that innovate).
Youtie asked participants, do all eight activities need to be covered in the questions?
Cohen asked what the unit of analysis was? From what is the data collected? Jankowski stated that the
Oslo Manual says to collect it at the enterprise level, not at the establishment level. Collect from the
lowest legal entity with decision-making authority. Cohen suggested that the unit of analysis should be
the innovation project. Jankowski thanked Cohen and his colleagues for bringing the object-based
approach to the community’s attention. He added that had it not been for the earlier research of Cohen
et al., the object-based approach would not be included in the manual but added that the focus should
be the subject-based approach; because the enterprise is the unit of analysis for all the international
SRI INTERNATIONAL
30
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
statistics. Cohen clarified that he was not suggesting object-based instead of subject based collection.
Youtie noted that the next slides cover detailed questions about the eight activities.
Poole added that if one is doing training or deployment of new technology or innovation, it could be in
an entirely different location from where the design is taking place. Typically, data collection is done at a
national level, but we must consider how multi-national organizations function as well.
Youtie noted that the rest of the slides contained more detailed questions about the eight activities
conducted in-house, as well as funding sources. She continued with her presentation.
Key Questions for Data Collection:
•
•
•
Were the eight activities conducted, regardless of the reason, with innovation as the aim? Was
each activity conducted in house or procured from external organizations?
What was the total expenditure for each of the seven activities, excluding innovation
management (not recommended by the Manual)?
What were the funding sources for innovation?
Sawhney mentioned the need to address co-development and co-creation. Everything in the world
today is built upon partnerships. The boundaries of a firm are no longer rigid. The Oslo Manual considers
the source(s) as a core question. Sawhney noted that there is a binary question on make or buy; he
asked, what about the co-developed or co-created partnerships? Youtie responded that there may be
other chapters that address co-development and cocreation in the manual.
Jankowski responded that when organizing the workshop, NCSES deliberately did not send the Oslo
manual to all the participants because of its length and density, so they have not been able to review it.
However, the manual does address co-creation, and he assured Sawhney that NCSES would consider it
when developing the questions. Sawhney suggested a three-part question: did you make it, did you buy
it, and/or did you partner?
Next, Youtie noted that although it is difficult to assign a dollar figure to innovation management
activities, it is important to capture general total expenditures and expenditures on R&D.
A participant asked if it was really necessary to ask for the total expenditures.
Adam Jaffe, Professor, Economics, Brandeis University
Jaffe noted that many firms are engaged in activities that meet the definition of R&D (e.g., novel and
creative), but they would not label or identify their work as R&D. Jaffe suggested using different,
perhaps more generic phrases, such as new activities, to capture novel activities. Youtie responded that
the Oslo Manual suggests more detailed questions beyond these, including separating intramural and
extramural activities, to try to get to the economics flows, but it acknowledges that some firms
(especially small firms) cannot use an accounting approach, in which cases it suggests using an FTE
approach.
Jankowski asked if there was a consensus about the size of the business NCSES should cover in a
national innovation survey. Youtie responded that her experience suggested 10 or more employees
SRI INTERNATIONAL
31
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
should be the standard. Her experience in the state of Georgia showed that most companies with fewer
than 10 employees do not primarily do manufacturing, but rather distribution or sales.
Ann Xu, Senior Technical Advisor for Impact and Assessment, ARPA-E
Xu stated that many of the performers with which she works at ARPA-E are businesses with only two or
three people.
Jankowski commented that there may be different standards for different sectors (and added
parenthetically that it may require the survey to reveal how many employees a company actually
employs, as opposed to what the business register says).
Landefeld added that it is better to collect information from them once every five years rather than not
collect at all. Small firms are involved in innovation.
Poole added that there is a numbers problem with a large quantity of smaller firms. Below a certain size,
some firms focus on R&D and some do not. It is necessary to focus on those that conduct R&D, but how
do we find them? Jankowski responded that the assumption is that R&D itself is not necessarily a good
descriptor for businesses engaged with innovation. Some small firms have people with multiple roles
who take care of traditional business activities as well as new things.
E.J. Reedy, Director, Polsky Center for Entrepreneurship and Innovation, The University of Chicago
Reedy noted that many of the small firms have no revenue yet; therefore, looking at small firms only
through innovations introduced into the market could be problematic. Industries in different areas have
huge differences in culture and response rates. Reedy also noted that he would want to know, at least
for a few industries, what is happening at the state and local level, and what the microdata show.
Jankowski commented that they will be sensitive to those issues, but asked that for today the
participants focus on what is most important at the national level.
Youtie added that those are all considered to be core questions. Should they ask about marketing and
other follow-up activities? Youtie addressed the supplementary questions for human resources by
activity, data collection, number of innovation projects by activity, follow-up activities after
implementation, (but within the period), and planned activities. She asked about planned activities. In
her experience in Georgia, there are many promises but little follow-up activity, so she does not think
this category will yield much useful data.
Supplementary Questions for Data Collection:
•
•
•
•
R&D continuous or occasional
Innovation expenditures by funding source
Follow-up activities
Planned innovation activities and expenditures
Youtie concluded with a summary for prioritization and asked participants what they thought about
these questions:
•
Qualitative data on whether or not each of the eight activities were conducted
SRI INTERNATIONAL
32
Attachment E
TO 19 | INNOVATION EXPERT PANEL
•
•
•
•
•
WORKSHOP SUMMARY
Whether the activity was conducted for innovation
Whether or not each activity was conducted in house or procured from external organizations
Total expenditures for each of seven activities
Total innovation expenditures using the accounting method (shown for R&D)
Funding sources for R&D
Stuart Graham, Associate Professor, Scheller College of Business, Georgia Institute of Technology
Graham asked about the thought process behind the prioritization of the list. Jankowski responded that
it was a collaborative consensus effort. There was no systematic approach to determine which questions
to add to the core list. Cohen added that he would not use the terms procure or acquire in prioritization.
Jankowski responded that the term procure was intentional, but he would take that into consideration.
The participants took a brief break and reconvened for Wes Cohen’s discussion.
Session 2: Business Capabilities for Innovation3
Cohen began the next discussion with a presentation on business capabilities. He stated that some of
the points are being addressed by other chapters. He gave an overview of resources contributed by the
firm. Cohen added that the recent work from his colleagues shows that an important driver for
innovation performance at the macro level is the relationships across nodes, not just the nodes
themselves (e.g., individual firms, universities). They had asked where the most important new products
came from; 49% of respondents said that it was acquired from an external source, and many said it was
acquired through co-development with other entities. It is reasonable to focus on the firm, but it is
important to think about relationships and capabilities, and how to find managers of cooperative
ventures, licensing deals, and other relationships. He and his colleagues have also collected data on the
relevance of those channels over time.
Sawhney commented that there may be a way to capture a proxy through a measure of resources in the
ecosystem. Youtie commented that the next presentation will address knowledge flows. Sawhney
mentioned that he has written an article on the concept of relationship capital and argues that the most
important factors will not appear on the balance sheet. He called this “relation capital.” Cohen added
related aspects to think about: Where did you get the idea? Who built the prototype? What are the
knowledge flows? Invention is much more tangible downstream.
Cohen mentioned the definition:
•
Knowledge, competencies, and resources that a firm accumulates over time and draws upon in
the pursuit of its objectives.
Cohen asked the participants, where do these capabilities/resources reside? At what level should these
types of questions be asked, and who can respond knowledgably to these things? Upper management
might be clueless, and the innovative department might have a different perspective from management.
It is also complicated because it varies across firms. Does anyone have ideas of how to get at this?
3
See Appendix for presentation slides.
SRI INTERNATIONAL
33
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Cohen presented the last two components of the definition, workforce skills and how a firm manages its
human capital; and the ability to use technological tools and digital resources, with the latter providing
an increasingly important source of information for innovation.
The aggregate statistics should be broken down by industry. Cohen began reviewing the key questions
for data collection. He asked: why is there a need for supplementary questions and how were they
sorted? By feasibility? Vagueness? Jankowski responded that questions that made it into the key groups
were generally agreed on by the whole steering group, or at least were not strongly opposed by anyone.
Supplementary questions were generally those that one or a few members thought were important (but
none opposed).
Cohen continued by asking if there is a vehicle that can be used to determine if those data are
important. Can we develop a method to address the question, “So what?” Perhaps this is achievable by
working with others at or organizations external to NSF? Jankowski responded that he will consider that
in the refinements and advancements of future innovation survey collections; however, today’s focus is
to guide the next survey because many of the stakeholders in the room have had much more experience
with these kinds of topics and issues than NCSES’s colleagues elsewhere.
Cohen discussed the following key questions for data collection:
•
•
•
•
•
•
•
•
•
•
Number of full-time employed persons
• Also, by business unit (not the firm as a whole)
Total sales
• Also, business unit sales—the key correlate of R&D
Firm age by year the firm began business activities
• Date of entry by market
Firm ownership status (e.g., stand-alone, part of a national group, part of a multinational group)
Geographical distribution of sales (e.g., local, national, international markets)
Export share of sales
• By business unit
Importance of cost versus quality for the firm’s competitive strategy
• Suggested measures: percentage of process versus product R&D, and, for latter,
percentage dedicated to developing altogether new products versus product
improvement
Share of employed persons with a tertiary education
Business level of design capability
Cash flow
Cohen then gave an overview of supplementary questions for data collection, as follows:
•
•
•
•
Family-owned firm status
Number of product lines
Innovation management: responsibility for innovation within the firm
Innovation management: methods to support internal knowledge exchange
SRI INTERNATIONAL
34
Attachment E
TO 19 | INNOVATION EXPERT PANEL
•
•
•
WORKSHOP SUMMARY
Number of employed persons by major field of education
Technical expertise in emerging technologies
Digital competences
Cohen asked who is the audience, and does this address policymakers, management, and other
stakeholders? Is there a clear statement on that in the Oslo Manual? Why are we doing this? Also, there
should be a vehicle within these surveys that allows analysts to tease out cause and effect, because this
will allow researchers to answer the question, “So what?” To do that, distinguish between innovating,
imitating, and firms that do neither. Cohen suggested that he is a bit worried about using Likert scale
questions to assess that. He would prefer to ask concrete questions about behaviors; for instance, did a
university suggest new products to a firm? Jankowski responded that there is a scale to provide a range
of importance.
Discussion - Business Capabilities for Innovation
Cohen asked, what capability should be identified? Is it the capacity to invent or to take a product to the
next step of commercialization, manufacturing, marketing, or sales? Pfizer does some R&D, but mostly
acquires inventions from others. Cohen did not see any discussion of commercialization or sales in the
business capabilities chapter. He noted that firms with weak inventive capability benefit from being
around other firms, which provide them with inventions; firms with strong inventive capability do not
benefit from non-inventive firms but do benefit from being near universities. The capabilities should be
distinguished across managerial, technical, and other divisions. Marketing, sales, and manufacturing are
essential.
Sawhney suggested looking for the capability to innovate. Cohen asked, but are firms’ inventions
exclusively originating from R&D? Sawhney asked, why do we care?
Youtie added that collecting data about secondary education would be nice for some companies. Cohen
added that advanced degrees are covered elsewhere under workforce, and Jankowski confirmed that
the manual addresses this.
Cohen stated that when measuring performance, he prefers financial performance. Regarding products,
Cohen asked about incremental versus altogether new products. He has found data—even self-reported
data—on behaviors to be very useful. Youtie confirmed that strategy is important, but that many firms
hire those with a tertiary education for the front office, but the back office is mostly those without a
tertiary education. Youtie added that it could be helpful if those individuals with a tertiary education in a
technical field (as opposed to, for instance, individuals with master’s degrees in business administration
or social scientists) could be separated out. Xu talked about the importance of digital capabilities. She
said that she did not see any questions or data on that; she then asked about collecting information on
data analytics. Cohen responded that this was a good question, as they are constrained in terms of the
number of questions that can be asked. Jankowski read a brief paragraph from the Oslo Manual in
response to the earlier questions. Jankowski added that many design activities, including user
functionality and aesthetics, could be categorized as R&D, but they shouldn’t be in that category. In the
manual, digital technologies and data analytics are specifically singled out. There are more definitional
aspects there; digitization is taking place across all sectors and they want to capture it along those lines.
SRI INTERNATIONAL
35
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Ken Poole, Executive Director of C2ER, The Council for Community and Economic Research
Poole stated that the key questions address things that people know about firms. How much of this can
be imported from administrative record sources (e.g., business demographic questions)? Poole added
that there is much muddling between business capabilities and individual capabilities. If asking about
people, then business may not be the right unit of measurement. Cohen continued that looking at new
products and financial performance gives a certain measure of success. But what about tech failures?
What about the lives affected? Did the firm learn anything from the experience? Cohen added that
relationships are important as well.
Youtie noted that Poole’s point was very important. In some regions, firms make a business decision to
hire people with no skills, so that is also a business-level issue. Poole added that the fundamental issue
is who understands that strategy. Who in the firm knows this? In terms of skills, try to get that
information by proxy. But some of the strategic decisions should be in the list of questions.
Stephen Ezell asked about innovation project management. Cohen wondered, what are the incentives
and what are the criteria of performance evaluation for individuals and projects? These trigger
incentives and may vary across type of innovation, even within the same firm. Reporting structure can
be like a matrix. Cohen offered that the key might be the basis for prioritizing projects. But economists
make many assumptions that may not hold in the real world. Is the key market performance or market
share?
Crowell works in life sciences and noted that great value was created at advancing stages of investment;
he asked about questions to capture that value (e.g. institutional value). The ability to quantify the
additional values around innovation is important. Cohen responded that others may be able to answer
those questions more concretely. There is an assumption-based approach, based on the notion that
innovation capabilities are not observable. They have used a finite mixture model4—which is new and to
be deployed in marketing and healthcare—that allows the ability to develop a measure of inventive (not
innovative) capabilities.
Stephen Ezell, Vice President, Information Technology and Innovation Foundation
Ezell added that in the book Ten Types of Innovation: The Discipline of Building Breakthroughs, author
Larry Keeley and others suggest 10 essential questions to be asked. Those might be helpful as well.
Sawhney added that different levels across firms garner different responses. Across functions,
management and marketing are more optimistic, and sales and engineering are less optimistic. There is
a lot of respondent bias because the whole thing is subjective. Crowell agreed. Sawhney added that his
4
In Sept. 2018, Cohen and his co-authors published a NBER paper applying a finite mixture model approach to
treat inventive capability as an unobserved, latent variable. This is in contrast to trying to directly measure
inventive capability using empirical proxies such as prior patents, R&D, innovation, or sales. Finite mixture model
models are similar to some types of unsupervised machine learning. A multinomial logit relates characteristics to
outcomes, and the coefficients can differ for firms of different capabilities (latent types). Cohen et al. state that
they leverage theory to identify the latent class associated with higher inventive capability, and use the predicted
firm-specific probability of belonging to that latent class as a measure of the firm’s inventive capability.
https://www.nber.org/papers/w25051
SRI INTERNATIONAL
36
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
work measures innovation in what, why, and how as well as relationships across 12 dimensions divided
into four quadrants. His data show that people who perform one function do not understand other
functions or the picture as a whole. For example, the functional head in sales does not necessarily
understand the logistics.
Cohen added that one study he worked on was conducted by phone precisely because they had to talk
to many people in different roles within a firm to find the right people to address the questions and to
minimize bias.
Jankowski added that knowing how businesses are organized at the functional level is critical for any
survey that reports quantitative data on R&D. Much R&D is performed by very large businesses, so what
is the cut-off size? Business with nine or fewer people typically do not have super-specialized roles, so
many of the staff may have a good idea of what’s going on. Large companies will have people with
specialized knowledge. National innovation statistics question capabilities and outcomes. Overall
business innovation activity might not be as important. The innovation statistics that come out of BRDIS
show that much business innovation activity did not have R&D.
Youtie asked Cohen a closing question about whether the business capability questions should be in or
out. Cohen responded, should they be in? Yes. Should they all be in? No. But a couple of years of
research is needed before settling on what two to four reasonable indicators would be.
Session 3: Business Innovation and Knowledge Flows5
Landefeld began his discussion of business innovation and knowledge flows. IP is key to addressing these
issues. Chapter 6 of the Oslo Manual provides definitions on knowledge flows, knowledge exchange, and
innovation diffusion.
Market transactions deliberately transfer from one knowledge area to another; and customers may
suggest improvements that are incorporated into products or processes; these will never be included in
transaction records. Offshoring of resources and jobs may be tech transfer or tax avoidance. Trade
regulations and tariffs are also very important to the flow of knowledge. Also, some flows are deliberate
whereas others are accidental. Sawhney felt that the term accidental sounded like a leak; unintentional
might be better. Landefeld acknowledged the point but noted that sometimes there are real leaks;
additionally, the language comes from the Oslo Manual. It is not necessary to agree with it, but that’s
why Landefeld is using it in the discussion.
Ezell suggested adding trade secrets as an object that captures knowledge. Value chains are critically
important today. It is difficult to collect information on value chains; the only way researchers have been
able to get those data is through the world input-output model.
Key questions for data collection address inbound knowledge flows to innovation. Landefeld added that
in this chapter, there are some more interesting distinctions about knowledge being collected (e.g.,
geography). He urged NCSES to consider those variables. Cohen asked about respondents considering
5
See Appendix for presentation slides.
SRI INTERNATIONAL
37
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
in-bound knowledge flows to innovation. He considered universities as a source of knowledge flows to
be important.
Landefeld discussed key questions for data collection:
•
•
•
•
What inbound knowledge flows contribute to innovation? (The Oslo Manual provides mainly a
qualitative yes/no response; however, NCSES has great experience in economic flows and can
use BRDS and other surveys as well as the Oslo Manual as a launching platform.)
Where are the firm’s collaborative partners for innovation by location? If firms provide access to
economic data, this would be very interesting in the GVC context.
What are the sources of ideas and information for innovation (excluding details on internal
sources)? Landefeld does not understand this: see Table 6.2, which includes internal and
external sources of innovation. Or, is this problem with the firm perceptions/bias?
What are the barriers to knowledge interactions? Landefeld noted that Table 6.9 provides an
interesting classification of barriers to transactions, many of which are amenable to policy action
(including less action).
Next, Landefeld discussed supplementary questions for data collection:
•
•
•
•
What are the sources of inbound knowledge flows by location? Landefeld notes that this
provides important GVCs, but it is hard to collect on surveys; it may only be possible with world
input-output tables.
What are the outbound knowledge flows? Landefeld notes that this is hard, but both sides of
flow and asset transfers are needed; build on existing economic data and surveys.
What are the channels for knowledge-based interactions between firms and Higher Education
Institutions and Public Research Institutions?
What are key uses of IPR for knowledge flows? Landefeld notes that tax and transfer pricing for
IP make this very hard but extremely important.
Jankowski added that they wanted to distinguish internal sources versus external sources. He added that
the key recommendations do not always reflect the ease of collection or the importance of the
information.
Landefeld added that he worries that NCSES will have an item response problem. The U.S. Census
Bureau surveys have those issues. There are various barriers, (e.g. regulatory, financing). The innovative
activities should be collected to address those areas. Youtie added that they had barriers to knowledge
interactions in her earlier survey; there were always barriers, such as time and money. They did not
learn many new things from the survey of manufacturers, and it did not always get to the new barriers.
Discussion - Business Innovation and Knowledge Flows
Jankowski mentioned the concerns listed in the manual, such as anti-trust policy enforcement, high
coordination costs, cost of dispute settlements, and financial resources. He added that time and
resources are the common barriers; however, time and money will not provide insight on much of
anything.
SRI INTERNATIONAL
38
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Cohen asked if the barriers were something best left to inference. Tangible measures are needed. He
asked if ties with universities, reportable behaviors, and other aspects should be considered? Jankowski
noted that, by design, the manual does not specify actual questions, but high-level coordination of
national statistical agencies ensures comparability of data.
Poole added that from a user and policy perspective, the value is in asking what the influencers of
innovation are. Landefeld added that there are other unintended outcomes, such as IP infringement,
counterfeiting a firm’s products, and patent trolls. Is this a serious barrier to innovation? It would be
helpful to find out what businesses think. Jankowski responded that NCSES staff would talk to some
potential respondents before designing the instrument.
Poole added that the source ideas should be labs and clusters. If these are not showing up as sources of
innovation, why is money being spent on them? Cohen responded that providing a measure and tying it
into policy can be subtle and difficult, even if the measurement problem is solved. If only observing, how
does one know if it is too little or too much? Poole added that the observations might help determine
better questions to ask.
Nico Thomas, Performance Analyst, National Institute of Standards and Technology
Nico Thomas added that this information is useful for the National Institute of Standards and
Technology because it focuses on industry questions about challenges; the issue with a question like this
is that it is hard to ask one question that is broad enough to address each industry. Some of the other
questions may not be as applicable.
Landefeld asked if the microdata would be available and if there would be access at Research Data
Centers (RDCs). Jankowski confirmed that researchers would have access to BRDIS data at RDCs.
Xu added that a significant challenge is tracking the follow-up funding. She asked if anyone else has had
success on that. In general, they do not know what happens to those projects. Ezell said that they may
not know the full chain, and it can be hard to collect those upstream and downstream pieces of
information; getting that information would dramatically improve the results. Outbound knowledge
flows consider both sides (i.e., flows and asset transfers), which he would rank as high priorities. Ezell
added that he may not believe the surveys, but he can look at the results.
Youtie added that the takeaway is that they are all important, but there are many barriers. She asked
Sawhney how he feels about these slides. Sawhney responded that co-creation is more than a flow; it is
more than transactional, it is persistent creation rather than an accidental leak. Landefeld added that
the rental payments and shared agreements are not mentioned in this chapter.
Cohen commented that it is not just co-creation; it is also co-development. He was surprised that they
found in his work that of the 49% of firms that report cooperative outside relationships, 60% do not
involve a license. There are other methods of invention and acquisition. Cohen noted that these data
were about the acquisition underpinning the respondent’s most important new product.
Jankowski asked Graham if he thought that information/IP rights flow should be included in the survey.
Graham responded that he does not know how to think about that. Is it a supplementary question from
the Oslo Manual? What does it mean? Jankowski responded that he would not focus on supplemental
SRI INTERNATIONAL
39
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
versus key questions; instead focus on chapter items of higher interest and value. Is this the type of
information we should bother to try to include in a survey with only around 15 core questions? Graham
responded that IP is observable in some places, as it underpins royalties and tangible flows. It teaches
something, but not everything. Crowell agreed that IP would be very important.
Graham added that this is a backdoor way to get at Landefeld’s concerns. Landefeld noted that
transaction data can be misleading (e.g., Apple has much lower tax rate through maintaining a
headquarters in Ireland).
The group broke for lunch and the Oslo Manual Revisions Briefing.
Session 4: External Factors in Influencing Innovation in Firms6
Youtie reconvened the discussion and introduced Stephen Ezell.
Ezell began his presentation on external factors, with the goal of presenting the content of the chapter
and soliciting the input of participants on the most critically important factors about the external
environment. The systems view of innovation includes a number of external factors. They are often
beyond the immediate control of the firm and may be either drivers or inhibitors of innovation. He
added that it is important to understand that external factors define the firm’s operating environment.
The external environment also defines the operating domain. A firm operating in the technology
innovation sector may be in a different position than a firm elsewhere in the economy. Innovation must
protect IP and manage R&D, design, innovation, and cloud computing services.
This chapter shows the main elements of external factors in the environment. The core content of the
chapter talks about the various components of human resources and other economic policies.
Moreover, spatial and locational factors define the locations. Markets include the markets for the firms’
products. Society and the natural environment include climate change, levels of trust, and innovation in
a society. The Oslo Manual suggests these questions for data collection:
Key questions for data collection include the following:
•
•
•
•
What is the firm’s industry and main market?
What is the firm’s competition and product market characteristics?
What government support is there for innovation?
What are the drivers or barriers to innovation?
Supplementary questions for data collection include the following:
•
•
•
6
What are the locations of business activities and value chains?
What is the main customer’s share of sales and the industry of the main customer?
What is the effect of regulations on innovation?
See Appendix for presentation slides.
SRI INTERNATIONAL
40
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
One factor the chapter does not seem to address is if firms can sell their innovations in foreign markets.
Ezell asked the room, what is the nature of the broad type of business that firms engage in? Who are the
competitors? There is a set of qualitative measures about potential intensity. Ezell’s boss has said that
the most important thing to ask for is new products introduced in the past 12 months. He focused on
intermediaries and digital platforms as focus of value creation.
Discussion - External Factors in Influencing Innovation in Firms
Ezell asked for comments on those proposed items. Graham asked if the respondents will be known.
Jankowski said, yes, but only to the collecting agency. Linking with other data sets is a goal within the
federal statistical system. Graham wondered about selecting a sample frame from Census or some other
source, making collection of some administrative data automatic.
Sawhney wondered how to most elegantly capture disruption. There are traditional competitors and
disruptive competitors; the latter play a different game. Sometimes, the actions or identity of your next
competitor cannot be predicted (e.g., Amazon getting into insurance).
Cohen added that the applied R&D provides three classes of variables that have different dimensions.
Everything is modular. Defining external can be pretty dicey. It requires a lot of attention to
measurement. It has only been in the past 15 years that it has been estimated in any rigorous fashion.
Aside from intensity of competition, there is demand, technological opportunity (i.e., technical
performance value for dollar in R&D), and finally, appropriability (i.e., the degree to which firms can
hope to profit from their innovations).
Cohen also brought up the issue of data linkages to other data sets. Many groups have endorsed this,
and he felt it was a positive recommendation this meeting should make to NSF. Landefeld noted that
there had been some very good work between the U.S. Bureau of Economic Analysis and the U.S.
Census Bureau, and it may be helpful to link to that data.
Cohen added that this is something that has been studied for decades; there are measures that are out
there. For example, one can ask how many firms can come up with a substitute to hurt profit. There are
other questions that could be framed. Ezell added that there are valuable questions around
competition. The master chart in the Oslo Manual provides more details on intermediaries and digital
platforms.
Sawhney added that the traditional analysis of innovation has looked at the firm as the unit of analysis
and the product as the output; but now the product is moving towards platform and the firm is moving
towards ecosystem. The problem is how to measure a network in which the product is the output. It
would be best to take the external view, considering it an ecosystem. Cohen asked, if the platform
enables markets where markets had not existed, how is that really different from the situation before?
The speed of information is faster, of course. But is this an evolution in the form of markets? Is this a
difference of degree or a difference of kind?
Sawhney responded that it is a difference of kind, if you are a marketplace operator; it is a two-sided
market with network externalities. Cohen suggested that there are also demand-side externalities.
SRI INTERNATIONAL
41
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Sawhney responded that there are both direct and indirect externalities. Today, there are marketplaces
mediated by platforms. The phenomenon of indirect externalities is new.
Jankowski responded that the Oslo steering group had discussed the idea of platforms but did not feel
that they had enough knowledge or actual experience to address it in the Manual. He asked, is it
important enough to pursue in the next survey, or should we wait to gain more knowledge from other
sources? Sawhney argued that it was not a binary question. Questions can be added today, even if they
are not as precise as we’d like. For example, are you dependent on other technology for how you work
and how you are paid?
Poole added that the Bureau of Labor Statistics created the Contingent Worker Survey that gets at some
of those questions from the worker’s perspective. Sawhney noted that someone may say, I am a oneperson company, but they write software for Uber, so actually they are involved in the platform. There
are many all-digital platforms like this. Cohen added that it is important to differentiate the digital
platform from other types. Sawhney suggested that it is necessary to move to where the profit is going
to be because value is de-materializing. Data and digitalization are capturing many things going forward.
Ezell continued, stating that government support for innovation should be a key component of the data
collection. He gave an overview of the main types of government policies supporting innovation: grants,
equity finance, debt finance, guarantees for debt financing, payments for goods and services, tax
incentives, and public infrastructure and services. It is helpful to tap into support for government
innovation activities.
Jankowski agreed that there are always those possibilities. There are other ways of linking those surveys
to bring in external non-Census data at a firm level to match survey answers with funding levels from
non-statistical agencies. It may be possible to use linkages as substitutes for some questions, but he is
not quite as optimistic about data linking activities becoming a reality soon. He would like for the
important items to include valuable real estate. Jankowski continued that most data linking has been on
the demographic surveys, rather than the establishment surveys.
Xu agreed. She added that this is important and from a grants agency perspective; she would like to see
these data, but having designed surveys, she knows that respondent burden must be considered. Linking
data is difficult, but can be done. The data community is affected by the same forces of disruption as
everyone else. She encourages NCSES to think about linkages ahead of time, as this will help researchers
in their work. She links self-reported data to two other data sets in her own research, but it is difficult.
Poole added that he is not delusional about what the issues are; unique identifiers are necessary. He
asked what variables are available to be linked? He does not just think of the federal level, there are
state data as well. Push the envelope on data linkage, but do not be misled about what the outcomes
will be. A big piece of this will be data quality improvements. This survey element is important, but a yes
or no is more important than a weighted number.
Reedy suggested that NCSES consider cycling survey components to add value to the data variables that
are already available for linkages across time, as has been done with the U.S. Census Bureau’s Annual
Survey of Entrepreneurs. He noted, however, that someone needs to go into the field to negotiate for
linkages and data improvements to support linkage.
SRI INTERNATIONAL
42
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Ezell continued, stating that there are various drivers across innovation, including working capital and
access to suppliers. Youtie asked, do we really need to ask for this information? When she has used CIS
in models, she has been okay using secondary data to control for how competitive a market is. Some of
the firms that she’s interviewed have had no idea of the number of competitors they have. In Georgia,
they think of the local market, not the global market.
Cohen asked, what are the impediments to knowledge flows? He does not want to ask firms, “Why do
they not innovate?” and “Is this a barrier?” Those questions are not valuable. In terms of external data,
competition is not always a great measure. In his work from the 1990s, it became clear that firms
believed that there were far more competitors than the actual number of businesses would suggest. The
broad question is to distinguish what to collect from what to infer.
Ezell added that there are several supplementary questions. The question is, is it worth investigating the
firm’s location and space domestically and globally in the survey? Youtie asked, could a firm answer this
question?
Poole said that some years ago the BLS tried to look at the issue of outsourcing. Either they were not
successful or did not report on it. It would be great to know the outcome of this work. Cohen asked,
what were the difficulties? Poole said that there were more bureaucratic issues than whether the data
was capturable. Budget resources in the statistical agencies are limiting this type of research.
Ezell continued, noting that there are questions about consumer responsiveness to innovation and trust.
He recommended not asking these questions, as the World Values Survey already collects similar data.
For the chapter overall, he noted that he liked the questions but not the way they were approached.
When Ezell asked about the importance of drivers, Cohen stated that the underlying science and
engineering base is a big driver not appearing on the list “Main elements of the external environment
for business innovation” or in the following presentation slides. Is there a profitable way to ask
something meaningful about the regulatory effects of product innovation? Youtie suggested asking
particular questions for particular industries. Asking it in a way that would be relevant to a particular
industry may enable them to answer yes. Graham felt that there would be no easy way to do this, and it
would lead back to inference.
Jankowski added that the chapter assumes that regulations can both inhibit or foster innovation. The
clean energy industry came about because of regulations. Graham asked but can that disinterestedness
be conveyed in a survey? He suspected not. Jankowski responded, let’s first look at the types of
questions to ask. Then, after the workshop, the cognitive folks can determine if there are questions that
cannot be asked without leading the respondent one way or the other. The question is, what should be
on the U.S. innovation survey? Do not worry about the other stuff yet. Ending up with 15 questions after
today may later change to only five after review and consideration; if that’s the case, everyone will
accept that.
Cohen added that, given that the role of regulation is extremely important, obtaining solid, objective
data on the effects is all the more important. Then, the question is, how is that achieved? There may be
an effect on innovation, but should it be left there? Folks should be careful and safe, but the social
SRI INTERNATIONAL
43
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
effects are far down the line, beyond NCSES’s remit. Also, he suggests not using the word regulation, as
it is a political hot potato. Perhaps talk about specific policies for different industries.
Landefeld added that yes/no questions are being asked for many variables, and it is necessary to think
about hard numbers. Over 75% of federally funded R&D is conducted by multinational companies. A
good chunk of multi-national data would be more valuable than 300,000 U.S. respondents. Ask them
these questions.
Youtie asked Landefeld what he recommends should be included and not included.
Landefeld responded, yes to firm’s industry, competition, and government support for innovation. He is
iffy about barriers. Data should be collected on barriers, but it is best not to ask explicitly if something is
a barrier.
Reedy added that a Kaufmann survey tried to assess competition from a regional versus a national
perspective and discovered that 25% of the economy will now say that Amazon is competition.
Session 5: Objectives and Outcomes of Business Innovation7
Reedy began the discussion by noting that he thought he’d answered many of the questions that had
come up while he reviewed the chapter, but after the day’s discussions, he was no longer so sure. He
gave an overview of the questions in the chapter. He added that this is much less prescriptive than the
previous version of the Oslo manual.
He reflected on a conversation he had with someone at the airport who works in the chemical
manufacturing industry. Reedy later inferred that this person was probably in charge of chemical
products. He talked about a product in the agricultural space that gave a 10% increase yield rate for
farms. Reedy noticed that the chemical industry employee jumped straight to what the Oslo Manual
describes as the object perspective. The industry employee then discussed difficulties getting his
products into the marketplace.
The average customer is around 65 years old, and, though they are becoming savvier, they do not
always believe statistics. Even with a 10% increase the industry employee doesn’t feel like he has a good
value proposition. In bad years, he must convince customers that their 50% lower yield would have been
60% lower without his product. He has some products with a 1-2% yield increase rate but does not feel
his customers will buy them because of the low increase rate, so they sit on the shelf. Reedy notes that
in this case there may be patents for inventions not taken to market that may still have tangential added
value at the firm but will not show up in measurements.
Cohen added, what this means is that no one sees the projects that fail. They will not be reflected at all
through the survey. They are very revealing about barriers and constraints. Some questions at a project
level could address those types of questions.
Jankowski added that the manual does address innovation activities, acknowledging that their result
could be ongoing or abandoned. Failure is an acceptable outcome. But comparing an innovative and a
7
See Appendix for presentation slides.
SRI INTERNATIONAL
44
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
non-innovative firm is different from comparing an innovative firm with innovative success and an
innovative firm with innovative failure. The ABS tries to capture the failed innovation activities as well as
non-innovative firm characteristics.
Reedy began going over the slides, starting with an overview of the definitions. Most of the chapter
used a similar terminology for objectives and outcomes, but Reedy was not convinced that the same
structure could be used for both; in some cases, it may not be possible or feasible. People often start
with outcomes for innovative firms, for which the manual suggests a three-year time period, then
transition into the objectives.
Jankowski added that the manual committee had left open the question of whether or not the threeyear time period was most appropriate, but the entire manual is built around a three-year time period.
Reedy agreed that three years seemed to be a good period to measure impact. Reedy then gave an
overview of the questions for data collection.
Key questions for data collection are as follows:
•
Innovation objectives and outcomes by area of influence (e.g., markets, production/delivery,
business organization, economy/society)
• Innovation objectives and outcomes for business strategies
• Innovation sales share in total business sales
Supplementary questions for data collection are as follows:
•
•
Counts of product innovations
Major impacts of innovations for markets
Discussion - Objectives and Outcomes of Business Innovation
Youtie asked about the three-year reporting period. Reedy responded that the three-year period allows
a good-sized window to measure impact. One year may not be enough time to measure impact. Cohen
asked about using impact as an outcome measure to determine how successful firms are at doing this?
It could also describe market-level intensity around innovation. Jankowski agreed with that
characterization. Sawhney added that they also perform three-year testing.
Reedy added that the manual can be applied to how outcomes are interpreted. Reedy cautioned that
some effects would only be observable to customers. Another area is the impact on society.
In terms of objectives, it was not clear to what extent the focus should be on the last three years.
Jankowski added that the manual committee was silent on that issue, but the starting point of
measurement is the original ideas that went into innovation activities. So, objectives are probably linked
to the three-year period, but it is probably not a tight timeline relationship. Reedy said that is how he’d
originally read the chapter, but now wonders, what if people want to look forward? What do are the x,
y, z objectives leading to in three years? The R&D lifecycle is likely longer than three years.
Cohen asked, what about the objectives today? Reedy said that the manual should be clearer. Overall,
this chapter talks generally about those variables. Sawhney added that the time horizon for innovation is
SRI INTERNATIONAL
45
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
a function of objectives and outcomes. The questions are, what are your time horizons, and how do you
allocate effort across different time horizons? Boeing might have a 60-year time horizon for an aircraft.
A small biotech firm might have tiny timelines.
Cohen added that this would require distinguishing different types of innovation. Some firms do not
care about the next year. This is not just a large-firm issue, even small companies may take 15 years to
get something to market. Ask about the distribution of projects across horizons or a metric of product
planning for product strategy. It should measure the life cycle. He suggested that having one question
that asks about lifecycle and another that asks about the types of innovation.
Reedy drew a diagram to illustrate that there are two types of taxonomies that apply to objectives and
outcomes, as follows:
•
•
By area of influence. These variables assess how much is it related to markets for products,
productivity and delivery, business organization, and economy, society, and environment
(quality of life and well-being).
By strategy. These variables assess, positioning, internal capabilities, and positioning firms
within the market.
Youtie mentioned that this is the first group of questions that she cannot relate to because they are
squishy, except for innovation sales as a percent of total business, and counts of product innovation.
Xu added that social impact is still a factor. Some nonprofits have both economic and social goals
(double or triple bottom line). Even if all these questions are not covered, consider asking, which do you
prioritize?
Jankowski cautioned against making presumptions about what the Manual steering group wanted. The
Manual’s questions were distilled from activities seen in surveys, Eurostat, and the literature. It is not so
much a hierarchy as a general list. The committee discussed combining outputs and outcomes, but
Jankowski did not like the idea. Environmental and societal outcomes may be squishy, but many
businesses cite these as part of their goals (objectives). They want to be good corporate citizens; make
money, but also have a societal impact. But it is a big basket; there was no assumption that all these
questions would be asked.
Poole felt that if this is added in the survey, people will say what they think others want to hear. It would
be useful as qualitative research. Unless someone can write questions that address those topics
appropriately (without respondents telling you want to hear), issues will have biases. Friends in the
corporate world talk about these things, but he does not know the cost of public relations.
Cohen asked if it worth inquiring about intermediate objectives? Often respondents do not know the
market or the demand and need to learn about these things. Ultimately the objective is to make money.
But it may be useful to inquire about more intermediate objectives. Moreover, how does one get to
objectives other than strictly commercial ones (e.g., the malaria vaccine). Jankowski responded that the
Oslo Manual addresses both response biases and non-market aspects of innovation.
SRI INTERNATIONAL
46
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Reedy added that he works with many students, mainly undergrads, and most have a hybrid business
vision, with a high level of intentionality about societal impact. Jankowski was not inclined to include
these on the first survey round. Youtie noted that these goals come from the European Union’s Horizon
2020 plan, but they are not yet really in the language of U.S. policy. Perhaps, it will happen in a few
years.
Cohen added that there may be a way to find out if they actually spend money on these things. Dollars
dedicated to these objectives may be more concrete below the public relations level. How much funding
do you dedicate to these objectives? Poole asked if that would require forcing a choice? People will pick
an attractive sounding answer because there is no cost for doing so. He suspects this would need a lot of
testing.
Xu suggested a potential way to check: ask how they measure their outcomes. Investments are
measurable. Graham suggested finding out what managers bonuses are linked to.
Reedy concluded by saying he was concerned about the granularity of the questions and would not
recommend implementing all of them; he also suggested asking at the firm rather than object level.
Discussion Summary, Recommendations, and Conclusion
In reviewing and summarizing the day’s discussions, Youtie said that the group appeared to have
become more intense and probing as the day progressed and that the business innovation activities
discussion might have happened too soon. However, generally, people are on board with asking the
questions in this section, especially if there are dollar signs associated with the expenditures and funds.
Cohen liked the idea of supplementary questions asking about number of innovation projects by activity,
especially innovation management activities. There were some issues about innovation management
activities. Poole suggested asking some human capital questions to get at capacity in management. For
example, ask how many people are doing this kind of work?
The business capabilities section had the second lowest ranking with the whole group. Because people
felt that the results were not at the firm level, it would be better to have another level to get new
information. Overall, the participants felt that several of the indicators would be useful. Youtie finds
some of the questions useful and has used them in the past herself (e.g., tertiary education).
Youtie added that there could be some questions in the CIS that are useful. The first demographic
questions will be asked anyway; tertiary education is a standard variable for innovation capability. The
design capability that Xu and Sawhney mentioned seemed vague, and participants could not get their
minds around them in a practical way. This was the chapter with the least amount of consensus. Poole
added that it might be easier to better understand a firm’s capabilities by looking at the occupations of
its employees. Youtie added that everything is on the table, so yes, this would provide a better sense of
what they are doing. Cohen added that experience and years of experience might cut a couple of
different directions. There is no standard CIS question about this. Landefeld added that labor data can
also provide information on the mix of occupation, rather than education.
Youtie noted everyone’s great interest in the chapter about Business Innovation and Knowledge Flows
section. She summarized that Landefeld’s priority is anything that can be monetized. Sawhney felt it got
SRI INTERNATIONAL
47
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
some but not all of co-creation. Of the suggested questions, the participants only excluded barriers to
knowledge interaction. Cohen suggested that researchers should not confine themselves solely to those
variables and consider spillovers, co-development activities, and upstream knowledge as well.
Youtie continued by summarizing the discussion of external factors presented by Ezell. The group liked
most of the questions, except for drivers and barriers to innovation. The group discussed linking to other
data sources. But it is also more difficult if local and national sources are preferred. (There is no magic
identification number to link everything). The group is still on the fence about drivers and barriers to
innovation. Participants understood that regulation could be positive as well as negative for innovation,
but it problematic to ask. External drivers to innovation was not on the list and it should be added.
NCSES should also consider the Yale survey.
Lastly, the group discussed the societal benefits in addition to the standard sales share. Youtie
confirmed that the group felt that questions in these areas should be asked, and they should figure out a
way to ask them.
Youtie asked for any final questions or feedback for Jankowski.
Sawhney asked about any next steps and the outcomes of this workshop. Jankowski said that they are
not well planned out yet. They are awaiting the manual release, which will be a major input on the
variables that NCSES will take into consideration. They will review the discussion feedback and contact
businesses about the topics that NCSES asks about. There are questions that NCSES will ultimately set
out to test with businesses. Next year, their colleagues at Eurostat will launch a survey that will include
Oslo 2018 input. They are doing the testing now. Moreover, Japan is launching a survey this winter that
addresses some of these concepts. Stats Canada will also be launching a survey in a year and a half. In
about a year, NCSES will have an instrument with Oslo 2018 questions; that survey will be launched in
2020, with a 2021 or 2022 data release.
Sawhney acknowledged that the following question would be unfair. Noting that in the past, when
Microsoft sold Office, they would release the product, not see what customers were doing with it, and
not release an update for another three years. But now, telemetry is built into the software and
Microsoft sees what customers are doing as the use the product. The software is now a service. That is
an analogy. Participants spent the day talking about the questions, but what is now known about the
return on investment for the questions in the survey, and what are the use cases? What will users do
with this information? That should be the focus of the survey questions; end users should be the key
focus of the survey adjustments.
Jankowski stated that NCSES would interact with the workshop participants again, and share NCSES’s
conclusions. He added that one of the biggest disappointments is that the workshop did not include
more people from the Federal Government to whom organizers would have reached out. This should
serve policy stakeholders as well.
Poole added that any increase in granularity in geography would increase the usage dramatically. There
is a lot of interest in these data the state and sub-state level, but the national data does not tell the story
that they need.
SRI INTERNATIONAL
48
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Landefeld added that NCSES might need to talk one-on-one with analysts. At the analyst level, there is a
lot of good feedback (in the government sector).
Cohen added that missing from the discussion was the adoption and diffusion of platforms, and nonhigh-tech innovations that have a huge economic impact (e.g., the supermarket, containerization). He is
concerned about that being captured, though this may be beyond what the Oslo Manual covers.
Jankowski responded that the Oslo Manual explicitly states that the innovation does not have to be high
tech, though the last version was clearer about this. The manual is meant not to have a bias toward
technology.
Youtie added that there was a technology use survey by Census in the 1990s that addressed that issue
as well, but they do not field it anymore.
Jaffe noted that on his own surveys, when he had asked respondents to answer questions with
subcategories, he would usually roll them back up for the analysis. However, this would be useful for
geographic breakouts. The subcategories take a lot of real estate and time.
Youtie asked for any final closing comments and suggestions for NCSES.
Mark Crowell, Executive Entrepreneur in Residence, Eshelman Institute for Innovation, UNC Eshelman
School of Pharmacy
Crowell stated the ways universities manage their innovation assets is shifting. The UNC Pharma school
is becoming much more like companies in developing partnerships and investments and taking products
almost all the way to market. There’s a need to capture that value as well.
Sawhney noted that Google came out of Stanford.
Youtie addressed all participants and thanked the stakeholders for their participation and comments.
She added that there had been wonderful diversity in the responses today. Youtie asked that the
stakeholders keep the group informed on any interesting developments in their work on innovation.
Jankowski will keep the group informed as NCSES makes future developments.
SRI INTERNATIONAL
49
Attachment E
TO 19 | INNOVATION EXPERT PANEL
WORKSHOP SUMMARY
Appendix: Presentation Slides
Continuing Examination of Innovation in the U.S. Business Sector by NCSES
John E. Jankowski, Program Director of NSF’s Research and Development Statistics Program, presented
NCSES’s Continuing Examination of Innovation in the U.S. Business Sector. He provided an overview of
the organizational background, NCSES innovation surveys, the Oslo Manual, and the purpose of the
workshop.
Session 1: Business Innovation Activities
Jan Youtie, Georgia Tech University, Workshop Chair, presented Discussion 1: Business Innovation
Activities. Business Innovation Activities are activities that firms may undertake in pursuit of innovation,
including R&D; engineering, design, and other creative work; marketing and brand equity activities;
intellectual property (IP); employee training; software development and databases; investment in
buildings, machinery, equipment and other tangible asset, and innovation management activities.
Session 2: Business Capabilities for Innovation
Wes Cohen, Duke University, presented Discussion 2: Business Capabilities for Innovation. Business
capabilities for innovation include the knowledge, competencies, and resources that a firm accumulates
over time and draws upon in the pursuit of its innovation objectives. Collecting data on business
capabilities is important to understand the effect of innovation on firm performance and why some
firms innovate, and others do not.
Session 3: Business Innovation and Knowledge Flows
Steve Landefeld, Bureau of Economic Analysis (Retired), presented Discussion 3: Business Innovation
and Knowledge Flows. Knowledge flows include how knowledge is accessed, how it is deployed, and
how it is exchanged between firms and other actors engaged in the innovation system.
Session 4: External Factors in Influencing Innovation in Firms
Stephen Ezell, Information Technology and Innovation Foundation, presented Discussion 4: External
Factors in Influencing Innovation in Firms. External factors can influence a firm’s incentive to innovate,
the types of innovation activities that it undertakes, and its innovation capabilities and outcomes.
External factors may also be the object of a business strategy, public policy, or concerted social action by
public interest groups.
Session 5: Objectives and Outcomes of Business Innovation
E.J. Reedy, University of Chicago, presented Discussion 5: Objective and Outcomes of Business
Innovation. The planning and development stage for an innovation includes the identification of a set of
one or more objectives that the innovation is expected to achieve. The objectives can concern the
characteristics of the innovation itself, such as its specifications, and its market and economic objectives.
The outcomes of an innovation are similar to the objectives but consist of the innovation’s realized
effects. These can include unexpected effects that were not identified among the firm’s initial
objectives.
SRI INTERNATIONAL
50
Attachment E
NCSES’s Continuing Examination of
Innovation in the U.S. Business Sector
John E. Jankowski
Program Director, R&D Statistics
(jjankows@nsf.gov)
Innovation Workshop
October 16, 2018
10/16/18
National Science Foundation
National Center for Science and Engineering Statistics
www.nsf.gov/statistics/
Attachment E
Structure of Presentation
• NCSES Organizational Background
• NCSES Innovation Surveys (brief history)
• Oslo Manual Revisions and Purpose of this
Innovation Workshop
10/16/18
Attachment E
National Center for Science and Engineering
Statistics (NCSES)
•
NCSES is a Division within the Directorate for Social, Behavioral and
Economic Sciences within the National Science Foundation.
•
NCSES is one of the 13 Federal Government’s principal statistical agencies.
•
Created by Congress in 1950, NSF is an independent federal agency “to
promote the progress of science; to advance the national health, prosperity,
and welfare; to secure the national defense…
• NSF is mandated ...to provide a central clearinghouse for the collection,
interpretation, and analysis of data on scientific and engineering
resources and to provide a source of information for policy formulation
by other agencies of the Federal Government…
• This mandate has been fulfilled by NCSES and predecessor agencies.
•
Our role to collect information on the science and engineering enterprise,
including on competitiveness, was further codified in the America Competes
Reauthorization Act of 2010 (December 6, 2010) which “created” NCSES.
10/16/18
Attachment E
NCSES Innovation Surveys
•
•
Surveys conducted in partnership with the Census Bureau
Mandatory survey; item nonresponse for innovation questions historically ~10%-20%
•
Business R&D and Innovation Survey (BRDIS) (Reference years: 2008-2016)
–
–
–
–
All industries with 5 or more employees; primarily a survey for R&D activities;
Sample size =~42,000; 2016 Response rate =~80%
Innovation data collected 2008-2016
Starting with 2017 data collection, innovation data no longer collected
•
•
•
Microbusiness R&D and Innovation Survey (MicroBRDIS) (Reference year 2016)
–
–
•
All industries with 9 or more employees; remains primarily an R&D survey
Survey now called the Business R&D Survey (BRDS)
All industries with 1-9 employees with a focus on 1-4; small business owner survey with questions on
R&D and innovation and owner characteristics
Sample size =~200,000; 2016 Response rate =~76% for 1-4 and ~78% for 5-9
Annual Business Survey (ABS) (Reference years 2017-2021 or longer)
–
–
–
–
–
10/16/18
All industries with 1 or more employees
Sample size in 2017 = 850,000; non benchmark year samples = 350,000
Rotating module approach combining previously stand-alone surveys
Collects R&D data for businesses with 1-9 employees (previously Micro-BRDIS)
Collects data on business owner characteristics (Survey of Business Owners), financing, technology
and intellectual property and (for 2017-2019) innovation
Attachment
E
National Science
Foundation
Division of Science Resources Statistics
What was Innovation per the old Oslo Manual
(2005) / European Community Innovation Survey?
During [the recent 3-year period], did your enterprise introduce…
• new or significantly improved goods or services with respect to their
capabilities, user friendliness, components or sub-systems?
(Product innovation)
• a new or significantly improved production process, distribution
method, or support activity for your goods or service?
(Process innovation)
• new business practices; new methods of organizing work
responsibilities and decision making; or new methods of organizing
external relations with other firms or public institutions?
(Organizational innovation)
• significant changes to the design or packaging of a good or service;
new media or techniques for product promotion; new methods for
product placement; or new methods of pricing goods or services?
(Marketing innovation)
10/16/18
Attachment E
2009-16 BRDIS Innovation Questions
•
•
BRDIS was to be a platform for innovation (including CIS-like) questions
Measures only product and process innovation
Yes/No questions: During 2007-2009, did you introduce…
- New or significantly improved goods? …services?
- New or significantly improved methods of manufacturing or producing goods or
services? …logistics, delivery or distribution methods? …support activities?
•
•
No specific guidance on defining innovation
Does not define innovative activity per se
(for example, does not include performance of R&D as an innovation activity)
•
Degree of novelty: Innovation sales as a percentage of total sales
- that were new to one of your markets
- that were only new to your firm
- that were unchanged or only marginally modified
10/16/18
Attachment E
Revised Approach to 2016 Micro-BRDIS
•
•
•
•
Respondents to this survey were very small businesses (1-9 employees) and
cognitive testing found that wording different from that used in BRDIS
worked better for them.
Specific elements of innovation from the Oslo Manual definitions were
incorporated into the question items.
Included questions on product, process, organizational and marketing
innovation. (BRDIS asked only about product and process innovation)
Each aspect of innovation (for each form of product, process, marketing, and
organization) was broken into individual yes/no questions.
10/16/18
Attachment E
Expanded Approach to reference year 2017 ABS
(based substantially on 2016 CIS)
•
First NCSES stand-alone innovation survey, which still did not
stand-alone
• Launched in mid-2018 and currently being fielded
• Specific innovation topics from the 2005 Oslo Manual were asked.
• Definitions were incorporated into the yes/no question items on
product, process, organizational and marketing innovation
• Included questions on
–
–
–
–
–
–
–
10/16/18
Novelty of innovation (new to firm vs new to market)
Percentage of innovation sales in total sales
Importance of various business strategies
Types of innovation activities engaged in
Results of innovation activities (abandonment, successful, ongoing)
Type and location of cooperation partner
Factors interfering with business innovation
Attachment E
Reference year 2018 Annual Business Survey
(based somewhat on 2018 CIS)
•
•
•
•
To be launched mid-2019 with reference year 2018
Extremely scaled back version – in order to incorporate findings from
this workshop and methodological testing
Incorporates new 2018 Oslo Manual definitions of innovation
(upcoming slides)
Includes questions on
– Percentage of innovation sales in total sales
– Types of innovation activities engaged in
– Total costs of innovation activities
10/16/18
Attachment E
2018 Oslo Manual Revision Process
•
•
•
•
•
•
•
•
Three year international effort, led by the OECD’s Working Party of National
Experts on Science and Technology Indicators (NESTI) and the European Union.
Resulted in the 4th edition of the Oslo Manual (2018), to be formally launched in
Paris on October 22nd.
Provides general definition of innovation and revised concepts for measuring
business innovation (upcoming slide)
Adaptation of terminology to be consistent with the System of National Accounts
(SNA) and with other statistical manuals such as the Frascati Manual (for collecting
R&D information)
Acknowledges cross-cutting issues: Digitalization and Globalization
Specific provision for experimentation in terms of types of questions and types of
data (e.g., administrative data)
Includes expanded methodological guidance
Recognizes the object-based approach for measuring innovation (focal innovation),
but emphasizes subject-based approach
Broad expansion of topic areas—basis for the 5 sessions of our Workshop
10/16/18
Attachment E
Contents of 2018 Oslo Manual
(What to Prioritize in the U.S. 2019 ABS Collection?)
•
•
•
•
•
•
•
•
•
•
•
Chapter 1. Introduction to innovation statistics
Chapter 2. Concepts for measuring innovation
Chapter 3. Concepts and definitions for measuring business innovation (John)
Chapter 4. Measuring business innovation activities (Jan)
Chapter 5. Measuring business capabilities for innovation (Wes)
Chapter 6. Business innovation and knowledge flows (Steve L)
Chapter 7. Measuring external factors influencing firm innovation (Stephen E)
Chapter 8. Objectives and outcomes of business innovation (E.J.)
Chapter 9. Methods for collecting data on business innovation
Chapter 10. The object method for innovation measurement
Chapter 11. Use of innovation data for statistical indicators and analysis
Each chapter has recommendations for key indicators and supplementary indicators
10/16/18
Attachment E
General Definition(s) of Innovation (1)
•
An Innovation is a new or improved product or process (or combination
thereof) that differs significantly from the unit’s previous products or
processes and that has been made available to potential users (product) or
brought into use by the unit (process).
–
•
This definition is to be the basis for measuring innovation in all sectors of the economy.
A business innovation is a new or improved product or business process
(or combination thereof) that differs significantly from the firm's previous
products or business processes and that has been introduced on the market
or brought into use by the firm.
– A product innovation is a new or improved good or service that differs
significantly from the firm’s previous goods or services and that has been
introduced on the market.
– A business process innovation is a new or improved business process for one
or more business functions that differs significantly from the firm’s previous
business processes and that has been brought into use by the firm.
10/16/18
Attachment E
General Definition(s) of Innovation (2)
•
There are two major types of innovation by object: innovations that change
the firm’s products (product innovations), and innovations that change the
firm’s business processes (business process innovations)
– Product innovations can involve two generic types of products: goods (tangible
objects) and services (intangible activities). (Note: Knowledge-capturing products,
as identified in the SNA, can have the characteristics of either a good or service
and concern the provision, storage, safekeeping, communication and
dissemination of digital information that users can access repeatedly.)
– There are six types of business process functions: (1) methods for producing
goods or providing services; (2) distribution and logistics; (3) marketing and sales;
(4) information and communication systems; (5) administration and management
activities; and (6) product and business process development (including activities
to identify, develop or adapt products or a firm’s processes).
•
The 2018 Oslo Manual cross-walks this revised set of innovation types to the
2005 Oslo Manual’s four types of innovation.
10/16/18
Attachment E
Thank you and let’s have at it!
John E. Jankowski
jjankows@nsf.gov
10/16/18
Attachment E
Discussion 1 – Business Innovation
Activities
National Center for Science and Engineering Statistics
National Science Foundation
Innovation Workshop
October 16, 2018
National Science Foundation
National Center for Science and Engineering Statistics
www.nsf.gov/statistics/
10/16/18
Attachment E
Definition
• All developmental, financial and commercial
activities that are intended to result in an
innovation
– Can be performed in-house or obtained from external
organizations
– May be postponed or abandoned during the
observation period due to multiple reasons
– Can create knowledge or information that is not used
to introduce an innovation during the observation
period. This includes knowledge from activities that
fail to meet their primary innovation goals
10/16/18
Attachment E
8 Types of Business Innovation Activities
Research and experimental development (R&D) activities
Engineering, design and other creative work activities
Marketing and brand equity activities
Intellectual property (IP) related activities
Employee training activities
Software development and database activities
Activities related to the acquisition or lease of tangible
assets (buildings, equipment, machinery and other tangible
assets)
• Innovation management activities
•
•
•
•
•
•
•
10/16/18
Attachment E
Innovation Activity
R&D
Includes
*1) novel, 2) creative, 3) uncertain outcome, 4)
systematic, 5) transferable, reproducible
*Basic, applied, development
Engineering, design, other creative work *5 R&D criteria not met
*Testing, set-up, tooling, production, quality
*Service firms use design for look, use
*Engineering may not be for innovation
Marketing, brand equity
*Pricing, placement, promotion advertising
*Usually if linked to a new, improved product
IP
*Patents, utility patents, industrial designs,
trademarks, copyright, IC designs, plants,
trade secrets for new products, process
Employee training
*Not for existing products, general skills
Software development, databases
*Not to maintain IT system, supporting
existing systems, routine functions
Tangible assets
*Buildings, instruments, equipment,
computer hardware (not replacement)
Innovation management
*Funding, organization, external
collaborations, performance
10/16/18
*For innovation active firms
Attachment E
Key questions for data collection
• Qualitative data on whether or not each of the 8
activities were conducted, regardless of the
reason, identifying whether innovation was the
aim
• Whether or not each activity was conducted inhouse or procured from external organizations
• Total expenditures for each of seven activities
– expenditures for innovation management not
recommended
• Funding sources for innovation
10/16/18
Attachment E
Source specifically relevant to innovation
Innovation Activity
Any activity, in- Activity conducted Activity procured
house or
in-house for
from external source
procured
innovation
for innovation
R&D
Engineering, design, other
creative work
Marketing, brand equity
IP
Employee training
Software development,
databases
Tangible assets
Innovation management
10/16/18
Oslo Manual, 2018, p. 82, Table 4.1
Attachment E
Expenditure data on activities & specifically
relevant to innovation
Innovation Activity
Total expenditures
(all firms)
R&D
$
Expenditures for
innovation (innovationactive firms only)
$
Engineering, design, other
creative work
Marketing, brand equity
$
$
$
$
IP
$
$
Employee training
$
$
Software development, databases $
$
Tangible assets
$
$
Firms that perform R&D as a service to other firms include under "Total
expenditures" and their own innovations under "Expenditures for innovation"
10/16/18
Oslo Manual, 2018, p. 84, Table 4.2
Attachment E
Accounting method for collecting data
specifically related to innovation
Expenditure on
1
R&D
1a
Intramural R&D
(include personnel cost, materials and other supplies,
purchase of capital goods for R&D activities)
1b Extramural R&D (purchase of R&D from other parties)
2
Innovation activities other than R&D
2a
Own personnel (excl. cost of R&D personnel)
2b Services purchased from other parties (excl. purchase of
R&D services)
2c Materials and supplies (excl. materials/supplies for R&D)
2d Capital goods (purchased tangible, intangible assets)
10/16/18
Oslo Manual, 2018, p. 86, Table 4.3
Total expenditures for
innovation (innovationactive firms only)
Attachment E
Supplementary: Human Resources by
Activity (FTE)
• Person-months for each activity
– Activities most likely to involve labor costs
– Small firms in service industries
• Can combine with monthly salary data
10/16/18
Attachment E
Disaggregation by Sources of Funds
•
•
•
•
•
•
•
•
•
•
•
Own funds (retained profits or income from asset disposal)
Transfers from affiliated firms (holding, subsidiary or associated companies located
in the domestic country or abroad)
Customer orders (including procurement contracts from governments or
international organizations)
Shareholder loans
Debt funding from commercial loans (banks, credit cards etc.), overdraft facilities,
suppliers' credit
Loans from governments
Loans from international organizations
Equity from private equity or venture capital firms, business angels or other
individuals (family and friends)
Grants or subsidies from governments, international organizations, NGOs, etc.
Bonds and obligations
Other sources (e.g. Crowdfunding)
10/16/18
Attachment E
Supplementary questions for data collection
Additional information for specific activities, such
• as whether R&D activities are conducted
continuously or on an occasional basis
• Follow-on activities
• Planned innovation activities and expenditures
10/16/18
Attachment E
Supplementary: Number of innovation
projects by activity
•
•
•
•
Undertaken during period
Completed during period
Ceased before completion during period
Ongoing projects by the end of period
10/16/18
Attachment E
Supplementary: Follow on activities after
implementation, but within the period
• Marketing: promote the sale of a product
innovation in the market
• Training: training of employees, customers in the
use of product or business process innovations
• After-sales services: installation, updating and
repair services, guarantee and return,
information
10/16/18
Attachment E
Supplementary: Planned activities
~Firm plans to conduct any innovation activities in the one or
two years after the reference year?
• Yes
• No
• Don’t know
~Total innovation expenditures compared to the reference
year (if any) are expected to
• Increase
• stay the same
• decrease
10/16/18
Attachment E
Summary for Prioritization
• Qualitative data on whether or not each of the eight activities were
conducted
– Whether the activity was conducted for innovation
• Whether or not each activity was conducted in-house or procured
from external organizations
• Total expenditures for each of seven activities
• Total innovation expenditures using the accounting method (shown
for R&D)
• Funding sources for R&D
• Supplementary questions
–
–
–
–
10/16/18
R&D continuous or occasional
Innovation expenditures by funding source
Follow-on activities
Planned innovation activities and expenditures
Attachment E
Discussion 2 – Business Capabilities
for Innovation
National Center for Science and Engineering Statistics
National Science Foundation
Innovation Workshop
October 16, 2018
National Science Foundation
National Center for Science and Engineering Statistics
www.nsf.gov/statistics/
Attachment E
Definition
• Knowledge, competencies and resources that a firm accumulates
over time and draws upon in the pursuit of its objectives
– Resources controlled by a firm
• Human, physical and intangible capital, accumulated experience, internal
sources of finance, ownership status, and access to outside resources
– General management capabilities of a firm, including capabilities
related to managing innovation activities
• Business strategy, for example, cost competitiveness or focus on quality
features (e.g. functionality, durability, flexibility of use, etc.)
• Organizational and managerial capabilities
• Characteristics of the business owner and top management
• Innovation management capabilities
• IP management and appropriation
• Where do these capabilities/resources reside?
=>Who can respond knowledgeably to questions—Upper management?
Innovation project managers? R&D managers?
2
Attachment E
Definition (continued)
– Workforce skills and how a firm manages its human
capital
• Workforce qualifications, occupational structure and
competences
• Human resource management
– Ability to use technological tools and digital resources,
with the latter providing an increasingly important
source of information for innovation
• Technical expertise
• Design capabilities
• Capabilities related to digital technologies and data analytics
3
Attachment E
Key questions for data collection
•
Number of employed persons (FTEs)
•
Total sales
– Also by business unit
– Also business unit sales—THE key correlate of R&D
•
•
•
•
Firm age by year the firm began business activities
– Date of entry by market
Firm ownership status (stand alone, part of a national group, part of a
multinational group)
Geographical distribution of sales (local, national, international markets)
Export share of sales
– By business unit
•
Importance of cost versus quality for the firm’s competitive strategy
•
•
Share of employed persons with a tertiary education
Business level of design capability
– Suggested measures: % process vs. product R&D, and, for latter, % dedicated to developing
altogether new products vs. product improvement
– ?
•
Cash flow?
4
Attachment E
Supplementary questions for data
collection
• Family owned firm status
• Number of product lines
• Innovation management: responsibility for innovation
within the firm
• Innovation management: methods to support internal
knowledge exchange
• Number of employed persons by major field of
education
• Technical expertise in emerging technologies
– Only?
• Digital competences
5
Attachment E
Issues/questions?
• Questions bearing on “business capabilities”
relegated to “supplementary questions?—Why?
• To understand drivers of innovation and innovative
performance, do not focus exclusively on innovating
firms—Surveying imitators and firms that do neither?
• What question do we want to answer with measures
of capabilities?
– Why are we doing this?
– If it’s the role of capabilities in driving firms’ innovative
performance, very difficult due to endogeneity
• Per above, to analyze effect of capabilities, need to identify not
only (N-t-M) innovators, but also imitators (i.e., N-t-F
“innovators”), as well as firms that do neither
6
Attachment E
• More questions
– Capability to do what?
• Starting with the coarse distinction between invention and
innovation (i.e., commercialization of new products)=>
– Invent? Develop?
– Commercialize?
» By bringing manufacturing, marketing and sales capabilities
to bear
• Example (Arora, Cohen and Cunningham, 2018)
– Firms with weak “inventive” (i.e., upstream) capability, benefit
from proximity to others’ inventions, while those with strong
inventive capability do not, though they benefit from proximity to
universities
• May want to distinguish capabilities by classes of activity
– Managerial
– Technical (invention and/or development)
– Commercialization (Mfg./Sales/Mkting)
7
Attachment E
• How to measure performance? And what kind?
– Financial performance?
• At what level? B.U.? Firm?
– Technological success?
– Commercial success?
– Learning?
• Coarsely distinguishing between invention and
innovation, move away from assumption that firms are
self-sufficient in innovation
– In U.S. mfg, 2010, 49% of innovating manufacturing firms
acquired the invention (i.e., overall concept, prototype)
underlying their most important new or significantly
improved product from an external source (Arora, Cohen and
Walsh, 2016)
=> Highlights importance of firms’ ability to search for and
manage external relationships
8
Attachment E
• Sample questions for “managerial capabilities” at upper
level of mgt
– Objectives?
– Incentives? Of what form?
• Pay, promotion, nonpecuniary, etc.?
– Criteria for personnel and project performance evaluation?
– Reporting structures?
• To whom do innovation project managers report?
• Delegation of decision rights to project managers?
– Basis for prioritizing projects
– Typical timelines?
– And for many of the above, would want to know variation
across business units and projects
• Also useful to distinguish across types of innovation?
– E.g., Incremental versus altogether new products, perhaps for
new markets
• Questions at level of innovation project managers?
9
Attachment E
A different approach: Assume innovation
capabilities are unobservable (Arora, Cohen and
Cunningham, 2018)
• Latent variable, discrete choice model
– Employs a finite mixture model employed in marketing
and health care literatures
• Assumes the data generated reflect mixture of
two or more distributions.
– Each data point has unknown probability of belonging
to one of the distributions.
– The unknown probabilities and the set of coefficients
from the model predicting innovation outcomes for
each distribution are jointly estimated.
10
Attachment E
• We distinguished firm types by their underlying
capability, allowing firm characteristics (e.g., size) and
supply of external inventions and knowledge,
respectively, to relate differently to innovation
outcomes (i.e., innovation, imitation, neither) for two
classes of firms distinguished by inventive capability
=> Simultaneous estimation of
– Probability that firm belongs to high (v. low) capability class
– Class-conditional probabilities of innovation outcomes
(innovate, imitate, neither)
– Class-specific coefficients
• Method provided a measure of “inventive capability”
(i.e., the probability of high inventive capability) that
can be used in subsequent analyses.
11
Attachment E
Discussion 3 – Business Innovation
and Knowledge Flows
National Center for Science and Engineering Statistics
National Science Foundation
Innovation Workshop
October 16, 2018
National Science Foundation
National Center for Science and Engineering Statistics
www.nsf.gov/statistics/
Attachment E
Definition (from Oslo Manual)
•
Knowledge flows encompass the deliberate or accidental transmission of
knowledge
– Deliberate: asset purchase or payment for services such as rental, royalty, or
profit)
– Accidental: Spillovers, externalities, public goods
•
Knowledge exchange (sometimes referred to as knowledge transfer) is the
deliberate transmission of knowledge from one entity to another
– Economic analysis needs prices and sales (weights)
•
Innovation Diffusion
– The process and the outcomes of innovation diffusion are of policy and
research interest because diffusion amplifies the economic and social impacts
of ideas and technology (Social vs. private rates of return)
– Innovation diffusion can also create knowledge flows that lead to further
innovations (spillovers/public goods)
– The expected speed and nature of innovation diffusion also shape the
incentives to innovate. (payback and ownership)
2
Attachment E
Definition (continued)
– Knowledge Flows (OSLO Manual - Ch
• Knowledge can be captured by “objects” such as databases,
software routines, patents, publications, public presentations
and know-how (embodied technical change)
• All organizations, agents or individuals can be involved in
knowledge flows (market transactions, GVC/MNE, World I-O)
– “Firms (internal/external, customers , universities, investors, experts
and other potential sources of knowledge” OSLO
3
Attachment E
Key questions for data collection
• Contribution of inbound knowledge flows to innovation (OSLO
mainly qualitative yes/no; NSF has lots of experience
economic flows, use BRDS other surveys and OSLO as
launching platform)
• Collaboration partners for innovation by location
– Really interesting in GVC context if can get economic data
• Sources of ideas and information for innovation, but excluding
details on internal sources
– Don’t understand: see Table 6.2, which includes internal/external
sources of innovation; or is this problem with own firm
perceptions/bias
• Barriers to knowledge interactions
– Table 6.9: Interesting classification of barriers to transactions, many
amenable to policy action (including less action)
4
Attachment E
Supplementary questions for data
collection
• Sources of inbound knowledge flows by location (Very
important GVCs, but hard to collect on surveys, may only
be possible with World I-O Tables)
• Outbound knowledge flows (Hard, but need both sides of
flows and asset transfers; build on existing economic data
and surveys)
• Channels for knowledge-based interactions between
firms and Higher Education Institutions (HEI) and Public
Research Institutions (PRI) (Yes!)
• Use of Intellectual Property Rights (IPR) for knowledge
flows (Tax, transfer pricing for IP make this very hard, but
extremely important)
5
Attachment E
Discussion 4 – External Factors
Influencing Innovation in Firms
National Center for Science and Engineering Statistics
National Science Foundation
Innovation Workshop
October 16, 2018
National Science Foundation
National Center for Science and Engineering Statistics
www.nsf.gov/statistics/
Attachment E
Definition
• External factors can:
– influence a firm’s incentives to innovate and its
innovation activities, capabilities and outcomes
– be the object of a business strategy, public policy or
concerted social action by public interest groups
– are often beyond the immediate control of management
– be drivers or barriers to innovation
– define the firm’s innovation operating environment
– depend on proximity to universities/clusters
– can we really sell our innovations into foreign markets?
2
Attachment E
3
Attachment E
Definition (continued)
– External factors may include:
• Spatial and locational factors that define the firm’s location
and its distance to product and labor markets which can
influence costs and consumer awareness
• Markets for the firm’s products; types of customers; the
influences of customers on innovation; competition and
collaboration in markets; coordination and standards in
markets
• Public Policy including regulations, government support,
public infrastructure
• Society and the natural environment
4
Attachment E
Questions for data collection
Key Questions
• Firm’s industry and main market
• Competition and product market characteristics
• Government support for innovation
• Drivers or barriers to innovation
5
Attachment E
Proposed Items for Inclusion in Questions on Competition
6
Attachment E
Main Types of Government Policy Instruments to Support Innovation
7
Attachment E
Questions for data collection
Key Questions
• Firm’s industry and main market
• Competition and product market characteristics
• Government support for innovation
• Drivers or barriers to innovation
Supplemental Questions???
• Location of business activities and value chains
• The main customer’s share of sales and the industry of
the main customer
• Effect of regulations on innovation
8
Attachment E
9
Attachment E
“Proposal for Integrated Collection of Data on External Innovation Drivers”
10
Attachment E
Discussion 5 – Objectives and
Outcomes of Business Innovation
National Center for Science and Engineering Statistics
National Science Foundation
Innovation Workshop
October 16, 2018
National Science Foundation
National Center for Science and Engineering Statistics
www.nsf.gov/statistics/
10/16/18
Attachment E
Definition
• Planning and development for an innovation
includes the identification of one or more
objectives that the innovation is expected to
achieve
– Objectives can concern the characteristics of the
innovation itself, such as its specifications, and its
market and economic objectives
• The outcomes of an innovation are similar to the
objectives, but consist of the innovation’s realized
effects.
• Outcomes can include unexpected effects that were
not identified among the firm’s initial objectives
10/16/18
Attachment E
Definition (continued)
– Objectives may include the following:
• Profit generation, cost savings, or improvements in productivity
• an increase in sales or brand awareness from product innovation
• changes to the firm’s capabilities, markets, or the types of
customers
• establishment of new external linkages
– Outcomes may include the following:
• extent to which a firm’s objectives are met
• broader effects of innovation on other organizations, the economy,
society, and the environment
– Compared with product innovations, respondents have
more difficulties in providing quantitative outcome data for
business process innovations
10/16/18
Attachment E
Questions for data collection
Key Questions
• Innovation objectives and outcomes by area of influence (e.g.,
markets, production/delivery, business organization,
economy/society)
• Innovation objectives and outcomes for business strategies
• Innovation sales share in total business sales
Supplemental Questions
• Counts of product innovations
• Major impacts of innovations for markets
10/16/18
| File Type | application/pdf |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |