Memo for Request for Approval for Cognitive Testing of Innovation Module for the 2017 Survey of Business Owners (SBO)

Memorandum

Date: July 7, 2017

To: Margo Schwab, Desk Officer

Office of Management and Budget

From: John R. Gawalt, Director

National Center for Science and Engineering Statistics

National Science Foundation

Via: Suzanne Plimpton, Reports Clearance Officer

National Science Foundation

Subject: Request for Approval for Cognitive Testing of Innovation Module for the 2017 Survey of Business Owners (SBO)

This memorandum is to request approval for the National Science Foundation’s (NSF) National Center for Science and Engineering Statistics’ (NCSES) plans to conduct cognitive interviews under the generic clearance for survey improvement projects (OMB number 3145-0174). This research will focus on the ability of small and medium sized companies (between five and 500 employees) to answer innovation-related questions. This activity is associated with a future survey to be conducted in partnership with NCSES and the US Census Bureau (Census).

Background

NCSES is broadly tasked with measuring the role of science and technology (S&T) in the United States’ economy and abroad. A major component of this activity is its sponsorship of the Business Research and Development (R&D) and Innovation Survey (BRDIS), which collects information annually on R&D and innovation by industrial firms within the United States. BRDIS consists of multiple questionnaires, the most prominent being the BRDI-1 questionnaire for companies with five or more employees. This year another questionnaire was added, the Business Research and Development and Innovation Survey for Microbusinesses (BRDI-M), for companies with fewer than ten employees.

The BRDI-1 and BRD1-M surveys are conducted through a partnership NCSES has with Census. NCSES is continuing this partnership through a planned survey module on innovation. Currently, NCSES and Census are discussing a revised survey where several questions from the Survey of Business Owners are combined with a module on innovation.

The innovation module that will be tested is largely based on questions that have been used on the Community Innovation Survey, which has been administered in all 28 European Union countries for multiple surveys. However, most of these questions have not been used in the United States. Cognitive interviews will help to improve and refine the existing questions, and ensure their suitability for an innovation related survey in the US.

The innovation questions from BRDI-1 and BRDI-M focus more on the incidence rate of innovation. The module on innovation that is being proposed will expand the scope of data beyond incidence rates to include:

• Incidence rate on product, process, marketing, and organization innovation

• New to market and new to firm product innovation

• Innovation activities

• Partnerships

• Reasons for not innovating, and barriers to innovation

Proposed Methodology

NCSES is working with cognitive researchers from Westat to conduct no more than 25 interviews. Interviews will be conducted either in person or utilizing WebEx conferencing, and will assess the new questions, the interest in new data points and the ability of respondents to provide data. All interviews will be based on the attached protocol (Attachment 3); however, because the survey protocol is too long to be completed in an hour, not every question in the protocol will be asked of every participant. Our plan is for every question to be examined in depth with at least several participants (e.g., some participants will be asked in depth about one subset of questions in the planned survey, others will be asked in depth about a different subset, and others will be asked to cover the entire module, with less depth). Also, some questions in the protocol are intended only as probes if the participants do not address the issue on their own initiative; ideally, it is preferred that the participants initiate the discussion so they do not feel they are being “corrected,” but the additional probes will be used when needed.

The interviews will be conducted with concurrent probing, with the participant first reading through a question and providing an answer (when possible; however, participants will not be asked to refer to records to look up answers). After a question is completed, the participant will be asked to describe his/her thought processes in interpreting the question and developing the answer, and probed about issues that are viewed as presenting potential concerns. It is possible that the protocol will be modified in minor ways as the interviews progress; e.g., if an item is not being understood properly, then alternative wordings may be offered to see if they are more effective.

These interviews will allow NCSES to better understand the question-response process, examining whether participants understand the questions, whether the questions are interpreted consistently across respondents, whether data are available for answering the questions, what strategies participants use to develop answers, and whether the participants are willing and able to respond accurately. In most cases, an NCSES subject matter expert will accompany the researchers to conduct the interviews.

The in-person and WebEx interviews will be conducted in the same manner and based on the same protocol, except that not all people will be in the same room. That is, WebEx participants will be shown a survey question on a computer screen rather than reading from a piece of paper. In-person interviews are an ideal format because it is possible to view a participant’s facial expressions, and it is easier to establish a personal relationship with the participant. However, WebEx interviews still are effective, they allow greater geographic disparity among the participants while controlling costs, and they are easy to schedule and less invasive for the participants. It is expected that in-person interviews will be conducted at the participants’ offices.

Businesses will be selected to ensure representation across a variety of characteristics such as geographic region, number of employees, and industry. It is anticipated that either the Small Business Administration (SBA) database (http://dsbs.sba.gov/dsbs/search/dsp_dsbs.cfm) or the Dun & Bradstreet (D&B) database will be used to select businesses to recruit. Preliminary review of each database suggests that while D&B’s database is more comprehensive, the SBA database may be updated more frequently. Thus, both will be used.

To recruit respondents, businesses will be contacted by phone and e-mail. The best respondent for the interview will be “the person in your company who is most knowledgeable about innovations your company has made or is working on”, which may or may not be the business owner. After respondents agree to participate, an e-mail will be forwarded thanking them for their willingness to participate and confirming the appointment. Interviews will be recorded, but only after written (Informed Consent Form - Attachment 4) or verbal consent from the respondent is obtained.

Incentive Payments

NCSES proposes offering respondents an incentive of $40 per interview. This is a standard amount for cognitive interviews, which are more intensive than ordinary surveys and which require people to schedule a significant amount of time to be away from their normal activities (in contrast, surveys can be completed in small portions at a time at a person’s convenience). For previous cognitive interviews with this population, a $40 incentive was offered.

The incentive payments will be made after the interview is completed. For in-person interviews, the participants will receive $40 in cash immediately at the end of the interview. For WebEx interviews, a check will be mailed after the interview is completed.

Response Burden

46 burden hours are estimated for this project. The estimated time for completion of the interview is about 60 minutes each for a total of 25 hours (25 interviews x 60 minutes = 25 hours). It is anticipated that 250 companies will be contacted for recruiting purposes. The recruiting process should take on average five minutes per company resulting in 20.8 hours (250 company contacts x 5 minutes = 21 hours).

Contact Person

Audrey Kindlon

Project Officer

Research and Development Statistics Program

National Center for Science and Engineering Statistics

National Science Foundation

akindlon@nsf.gov

703-292-2332

Attachment 1: Recruiting Script

Attachment 2: Interview Questions

Attachment 3: Innovation Module Cognitive Testing Interview Protocol

Attachment 4: NSF/NCSES Consent Form

3